Extension talk:Cargo

Query non cargo tables
Hi, there is any option to do this? (from #cargo_query)? Ty


 * No - if you want to include non-Cargo DB tables, the best way to do is via the External Data extension. You can use it to store that outside data in Cargo, and then query everything together. Yaron Koren (talk) 03:14, 5 August 2019 (UTC)

Display format: calendar not working with mobile view
When using the mobile view implemented by the MobileFrontend extension, somehow the format=calendar cannot be shown, regardless of the used skin. Query as format=list or ul works fina, as well as Special:ViewData and Special:CargoTables.

Same issue with format=timeline, although I'm personally mostly interested in the calendar.

Recurring Event Syntax
What should the syntax for a recurring event be for a table with a single field called "Birthday" where the value is fed from a form and which normally would look like this:

Something like this?

Thanks! Bryandamon (talk) 03:37, 7 August 2019 (UTC)


 * That seems right. Yaron Koren (talk) 13:34, 7 August 2019 (UTC)

So, Cargo queries aren't dynamic?
This may sound like a stupid question, but I don't see this stated anywhere on the wiki.

I have noticed that when I add a new item, any cargo queries I have in the wiki don't get updated. I have to re-edit the page with the query to see the new item.

This seems to really defeat the purpose of the #cargo_query function. So, am I doing something wrong or is it supposed to work like that?

Metalliqaz (talk) 16:21, 14 August 2019 (UTC)


 * See here. Yaron Koren (talk) 20:29, 14 August 2019 (UTC)


 * Understood, thanks. Metalliqaz (talk) 02:52, 15 August 2019 (UTC)

Underscores in field names cause issues
I have a Cargo table with a field called "Case". Since "Case" is a reserved word I had to rename it to "Case_". This works fine, except for one thing. In Special:CargoTables, when viewing the table, that column is broken. The table shows the column as "Case", and it is empty. The Table Structure list correctly shows the field as "Case_ - String". Cargo queries work fine. Metalliqaz (talk) 16:30, 14 August 2019 (UTC)


 * The issue there is that the field name ends with an underscore. I should put in a check to not allow that, because it will lead to problems, as you discovered. Yaron Koren (talk) 20:28, 14 August 2019 (UTC)


 * I changed the field from "Case_" to "CaseType" and rebuilt the data. It took two tries for the table to show up correctly.  But now my wiki is essentially broken.  Any new items added since then cause Visual editor to throw an error trying to save, the new pages aren't showing up in the Cargo tables and they aren't showing up in the category pages. Is there any way to recover this?
 * Edit: nevermind, php cargoRecreateData.php seems to have fixed it.Metalliqaz (talk) 19:11, 15 August 2019 (UTC)

VisualEditor compatibility with #cargo_query
Back again, I'm afraid. I've got everything working pretty well. I have a page that contains a #cargo_query table, and it works as expected. However, I've noticed that if I edit that page with VisualEditor, it will wrap the parser function in  tags and thus break the page if I save it. Is there any setting I have to change to prevent this?


 * Update: I played around on Wikipedia, and even their VisualEditor doesn't seem to really handle parser functions very well. It thinks they are templates.  In my case, I had to move some text after the ":" on the first line, and then it at least recognized my #cargo_query as a template and didn't wrap it in  .  So I guess this is a VisualEditor problem (or maybe Parsoid) Metalliqaz (talk) 02:34, 17 August 2019 (UTC)


 * Not a visual editor solution specifically, but I'd suggest as a general style rule always wrapping cargo_query calls in templates, this should have the side effect of hiding them from VE issues --RheingoldRiver (talk) 08:31, 17 August 2019 (UTC)

Make - a disallowed character in field names
Maybe I'm doing this wrong but I've never been able to do a query on a field with - in the name if a table specification is required to avoid ambiguity (e.g.  is fine but   doesn't work). If there's some easy way to fix this that idk about then never mind (but how??) - but assuming this is impossible, I think - should just be a global disallowed character in field names. Maybe a setting for backwards compatibility regarding this that defaults to enabling the disallow? --RheingoldRiver (talk)


 * This is also a good idea - dashes in table and field names tend to cause problems. I just checked in a change so now they're disallowed. I didn't add in any backward compatibility - hopefully not too many people are using them now. Yaron Koren (talk) 18:14, 12 September 2019 (UTC)

Use replacement table when creating _pageData?
Is this possible? If not could you add support for that? With the _pageNameOrRedirect field I'm going to add a ton of dependencies on this table that would break for any recreate (recreated today so I found out it doesn't use a replacement table now). Thanks! (actually still don't have this field, we upgraded just to 2.2 today but will have that soon hopefully) --RheingoldRiver (talk) 23:55, 20 August 2019 (UTC)


 * Good idea - I just checked in "--replacement" flags for the setCargoPageData.php and setCargoFileData.php scripts - so now, replacement tables can be created for _pageName and _fileName (at _pageName__NEXT and _fileName__NEXT), and they work the same way as standard replacement tables. Yaron Koren (talk) 23:02, 9 September 2019 (UTC)

Template to test whether page contains Cargo query
At the minute I've come up with :

I can test for an empty/non-empty result but that doesn't always work...


 * 1) Page does not have query on the relevant table - returns empty result
 * 2) Page does have ... - returns page name (non empty)
 * 3) Mistake, e.g. page or table name typed incorrectly - returns error message (non empty)

What would be a better way? Thanks. Jonathan3 (talk) 23:17, 25 August 2019 (UTC)


 * I don't understand - what are you trying to do, and what have you tried? Yaron Koren (talk) 00:04, 26 August 2019 (UTC)


 * I previously used Cargo in a template to test whether a page was in a certain category, but then learnt that unless you save the page twice the categories don't get saved by Cargo to the _pageData table.


 * I'd like to replace that test (i.e. is the page in a category that only a Cargo template would put it in?) with another test (i.e. does the page contain a Cargo query on the relevant table?).


 * The code above sort-of works but doesn't discriminate between a page title being returned and an error message being returned...


 * (The LIKE thing is because some of my page titles have both ' and " characters in them, so isn't directly relevant to the question.)


 * Thanks. Jonathan3 (talk) 00:16, 26 August 2019 (UTC)


 * I still don't understand.. you want to query for the existence of another query? Why? And how would the query above find that? It seems like just a normal query, not something that would find specific wikitext or some such. Yaron Koren (talk) 23:25, 26 August 2019 (UTC)


 * I have a template for displaying a page summary, which used DPL3 to display the introductory section of a page. Eventually all these pages will use Cargo; currently only a minority do. The template needs to work out whether to use Cargo to display the summary (from a database field relevant to whichever table the page uses) or DPL3 (from the first section). Using categories is currently unreliable so I just wondered if there is an ideal way of checking whether there is a Cargo query on the page... if there is, then the page summary template would use Cargo, and otherwise it would use DPL3. Maybe this is a more helpful way of describing the problem and/or maybe I am looking for a solution to the wrong problem (see XY problem!). Jonathan3 (talk) 01:38, 28 August 2019 (UTC)


 * Sorry, I still don't understand. Though I'm guessing that there is indeed a simpler solution. Hopefully someone else here can help with this. Yaron Koren (talk) 15:02, 28 August 2019 (UTC)


 * You're trying to push templates and Cargo way beyond their intended use. In my humble opinion, the correct answer is to properly plan your transition to the new scheme.  Make your template follow the old way of doing things (the current majority).  Add the new Cargo tables on top of that to each page at your leisure, then change over the template to the new method when you're done.  This sounds like a good place to use a bot.  Bots are useful and pretty easy to write for a moderately experience programmer. Metalliqaz (talk) 13:47, 29 August 2019‎


 * I don't think "to push templates and Cargo way beyond their intended use" is necessarily a bad thing, as long as it works (which it seems to now). I will think through your suggestions, though! I didn't want to use a bot to move every page to use the new Cargo template, as I want to add and check data manually for each page, which will take ages, and I wanted the new pages to have been perfected and to be easily identifiable as such. I didn't want the page summary template to do the same with both types of page, because it can show more useful information from the Cargo pages. However, I can see that there could be some benefits to transferring every page and possibly having an "incomplete" field to identify which pages need further work. Thanks :-) Jonathan3 (talk) 00:33, 30 August 2019 (UTC)


 * Don't get what you are trying to do here either, but for botting you can use AutoWikiBrowser. You can still check all changes manually, but much of the legwork can be done by the program, for instance using regular expressions or simple text replacement. If you only want to check for the existence of a cargo query, you can proably use AWB for that too. Perhaps this could be useful? Pangaearocks (talk) 18:41, 30 August 2019 (UTC)


 * Thanks. I'll look into AWB. Years ago I used Pywikibot for a simpler task (recategorising pages) so I'll look into that again too. Jonathan3 (talk) 20:56, 30 August 2019 (UTC)

Returning wikitext from Cargo queries
Suppose I have a table T, and a template A which generates wikitext using T, including links and images:

I would like to store A into another Cargo table as a wikitext field, say T2:

for example by calling A directly inside :

But the problem is, if A doesn't have, then T2 will receive strip markers (and sometimes store the row twice), yet if A uses  , the link disappears while the image is turned into a raw &lt;img&gt; tag, exposing the URL of the cached image file. This makes A unusable as the input to any wikitext field of a Cargo table. One solution is to replace A with a Lua version, for example: A's body then becomes just. This allows A to work properly, but in the general case T2.desc may accept any Cargo-based template as its input, and it is simply infeasible to repeat the same conversion on every template that could possibly be used for T2.desc. Is it possible to fix this from the parser functions alone, without using Lua? --HertzDevil (talk) 13:00, 27 August 2019 (UTC)


 * That's a tricky one, and you may have found a bug. But is all this really necessary? Why do you need to store that concatenated string at all? And if you need to store it, why not store it in the page corresponding to "_pageName", so that you can do it without calling a query? Yaron Koren (talk) 13:46, 27 August 2019 (UTC)

_pageData take columns from actual table, not the setting
Right now if you add columns to _pageData in localsettings but don't immediately rebuild the table, everything breaks because it tries to add the extra columns immediately. Could you make it instead look at the existing table to discover columns so that if there's some delay between columns being added and the table being recraeted, it's still working? --RheingoldRiver (talk) 20:08, 6 September 2019 (UTC)


 * I thought about doing it that way when first implementing _pageData - I decided against it because it adds a query every time someone saves a page, which is not a big deal but seems like overkill. Why can't you rebuild the table right away? (And would that "replacement" option help?) Yaron Koren (talk) 20:19, 6 September 2019 (UTC)


 * Now that we know it works this way, we'll be ok, but we had a ton of wikis break recently because of a change to the default _pageData setting that we wanted to take effect the next time tables were recreated, but not to run the recreate immediately. Is one extra read query that much of a problem? A replacement table actually would help this somewhat since the rebuilds would be lower-impact and could be safely done sooner, but this seems like a pretty unsafe way to do things as-is; _pageData failing is super high-impact since it causes the rest of the tables to be unable to store as well. --RheingoldRiver (talk) 21:44, 6 September 2019 (UTC)

Bolded Nos
How about reordering the table then so the two "No" lines are next to each other? Every time I've looked at that table it's been hard to find which lines say No so I think it does need some kind of improved visibility - fwiw to me indexed or not is the most important quality of the field type I chose, so I think bold makes sense, but something else could work instead --RheingoldRiver (talk) 06:59, 21 September 2019 (UTC)


 * I agree that "Yes" vs. "No" doesn't pop out at all. How about a check (&#10004;) and blank, instead? Yaron Koren (talk) 03:27, 25 September 2019 (UTC)


 * Good idea - I changed it to that and also indicated unindexed instead of indexed, so it's super clear now --RheingoldRiver (talk) 17:07, 27 September 2019 (UTC)


 * I thought it was going to be check marks? Yaron Koren (talk) 13:38, 29 September 2019 (UTC)


 * Oops forgot, updated again --RheingoldRiver (talk) 14:04, 29 September 2019 (UTC)


 * Alright - it looks pretty good. I'm not sure how I feel about having the column be "Unindexed?" rather than "Indexed?" - I know why you did it, but it still seems confusing. I'll see how I feel about it. Yaron Koren (talk) 14:47, 29 September 2019 (UTC)

More graceful failure if storing to a table not created yet
This is one of the most commonly aske FAQ for us, "why is my template erroring" when people don't put a cargo_store inside of includeonly when declaring a table, or when someone's copying code from one wiki to another. Ideally I think an attempt to store to a non-existent table should just be ignored silently, but maybe an in-between option would be to generate error text in place of the template itself instructing the user to create the table prior to storing to it would work better? As-is people just get a database error, nothing is saved, and they don't even know it's a Cargo error. --RheingoldRiver (talk) 20:14, 24 September 2019 (UTC)


 * That sounds like a good idea - and it's similar to your request above to do checking on the set of columns before saving to _pageData, etc. It sounds like ideally, the #cargo_store code should check for both the existence of the table, and the existence of all relevant columns, before trying to save the data. There's really no point trying to maximize the efficiency of page saves, since they're pretty rare compared to page reads. Yaron Koren (talk) 03:31, 25 September 2019 (UTC)


 * It already deals with incorrect columns in user-stored tables fine actually, it just does nothing with them - this is extremely helpful behavior because it means I can keep not-indended-for-store fields in my Lua tables and not bother doing any validation of what actually gets sent to Cargo, I just use a convention of uppercase = Cargo field and lowercase = display or other processing field, and then one data structure has everything. The problems are when the entire table doesn't exist or when _pageData specifically is "declared" with a different set of columns from what the live table has --RheingoldRiver (talk) 17:12, 27 September 2019 (UTC)

Lag in _pageData updating categories
When a category is added to a page, the page needs to be blank edited a second time for the _pageData categories field to show up updated, instead of just immediately being updated when the page is edited the first time. Is this possible to fix? --RheingoldRiver (talk) 14:10, 29 September 2019 (UTC)


 * This came up a few months ago - the conclusion was that Cargo should use some MediaWiki hook to update the categories data once it's been set in MediaWiki itself, if that's possible. This hasn't been done yet, though. Yaron Koren (talk) 14:50, 29 September 2019 (UTC)

Fatal exception of type "Error"
Cargo is throwing the following error when I click on the "page values" link: [XZUO-EWj@38AAEV8NL8AAAAA] 2019-10-02 20:56:28: Fatal exception of type "Error"

When I enable debug in Localsettings, I get more info. Top of the page says: Notice: Undefined variable: cdb in /home/xxxxx/extensions/Cargo/specials/CargoPageValues.php on line 147

Body of page says: [XZUP6EWj@38AAGblKQkAAAAA] /index.php?title=Tina_Tenbergen&action=pagevalues Error from line 147 of /home/xxxxx/extensions/Cargo/specials/CargoPageValues.php: Call to a member function addIdentifierQuotes on null Backtrace:
 * 1) 0 /home/xxxxx/extensions/Cargo/specials/CargoPageValues.php(66): CargoPageValues->getRowsForPageInTable(string)
 * 2) 1 /home/xxxxx/extensions/Cargo/includes/CargoPageValuesAction.php(26): CargoPageValues->execute
 * 3) 2 /home/xxxxx/includes/MediaWiki.php(499): CargoPageValuesAction->show
 * 4) 3 /home/xxxxx/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
 * 5) 4 /home/xxxxx/includes/MediaWiki.php(865): MediaWiki->performRequest
 * 6) 5 /home/xxxxx/includes/MediaWiki.php(515): MediaWiki->main
 * 7) 6 /home/xxxxx/index.php(42): MediaWiki->run
 * 8) 7 {main}

What do I need to do to fix this? Thanks! Tenbergen (talk) 21:14, 2 October 2019 (UTC)


 * Sorry about that - that was a bug that unfortunately got into version 2.3 of Cargo. I released a new version, 2.3.1, a few days ago to fix this problem - so you just need to upgrade. Yaron Koren (talk) 18:37, 3 October 2019 (UTC)

List field subtables don't get switched in along with the main table when using replacement tables
I think we finally found the cause of HOLDS being buggy, list-type fields in general being unreliable, etc. When a table is recreated with replacement, and then switched in, the replacement for the list-type fields' tables isn't switched in along with it properly. Recreating without replacement doesn't have this issue, but recreating without replacement is also extremely disruptive to the wiki.

The problem is at  it gets limited by   which is declared as. Could you make sure this column gets updated to be the correct type during update.php? (this request is from Robert, I'm just relaying it) --RheingoldRiver (talk) 15:58, 6 October 2019 (UTC)


 * Right, that was a problem that was fixed back in 2016, but I never made a separate patch for it. This wiki must have had Cargo installed on it pretty early on. I don't know if it makes a sense to create a patch for the problem at this point... hopefully it can be fixed manually on this wiki. Yaron Koren (talk) 17:22, 6 October 2019 (UTC)


 * Ahh ok, that makes sense why we've had these problems for forever but they aren't coming up on other platforms. Robert fixed this morning on my wiki and Path of Exile which were the two that had the most problems since we use the most Cargo, I'm not sure what the plan is for the rest of the Cargo wikis (there's a lot at this point). Hopefully this was the only remaining large issue and everything is a lot smoother now --RheingoldRiver (talk) 18:02, 6 October 2019 (UTC)

Two table recreation suggestions/requests
1) On replacement table deletion, could the remaining associated jobs to process those pages be killed? As-is, if you recreate a large table & notice a mistake, you have to wait for the queue to finish because if you delete & then re-recreate the new replacement table will have duplicate rows sometimes due to the leftover jobs from the previous recreate (this is the only time duplicate rows seem to be a thing anymore, when a page is either edited/blank edited or has jobs left over from a previous recreate, during repopulation of a replacement table)

2) Could you hook into Echo to send a notification on replacement table jobs finishing? A lot of my recreates (and also on other GP wikis) take 30 minutes or more, and on occasion I've forgotten for an entire day or longer to switch it in, until someone error reports to me that nothing is updating. Not sure how many wikis other than GP have enough Cargo usage that this is an issue, but it definitely would be useful on several of our wikis.

--RheingoldRiver (talk) 11:01, 9 October 2019 (UTC)


 * I'm not sure if either of these is really doable. Handling the job queue is tricky. If the 2nd one were doable, then the code could do the switch automatically - that's actually how I originally wanted the code to do it, but I couldn't figure out how it could be done without doing an expensive query after every job is run. The first one is more doable, but I'm not 100% sure that this is always the correct behavior. But I haven't thought about it that much. What do you think? Yaron Koren (talk) 03:27, 11 October 2019 (UTC)


 * I think for sure if a replacement is cancelled, you'll for sure want those jobs cancelled as well - the jobs were intended to repopulate a table that now doesn't even exist. I guess maybe someone was using a really broken workflow where to force a table to update, they recreated with replacement then immediately deleted to get the jobs to continue, but if anyone is doing that then probably "start jobs to update this table" as a separate option would be reasonable, rather then maintaining the current setup.


 * For the second part, how does Special:CargoTables know when to stop showing the message "one or more of these tables is currently being repopulated, via the job queue?" Couldn't the same process that hides that also hook into Echo? FWIW though I wouldn't be in favor of automatic switches (or at least have it be an option to be automatic or not) since often I want to check that I did the right thing with a query on __NEXT prior to switching in myself --RheingoldRiver (talk) 11:30, 11 October 2019 (UTC)


 * For #1: that's fair - it does sound like a good idea. For #2 - the issue is that, in the job queue - which is just a database table called "job" - there's a column for the job type, then another column for all the other info. So the Cargo code can look through the table and see if there are any jobs of that type ("cargoRecreateData", or something like that), but it can't easily query to see which of them are for a specific Cargo table. For that, it would need to go through each "cargoRecreateData" entry/job, and parse its info manually. Which may not take all that long to do, but it's not a single SQL query. You do make a good point that the switch should not (always) be done automatically even if it could be, but I'm not sure it makes sense for the Cargo code to do a check like this in the first place. Yaron Koren (talk) 18:01, 11 October 2019 (UTC)


 * Ah, I'd be totally happy with a notification that "all currently-recreating tables are now complete," that's pretty much equivalent (I avoid recreating two tables at the same time regardless, seems safer). Would that be doable? --RheingoldRiver (talk) 18:41, 11 October 2019 (UTC)


 * That's interesting. It's doable, yes. And sounds like a pretty good idea. Yaron Koren (talk) 18:52, 11 October 2019 (UTC)

Another _pageData request (sorry)
Could you add all of the fields from Help:Magic words ? Right now ROOTPAGENAME is the one I'm interested in (for a join condition). I also tried to use a CONCAT inside of a join which afaik is valid in SQL but not Cargo? Is there a reason this isn't allowed in Cargo? --RheingoldRiver (talk) 09:29, 21 October 2019 (UTC)

Actually maybe there's a lot of magic words that could be included in general...I could go through and make a proposal of which ones seem useful? Or how possible would it be to just add in general, "any magic word on a page can be added as a field in _pageData" ? --RheingoldRiver (talk) 09:31, 21 October 2019 (UTC)


 * Do you mean a separate column for each one? If so, then yes, a proposal of the specific fields you want would be useful. It's hard to imagine you really want seven different variations on the page name... Yaron Koren (talk) 18:49, 21 October 2019 (UTC)


 * Yeah a separate column for each. Because of inability to concat in joins a lot of random columns can end up being really useful. Is the "any magic word" (i.e. not just the ones in that section, but any magic word you can evaluate on the page) thing not practical to do? If so I'll go through and make a list. --RheingoldRiver (talk) 19:47, 21 October 2019 (UTC)


 * Well, "any magic word" doesn't really make sense, since most of the magic words don't apply to a specific page (like "SITENAME"). Yes, a list would be helpful. Yaron Koren (talk) 21:03, 21 October 2019 (UTC)


 * If it could be written in a general way though, then up to the user what they want, even if they could do something silly like store a row that's constant, that wouldn't be a problem right? I was thinking something like "wgCargoPageDataMagicWordFields" where you list the magic words you want, and then it evaluates them on each page to store to the table in those columns, and the name of the column could just be the magic word (maybe all lowercase instead of all caps). Is that possible? --RheingoldRiver (talk) 09:40, 22 October 2019 (UTC)


 * I don't know. But even if it's possible, it may not be worthwhile, if we're only talking about a few additional fields. It seems to me that, as far as the page name is concerned, there might be three additional fields that make sense: for a page titled "Help:A/B/C/D", it could be useful to have fields that store "A", "A/B/C", and "D". (It looks like those correspond to ROOTPAGENAME, BASEPAGENAME and SUBPAGENAME.) Yaron Koren (talk) 16:28, 22 October 2019 (UTC)


 * Okay, yeah I think those would be nice, along with NAMESPACE. Are the revision-specific ones possible, or would that run into the same issue categories have currently? In particular REVISIONUSER and REVISIONID I think aren't possible with existing _pageData fields. In particular for the revision properties, they would be useful for creating anti-vandalism (including accidental vandalism by user script (not that I ever need that.....)) tools, without having to use the recentchanges api which is unreliable if something goes unnoticed for a while - I'm not sure if this would open up completely new functionality that's currently impossible (other than by making multiple api calls), but it would definitely be a lot easier to simply run api cargo queries to get page lists. Part of the reason I was thinking an arbitrary list of magic words could be cool is that it would open up extension magic words too, but I can't think of any specifically I'd use immediately. --RheingoldRiver (talk) 21:27, 22 October 2019 (UTC)


 * The namespace is already stored via the _pageNamespace field in every regular Cargo table - is that good enough? REVISIONUSER could be useful - presumably it would be stored via some field like _lastEditor - but what would be the point of storing REVISIONID? Yaron Koren (talk) 02:48, 23 October 2019 (UTC)


 * Oh cool I didn't realize _pageNamespace was a thing, yeah that's for sure enough. Getting the revision ID seems pretty useful for external maintenance scripts, currently the only ways to get rev ids are either recentchanges or individually querying one single page. I doubt I'd ever use it on-wiki though, only through the cargoquery API. --RheingoldRiver (talk) 06:39, 23 October 2019 (UTC)


 * But wouldn't you need to join the revision ID in a query with some non-Cargo tables for it to be useful? Yaron Koren (talk) 14:17, 23 October 2019 (UTC)


 * No, I'd be using Cargo to discover revision IDs to use somehow, e.g. to rollback in a cleaning-up-problematic-bot-edits script. --RheingoldRiver (talk) 19:12, 23 October 2019 (UTC)


 * If it's from a script, couldn't you get the revision ID through a regular SQL query on the main MediaWiki database? (And actually, couldn't you get REVISIONUSER the same way?) Yaron Koren (talk) 19:45, 23 October 2019 (UTC)