Extension talk:Cargo

Query non cargo tables
Hi, there is any option to do this? (from #cargo_query)? Ty


 * No - if you want to include non-Cargo DB tables, the best way to do is via the External Data extension. You can use it to store that outside data in Cargo, and then query everything together. Yaron Koren (talk) 03:14, 5 August 2019 (UTC)

Display format: calendar not working with mobile view
When using the mobile view implemented by the MobileFrontend extension, somehow the format=calendar cannot be shown, regardless of the used skin. Query as format=list or ul works fina, as well as Special:ViewData and Special:CargoTables.

Same issue with format=timeline, although I'm personally mostly interested in the calendar.

Recurring Event Syntax
What should the syntax for a recurring event be for a table with a single field called "Birthday" where the value is fed from a form and which normally would look like this:

Something like this?

Thanks! Bryandamon (talk) 03:37, 7 August 2019 (UTC)


 * That seems right. Yaron Koren (talk) 13:34, 7 August 2019 (UTC)

So, Cargo queries aren't dynamic?
This may sound like a stupid question, but I don't see this stated anywhere on the wiki.

I have noticed that when I add a new item, any cargo queries I have in the wiki don't get updated. I have to re-edit the page with the query to see the new item.

This seems to really defeat the purpose of the #cargo_query function. So, am I doing something wrong or is it supposed to work like that?

Metalliqaz (talk) 16:21, 14 August 2019 (UTC)


 * See here. Yaron Koren (talk) 20:29, 14 August 2019 (UTC)


 * Understood, thanks. Metalliqaz (talk) 02:52, 15 August 2019 (UTC)

Underscores in field names cause issues
I have a Cargo table with a field called "Case". Since "Case" is a reserved word I had to rename it to "Case_". This works fine, except for one thing. In Special:CargoTables, when viewing the table, that column is broken. The table shows the column as "Case", and it is empty. The Table Structure list correctly shows the field as "Case_ - String". Cargo queries work fine. Metalliqaz (talk) 16:30, 14 August 2019 (UTC)


 * The issue there is that the field name ends with an underscore. I should put in a check to not allow that, because it will lead to problems, as you discovered. Yaron Koren (talk) 20:28, 14 August 2019 (UTC)


 * I changed the field from "Case_" to "CaseType" and rebuilt the data. It took two tries for the table to show up correctly.  But now my wiki is essentially broken.  Any new items added since then cause Visual editor to throw an error trying to save, the new pages aren't showing up in the Cargo tables and they aren't showing up in the category pages. Is there any way to recover this?
 * Edit: nevermind, php cargoRecreateData.php seems to have fixed it.Metalliqaz (talk) 19:11, 15 August 2019 (UTC)

VisualEditor compatibility with #cargo_query
Back again, I'm afraid. I've got everything working pretty well. I have a page that contains a #cargo_query table, and it works as expected. However, I've noticed that if I edit that page with VisualEditor, it will wrap the parser function in  tags and thus break the page if I save it. Is there any setting I have to change to prevent this?


 * Update: I played around on Wikipedia, and even their VisualEditor doesn't seem to really handle parser functions very well. It thinks they are templates.  In my case, I had to move some text after the ":" on the first line, and then it at least recognized my #cargo_query as a template and didn't wrap it in  .  So I guess this is a VisualEditor problem (or maybe Parsoid) Metalliqaz (talk) 02:34, 17 August 2019 (UTC)


 * Not a visual editor solution specifically, but I'd suggest as a general style rule always wrapping cargo_query calls in templates, this should have the side effect of hiding them from VE issues --RheingoldRiver (talk) 08:31, 17 August 2019 (UTC)

Make - a disallowed character in field names
Maybe I'm doing this wrong but I've never been able to do a query on a field with - in the name if a table specification is required to avoid ambiguity (e.g.  is fine but   doesn't work). If there's some easy way to fix this that idk about then never mind (but how??) - but assuming this is impossible, I think - should just be a global disallowed character in field names. Maybe a setting for backwards compatibility regarding this that defaults to enabling the disallow? --RheingoldRiver (talk)


 * This is also a good idea - dashes in table and field names tend to cause problems. I just checked in a change so now they're disallowed. I didn't add in any backward compatibility - hopefully not too many people are using them now. Yaron Koren (talk) 18:14, 12 September 2019 (UTC)

Use replacement table when creating _pageData?
Is this possible? If not could you add support for that? With the _pageNameOrRedirect field I'm going to add a ton of dependencies on this table that would break for any recreate (recreated today so I found out it doesn't use a replacement table now). Thanks! (actually still don't have this field, we upgraded just to 2.2 today but will have that soon hopefully) --RheingoldRiver (talk) 23:55, 20 August 2019 (UTC)


 * Good idea - I just checked in "--replacement" flags for the setCargoPageData.php and setCargoFileData.php scripts - so now, replacement tables can be created for _pageName and _fileName (at _pageName__NEXT and _fileName__NEXT), and they work the same way as standard replacement tables. Yaron Koren (talk) 23:02, 9 September 2019 (UTC)

Template to test whether page contains Cargo query
At the minute I've come up with :

I can test for an empty/non-empty result but that doesn't always work...


 * 1) Page does not have query on the relevant table - returns empty result
 * 2) Page does have ... - returns page name (non empty)
 * 3) Mistake, e.g. page or table name typed incorrectly - returns error message (non empty)

What would be a better way? Thanks. Jonathan3 (talk) 23:17, 25 August 2019 (UTC)


 * I don't understand - what are you trying to do, and what have you tried? Yaron Koren (talk) 00:04, 26 August 2019 (UTC)


 * I previously used Cargo in a template to test whether a page was in a certain category, but then learnt that unless you save the page twice the categories don't get saved by Cargo to the _pageData table.


 * I'd like to replace that test (i.e. is the page in a category that only a Cargo template would put it in?) with another test (i.e. does the page contain a Cargo query on the relevant table?).


 * The code above sort-of works but doesn't discriminate between a page title being returned and an error message being returned...


 * (The LIKE thing is because some of my page titles have both ' and " characters in them, so isn't directly relevant to the question.)


 * Thanks. Jonathan3 (talk) 00:16, 26 August 2019 (UTC)


 * I still don't understand.. you want to query for the existence of another query? Why? And how would the query above find that? It seems like just a normal query, not something that would find specific wikitext or some such. Yaron Koren (talk) 23:25, 26 August 2019 (UTC)


 * I have a template for displaying a page summary, which used DPL3 to display the introductory section of a page. Eventually all these pages will use Cargo; currently only a minority do. The template needs to work out whether to use Cargo to display the summary (from a database field relevant to whichever table the page uses) or DPL3 (from the first section). Using categories is currently unreliable so I just wondered if there is an ideal way of checking whether there is a Cargo query on the page... if there is, then the page summary template would use Cargo, and otherwise it would use DPL3. Maybe this is a more helpful way of describing the problem and/or maybe I am looking for a solution to the wrong problem (see XY problem!). Jonathan3 (talk) 01:38, 28 August 2019 (UTC)


 * Sorry, I still don't understand. Though I'm guessing that there is indeed a simpler solution. Hopefully someone else here can help with this. Yaron Koren (talk) 15:02, 28 August 2019 (UTC)


 * You're trying to push templates and Cargo way beyond their intended use. In my humble opinion, the correct answer is to properly plan your transition to the new scheme.  Make your template follow the old way of doing things (the current majority).  Add the new Cargo tables on top of that to each page at your leisure, then change over the template to the new method when you're done.  This sounds like a good place to use a bot.  Bots are useful and pretty easy to write for a moderately experience programmer. Metalliqaz (talk) 13:47, 29 August 2019‎


 * I don't think "to push templates and Cargo way beyond their intended use" is necessarily a bad thing, as long as it works (which it seems to now). I will think through your suggestions, though! I didn't want to use a bot to move every page to use the new Cargo template, as I want to add and check data manually for each page, which will take ages, and I wanted the new pages to have been perfected and to be easily identifiable as such. I didn't want the page summary template to do the same with both types of page, because it can show more useful information from the Cargo pages. However, I can see that there could be some benefits to transferring every page and possibly having an "incomplete" field to identify which pages need further work. Thanks :-) Jonathan3 (talk) 00:33, 30 August 2019 (UTC)


 * Don't get what you are trying to do here either, but for botting you can use AutoWikiBrowser. You can still check all changes manually, but much of the legwork can be done by the program, for instance using regular expressions or simple text replacement. If you only want to check for the existence of a cargo query, you can proably use AWB for that too. Perhaps this could be useful? Pangaearocks (talk) 18:41, 30 August 2019 (UTC)


 * Thanks. I'll look into AWB. Years ago I used Pywikibot for a simpler task (recategorising pages) so I'll look into that again too. Jonathan3 (talk) 20:56, 30 August 2019 (UTC)

Returning wikitext from Cargo queries
Suppose I have a table T, and a template A which generates wikitext using T, including links and images:

I would like to store A into another Cargo table as a wikitext field, say T2:

for example by calling A directly inside :

But the problem is, if A doesn't have, then T2 will receive strip markers (and sometimes store the row twice), yet if A uses  , the link disappears while the image is turned into a raw &lt;img&gt; tag, exposing the URL of the cached image file. This makes A unusable as the input to any wikitext field of a Cargo table. One solution is to replace A with a Lua version, for example: A's body then becomes just. This allows A to work properly, but in the general case T2.desc may accept any Cargo-based template as its input, and it is simply infeasible to repeat the same conversion on every template that could possibly be used for T2.desc. Is it possible to fix this from the parser functions alone, without using Lua? --HertzDevil (talk) 13:00, 27 August 2019 (UTC)


 * That's a tricky one, and you may have found a bug. But is all this really necessary? Why do you need to store that concatenated string at all? And if you need to store it, why not store it in the page corresponding to "_pageName", so that you can do it without calling a query? Yaron Koren (talk) 13:46, 27 August 2019 (UTC)

_pageData take columns from actual table, not the setting
Right now if you add columns to _pageData in localsettings but don't immediately rebuild the table, everything breaks because it tries to add the extra columns immediately. Could you make it instead look at the existing table to discover columns so that if there's some delay between columns being added and the table being recraeted, it's still working? --RheingoldRiver (talk) 20:08, 6 September 2019 (UTC)


 * I thought about doing it that way when first implementing _pageData - I decided against it because it adds a query every time someone saves a page, which is not a big deal but seems like overkill. Why can't you rebuild the table right away? (And would that "replacement" option help?) Yaron Koren (talk) 20:19, 6 September 2019 (UTC)


 * Now that we know it works this way, we'll be ok, but we had a ton of wikis break recently because of a change to the default _pageData setting that we wanted to take effect the next time tables were recreated, but not to run the recreate immediately. Is one extra read query that much of a problem? A replacement table actually would help this somewhat since the rebuilds would be lower-impact and could be safely done sooner, but this seems like a pretty unsafe way to do things as-is; _pageData failing is super high-impact since it causes the rest of the tables to be unable to store as well. --RheingoldRiver (talk) 21:44, 6 September 2019 (UTC)

Bolded Nos
How about reordering the table then so the two "No" lines are next to each other? Every time I've looked at that table it's been hard to find which lines say No so I think it does need some kind of improved visibility - fwiw to me indexed or not is the most important quality of the field type I chose, so I think bold makes sense, but something else could work instead --RheingoldRiver (talk) 06:59, 21 September 2019 (UTC)


 * I agree that "Yes" vs. "No" doesn't pop out at all. How about a check (&#10004;) and blank, instead? Yaron Koren (talk) 03:27, 25 September 2019 (UTC)


 * Good idea - I changed it to that and also indicated unindexed instead of indexed, so it's super clear now --RheingoldRiver (talk) 17:07, 27 September 2019 (UTC)


 * I thought it was going to be check marks? Yaron Koren (talk) 13:38, 29 September 2019 (UTC)


 * Oops forgot, updated again --RheingoldRiver (talk) 14:04, 29 September 2019 (UTC)


 * Alright - it looks pretty good. I'm not sure how I feel about having the column be "Unindexed?" rather than "Indexed?" - I know why you did it, but it still seems confusing. I'll see how I feel about it. Yaron Koren (talk) 14:47, 29 September 2019 (UTC)

More graceful failure if storing to a table not created yet
This is one of the most commonly aske FAQ for us, "why is my template erroring" when people don't put a cargo_store inside of includeonly when declaring a table, or when someone's copying code from one wiki to another. Ideally I think an attempt to store to a non-existent table should just be ignored silently, but maybe an in-between option would be to generate error text in place of the template itself instructing the user to create the table prior to storing to it would work better? As-is people just get a database error, nothing is saved, and they don't even know it's a Cargo error. --RheingoldRiver (talk) 20:14, 24 September 2019 (UTC)


 * That sounds like a good idea - and it's similar to your request above to do checking on the set of columns before saving to _pageData, etc. It sounds like ideally, the #cargo_store code should check for both the existence of the table, and the existence of all relevant columns, before trying to save the data. There's really no point trying to maximize the efficiency of page saves, since they're pretty rare compared to page reads. Yaron Koren (talk) 03:31, 25 September 2019 (UTC)


 * It already deals with incorrect columns in user-stored tables fine actually, it just does nothing with them - this is extremely helpful behavior because it means I can keep not-indended-for-store fields in my Lua tables and not bother doing any validation of what actually gets sent to Cargo, I just use a convention of uppercase = Cargo field and lowercase = display or other processing field, and then one data structure has everything. The problems are when the entire table doesn't exist or when _pageData specifically is "declared" with a different set of columns from what the live table has --RheingoldRiver (talk) 17:12, 27 September 2019 (UTC)

Lag in _pageData updating categories
When a category is added to a page, the page needs to be blank edited a second time for the _pageData categories field to show up updated, instead of just immediately being updated when the page is edited the first time. Is this possible to fix? --RheingoldRiver (talk) 14:10, 29 September 2019 (UTC)


 * This came up a few months ago - the conclusion was that Cargo should use some MediaWiki hook to update the categories data once it's been set in MediaWiki itself, if that's possible. This hasn't been done yet, though. Yaron Koren (talk) 14:50, 29 September 2019 (UTC)

Fatal exception of type "Error"
Cargo is throwing the following error when I click on the "page values" link: [XZUO-EWj@38AAEV8NL8AAAAA] 2019-10-02 20:56:28: Fatal exception of type "Error"

When I enable debug in Localsettings, I get more info. Top of the page says: Notice: Undefined variable: cdb in /home/xxxxx/extensions/Cargo/specials/CargoPageValues.php on line 147

Body of page says: [XZUP6EWj@38AAGblKQkAAAAA] /index.php?title=Tina_Tenbergen&action=pagevalues Error from line 147 of /home/xxxxx/extensions/Cargo/specials/CargoPageValues.php: Call to a member function addIdentifierQuotes on null Backtrace:
 * 1) 0 /home/xxxxx/extensions/Cargo/specials/CargoPageValues.php(66): CargoPageValues->getRowsForPageInTable(string)
 * 2) 1 /home/xxxxx/extensions/Cargo/includes/CargoPageValuesAction.php(26): CargoPageValues->execute
 * 3) 2 /home/xxxxx/includes/MediaWiki.php(499): CargoPageValuesAction->show
 * 4) 3 /home/xxxxx/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
 * 5) 4 /home/xxxxx/includes/MediaWiki.php(865): MediaWiki->performRequest
 * 6) 5 /home/xxxxx/includes/MediaWiki.php(515): MediaWiki->main
 * 7) 6 /home/xxxxx/index.php(42): MediaWiki->run
 * 8) 7 {main}

What do I need to do to fix this? Thanks! Tenbergen (talk) 21:14, 2 October 2019 (UTC)


 * Sorry about that - that was a bug that unfortunately got into version 2.3 of Cargo. I released a new version, 2.3.1, a few days ago to fix this problem - so you just need to upgrade. Yaron Koren (talk) 18:37, 3 October 2019 (UTC)
 * Thanks Yaron! Tenbergen (talk) 15:01, 29 October 2019 (UTC)

List field subtables don't get switched in along with the main table when using replacement tables
I think we finally found the cause of HOLDS being buggy, list-type fields in general being unreliable, etc. When a table is recreated with replacement, and then switched in, the replacement for the list-type fields' tables isn't switched in along with it properly. Recreating without replacement doesn't have this issue, but recreating without replacement is also extremely disruptive to the wiki.

The problem is at  it gets limited by   which is declared as. Could you make sure this column gets updated to be the correct type during update.php? (this request is from Robert, I'm just relaying it) --RheingoldRiver (talk) 15:58, 6 October 2019 (UTC)


 * Right, that was a problem that was fixed back in 2016, but I never made a separate patch for it. This wiki must have had Cargo installed on it pretty early on. I don't know if it makes a sense to create a patch for the problem at this point... hopefully it can be fixed manually on this wiki. Yaron Koren (talk) 17:22, 6 October 2019 (UTC)


 * Ahh ok, that makes sense why we've had these problems for forever but they aren't coming up on other platforms. Robert fixed this morning on my wiki and Path of Exile which were the two that had the most problems since we use the most Cargo, I'm not sure what the plan is for the rest of the Cargo wikis (there's a lot at this point). Hopefully this was the only remaining large issue and everything is a lot smoother now --RheingoldRiver (talk) 18:02, 6 October 2019 (UTC)

Two table recreation suggestions/requests
1) On replacement table deletion, could the remaining associated jobs to process those pages be killed? As-is, if you recreate a large table & notice a mistake, you have to wait for the queue to finish because if you delete & then re-recreate the new replacement table will have duplicate rows sometimes due to the leftover jobs from the previous recreate (this is the only time duplicate rows seem to be a thing anymore, when a page is either edited/blank edited or has jobs left over from a previous recreate, during repopulation of a replacement table)

2) Could you hook into Echo to send a notification on replacement table jobs finishing? A lot of my recreates (and also on other GP wikis) take 30 minutes or more, and on occasion I've forgotten for an entire day or longer to switch it in, until someone error reports to me that nothing is updating. Not sure how many wikis other than GP have enough Cargo usage that this is an issue, but it definitely would be useful on several of our wikis.

--RheingoldRiver (talk) 11:01, 9 October 2019 (UTC)


 * I'm not sure if either of these is really doable. Handling the job queue is tricky. If the 2nd one were doable, then the code could do the switch automatically - that's actually how I originally wanted the code to do it, but I couldn't figure out how it could be done without doing an expensive query after every job is run. The first one is more doable, but I'm not 100% sure that this is always the correct behavior. But I haven't thought about it that much. What do you think? Yaron Koren (talk) 03:27, 11 October 2019 (UTC)


 * I think for sure if a replacement is cancelled, you'll for sure want those jobs cancelled as well - the jobs were intended to repopulate a table that now doesn't even exist. I guess maybe someone was using a really broken workflow where to force a table to update, they recreated with replacement then immediately deleted to get the jobs to continue, but if anyone is doing that then probably "start jobs to update this table" as a separate option would be reasonable, rather then maintaining the current setup.


 * For the second part, how does Special:CargoTables know when to stop showing the message "one or more of these tables is currently being repopulated, via the job queue?" Couldn't the same process that hides that also hook into Echo? FWIW though I wouldn't be in favor of automatic switches (or at least have it be an option to be automatic or not) since often I want to check that I did the right thing with a query on __NEXT prior to switching in myself --RheingoldRiver (talk) 11:30, 11 October 2019 (UTC)


 * For #1: that's fair - it does sound like a good idea. For #2 - the issue is that, in the job queue - which is just a database table called "job" - there's a column for the job type, then another column for all the other info. So the Cargo code can look through the table and see if there are any jobs of that type ("cargoRecreateData", or something like that), but it can't easily query to see which of them are for a specific Cargo table. For that, it would need to go through each "cargoRecreateData" entry/job, and parse its info manually. Which may not take all that long to do, but it's not a single SQL query. You do make a good point that the switch should not (always) be done automatically even if it could be, but I'm not sure it makes sense for the Cargo code to do a check like this in the first place. Yaron Koren (talk) 18:01, 11 October 2019 (UTC)


 * Ah, I'd be totally happy with a notification that "all currently-recreating tables are now complete," that's pretty much equivalent (I avoid recreating two tables at the same time regardless, seems safer). Would that be doable? --RheingoldRiver (talk) 18:41, 11 October 2019 (UTC)


 * That's interesting. It's doable, yes. And sounds like a pretty good idea. Yaron Koren (talk) 18:52, 11 October 2019 (UTC)

Another _pageData request (sorry)
Could you add all of the fields from Help:Magic words ? Right now ROOTPAGENAME is the one I'm interested in (for a join condition). I also tried to use a CONCAT inside of a join which afaik is valid in SQL but not Cargo? Is there a reason this isn't allowed in Cargo? --RheingoldRiver (talk) 09:29, 21 October 2019 (UTC)

Actually maybe there's a lot of magic words that could be included in general...I could go through and make a proposal of which ones seem useful? Or how possible would it be to just add in general, "any magic word on a page can be added as a field in _pageData" ? --RheingoldRiver (talk) 09:31, 21 October 2019 (UTC)


 * Do you mean a separate column for each one? If so, then yes, a proposal of the specific fields you want would be useful. It's hard to imagine you really want seven different variations on the page name... Yaron Koren (talk) 18:49, 21 October 2019 (UTC)


 * Yeah a separate column for each. Because of inability to concat in joins a lot of random columns can end up being really useful. Is the "any magic word" (i.e. not just the ones in that section, but any magic word you can evaluate on the page) thing not practical to do? If so I'll go through and make a list. --RheingoldRiver (talk) 19:47, 21 October 2019 (UTC)


 * Well, "any magic word" doesn't really make sense, since most of the magic words don't apply to a specific page (like "SITENAME"). Yes, a list would be helpful. Yaron Koren (talk) 21:03, 21 October 2019 (UTC)


 * If it could be written in a general way though, then up to the user what they want, even if they could do something silly like store a row that's constant, that wouldn't be a problem right? I was thinking something like "wgCargoPageDataMagicWordFields" where you list the magic words you want, and then it evaluates them on each page to store to the table in those columns, and the name of the column could just be the magic word (maybe all lowercase instead of all caps). Is that possible? --RheingoldRiver (talk) 09:40, 22 October 2019 (UTC)


 * I don't know. But even if it's possible, it may not be worthwhile, if we're only talking about a few additional fields. It seems to me that, as far as the page name is concerned, there might be three additional fields that make sense: for a page titled "Help:A/B/C/D", it could be useful to have fields that store "A", "A/B/C", and "D". (It looks like those correspond to ROOTPAGENAME, BASEPAGENAME and SUBPAGENAME.) Yaron Koren (talk) 16:28, 22 October 2019 (UTC)


 * Okay, yeah I think those would be nice, along with NAMESPACE. Are the revision-specific ones possible, or would that run into the same issue categories have currently? In particular REVISIONUSER and REVISIONID I think aren't possible with existing _pageData fields. In particular for the revision properties, they would be useful for creating anti-vandalism (including accidental vandalism by user script (not that I ever need that.....)) tools, without having to use the recentchanges api which is unreliable if something goes unnoticed for a while - I'm not sure if this would open up completely new functionality that's currently impossible (other than by making multiple api calls), but it would definitely be a lot easier to simply run api cargo queries to get page lists. Part of the reason I was thinking an arbitrary list of magic words could be cool is that it would open up extension magic words too, but I can't think of any specifically I'd use immediately. --RheingoldRiver (talk) 21:27, 22 October 2019 (UTC)


 * The namespace is already stored via the _pageNamespace field in every regular Cargo table - is that good enough? REVISIONUSER could be useful - presumably it would be stored via some field like _lastEditor - but what would be the point of storing REVISIONID? Yaron Koren (talk) 02:48, 23 October 2019 (UTC)


 * Oh cool I didn't realize _pageNamespace was a thing, yeah that's for sure enough. Getting the revision ID seems pretty useful for external maintenance scripts, currently the only ways to get rev ids are either recentchanges or individually querying one single page. I doubt I'd ever use it on-wiki though, only through the cargoquery API. --RheingoldRiver (talk) 06:39, 23 October 2019 (UTC)


 * But wouldn't you need to join the revision ID in a query with some non-Cargo tables for it to be useful? Yaron Koren (talk) 14:17, 23 October 2019 (UTC)


 * No, I'd be using Cargo to discover revision IDs to use somehow, e.g. to rollback in a cleaning-up-problematic-bot-edits script. --RheingoldRiver (talk) 19:12, 23 October 2019 (UTC)


 * If it's from a script, couldn't you get the revision ID through a regular SQL query on the main MediaWiki database? (And actually, couldn't you get REVISIONUSER the same way?) Yaron Koren (talk) 19:45, 23 October 2019 (UTC)


 * I was thinking to join other Cargo tables to get to the revision ID + User. I don't have a specific script in mind that I'd use this for right now, but if I wanted to make an exhaustive list of what might be useful to have, I'd definitely include it. But I'm not sure how involved it is to add additional fields here, so I'd agree revisionid is probably the least likely to be useful out of everything and could definitely be left out if each field is significant work to make available. --RheingoldRiver (talk) 07:40, 24 October 2019 (UTC)


 * It's not significant work to add any such field, but there's a cost in terms of additional complexity of the software and documentation. Yaron Koren (talk) 16:19, 24 October 2019 (UTC)


 * Ok, I think it would be nice to have, but it's ok to skip that one. --RheingoldRiver (talk) 19:54, 24 October 2019 (UTC)


 * It sounds like the request here is to add fields called something like _rootPageName, _basePageName and _subPageName to the _pageData table. Is that correct? It would be a little strange to have some of the page name-related fields contained within regular Cargo tables, while others (these three) would go into _pageData, but maybe not that strange. Yaron Koren (talk) 15:41, 25 October 2019 (UTC)


 * Can you do those three and _lastEditedBy (REVISIONUSER)? I could definitely see printing last editor on a page, or even doing something like a gadget that shows the last editor of each page in someone's contribution history (actually I really want that now...). --RheingoldRiver (talk) 18:41, 25 October 2019 (UTC)

Error message about forbidden characters doesn't mention -
Unrelated to the above, someone sent me this screenshot just now, with - not included in the list of disallowed characters --RheingoldRiver (talk) 19:54, 24 October 2019 (UTC)


 * Sorry about that - I just checked in a fix for this. Yaron Koren (talk) 15:14, 25 October 2019 (UTC)

Empty cargo table creates error in cargoRecreateData.php --quiet
I had created a template/table for a project that did not pan out. I had never finalized it and it was creating errors on pages, so I figured I would just comment out the table declaration etc and all template calls. That worked as far as getting rid of errors on pages. However, the table still exists, even after deleting it from the Cargo table listing in special pages, and after running cargoRecreateData.php. Shouldn't a table no longer declared be removed by a recreate? The continued presence of the table raises an error during our nightly maintenance script: when we run cargoRecreateData.php --quiet we get "Table "xxxx" is not declared in any template." If that is not an error, then should it be output when running in --quiet mode? I am not sure if this is a bug that needs changes or not. If it is functionality as intended, then I need a way to remove that table so the recreate script stops giving an error. Would it break anything if I just deleted the table manually in the database? Tenbergen (talk) 15:11, 29 October 2019 (UTC)


 * Yes, it shouldn't be displaying that error message. If you go to Special:CargoTables, can you see the table there (and delete it)? Regardless, yes, deleting the table manually should also work fine. Yaron Koren (talk) 16:08, 29 October 2019 (UTC)
 * I tried to delete the table manually in Special:CargoTables. After that the script ran without the message. Thanks, Yaron!Tenbergen (talk) 01:48, 30 October 2019 (UTC)

Error when used with Approved Revs
Hello, I have "Approved Revs" installed and when hitting "Approve" for an edit I get an error: Notice:  Undefined variable: cdb in /opt/bitnami/apps/mediawiki/htdocs/extensions/Cargo/Cargo.hooks.php on line 245

Using: MediaWiki: 1.33.1, Approved Revs: 1.2, Cargo: 2.3.1 If Cargo is disabled the "Approved Revs" extension works as expected. Is this a configuration issue or something else?--Greg Hitchon (talk) 15:52, 18 November 2019 (UTC)
 * It seems like if you add  to the onARRevisionApproved function it resolves the issue.--Greg Hitchon (talk) 16:04, 18 November 2019 (UTC)
 * However this still seems like not the correct fix as when approving a page it gets deleted from the Cargo table. Any advice welcome! --Greg Hitchon (talk) 16:50, 18 November 2019 (UTC)


 * Sorry about the first problem - that's a bug from September. I just checked in a fix for it. I don't know what's causing that second problem, though. It doesn't happen for me - with this fix in place, Cargo data seems to get set correctly after every approval (and unapproval). Does the problem happen for you consistently? Yaron Koren (talk) 17:58, 18 November 2019 (UTC)
 * Thanks for the quick fix!!! Maybe it is due to having . If an approved rev exists and a new edit is made the "approved" cargo data is removed. The expected behaviour in this case would be for nothing to happen.--Greg Hitchon (talk) 19:03, 18 November 2019 (UTC)
 * Sorry for the multiple edits here (new to this so not sure of the etiquette/if this is the right place for this convo) Tried with  and had a similar result of the Cargo data being removed upon save. It is properly saved if I 'Approve' another revision --Greg Hitchon (talk) 19:11, 18 November 2019 (UTC)


 * The edits here are fine. And thanks for clarifying that the problem came about when doing an edit after an approval, not just the approval itself. I was indeed able to replicate that problem, and it turned out to be a bug due to a change in the Approved Revs code about a year ago. (I guess the combination of Approved Revs + Cargo is still not that popular, since no one reported this problem until now.) I just checked in another fix to the Cargo code - hopefully everything works now. Please let me know if there are any more problems. Yaron Koren (talk) 20:23, 18 November 2019 (UTC)
 * Thanks again for the quick fixes here! The update has fixed that issue generally however one reproducible issue is:
 * Start with a page with an Approved Rev
 * Make an edit (after the edit everything is fine)
 * Approve the latest approval (any other approval seems to work)
 * Cargo data is deleted
 * Little disconcerting that no one has reported the issue in 1+ years. It seems like those extensions fulfill our business needs perfectly (thanks for the work on them) so really want to get it sorted out. I can spend some time getting familiar with the code but am new to the php/wiki world so not sure how much help I will be off the bat here.--Greg Hitchon (talk) 01:17, 19 November 2019 (UTC)
 * Looked into this a little further. It looks like in a few cases the parser is not running after the options are set and so skipping adding data. If you manually trigger these then it seems to work:
 * onARRevisionApproved: Add  after the "CargoStore::settings..." line
 * onARRevisionUnapproved: Add  after the "CargoStore::settings..." line
 * Hope this helps! --Greg Hitchon (talk) 16:25, 19 November 2019 (UTC)

Yes, it's strange that these bugs had not been caught (or at least reported) until now - Cargo and Approved Revs both have significant usage. Maybe not together, though; I don't know. Anyway, sorry about that problem. And good sleuthing! I just checked in what I think is a fix for this bug, based in part on your suggested change. Yaron Koren (talk) 20:47, 19 November 2019 (UTC)
 * Great thanks again! Tested and can confirm this fixes the issue. Thought there was a similar issue with "unapprove" but can't replicate it so might have been something I had changed. Will hopefully put the 'Approved Revs'/'Cargo' combination through the paces and make it easier for others to adopt! --Greg Hitchon (talk) 20:59, 19 November 2019 (UTC)

Display format pie chart
Hi all, I am trying to build some pie charts with the display format=pie chart. The documentation, however, only mentions very few parameters. To allow customization, are there any more? I would be really happy to see parameters like these ones for jpplotseries in SMW, specifically I would like to customize chart clour and display of data labels (can't enter an external link, but for jgplotseries, they were called "datalabels" and "colorscheme"). Any hope?


 * Let me first note that I think pie charts are a bad idea, and you should use bar charts instead! I know Cargo has a "pie chart" format, but that's only because I was pressured into adding it. Of course, this same request could apply to bar charts too. I just wanted to note that. Anyway, I think you can set the data labels by just setting aliases for each field name - do you want something more than that? Setting colors (or a color scheme) is a reasonable idea, though. Which would you prefer - manual control of colors, or a color scheme? Yaron Koren (talk) 16:10, 20 November 2019 (UTC)


 * Hi, sure, bar charts would work too in one of my cases, though in the second case the focus is really on shares, so pie charts seem to be the better choice in principle. I see that the bar chart format automatically shows the data value next to the bar. I meant the display of data values (absolute, or perent), not data labels, sorry about that. But when choosing bar chart, I get a legend that says "Blank value 1". So, what I need would be (in that order of priority):

If you could get around to that, that would be wonderful. If you don't want to invest in the pie chart, I can make do with the bar chart only. Thank you so much! --MelanieUh (talk) 10:23, 21 November 2019 (UTC)
 * 1) Customized colour (better than color scheme) per value
 * 2) Suppress the legend
 * 3) Pie chart: Choose to display absolute or percentage data for each value
 * Hi Yaron, any chance you have this issue on your todo list yet and an idea when you might be able to do it? Much appreciated!

--MelanieUh (talk) 05:25, 3 December 2019 (UTC)


 * Sorry for the delay. I understand the first two requests - which could probably be handled with parameters like "colors=" and "show legend=". But can you explain the last one, with absolute or percentage data? Yaron Koren (talk) 18:18, 4 December 2019 (UTC)


 * Okay, having looked into it more, I now understand the last one as well - the NVD3 library lets you display the number or percentage of each "slice", instead of its name. My new question is: can there be any kind of intelligent defaults made for all these settings? Not for colors, but for whether to show the legend, and which text to show for the labels (or whether to show the labels at all). Like, for instance - if the legend is shown, the labels should be numbers, but if there's no legend, they should always be the names? And is there any logic for whether it should be numbers or percentages? It seems to me like absolute numbers are always better to show, since the pie chart itself is supposed to give a sense for the percentage. Yaron Koren (talk) 04:09, 5 December 2019 (UTC)


 * Hey! My suggestions for default would be: Legend=on, Labels= numbers (and can be changed to percent, name or turned off completely). If no legend, Labels can indeed be name AND number. Would it be complicated to have two things in the Labels parameter, separated by comma? In Excel I often see that the series name AND the number is shown, or number AND percentage, separated by comma. --MelanieUh (talk) 07:45, 5 December 2019 (UTC)


 * Okay, thanks. The defaults of "legends on, labels = numbers; legends turned off, labels = text" make sense, I think. Yes, you could have both text and numbers for the labels, but (a) it might get too crowded in the chart (unless it's huge), and (b) I generally trust the makers of the NVD3 library, who have dealt with a lot more use cases - they only set the options of name, number and percentage. Though come to think of it, the size set for the pie chart could affect the defaults as well... (My goal with Cargo is to make the defaults as "smart" as possible.) Yaron Koren (talk) 13:55, 5 December 2019 (UTC)

Cause of multiple rows for the same item
I've had problems occasionally where a row is repeated in a Cargo (2.3.1 - 58278ee) table and I think I found the cause. I have a Text field with:

If the user includes an asterisk (*) at the beginning of a line, it seems to cause the issue, removing the asterisk (or moving from it the beginning of the line) seems to solve the problem. — Bryandamon (talk) 01:40, 27 November 2019 (UTC)


 * That’s really interesting, as I still get duplicate rows, despite the problem having been solved for most people, and I fairly often have an asterisk (for a bullet point) at the start of a line in a textarea... Jonathan3 (talk) 10:58, 27 November 2019 (UTC)


 * Interesting indeed. Out of curiosity, are you adding those asterisk bullet points with mobile?  Have you tried removing (or moving) the asterisk and if so, did it fix your duplicate row? — Bryandamon (talk) 23:22, 27 November 2019 (UTC)


 * I nearly always edit using the PC. I have never been able to sit down and work anything out for sure. I have a MySQL query which identifies duplicate rows (incidentally, this might be something useful to have on the Cargo web interface) and when I notice duplicate rows I just run the command line script to recreate the database. Jonathan3 (talk) 23:55, 27 November 2019 (UTC)


 * I'm talking with Bryan about this right now, here's a reproduction of the duplicate rows issue - link - except using a nowiki tag instead of a bullet point. My immediate guess is this is a VE issue where bullet points are parsed later than the Cargo store that's causing the problem. I'm not sure how to fix that part, it sounds potentially really complicated to deal with, but there's definitely an issue where problematic tags cause duplicate rows at least. --RheingoldRiver (talk) 18:32, 4 December 2019 (UTC)


 * This patch would probably help in some of these cases - I really should just check it in. I don't know if it would help with that starting asterisk case, though... I don't know what would be causing that. Yaron Koren (talk) 23:25, 4 December 2019 (UTC)

Error with Cargo table
Recently I openedCargo extension on rs.miraheze.org. This is the first time I have used Cargo. However, after editing a template decaling Cargo, error happens. See Special:CargoTables and Template:关卡信息. And Special:CargoTables is currently empty. --SolidBlock (talk) 11:50, 5 December 2019 (UTC)


 * I don't know what's causing that error. You would need to get an admin to add "$wgShowExceptionDetails = true;" to the settings for your wiki to see the actual error message, if that's possible. Yaron Koren (talk) 13:47, 5 December 2019 (UTC)
 * I've informed miraheze stewards to do that. Thanks. --SolidBlock (talk) 02:47, 15 December 2019 (UTC)

populating _pageData
I have been finding data, especially about categories, not being in table _pageData where I would expect it. I realize that relying less on categories and more on cargo data would work around this, but I have some older wikis that used to rely more on SMW which worked ok with categories, so changing this retrospectively would take a fair bit of content change.

Something similar was mentioned here before, with a suggested solution, but no sign if something was done with it.

I would be prepared to add a  to our overnight maintenance script, but then the table still would need to be manually replaced in Special:CargoTables.

Should it be necessary to run setCargoPageData.php regularly, or is there another solution? An if setCargoPageData.php, then how do I automate the "replace"?

Actually, I just realized a second issue: I guess the --repacement should deal with the replace, but for me does not: the table gets generated, but I have to manually replace it.


 * Right, this issue hasn't been fixed yet - it would require making use of some other MediaWiki hook to update just the "categories" field. Until then, a cron job is probably the way to go, sadly. The "--replacement" option deliberately doesn't replace the table at thend, in case the admin wants to double-check the new data first, but it does make sense to add an option to actually do the replacement, like "--doReplace". Yaron Koren (talk) 16:41, 12 December 2019 (UTC)


 * I would have expected the "--replacement" switch to cause the replacement to happen. What happens if I run  without the switch, then, it doesn't seem to do the update either.

If replacement is currently not available via command line switch, then is there a way to run whatever the URL  does from the command line? I could easily schedule a second line, but if it actually has to go to the URL then I would also have to manage login sessions and that quickly gets out of my knowledge range... Tenbergen (talk) 18:21, 12 December 2019 (UTC)


 * Alright, you convinced me to actually fix this issue. I just checked in to Cargo a patch that uses hooks to save (hopefully) the correct category data after every page save. If you can, please try out the latest code and let me know if it works for you! Yaron Koren (talk) 18:05, 13 December 2019 (UTC)

"In the future" and "In the past" filters on Special:Drilldown
I would find it useful to have these two options, for dates, in addition to the usual grouped date ranges. Would this be possible? Thanks. Jonathan3 (talk) 19:37, 11 December 2019 (UTC)


 * It's an interesting idea. I wonder if this kind of thing can be extended in some way... is there any other special behavior that would be useful for date fields where you want this kind of filtering? Email notification when they occur? Email reminders beforehand? Color-coding within tables, depending on whether they're in the past or future? Yaron Koren (talk) 16:48, 12 December 2019 (UTC)


 * Thanks. It would be useful for events, or anything with a closing date, for example, if the URL parameters could ensure only events "in the future" are displayed. It would be good if (by default, without that option used) the past and future events were displayed differently - this could be an option when the table is declared, though that might overcomplicate things. Jonathan3 (talk) 00:48, 14 December 2019 (UTC)

Error: operator for the virtual field must be "HOLDS" etc., if field name in WHERE clause and field value are the same or nearly the same
Hi everybody,

I face the following problem.

In a field named "country" I store a list of strings. The values to be stored are "country-1", "country-2", "country-3" and "abc" (abc is just for testing purpose)

Now, I want to query the database for all projects which hold for instance "country-1". The query looks like this:

Regrettably, I get this error message, although I used "HOLDS" as indicated

Error: operator for the virtual field 'webmo_project.country' must be 'HOLDS', 'HOLDS NOT', 'HOLDS LIKE' or 'HOLDS NOT LIKE'.

But, if I query for the test value "abc" everything works fine and I get the expected results from the query

I guess, that using similar names for the field name "country" and the field values like "country-1" is maybe causing this problem. Could this be a bug?

--Shuitavsshente (talk) 09:54, 12 December 2019 (UTC)


 * Might the problem be the hyphens in the field names? Jonathan3 (talk) 13:04, 12 December 2019 (UTC)


 * No, there are no hyphens in the field names - just in the string being looked for. Yes, I think this is a bug, due to overly-aggressive validation in Cargo. I can think of two ways to get around it: one is to change the query to "country__full LIKE '%country-1%'" ("HOLDS" won't work there), and the other is to rename the "country" field - to "country_name" or anything else. Yaron Koren (talk) 14:14, 12 December 2019 (UTC)


 * Thank you very much for the quick response. Regrettably, switching to HOLDS LIKE '%country-1%' does not work either. But somehow a funny fact: I adjusted the where-clause to "country HOLDS LIKE '%ountry-1%' (that means, I have deleted the leading "c" of country) and afterwards it is working fine. Seems to affirm my initial assumption, that similar wording for field names and field values causes this problem. --Shuitavsshente (talk) 15:13, 12 December 2019 (UTC)


 * Oh, right, that would work too. I think you misunderstood my first suggestion, but as long as it works, there's no problem. Yaron Koren (talk) 16:28, 12 December 2019 (UTC)


 * "No, there are no hyphens in the field names - just in the string being looked for" - Sorry - I typed my message on the phone and hadn't read the question properly. Jonathan3 (talk) 00:43, 14 December 2019 (UTC)

Error handling stores duplicate rows
I think there's some kind of error handling that leads to duplication of rows. Here is a reproduction of one where a floating point is tried to stored in an Integer field, and it ends up storing the row twice. I think there might be some handling where it thinks it wasn't stored in the first place and then goes back and re-stores it again later on (given the order that the stores are done in).

I'd propose a new default column added to every table that's called _warnings, or maybe 2, called _warningCodes (List of String indexed) and _warningText (not indexed) so you can query codes and also read full text. I'd definitely be interested in helping to write this though I think I'm pretty far away from understanding the overall extension to do it soon. But, in the meantime, if you know what might be going on to cause these duplicates, I think that would be nice to fix (maybe it could, for now, just enter the page into an error category kinda like Max Loops Exceeded that the Loops extension has, so that it's not failing silently, and also not duplicating rows). --RheingoldRiver (talk) 22:34, 12 December 2019 (UTC)

Oh, I also forgot to say, that reproduction did not work when I had only 2 columns in the table, see: here --RheingoldRiver (talk) 01:21, 13 December 2019 (UTC)


 * I'm guessing that's due to the fact that, e.g., a value of "1.97" is being compared to "2", the code sees they're not the same, and assumes that it should be a separate row. I'll have to add in a fix for this, along the lines of the "check the substring" patch that I also need to check in. :( I don't understand the "warning" stuff... is that related to this duplicates problem? And would you still want those added, if this specific issue is fixed? Yaron Koren (talk) 18:09, 13 December 2019 (UTC)


 * So the point of the warning stuff is like, atm the duplication only happens when there's an actual problem with the input, e.g. I was trying to save a floating point in an Integer column. Currently the only way to discover problems like this is to notice the duplication and fix it. So if the duplication is gone (which I really think it should be, even if it leaves us without any discovery method), then we'll just have inputs silently failing and possibly never getting noticed. In a lot of cases this won't matter, but I think there should at least be SOME ability to detect warnings. So I was thinking one of the better ways might be to add columns to each row for the purpose of discovering errors - if there's longer explanations those can be in a text-type field called _warningTexts, and then, in order to make queries on types of errors, there can be a List of String (or Int) column called _warningCodes, or maybe _warnings and _warningCodes or something. Then stuff will work as best as it can without adding incorrect duplicate lines, and for the most part stuff will "just work", but then there's also a discovery method present for locating issues.


 * An (imo worse) alternative to putting warnings directly into the rows would be to have a cargo_store automatically populate categories on the page with the types of warnings - but I think this isn't as good an option because you don't get specificity to the individual line of where stuff went wrong. --RheingoldRiver (talk) 03:19, 14 December 2019 (UTC)


 * Well, this sounds like two separate, and basically unrelated, discussions. I'm pretty sure that the duplication problem can be fixed in one way or another. But the handling of un-allowed values is an interesting question. There already is handling of error values in the case of "enumeration" fields - fields that have a specified list of allowed values. For those, values that aren't in this list are simply ignored - not stored at all, while everything else gets stored normally. For these and other cases (like trying to store a float in an integer field, or storing a too-large text in a string field), there are a number of possibilities: (a) ignore the bad data (as is sometimes done now), (b) don't save that row at all, (c) try to "massage" the data as best as possible (like rounding numbers, and truncating strings), (d) display an error message on the screen, (e) add the page to one or more special categories, (f) add one or more fields to every Cargo table to store the precise error information. And in theory, I think any of (a), (b) and (c) can be done with any of (d), (e) and (f). What do you think? Yaron Koren (talk) 04:27, 16 December 2019 (UTC)


 * I think in general they are related just because it would be good to push both updates at the same time, or at least the error discovery one first, so that discovery isn't lost after duplication is fixed. But in practice for writing code they're probably unrelated, yeah. Those options sound like all of the options, yeah. I think if (c) is done, that should be settable via a preference, so there's an option to have an actual type check implemented, so (c) with a fallback to (a) in the case that either the data can't be fixed or that preference is set to "don't attempt to save anyway" (atm I think it's maybe sometimes doing this? Since the floating point saved as 0 when there were few columns in the table but then broke everything when there were more columns? Does that make sense for how the code is?) For reporting, maybe doing both (e) and (f) would work - people can choose to HIDDENCAT the reporting category(ies?) if they don't care to be notified of warnings, but it'll still be there to check for issues. I think a category is a lot better than a message to a page, since you can easily see all pages in the category, also that makes it easily accessible via api. --RheingoldRiver (talk) 07:50, 16 December 2019 (UTC)


 * What about (b) (don't save row), combined with (d) (error message on screen) and (e) (error categories)? I think we're better off knowing that what we have typed isn't being entered into the database exactly at the outset (so not (a) or (c)). Categorisation would be useful for dealing with errors later (e.g. time constraints, or maybe for the administrator to deal with errors that editors have ignored). This would probably make it unnecessary to have separate table rows for error messages (so not (f)). Jonathan3 (talk) 00:06, 17 December 2019 (UTC)


 * Thanks for the feedback. I actually thought of another option: (g) have a separate special Cargo table, called "_storageErrors" or something, which just held the errors for bad data. It could have fields like _pageID, _tableName, _rowID (only if the data is getting saved, though), _fieldName, _value, _errorID - something like that. Does that sound appealing? Yaron Koren (talk) 00:18, 17 December 2019 (UTC)


 * I'm pretty opposed to on-screen errors tbh, Loops extension does that and it's the worst, I ended up changing my i18n for it to print one word as LoopsErrorOnThisPage so I had that as a search string for discovery (can't do regex cos it's not insource!), which is a ridiculous workaround and imo shouldn't ever be an intended way to do things. Printing error messages is more likely to confuse users & make them scared to do edits in the case that they aren't able to do anything, and users who ARE able to do things can just check error population pages. A category is a much better option imo. Regarding a separate table, hm...that could also be possible to create it like _pageData, and then it's possible to do a display with all warnings across all tables instead of needing to enumerate for each table. The first reservation I have is that it might not be clear how to join the two together, since you need both a join with _rowID and also a where (to ignore identical _rowID but from other tables), but I think that could be provided in documentation. And the other concern is, would this write all of the lines all at once in the case there are errors/warnings? Or could this potentially double save time for a page? --RheingoldRiver (talk) 00:37, 17 December 2019 (UTC)

I thought more about this, based in part on all the feedback. Here are some of my thoughts: Any thoughts? Yaron Koren (talk) 17:39, 17 December 2019 (UTC)
 * The category system, (e), doesn't seem "granular" enough. If there are 100 fields on a page, and the page gets added to a category called "Pages with an invalid value for an integer field", how helpful is that to users?
 * The original idea of adding fields like "_warningCodes", which I listed as (f), similarly doesn't seem granular enough, surprisingly. Again, if there are dozens of fields in a table, and the _warningCodes lists some error codes for that row, there's no obvious way to know which fields/values those apply to.
 * I guess that leaves a dedicated DB table, (g), as the error storage solution. It's true that doing joins on it would be tricky, but it's doable - especially in PHP or Lua - and I don't see an alternative.
 * I still think having onscreen errors, (d), could make sense. I don't see error messages as necessarily confusing or scary, if they're clearly written and displayed in neutral colors. Something like "Field 'Number_of_employees' in table 'Company' must be an integer" is fairly clear, and not so different from the error/warning tags that appear at the top of some Wikipedia articles ("The neutrality of this article is disputed").
 * As for the saving of the data, I think I agree with RheingoldRiver's suggested approach of "(c) with a fallback to (a)". (b), or not saving the row at all, seems too extreme - especially if the value can't easily be changed. (Though I will note that (b) would make the row duplication problem easier.)