Extension talk:Cargo

Self-Inflicted Duplicate Issue
Putting the solution at the top! Don't do this: My MediaWiki Page -> My Template #1 -> Cargo Query using Template #2 for display -> Template #2 stores Cargo data. Re-using Template #2 caused rebuilding of the associated Cargo database to pick up on My Mediawiki Page as something to include in Cargo despite the exact same data being present in Template #2. Thanks to Yaron Koren for patiently debugging my recursive template usage issue.

Original problem follows... This isn't a recurrence of the duplicate rows issue that cropped up in 2021. I am 99.9% sure it's self-inflicted.

Steps
 * 1) Import page data from Excel into MyCargoDB.  All pages are prefixed with CARGO ("CARGO_MyPageName") because they're data, not to be looked at directly by readers.
 * 2) The template MyTemplate queries data pulled into Cargo from the imported CARGO pages.
 * 3) Page called MyPageName uses MyTemplate to display data pulled into Cargo from CARGO_MyPageName.
 * 4) Rebuild MyCargoDB.
 * 5) Now MyPageName using MyTemplate displays Cargo data twice.
 * 6) The view in Cargo Tables of MyCargoDB shows CARGO_MyPageName AND now MyPageName which explains why it's appearing twice.
 * 7) If this keeps up I'll have 150 duplicates.

I'm not sure what mistake I've made to create this circular import into Cargo.

SovereignGFC (talk) 15:06, 2 May 2023 (UTC)


 * That first step sounds odd - are you directly creating a table within the database? If so, you shouldn't do that - the data stored by Cargo should always reflect what's in the wikitext. Could that be the issue? Yaron Koren (talk) 02:56, 3 May 2023 (UTC)
 * Apologies, I am not directly creating a table. I import through an extension that creates wiki pages from said Excel sheet that match MyTemplate.
 * After importing, I immediately execute runJobs.php.
 * I then Recreate data on MyCargoDB.
 * If any MyPageName using MyTemplate that got put into MyCargoDB exists at that point, MyCargoDB gains an entry from MyPageName even though MyCargoDB was already populated by MyTemplate. SovereignGFC (talk) 22:02, 3 May 2023 (UTC)
 * Okay, never mind. Well, it's strange that pages that look simply like "MyPageName" are showing up as holding Cargo data - even though they're meant to only display, not store, Cargo data. What exactly is in the template "MyTemplate"? Yaron Koren (talk) 14:30, 4 May 2023 (UTC)
 * MyTemplate is NOT a Cargo-generated template. It is an actual page template that DOES contain Cargo queries using templates created by Cargo's Create a class. SovereignGFC (talk) 01:11, 5 May 2023 (UTC)
 * Well, does MyTemplate contain a call to #cargo_store? And if not, how are the pages that call it getting Cargo data stored for them? Yaron Koren (talk) 17:32, 5 May 2023 (UTC)
 * MyTemplate does not have calls to #cargo_store.
 * MyPageName -> MyTemplate -> Cargo query in MyTemplate -> Separate Cargo-specific pages/templates referenced in query. SovereignGFC (talk) 18:29, 5 May 2023 (UTC)
 * Alright. Well, barring more information about your setup, I'm totally stumped - I have no idea how pages like MyPageName could show up in Special:CargoTables. Yaron Koren (talk) 18:43, 5 May 2023 (UTC)
 * What additional information might be helpful? SovereignGFC (talk) 00:50, 6 May 2023 (UTC)
 * The contents of MyPageName and MyTemplate, I suppose. Yaron Koren (talk) 13:29, 7 May 2023 (UTC)

Header in dynamic table
How to change header in dynamic table?

Tamplate:

Table header looks like this:

Thanks! -- Oleksii 212.80.47.83 06:58, 6 May 2023 (UTC)


 * Add something like "=ColumnName" after the final ")" - i.e., set an alias for the field. Yaron Koren (talk) 13:30, 7 May 2023 (UTC)
 * how to display format=dynamic table in Lua?
 * the word 'format' seems to be a Lua reserved word
 * --Oleksii 212.80.47.83 15:55, 7 May 2023 (UTC)
 * There might not be any way to display result formats from within a Lua Cargo query - I'm not sure. Yaron Koren (talk) 02:42, 8 May 2023 (UTC)
 * It looks like maybe Lua wouldn't mind you using "Format" (see https://stackoverflow.com/questions/5530039/make-a-reserved-word-an-identifier-variable-in-lua) - but Cargo seems to want "format". Jonathan3 (talk) 14:24, 29 June 2023 (UTC)
 * It looks like maybe Lua wouldn't mind you using "Format" (see https://stackoverflow.com/questions/5530039/make-a-reserved-word-an-identifier-variable-in-lua) - but Cargo seems to want "format". Jonathan3 (talk) 14:24, 29 June 2023 (UTC)

WITHIN statement error when a Hierarchy item has an apostrophe
I have a hierarchy definition in a template that includes the item "St. Mary's Anglican Church". When I try to query this with a WITHIN statement, I get an error that appears to split this at the apostrophe: WHERE Collection IN ( SELECT `_value` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_left` >= ( SELECT `_left` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_value` = "St. Mary' ) AND `_right` <= ( SELECT `_right` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_value` = "St. Mary' ) )s Anglican Church") ORDER BY `cargo__Collection`.`_pageID`,`cargo__Collection`.`_pageName` . When I put double-quotes around a string with an apostrophe in a WHERE clause this seems to work fine. Is this a bug or am I writing the query wrong? Thanks! Tenbergen (talk) 04:10, 9 May 2023 (UTC)


 * Sorry for the delay. That's definitely a parsing bug in Cargo. Yaron Koren (talk) 12:54, 7 June 2023 (UTC)

Recreate and 500 Line Bug Again

 * MediaWiki 1.39.3
 * Cargo 3.4.2

When I kick off re-create data from the Cargo tables page, it gets stuck at 500 lines.

When I import a spreadsheet to that same table/template with 500+ lines and execute runJobs.php, the table populates.

What additional diagnostic or debugging data can I provide? SovereignGFC (talk) 03:40, 13 May 2023 (UTC)


 * Sorry, I missed this before. Do you see any errors in the browser's JavaScript console? Yaron Koren (talk) 02:37, 29 May 2023 (UTC)
 * Now that the backup function (export to Excel which keeps my special character filled text intact) works thanks to the phpspreadsheet update, I was able to test this.
 * Open browser dev/debug tools.
 * Recreate data link on template page, Cargo view all tables page (line of the table), or Cargo table (table to be recreated).
 * ...no errors detected. The command to recreate data just returns a 200 OK.
 * Invoke runJobs.php. Observe that only 500 rows exist.
 * SovereignGFC (talk) 01:32, 2 June 2023 (UTC)
 * If you run cargoRecreateData.php from the command line on that table, does that work? Yaron Koren (talk) 13:09, 7 June 2023 (UTC)

Vertical list of names be grouped by the initial letters
Sorry if my question has already been asked here, but I have just started using Cargo. How could a vertical list of names (from Cargo databases) be grouped by the letters of the alphabet using a Cargo query? Lalo5555 (talk) 08:49, 27 May 2023 (UTC)


 * If you use the "category" format, it will provide a display that's hopefully close enough to what you are looking for. Yaron Koren (talk) 02:36, 29 May 2023 (UTC)

Export as Excel, Composer, What Did I Miss?
(This is preventing me from properly testing the 500-line recreate bug as I want to back up my big table beforehand!)

When I try to export as Excel, I get a 500 internal server error when I click on the link to download. Unfortunately, even with MediaWiki debug on, that's all I do get: HTTP/1.1 500 Internal Server Error Date: Wed, 31 May 2023 02:41:54 GMT Server: Apache X-Powered-By: PHP/7.4.33 X-Content-Type-Options: nosniff X-Request-Id: ZHaz8pqI3nmiV89-O7ng8wAAAss Strict-Transport-Security: max-age=63072000; includeSubDomains X-Frame-Options: SAMEORIGIN Content-Length: 0 Connection: close Content-Type: text/html; charset=UTF-8

I have Composer installed at the same level of directory as MediaWiki and I confirm that phpoffice/phpspreadsheet is installed (1.19.0). I know it works because I can import spreadsheets just fine...but not export for some reason. SovereignGFC (talk) 02:46, 31 May 2023 (UTC)


 * The "excel" format code is very old, and it actually uses the PHPExcel library, not PhpSpreadsheet. Yaron Koren (talk) 13:04, 31 May 2023 (UTC)
 * Inspired by this discussion, I just added support for PhpSpreadsheet to the "excel" format. At least, in theory - I haven't actually tried running it. But there's a chance that it works! Yaron Koren (talk) 15:54, 1 June 2023 (UTC)
 * Can confirm this works!
 * I did the very proper and sophisticated thing of "copy entire single changed file and paste text into existing version" before running the query.
 * All 800+ items exported without issue. SovereignGFC (talk) 00:51, 2 June 2023 (UTC)
 * That's great to hear! Yaron Koren (talk) 02:44, 2 June 2023 (UTC)
 * One item of note: It defaults to the XLS extension, not XLSX. Excel 2021 complains the file doesn't match the extension and it may be a security issue.
 * I'm guessing that either specifying XLS format and keeping the XLS extension or changing it to use XLSX would make this nag go away. SovereignGFC (talk) 12:12, 5 June 2023 (UTC)
 * Ah, good point. XLSX seems to be the superior format, so I just changed the default filename to be "results.xlsx". Yaron Koren (talk) 15:03, 5 June 2023 (UTC)

Remove unwanted comma from results if fields are empty
I have this table with the fields ImageText and ImageFile being optional:

If ImageText and ImageFile have values, they display framed images with captions using this query:

This works fine, but how do I remove the commas from the display when ImageText and ImageFile are empty? Tahoma403 (talk) 04:22, 7 June 2023 (UTC)


 * It could that just adding  to the #cargo_query call will fix the problem. Yaron Koren (talk) 13:10, 7 June 2023 (UTC)
 * That removed,  ,  ,  ,  ,   from the top of the results, but in between each  , I still get one  ,   for some reason. Any idea? Tahoma403 (talk) 06:23, 8 June 2023 (UTC)
 * I think that's just due to MediaWiki's own display of images, though I could be wrong. Have you considered using the "gallery" format for this data, by the way, instead of assembling wikitext? It might be a nicer-looking output. Yaron Koren (talk) 12:32, 8 June 2023 (UTC)
 * Yes, that must be the reason then. I have looked at format=gallery before, but couldn't figure out where to put ImageFile in the query - would you mind showing where/how to put it in the fields= parameter to display the image and caption? Tahoma403 (talk) 13:09, 10 June 2023 (UTC)
 * Well, now that I think about it, to use "format=gallery", you would first need to change the type of "ImageFile" from String to File in the #cargo_declare call. Once that is done (and the table is recreated), a simple query like this might work: . Yaron Koren (talk) 13:13, 13 June 2023 (UTC)

Output as RSS (or Atom or whatever)?
I'm hoping to create RSS/whatever from the output of wikitext. My first thought was that Cargo might do that, so I checked the documentation pages, but didn't notice anything about it. So, I asked about it on the main MediaWiki support desk (here). One of the suggestions was that Cargo can do it. So I looked around here some more, but still haven't been able to find anything about it.

Can Cargo do this? If so, could you please point me towards info on how to make it do so? Thanks! -Rwv37 (talk) 03:50, 19 June 2023 (UTC)


 * It's a new feature, and hasn't been included in a release yet. If you're running the master version of Cargo, then you can do  (or  ) in a query, to get a link to an RSS feed. Sam Wilson 06:12, 19 June 2023 (UTC)
 * I meant feed (the type can be specified with atom). Sam Wilson 07:25, 19 June 2023 (UTC)
 * @Samwilson: Great - thank you! Rwv37 (talk) 22:04, 19 June 2023 (UTC)
 * This is great news. Thanks for working on it. Jonathan3 (talk) 13:52, 20 June 2023 (UTC)


 * Just for the benefit of anyone else who wants to try this out: It seems like you probably need to use MediaWiki 1.39 or higher. I'm using 1.38, and I get an error indicating that it can't find the class "MainConfigNames", which (it looks to me) was added in 1.39 -Rwv37 (talk) 04:13, 21 June 2023 (UTC)
 * That's odd. was added in 1.38, so it should be working. Sam Wilson 04:51, 21 June 2023 (UTC)
 * @Samwilson, I see that that page says 1.38, but: I'm largely ignorant about the internals of MediaWiki, and maybe I'm misinterpreting something, but is it possible that's just a mistake? I ask for a few reasons:
 * (1) When I actually try it, I get an error saying it can't find that class.
 * (2) This page (for MainConfigNames.template) says 1.39+.
 * (3) Also on that page, there are links for its source code for various MW versions; if you click on the 1.39 one, you're brought to code, but if you click on 1.38 or earlier, you're brought to a "no such file" page.
 * (4) My 1.38 installation does not have any file of that name (or any *MainConfigNames* file, case insensitive).
 * -Rwv37 (talk) 19:25, 21 June 2023 (UTC)
 * You're quite right! That file was introduced in and is not included in 1.38 (but only 1.39 and above). Here's a patch for Cargo:  ( could you have a look?), and another one for bumping the @since for MainConfigNames: . Sam Wilson 22:38, 21 June 2023 (UTC)
 * @Samwilson Thanks! I'm not sure of something, though: If I check out the latest git master branch now (as per the installation instructions), will this patch be included? Or would it have to wait for @Yaron Koren to merge it in or something? Thanks again. -Rwv37 (talk) 04:00, 23 June 2023 (UTC)
 * Yep, Yaron has already merged 932020 so if you pull the latest master now then it should be okay for 1.38. Sam Wilson 04:03, 23 June 2023 (UTC)
 * @Samwilson Just got a chance to give the latest a try with 1.38 - only a little bit, and I'll definitely be playing around with it more, but it seems to be working as far as I've checked! I have noticed one minor bit of seeming weirdness, but it may be due to how I'm using it. I'll eventually open up a new section on this page for it if I can't figure it out myself.
 * Thanks again! -Rwv37 (talk) 01:10, 29 June 2023 (UTC)
 * Great! Let me know. I wonder if the weirdness is related to timezones… :-P Sam Wilson 01:31, 29 June 2023 (UTC)

How is the  set? It looks like it's just using the URL of the page (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cargo/+/888110/18/includes/formats/CargoFeedFormat.php#125) I seem to remember having a problem with the old "BulletFeed" extension that RSS readers wouldn't know which were new items. Sometimes my items were different but shared the same URL. I ended up editing my copy of it to specify that  was the URL plus a URL parameter based on the. Jonathan3 (talk) 10:59, 21 June 2023 (UTC)


 * Yes, that's right, it's the URL of the page, but can be overridden by a query's  field. I think perhaps it should use the revision ID instead, as the default link? But that doesn't look as nice for consumers of the feed. I wondered about using   but then it might not link to the current version. Open to suggestions either way! :-) Sam Wilson 11:07, 21 June 2023 (UTC)
 * I think people are suspicious of URL parameters that aren't obvious (are they tracking? etc) so I used something clearly based on the date and time. The  and   are the same - I can't remember why I didn't have the date/time only in the   - maybe that would just have entailed more work on my part. I had to update the BulletFeed extension most times I upgraded MediaWiki. It bases its RSS feed on Cargo query results, so it will be good to cut out the middle man! I'll experiment with Cargo once I've upgraded to 1.38 and got confirmation that it works with that version :-)
 * The default you have is probably all right for most people. Especially as the  parameter which seems to set both   and   can be modified in the Cargo call. That might be enough for me too. Jonathan3 (talk) 12:35, 21 June 2023 (UTC)
 * I've been wondering a bit about this, and think that perhaps the  should be the canonical page URL, and the   should use the revision ID (e.g.  . That way, consumers know that the item is 'new' and will re-fetch it. Because often the publication date won't be updated even if the content is. Sam Wilson 01:30, 29 June 2023 (UTC)
 * That sounds good. In my case the RSS  is saved as a Cargo date/time field, and is when the page was originally created. I can't remember why I did this but probably to get round a weakness in the original BulletFeed extension. How will Cargo set it in the RSS feed output format - page creation date or something else? Jonathan3 (talk) 11:09, 29 June 2023 (UTC)

Duplicate results in Special:Drilldown
Have you had a chance to look at this yet (Extension_talk:Cargo/Archive_January_to_April_2023)?

It would be great if you could fix it. I'd be happy to try to help. I came up with an idea on the Phabricator task but couldn't get any further than that with it. Basically I wondered whether a  could be added to the Drilldown page's query.

Thanks. Jonathan3 (talk) 12:43, 21 June 2023 (UTC)


 * Sorry for the long delay in responding to this. Yes, I'm aware of the problem, and yes, adding something like "GROUP BY" or "DISTINCT" is probably the right solution. My initial thought was that, if there are multiple items, i.e. template calls, on a page that match the current set of filters, it could be helpful for users to see that there are multiple matches. But that's probably silly. Yaron Koren (talk) 21:11, 22 June 2023 (UTC)
 * Yes, I think your second thought is correct :-) At most it might be useful to have a number in brackets after the page name to indicate how many times it meets the various filter values, but I wouldn't find that useful myself.
 * Where does the MySQL query used on the Drilldown page come from? I could experiment with changing it. Jonathan3 (talk) 21:16, 26 June 2023 (UTC)

Page name search on Drilldown page
There is a "Full text" search available, but it's for the contents of the pages rather than the page names. Is it possible to add a page name search? Thanks. Jonathan3 (talk) 21:14, 26 June 2023 (UTC)

Hide some tables from Drilldown page
I've worked out how to do this with nth-child.

But it would be nice if each table tab header had a unique css id. Could you possibly add this?

Thanks. Jonathan3 (talk) 21:57, 27 June 2023 (UTC)

[SOLVED] Remove "Choose a table:" from Drilldown page
This comes from (cargo-drilldown-choosetable)(colon-separator).

I can blank MediaWiki:cargo-drilldown-choosetable but don't want to do that with the colon-separator in case it's used elsewhere in the site.

Is CSS the only way round this? Thanks. Jonathan3 (talk) 20:27, 28 June 2023 (UTC)


 * Probably. Yaron Koren (talk) 00:12, 4 July 2023 (UTC)
 * Cheers. I used  Jonathan3 (talk) 00:26, 4 July 2023 (UTC)

"Order by" in Drilldown tabs?
I've tried this but it doesn't work. Should it?

Jonathan3 (talk) 20:40, 28 June 2023 (UTC)

I'd like to show the results in date order (newest first). Jonathan3 (talk) 20:45, 28 June 2023 (UTC)

I think ORDER BY wouldn't work because the tab display code will rely on whatever rows are given it by the main Drilldown code. I can get it to order by Date by putting Date as the first field. But then it's ordered Oldest first, which is the opposite of what I want :-) Is there any way round that maybe? Jonathan3 (talk) 20:54, 28 June 2023 (UTC)

Custom text at bottom of Special:Drilldown
Would you consider adding a MediaWiki: message to go at the bottom of the drilldown page?

After this part:

Thanks. Jonathan3 (talk) 13:54, 29 June 2023 (UTC)

Database Error: Unknown column 'tl_namespace' in 'where clause'
I was trying out both SemanticMediaWiki and Cargo on my wiki. After trying each one I removed them. I decided to go with Cargo and tried to reinstall it again. It seemed like it went well until I tried to create a data table.

The error is saying "Did you forget to run your application's database schema updater after upgrading or after adding a new extension?" which I have already ran php update.php.

The full error is:

Error 1054: Unknown column 'tl_namespace' in 'where clause'

Function: SpecialCargoRecreateData::getNumPagesThatCallTemplate

Query: SELECT COUNT(*) AS total FROM `page`,`templatelinks` WHERE (tl_from=page_id) AND tl_namespace = 10 AND tl_title = 'Person'

Any suggestions on how to overcome this error? 70.160.223.43 22:43, 3 July 2023 (UTC)


 * What version of MediaWiki and Page Forms are you running? Yaron Koren (talk) 00:10, 4 July 2023 (UTC)
 * I typed my reply before reading yours! :-) In my head I still was on the Support Desk page. If I'd realised I'd have left the replying to you. Jonathan3 (talk) 00:19, 4 July 2023 (UTC)
 * Thanks, but I just got it back to work. I'm using MW 1.39.3 and when I downloaded Cargo I used Special:ExtensionDistributor/Cargo for 1.39 (but I think that version is what was messing it up). When I removed everything and downloaded from the link on Extension:Cargo/Download_and_installation the error was gone.
 * It seems the tables that first version of Cargo had was for MW 1.38 Manual:Templatelinks_table. 70.160.223.43 00:19, 4 July 2023 (UTC)
 * While I'm here, I should predict that Yaron is going to say that he doesn't recommend downloading from Extension Distributor and you should just run the latest code of his extensions :-) Jonathan3 (talk) 00:22, 4 July 2023 (UTC)
 * You predicted well. :) Yaron Koren (talk) 00:46, 4 July 2023 (UTC)
 * Something similar here:.
 * This seems to be the relevant Cargo code: https://github.com/wikimedia/mediawiki-extensions-Cargo/blob/master/includes/specials/SpecialCargoRecreateData.php#L168
 * What versions of MW and Cargo are you using? Jonathan3 (talk) 00:17, 4 July 2023 (UTC)

API Get Page Values
Using MediaWiki's API, I'd like to look up the page values for a specific page. I'm looking at the list of Cargo's APIs but I'm not finding which one to use. 70.160.223.43 01:48, 4 July 2023 (UTC)


 * I don't think there's a way to get all the Cargo values, across all of the tables, for a page via the API, unfortunately. Yaron Koren (talk) 14:52, 5 July 2023 (UTC)
 * Thank you very much. For anyone that comes in after looking for the same type of setup, I ended up having my script directly reading from the database tables. I know its probably not good practice or recommended for the vast majority, but it seems like it will work perfectly for me. 70.160.223.43 00:51, 6 July 2023 (UTC)

Duplicate entry 'XXXX' for key 'PRIMARY'
I'm getting a lot of duplicate errors for for my tables. I'm running MW 1.39.3 with Cargo 3.4.2.

To troubleshoot I've set $wgJobRunRate = 0; and I don't have any cronjobs running runJobs.php.

Each time I've gotten the error I've deleted the table using Special:CargoTables and then I recreated the table on Template:XXX.

To populate the table I'm using php cargoRecreateData.php.

Sometimes it's able to finish successfully, or sometimes it gets a fatal error for duplicate entry.

Is there something I can do to prevent this? How should I proceed each time this happens? And is it possible to change the code to an INSERT IGNORE instead of an INSERT command to prevent having that error be fatal? 70.160.223.43 21:15, 4 July 2023 (UTC)


 * There was actually one change to the Cargo code, in May, after the release of version 3.4.2, that might help with this problem - which reminds me that I should release a new version of Cargo. If you are using the actual version 3.4.2, from April, I would recommend upgrading to the very latest Cargo code, to see if that fixes the issue. If that doesn't work (or if you were already using more recent code), another option to try is to add the following to LocalSettings.php:  . Yaron Koren (talk) 14:50, 5 July 2023 (UTC)
 * I used the zip link on Extension:Cargo/Download_and_installation which I downloaded a few days ago. Is there a way I can check to see if I have the correct version?
 * For $wgCargoIgnoreBacklinks, would there be any downsides or things to be concerned of if I set this to true? 70.160.223.43 00:55, 6 July 2023 (UTC)