Extension talk:Cargo

Self-Inflicted Duplicate Issue
Putting the solution at the top! Don't do this: My MediaWiki Page -> My Template #1 -> Cargo Query using Template #2 for display -> Template #2 stores Cargo data. Re-using Template #2 caused rebuilding of the associated Cargo database to pick up on My Mediawiki Page as something to include in Cargo despite the exact same data being present in Template #2. Thanks to Yaron Koren for patiently debugging my recursive template usage issue.

Original problem follows... This isn't a recurrence of the duplicate rows issue that cropped up in 2021. I am 99.9% sure it's self-inflicted.

Steps
 * 1) Import page data from Excel into MyCargoDB.  All pages are prefixed with CARGO ("CARGO_MyPageName") because they're data, not to be looked at directly by readers.
 * 2) The template MyTemplate queries data pulled into Cargo from the imported CARGO pages.
 * 3) Page called MyPageName uses MyTemplate to display data pulled into Cargo from CARGO_MyPageName.
 * 4) Rebuild MyCargoDB.
 * 5) Now MyPageName using MyTemplate displays Cargo data twice.
 * 6) The view in Cargo Tables of MyCargoDB shows CARGO_MyPageName AND now MyPageName which explains why it's appearing twice.
 * 7) If this keeps up I'll have 150 duplicates.

I'm not sure what mistake I've made to create this circular import into Cargo.

SovereignGFC (talk) 15:06, 2 May 2023 (UTC)


 * That first step sounds odd - are you directly creating a table within the database? If so, you shouldn't do that - the data stored by Cargo should always reflect what's in the wikitext. Could that be the issue? Yaron Koren (talk) 02:56, 3 May 2023 (UTC)
 * Apologies, I am not directly creating a table. I import through an extension that creates wiki pages from said Excel sheet that match MyTemplate.
 * After importing, I immediately execute runJobs.php.
 * I then Recreate data on MyCargoDB.
 * If any MyPageName using MyTemplate that got put into MyCargoDB exists at that point, MyCargoDB gains an entry from MyPageName even though MyCargoDB was already populated by MyTemplate. SovereignGFC (talk) 22:02, 3 May 2023 (UTC)
 * Okay, never mind. Well, it's strange that pages that look simply like "MyPageName" are showing up as holding Cargo data - even though they're meant to only display, not store, Cargo data. What exactly is in the template "MyTemplate"? Yaron Koren (talk) 14:30, 4 May 2023 (UTC)
 * MyTemplate is NOT a Cargo-generated template. It is an actual page template that DOES contain Cargo queries using templates created by Cargo's Create a class. SovereignGFC (talk) 01:11, 5 May 2023 (UTC)
 * Well, does MyTemplate contain a call to #cargo_store? And if not, how are the pages that call it getting Cargo data stored for them? Yaron Koren (talk) 17:32, 5 May 2023 (UTC)
 * MyTemplate does not have calls to #cargo_store.
 * MyPageName -> MyTemplate -> Cargo query in MyTemplate -> Separate Cargo-specific pages/templates referenced in query. SovereignGFC (talk) 18:29, 5 May 2023 (UTC)
 * Alright. Well, barring more information about your setup, I'm totally stumped - I have no idea how pages like MyPageName could show up in Special:CargoTables. Yaron Koren (talk) 18:43, 5 May 2023 (UTC)
 * What additional information might be helpful? SovereignGFC (talk) 00:50, 6 May 2023 (UTC)
 * The contents of MyPageName and MyTemplate, I suppose. Yaron Koren (talk) 13:29, 7 May 2023 (UTC)

Header in dynamic table
How to change header in dynamic table?

Tamplate:

Table header looks like this:

Thanks! -- Oleksii 212.80.47.83 06:58, 6 May 2023 (UTC)


 * Add something like "=ColumnName" after the final ")" - i.e., set an alias for the field. Yaron Koren (talk) 13:30, 7 May 2023 (UTC)
 * how to display format=dynamic table in Lua?
 * the word 'format' seems to be a Lua reserved word
 * --Oleksii 212.80.47.83 15:55, 7 May 2023 (UTC)
 * There might not be any way to display result formats from within a Lua Cargo query - I'm not sure. Yaron Koren (talk) 02:42, 8 May 2023 (UTC)
 * There might not be any way to display result formats from within a Lua Cargo query - I'm not sure. Yaron Koren (talk) 02:42, 8 May 2023 (UTC)

WITHIN statement error when a Hierarchy item has an apostrophe
I have a hierarchy definition in a template that includes the item "St. Mary's Anglican Church". When I try to query this with a WITHIN statement, I get an error that appears to split this at the apostrophe: WHERE Collection IN ( SELECT `_value` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_left` >= ( SELECT `_left` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_value` = "St. Mary' ) AND `_right` <= ( SELECT `_right` FROM `cargo__CollectedIn__Collection__hierarchy` WHERE `_value` = "St. Mary' ) )s Anglican Church") ORDER BY `cargo__Collection`.`_pageID`,`cargo__Collection`.`_pageName` . When I put double-quotes around a string with an apostrophe in a WHERE clause this seems to work fine. Is this a bug or am I writing the query wrong? Thanks! Tenbergen (talk) 04:10, 9 May 2023 (UTC)


 * Sorry for the delay. That's definitely a parsing bug in Cargo. Yaron Koren (talk) 12:54, 7 June 2023 (UTC)

Recreate and 500 Line Bug Again

 * MediaWiki 1.39.3
 * Cargo 3.4.2

When I kick off re-create data from the Cargo tables page, it gets stuck at 500 lines.

When I import a spreadsheet to that same table/template with 500+ lines and execute runJobs.php, the table populates.

What additional diagnostic or debugging data can I provide? SovereignGFC (talk) 03:40, 13 May 2023 (UTC)


 * Sorry, I missed this before. Do you see any errors in the browser's JavaScript console? Yaron Koren (talk) 02:37, 29 May 2023 (UTC)
 * Now that the backup function (export to Excel which keeps my special character filled text intact) works thanks to the phpspreadsheet update, I was able to test this.
 * Open browser dev/debug tools.
 * Recreate data link on template page, Cargo view all tables page (line of the table), or Cargo table (table to be recreated).
 * ...no errors detected. The command to recreate data just returns a 200 OK.
 * Invoke runJobs.php. Observe that only 500 rows exist.
 * SovereignGFC (talk) 01:32, 2 June 2023 (UTC)
 * If you run cargoRecreateData.php from the command line on that table, does that work? Yaron Koren (talk) 13:09, 7 June 2023 (UTC)

Vertical list of names be grouped by the initial letters
Sorry if my question has already been asked here, but I have just started using Cargo. How could a vertical list of names (from Cargo databases) be grouped by the letters of the alphabet using a Cargo query? Lalo5555 (talk) 08:49, 27 May 2023 (UTC)


 * If you use the "category" format, it will provide a display that's hopefully close enough to what you are looking for. Yaron Koren (talk) 02:36, 29 May 2023 (UTC)

Export as Excel, Composer, What Did I Miss?
(This is preventing me from properly testing the 500-line recreate bug as I want to back up my big table beforehand!)

When I try to export as Excel, I get a 500 internal server error when I click on the link to download. Unfortunately, even with MediaWiki debug on, that's all I do get: HTTP/1.1 500 Internal Server Error Date: Wed, 31 May 2023 02:41:54 GMT Server: Apache X-Powered-By: PHP/7.4.33 X-Content-Type-Options: nosniff X-Request-Id: ZHaz8pqI3nmiV89-O7ng8wAAAss Strict-Transport-Security: max-age=63072000; includeSubDomains X-Frame-Options: SAMEORIGIN Content-Length: 0 Connection: close Content-Type: text/html; charset=UTF-8

I have Composer installed at the same level of directory as MediaWiki and I confirm that phpoffice/phpspreadsheet is installed (1.19.0). I know it works because I can import spreadsheets just fine...but not export for some reason. SovereignGFC (talk) 02:46, 31 May 2023 (UTC)


 * The "excel" format code is very old, and it actually uses the PHPExcel library, not PhpSpreadsheet. Yaron Koren (talk) 13:04, 31 May 2023 (UTC)
 * Inspired by this discussion, I just added support for PhpSpreadsheet to the "excel" format. At least, in theory - I haven't actually tried running it. But there's a chance that it works! Yaron Koren (talk) 15:54, 1 June 2023 (UTC)
 * Can confirm this works!
 * I did the very proper and sophisticated thing of "copy entire single changed file and paste text into existing version" before running the query.
 * All 800+ items exported without issue. SovereignGFC (talk) 00:51, 2 June 2023 (UTC)
 * That's great to hear! Yaron Koren (talk) 02:44, 2 June 2023 (UTC)
 * One item of note: It defaults to the XLS extension, not XLSX. Excel 2021 complains the file doesn't match the extension and it may be a security issue.
 * I'm guessing that either specifying XLS format and keeping the XLS extension or changing it to use XLSX would make this nag go away. SovereignGFC (talk) 12:12, 5 June 2023 (UTC)
 * Ah, good point. XLSX seems to be the superior format, so I just changed the default filename to be "results.xlsx". Yaron Koren (talk) 15:03, 5 June 2023 (UTC)

Remove unwanted comma from results if fields are empty
I have this table with the fields ImageText and ImageFile being optional:

If ImageText and ImageFile have values, they display framed images with captions using this query:

This works fine, but how do I remove the commas from the display when ImageText and ImageFile are empty? Tahoma403 (talk) 04:22, 7 June 2023 (UTC)


 * It could that just adding  to the #cargo_query call will fix the problem. Yaron Koren (talk) 13:10, 7 June 2023 (UTC)
 * That removed,  ,  ,  ,  ,   from the top of the results, but in between each  , I still get one  ,   for some reason. Any idea? Tahoma403 (talk) 06:23, 8 June 2023 (UTC)
 * I think that's just due to MediaWiki's own display of images, though I could be wrong. Have you considered using the "gallery" format for this data, by the way, instead of assembling wikitext? It might be a nicer-looking output. Yaron Koren (talk) 12:32, 8 June 2023 (UTC)
 * Yes, that must be the reason then. I have looked at format=gallery before, but couldn't figure out where to put ImageFile in the query - would you mind showing where/how to put it in the fields= parameter to display the image and caption? Tahoma403 (talk) 13:09, 10 June 2023 (UTC)
 * Well, now that I think about it, to use "format=gallery", you would first need to change the type of "ImageFile" from String to File in the #cargo_declare call. Once that is done (and the table is recreated), a simple query like this might work: . Yaron Koren (talk) 13:13, 13 June 2023 (UTC)

Output as RSS (or Atom or whatever)?
I'm hoping to create RSS/whatever from the output of wikitext. My first thought was that Cargo might do that, so I checked the documentation pages, but didn't notice anything about it. So, I asked about it on the main MediaWiki support desk (here). One of the suggestions was that Cargo can do it. So I looked around here some more, but still haven't been able to find anything about it.

Can Cargo do this? If so, could you please point me towards info on how to make it do so? Thanks! -Rwv37 (talk) 03:50, 19 June 2023 (UTC)


 * It's a new feature, and hasn't been included in a release yet. If you're running the master version of Cargo, then you can do  (or  ) in a query, to get a link to an RSS feed. Sam Wilson 06:12, 19 June 2023 (UTC)
 * I meant feed (the type can be specified with atom). Sam Wilson 07:25, 19 June 2023 (UTC)
 * @Samwilson: Great - thank you! Rwv37 (talk) 22:04, 19 June 2023 (UTC)
 * This is great news. Thanks for working on it. Jonathan3 (talk) 13:52, 20 June 2023 (UTC)


 * Just for the benefit of anyone else who wants to try this out: It seems like you probably need to use MediaWiki 1.39 or higher. I'm using 1.38, and I get an error indicating that it can't find the class "MainConfigNames", which (it looks to me) was added in 1.39 -Rwv37 (talk) 04:13, 21 June 2023 (UTC)
 * That's odd. was added in 1.38, so it should be working. Sam Wilson 04:51, 21 June 2023 (UTC)
 * @Samwilson, I see that that page says 1.38, but: I'm largely ignorant about the internals of MediaWiki, and maybe I'm misinterpreting something, but is it possible that's just a mistake? I ask for a few reasons:
 * (1) When I actually try it, I get an error saying it can't find that class.
 * (2) This page (for MainConfigNames.template) says 1.39+.
 * (3) Also on that page, there are links for its source code for various MW versions; if you click on the 1.39 one, you're brought to code, but if you click on 1.38 or earlier, you're brought to a "no such file" page.
 * (4) My 1.38 installation does not have any file of that name (or any *MainConfigNames* file, case insensitive).
 * -Rwv37 (talk) 19:25, 21 June 2023 (UTC)
 * You're quite right! That file was introduced in and is not included in 1.38 (but only 1.39 and above). Here's a patch for Cargo:  ( could you have a look?), and another one for bumping the @since for MainConfigNames: . Sam Wilson 22:38, 21 June 2023 (UTC)

How is the  set? It looks like it's just using the URL of the page (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cargo/+/888110/18/includes/formats/CargoFeedFormat.php#125) I seem to remember having a problem with the old "BulletFeed" extension that RSS readers wouldn't know which were new items. Sometimes my items were different but shared the same URL. I ended up editing my copy of it to specify that  was the URL plus a URL parameter based on the. Jonathan3 (talk) 10:59, 21 June 2023 (UTC)


 * Yes, that's right, it's the URL of the page, but can be overridden by a query's  field. I think perhaps it should use the revision ID instead, as the default link? But that doesn't look as nice for consumers of the feed. I wondered about using   but then it might not link to the current version. Open to suggestions either way! :-) Sam Wilson 11:07, 21 June 2023 (UTC)
 * I think people are suspicious of URL parameters that aren't obvious (are they tracking? etc) so I used something clearly based on the date and time. The  and   are the same - I can't remember why I didn't have the date/time only in the   - maybe that would just have entailed more work on my part. I had to update the BulletFeed extension most times I upgraded MediaWiki. It bases its RSS feed on Cargo query results, so it will be good to cut out the middle man! I'll experiment with Cargo once I've upgraded to 1.38 and got confirmation that it works with that version :-)
 * The default you have is probably all right for most people. Especially as the  parameter which seems to set both   and   can be modified in the Cargo call. That might be enough for me too. Jonathan3 (talk) 12:35, 21 June 2023 (UTC)

Duplicate results in Special:Drilldown
Have you had a chance to look at this yet (Extension_talk:Cargo/Archive_January_to_April_2023)?

It would be great if you could fix it. I'd be happy to try to help. I came up with an idea on the Phabricator task but couldn't get any further than that with it. Basically I wondered whether a  could be added to the Drilldown page's query.

Thanks. Jonathan3 (talk) 12:43, 21 June 2023 (UTC)


 * Sorry for the long delay in responding to this. Yes, I'm aware of the problem, and yes, adding something like "GROUP BY" or "DISTINCT" is probably the right solution. My initial thought was that, if there are multiple items, i.e. template calls, on a page that match the current set of filters, it could be helpful for users to see that there are multiple matches. But that's probably silly. Yaron Koren (talk) 21:11, 22 June 2023 (UTC)