Extension talk:Cargo

Table recreation doesn't seem to work in 1.37
If I recreate a table in MW1.37 and then run the runJobs.php to speed it up I get the following error, repeatedly: 2021-11-23 05:50:22 cargoPopulateTable dbTableName=Project replaceOldRows= namespace=0 title= requestId=YZyBEawni10429LV0Poz2AAAAFM (id=14240,timestamp=20211123055010) STARTING 2021-11-23 05:50:22 cargoPopulateTable dbTableName=Project replaceOldRows= namespace=0 title= requestId=YZyBEawni10429LV0Poz2AAAAFM (id=14240,timestamp=20211123055010) t=0 error=InvalidArgumentException: The given PageIdentity does not represent a proper page The table never gets populated. If I switch back to my backup directory from pre-upgrade (MW1.36.2) the table recreate works. This is using Cargo	3.0 (ee250d4) 07:36, 22 November 2021 Tenbergen (talk) 06:00, 23 November 2021 (UTC)


 * Thanks for that bug report. This was indeed a real issue, and it appears to have been caused by a bug in MediaWiki itself, surprisingly enough. I just checked in a fix to Cargo to get around it, so if you get the latest Cargo code, it should work with MW 1.37. Yaron Koren (talk) 16:09, 25 November 2021 (UTC)

Problem with multiple joins
Hello,

I have been working on code for a book catalog. I have hit the following small situation where I cannot seen to add another table in a cargo_query to provide the information I need.

The query

works and produces the expected output.

When I add the additional table Books, and the additional join expression Books._ID=Books__Genres._rowID  in order to extract the book title it likes like this:

it throws an error Error 42P01: ERROR: invalid reference to FROM-clause entry for table "cargo__Genres" LINE 1: ... LEFT OUTER JOIN "cargo__Genres__Gen_parent" ON (("cargo__Ge...

^

HINT: There is an entry for table "cargo__Genres", but it cannot be referenced from this part of the query. Function: CargoSQLQuery::run Query: SELECT "_pageName" AS "Genre","cargo__Books__Genres"."_rowID" AS "_rowID","cargo__Books"."_pageName" AS "_pageName" FROM "cargo__Genres","cargo__Books" LEFT OUTER JOIN "cargo__Genres__Gen_parent" ON (("cargo__Genres"."_ID"="cargo__Genres__Gen_parent"."_rowID")) LEFT OUTER JOIN "cargo__Books__Genres" ON (("cargo__Genres"."_pageName"="cargo__Books__Genres"."_value") AND ("cargo__Books"."_ID"="cargo__Books__Genres"."_rowID")) WHERE "cargo__Genres__Gen_parent"."_value" = 'math' ORDER BY "_pageName","cargo__Books__Genres"."_rowID","cargo__Books"."_pageName" LIMIT 100

Could you please help. I am not sure how to fix this. I have googled and the best I saw is that I am mixing explicit and implicit joins andI am now really confused.

Thanks. Lbundle (talk) 02:13, 23 November 2021 (UTC)


 * Sorry about that - there's nothing actually wrong with your query (and I'm glad you figured out how to get around the limitations of HOLDS). Sometimes the handling of #cargo_query doesn't work correctly if the "tables" and "join on" arguments are not in a specific order, and that is the case here. After some trial and error, I believe the following values for those two parameters will work:

tables=Books, Books__Genres, Genres, Genres__Gen_parent |join on=Books._ID=Books__Genres._rowID, Books__Genres._value=Genres._pageName, Genres._ID=Genres__Gen_parent._rowID


 * This is obviously a bug in Cargo, which I hope will get fixed soon. Yaron Koren (talk) 18:08, 24 November 2021 (UTC)


 * Thank you very much! This ordering you proposed worked. Lbundle (talk) 03:26, 28 November 2021 (UTC)

Using DataTables Styling Options
Currently, the "dynamic table" display option uses mostly default options for DataTables display format. It is highly preferrable to be able to customize DataTables' default styling options. For example: set "compact" mode for DataTables.

Is it possible to enable these options in any way?

Centering Map at Particular Location in Query
Is there a way to force a centering of a map to a particular point in a cargo query? Usually the map is zoomed and positioned such that all the "hits" of the query show and are centered. However, one can force a zoom level to override the computed scale. Is there a way to force a center point as well? That can be done using {{#display_map|center= ...} but I haven't found a way to incorporate that into a cargo query. Parma100 (talk) 13:55, 8 July 2021 (UTC)


 * Unfortunately, no; but a "center" parameter (or "starting bounds", which is what Page Forms has) would be a good idea. Yaron Koren (talk) 15:28, 9 July 2021 (UTC)


 * Okay. Weekend project. I'll see what I can come up with. Parma100 (talk) 16:01, 9 July 2021 (UTC)


 * I've got centering working for all three mapping services. Adds a  parameter and, if no   is specified, adds a zoom parameter to flesh out the services' requirements. Now I'm working on Leaflet clustering. I figure I'll wait until both tasks are working before I submit them for merging. Parma100 (talk) 13:26, 13 July 2021 (UTC)


 * Leaflet clustering now works. I'll be initiating a merge request directly. Parma100 (talk) 03:45, 15 July 2021 (UTC)


 * That's great - I'm looking forward to both patches! Yaron Koren (talk) 00:35, 16 July 2021 (UTC)

Query time span for calendar doesn't quite fit
Hi there! First of all: I love this extension. It is quite powerful.

Though, I have to admit that I only worked with this extension for roughly four hours and already hit what I would call a bug.

The situation given is like this:
 * I've got a simple table for events (Page, Start, End, Server (short name))
 * I would like to display the table in a calendar format
 * The calendar gets shown and some of the data displayed
 * The events in the table may have a time span of a single day or a whole month
 * Wiki is hosted by Miraheze so I don't have any file or database access

The problem is, some events are missing in the calendar. For example, I've got an event which takes place 2021-06-22 to 2021-07-20 and in the month July 2021 it doesn't get displayed. A similar problem occurs in weekly view, too.

The problem at hand (as far as I could analyze it) is with the way the data is being fetched. The GET looks like this:

As you can see, the initial query (and the previous&next buttons) set a limit (start=2021-06-27&end=2021-08-08) to the fetched data. In my eyes, my example event would partly take place in that limit. This might be a bug on fetching the dates within a given time frame. For me myself it would be enough if I could just set a higher threshold for start and end (is there a parameter for this?).

I hope I was able to describe my problem. Please don't hesitate to ask if you need further information or need to see our wiki. Thanks in advance!


 * You did in fact hit a bug! Sorry about that. I just checked in a fix for it - hopefully it will make its way to Miraheze before too long. Yaron Koren (talk) 17:39, 23 July 2021 (UTC)


 * Thanks for the quick fix! Miraheze merged the change into their productive system, too. I can confirm that the fix works fine. I'm looking forward to use Cargo more often in the future! Random visitor 12:48, 02 August 2021 (UTC)

Parsing order effect? Help welcome.
Using semantic mediawiki, I used to have a display programmed as :

I used link= in : http://dufal.fr/w/index.php?title=Mod%C3%A8le:ItemSuite&oldid=17785

Now replaced by link=

but I get a text version like

It seems to me that the parsing of [[Image: happens before the #cargo_query is extracted.

As a result, I cannot replicate with Cargo what used to work very well with SMW.

More specifically,
 * when the TOMEouLIVRE defines the field Image as a string, I see the name of the picture but not the image itself.
 * when the TOMEouLIVRE defines the field Image as a File, I see the image in the standard wikimedia format (framed, and with name under) which breaks my initial format.

Do I get something wrong ?

The first 4 highest level of page http://dufal.fr/wiki/Test use a series of formating of the field in TOMEouLIVRE table which I created to debug. The picture before the first title on that Test page is what I would expect to get (what I used to get with SMW's #show)

Many thanks.

Frederic.


 * Try adding "|no html" to the #cargo_query call. Yaron Koren (talk) 02:57, 8 August 2021 (UTC)

Also suffering from duplicate rows
I have recently forked a wiki from Fandom to my own host which makes use of extensive Cargo templates to store and query data. Unfortunately I'm finding that when I try save a page (via null edit) to create a new entry in a Cargo table the entry is being duplicated. An example can be seen on this page https://www.poewiki.net/wiki/Modifier:AbyssFireColdResistanceJewel1 When I run a null edit this entry gets duplicated and breaks subsequent queries.

Any idea why this might be happening?

Iamacyborg (talk) 21:15, 9 August 2021 (UTC)


 * I can't see this right now - right now there's no real data being stored for that page. Could you replicate the problem? Yaron Koren (talk) 03:14, 10 August 2021 (UTC)


 * You can see the dupes here https://www.poewiki.net/wiki/Special:CargoTables/mods Iamacyborg (talk) 07:18, 10 August 2021 (UTC)
 * Is there any data or additional information I could provide that might make debugging this easier? Iamacyborg (talk) 13:21, 10 August 2021 (UTC)


 * Okay, yes, I see the problem. That's very strange, and I don't know why it's happening. Does it only happen for null edits, or does it persist if you make some actual edit to a page? In any case, recreating the table will probably fix the issue, at least temporarily. Yaron Koren (talk) 14:01, 10 August 2021 (UTC)


 * I noticed in our templates we don't set any particular field as unique, might that be causing these dupes to happen? Iamacyborg (talk) 15:55, 10 August 2021 (UTC)


 * No, that shouldn't affect things - the duplicate check happens (or is supposed to happen) for all rows. Yaron Koren (talk) 21:40, 10 August 2021 (UTC)


 * Hmm, any idea how we might proceed? We have quite a few tables and having to rebuild them daily might get pretty slow, particularly if things break if someone inadvertently modifies or does a null edit of a page. Iamacyborg (talk) 07:22, 11 August 2021 (UTC)


 * I doubt you would have to rebuild them daily - you might only have to rebuild them once, unless null edits are planned to be a regular thing. Yaron Koren (talk) 19:42, 11 August 2021 (UTC)


 * Here's a weird one, we've found when editing pages that store data in our versions table that pages that end with a letter are not being duplicated https://www.poewiki.net/wiki/Special:CargoTables/versions You can see the duplicates in that table here https://www.poewiki.net/wiki/User:Iamacyborg/test Iamacyborg (talk) 07:17, 12 August 2021 (UTC)


 * After some further debugging, this seems related to Extension:PageImages, which when active, causes duplicate items to appear when a page is saved. Disabling the extension appears to have fixed the problem.


 * That's a great find - I just added a note about that to the "Known bugs" page. Yaron Koren (talk) 20:45, 19 August 2021 (UTC)


 * Is there any way to use both extensions? I started using PageImages fairly recently and have noticed more Cargo duplicates than usual. I don't know if it's related but it sounds likely. I like both extensions (though It's PageImages I'd give up if I had to choose one). Jonathan3 (talk) 22:35, 13 September 2021 (UTC)

We were having this issue over on Nookipedia and I found that performing a cargo_store without explicitly defining every field will lead to duplicates. I've created a task on Phabricator with more details. Thanks, ~ Super  Hamster  Talk Contribs 14:38, 1 September 2021 (UTC)

Calender colours don't fit for a compound query
Hi, I hit a bug when using the calendar output combined with a compound query.

The compound query is defined to do two queries. Each query defines a colour for the calendar like. The REST requests fits but the response doesn't fit as all results have the same colour.

Looking at the code I guess it's caused by the reuse of the variable  in CargoExport.php around line 122 and 148.

This bug might be lower priority as it is possible to walk around this problem by defining the colour as field:

--Random Miraheze user 20:38, 18 August 2021 (UTC)


 * That's strange; you can see the "color" parameter working correctly here. Maybe this is some bug that has since been fixed? Yaron Koren (talk) 19:05, 19 August 2021 (UTC)


 * Now that's interesting. Maybe it's a problem on our host's side. Sadly, I can't debug the code he's using. Well, we'll work with the workaround then. For documentation reasons, here is the request: https://counterside.miraheze.org/w/index.php?title=Special:CargoExport&tables[0]=EventPeriods&tables[1]=EventPeriods&join on[0]=&join on[1]=&fields[0]=Title,Start,End,Event=_pageName&fields[1]=Title,Start,End,Event=_pageName&where[0]=Server='SEA'&where[1]=Server='KR'&group by[0]=&group by[1]=&order by[0]=`Title`,`Start`,`End`,`Event`&order by[1]=`Title`,`Start`,`End`,`Event`&limit[0]=100&limit[1]=100&format=fullcalendar&color[0]=blue&color[1]=red&text color[0]=white&text color[1]=white&start=2021-05-30&end=2021-07-11&_=1629584478634 and the response: [{"title":"Circuit Link System","start":"2021-06-09 21:00:00","end":"2021-06-30 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Lee_Jisoo"},{"title":"Old Fear","start":"2021-07-07 21:00:00","end":"2021-07-28 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Old_Fear"},{"title":"Sweet Promotion","start":"2021-06-23 21:00:00","end":"2021-07-07 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Sweet_Promotion"},{"title":"Yoo Mina Recruitment Rate Up","start":"2021-06-30 21:00:00","end":"2021-07-21 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Yoo_Mina"},{"title":"Fenrir Yoo Mina Growth Limited Mission","start":"2021-06-30 21:00:00","end":"2021-07-21 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Yoo_Mina"},{"title":"Smell of the Game","start":"2021-06-21 19:00:00","end":"2021-07-19 19:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Smell_of_the_Game"}] In this example, the "Smell of the Game" is the one which should be red, but it is not. In any case, thanks for your time! --Random Miraheze user22:38, 21 August 2021 (UTC)

Using dynamic data for the allowed values field parameters in #cargo_declare?
Hi, is it possible to use dynamic data, something like a Category: for example, to feed the  field parameter?

My goal here is that the end user could add new values to the  using a Form (created using PageForms) instead of manually editing the template where the   is called.

Igorabsorto (talk) 07:36, 26 August 2021 (UTC)


 * Hi! That's not recommended, because you would then have to recreate the table every time the set of values changed. It's better to not have "allowed values" in the Cargo declaration, and instead handle it in the form, with "value from category" or something like that. Yaron Koren (talk) 15:35, 27 August 2021 (UTC)


 * Great, that makes sense. Thank you, Yaron! Igorabsorto (talk) 09:17, 30 August 2021 (UTC)

Date display (again)
I have a similar query to that asked by Parma100 in Date Display - I have a date field, that I would like to change the format of the output. If the stored date has only year or month/year, the format is great (e.g. 1919 / March 1919) respectively), but if it also has day then the format is `Y/m/d` (e.g. 1919/03/14) which is not particularly 'friendly' and could be ambiguous - I would like the display to be the same as MW default i.e. 14 March 1919. I have not been able to decode what the solution was in that topic.

Playing with the code sample in topic linked above I can force the date to `j F Y` but that stops dates with less precision working as required - I've not been able to get a value from `Fieldname__precision` to use in a condition.

This is what I currently have in my template:

Many thanks, --Jaydublu (talk) 17:32, 28 August 2021 (UTC)


 * You could try something like this in a template:
 * Jonathan3 (talk) 22:43, 13 September 2021 (UTC)
 * Jonathan3 (talk) 22:43, 13 September 2021 (UTC)

One template that attach to multiple tables
Hi, is there a way that the recreate table function of "all" attached templates recognize them ? Actually only the last call to attach is recognized.

I know in "Storing data" it is written not to attach to more than one table. But nevertheless it seems that atleast the template itself gets everything right (i.e. it shows at the top that it writes to table A, writes to table B and so on).

But at the special page cargo-tables only the last is shown to write to the corresponding attached table. And only this is recreated if we redo the tables via web or php command.

In our case we want to have some templates that combine several cargo tables/templates (i.e. attach to) and write some fixed values at some fields to the tables and only some as template parameters are passed. Or is there a better way to do this? Hope the question was somehow clear ;)

Cheers, Brabi81 (talk) 13:24, 20 September 2021 (UTC)


 * In theory, there's nothing wrong with what you're doing; but in practice, I think there's a bug that prevents what you are trying to do from working. If that's true, then, until the bug is fixed, one thing you can do is have that template itself call other templates, each of which either declares or attaches to a single Cargo table. Yaron Koren (talk) 18:23, 20 September 2021 (UTC)


 * The thing with one template that call other templates works fine. Thanks for that idea! But I'll also try to keep one eye open to see if the bug is fixed in future patch notes. Thanks Brabi81 (talk) 05:43, 24 September 2021 (UTC)

Can't populate Cargo db fields with empty data
Hello, I've been running into this for weeks now. Basically, all Cargo fields appears to be mandatory. Running cargoRecreateData.php on a Table with a Page with some empty fields throws this:


 * What version of Cargo are you running? I think this problem has been fixed already. Yaron Koren (talk) 16:28, 22 September 2021 (UTC)


 * Hi, it's 2.8 --Carloposo (talk) 17:00, 22 September 2021 (UTC)
 * Hi Yaron, changing field types (Integer and Floats to String) fixes the issue. Does this ring a bell? BTW, the same issue is present with empty Date fields. --Carloposo (talk) 11:00, 23 September 2021 (UTC)


 * There are a few pages about this error and strict SQL mode, eg https://stackoverflow.com/questions/28606483/how-to-allow-empty-string-for-integer-in-mysql#39409037


 * I often save Cargo pages with empty date fields and it's ok, so maybe when Cargo creates tables it uses the database default which is different in your case. Jonathan3 (talk) 11:37, 23 September 2021 (UTC)


 * Thank you Jonathan, that made my day! --Carloposo (talk) 13:18, 23 September 2021 (UTC)


 * When I tried this out, all empty values were saved as "null", including for Integer fields. By coincidence, I just released version 3.0 of Cargo - I would try upgrading to this, because it could be that there was some recent change that is resulting in the different behavior. Of course, if you already modified your database settings, you might not able to know (and might not care) whether the code is acting differently. Yaron Koren (talk) 19:25, 23 September 2021 (UTC)


 * Thank you, Yaron. Is upgrading as easy as replacing the Cargo folder with the new one? --Carloposo (talk) 07:44, 24 September 2021 (UTC)
 * Yes. Yaron Koren (talk) 14:19, 24 September 2021 (UTC)

Quick way to switch in multiple replacement tables
Because of the duplicate row thing, I've set up a handy Windows shortcut using plink to log in and create replacement tables - but then it's a pain having to go to the website to click the ticks for several tables. It would be nice to be able to do them all at once.


 * Maybe there could be a button on Special:CargoTables to do that?
 * Or maybe a new command line script?
 * Or maybe the existing script could have an extra parameter to switch in the replacement tables at the end?

Thanks Jonathan3 (talk) 10:01, 24 September 2021 (UTC)


 * I've just saved all the pages as bookmarks that can be opened all at once in different tabs, then I can click "Switch" for each. It's quick enough, and uses replacement tables. (I found that when I recreated the tables nightly without a replacement table, they still would have duplicates, which maybe was because of the clashing with the job queue problem.) Jonathan3 (talk) 20:55, 5 October 2021 (UTC)

Mediawiki still showing data even though Cargo database deleted
I am making a catalog for my books and have hit the data duplication bug. I have mediawiki 1.36.1 and Cargo 3.0, on PstgreSQL 13. In the process if trying to debug the issue I erased all the cargo data related to the catalog. However the data still shows up in mediawiki even after purging. I am not seeing where it is getting the data from. Can you please help? Also, is there any progress on the data duplication bug?


 * What do you mean by the data showing up? And what data duplication are you seeing? I personally haven't seen this problem in a long time. Yaron Koren (talk) 01:40, 7 October 2021 (UTC)


 * When I go to Catalog:Books, the book shows up and I can still view its details.


 * As for data duplication, when I edit a book, a duplicate appears in the sql database and in the cargo table.


 * I am working on a smallest case example which may help clarify.


 * What's on the page Catalog:Books? And an example for the duplication problem would help a lot, yes. Yaron Koren (talk) 13:08, 7 October 2021 (UTC)

Example of duplicate data
As promised, here is an example of my book catalog.

The main Book template is the following:

This is the "Book" template.

The Book form is

This is the "Book" form. To create a page with this form, enter the page name below; if a page with that name already exists, you will be sent to a form to edit that page.

Title:

Subtitle:

Author(s):

Genre(s):

Publisher:

Year of publication:

ISBN:

Number of pages:

Owner:

I first create the tables which are created appropriately. I then create a very simple book with just a title, and everything else is blank.

In 'Category Books' only one book shows up as expected (sorry, cannot seem to be able to insert screenshot).

Category:Books Jump to navigation Jump to search Pages in category "Books"

This category contains only the following page. A

A Book

But in the Books Table I have two books listed: Table structure:

Title - String Subtitle - String Authors - List of Page Genres - List of Page Publishers - List of Page Year_of_publication - Date Number_of_pages - Integer ISBN - Integer Owners - List of Page

This table has 2 rows altogether.

Recreate data. Page 	Title 	Subtitle 	Authors 	Genres 	Publishers 	Year of publication 	Number of pages 	ISBN 	Owners A Book 	A Book A Book 	A Book

In the PostgreSQL database there are two entries with different _IDs:

_ID 	_pageName 	_pageTitle 	_pageNamespace 	_pageID 	Title 	Subtitle ...... 1       a book	     a book	                 0      201     A Book	         ...... 2       a book	     a book	                 0      201     A Book	         ...... This is the duplication problem I am trying to fix.


 * Your data structure looks fine. I didn't notice before that this was a PostgreSQL database - I'm guessing that is part of the issue; Cargo works with PostgreSQL, though it's much less well-tested than with MySQL, unfortunately. If you recreate the table, does the duplication go away? And if so, does it come back every time you resave the page? Yaron Koren (talk) 14:44, 11 October 2021 (UTC)


 * I went through the data structure line by line to simplify the problem to the bare minimum. The bug seems to be in the template. in #cargo_declare I declared a Subtitle, but in #cargo_store I did not use it. When I made that fix, everything works now. I tried it on a three line data structure and the same thing shows up. So I believe this is a bug and it should have raised an error or warning. --Lbundle (talk) 19:41, 11 October 2021 (UTC)

Critical performance drop after version 2.4
Yaron, hello! After a long break, I decided to update my wiki. Its peculiarity is very large pages that generate a lot of records. After deploying the update on the auxiliary test server, I decided to try to overwrite one page. The result was either a 504 error, or only partial saving of records (from 150 to 300, although the page generates about 1500 records). I tried rolling back from the Master version to previous versions. It worked (and significantly faster) only when returning to version 2.4.

The test server runs Debian 11, MariaDB 10.5.11 and MW 1.35.4. --StasR (talk) 22:23, 11 October 2021 (UTC)


 * That's very interesting; it might be due to the data duplication check, which Cargo right now runs before every insert. With the old code, does it really save all 1,500 records? Yaron Koren (talk) 23:33, 11 October 2021 (UTC)


 * Here are the PageValues of the largest page. It contains 1991 entries. Yes, sometimes there are errors when recording. I have scripts that check that the values are written to the database to the end and without duplicates. But in general, everything works. --StasR (talk) 08:04, 12 October 2021 (UTC)


 * That's very interesting. If you're willing, it would be great if you could try it again with the latest code, but making a small change to the code, to see if the duplication check is really the issue. In the file includes/parserfunctions/CargoStore.php, just change this line (line 328 or so):

$rowAlreadyExists = self::doesRowAlreadyExist( $cdb, $title, $tableName, $tableFieldValues, $tableSchema );


 * ...to this:

$rowAlreadyExists = false; // self::doesRowAlreadyExist( $cdb, $title, $tableName, $tableFieldValues, $tableSchema );


 * If you try that, please let me know if that fixes the performance problem. Yaron Koren (talk) 14:12, 12 October 2021 (UTC)


 * I checked the change on version 2.5. It didn't help. --StasR (talk) 14:51, 12 October 2021 (UTC)


 * As a result of these experiments, I lost the “Page Values” menu item. Do you have any ideas how to restore it? --StasR (talk) 15:21, 12 October 2021 (UTC)


 * It turns out that this happens after executing “git checkout 2.4”. --StasR (talk) 15:49, 12 October 2021 (UTC)


 * Okay, thank you for trying that. And thanks for clarifying that the problem came in version 2.5. Given that, I have a new theory as to what is causing the slowness: this change, which added the $cdb->lockForUpdate call to CargoStore.php. Could you try commenting out that line in later versions, and see if that speeds things up? Yaron Koren (talk) 16:27, 12 October 2021 (UTC)


 * YESSSS! ) Checked for REL1_35 (aka 2.6) and for ‘master’ aka 3.0 (but I am not ready for ‘master’ due to changes in the storing of empty values). Thanks. --StasR (talk) 17:04, 12 October 2021 (UTC)

After a month of using the version 2.6, I can say with confidence that this version is faster and more reliable. I'm a little worried about what a commented-out line (lockForUpdate) might entail. --StasR (talk) 13:18, 17 November 2021 (UTC)


 * Okay, that's good to hear. I just removed that line from the code (which I had been planning to do anyway). I think it'll be fine - that line was only added about a year ago, and the code worked fine beforehand. Yaron Koren (talk) 15:16, 17 November 2021 (UTC)

Not show pages' namespaces on Special:Drilldown
Is this possible?

The problem (for me) currently is that they're all listed alphabetically according to namespace, which means that each column is like A, A cont'd, A cont'd...

Thanks. Jonathan3 (talk) 21:03, 12 October 2021 (UTC)


 * Bizarrely, today I have just remembered these things from 2017:


 * Extension_talk:Cargo/Archive_January_to_February_2017
 * Extension_talk:Cargo/Archive_January_to_February_2017
 * T157849: Add option to hide namespace in Special:Drilldown and/or query results
 * T157754: Fix the way namespaces are treated in the headings of Special:Drilldown


 * These were two patches I submitted that you used, firstly to be able to hide the namespace and secondly to use alphabetic headings based on page name (ignoring namespace).


 * The first ($wgCargoHideNamespaceName) still works, but the second doesn't any more. Jonathan3 (talk) 09:12, 21 October 2021 (UTC)

Help with conceptual knowledge gap with regards to implementing Cargo
Very naive question—I have have read many of the "getting started" type guides (like the link below) and watched several videos to wrap my mind around how Cargo works and even produced a simple working example on my wiki but I am still confused by one point that I can't quite seem to find information on in these guides (maybe only puzzling to me). I don't understand how one is supposed to 'populate' the Cargo table once it has been declared and stored. For example, with a simple example like an info box with authors and books) or something like that, what if you had 100s of entries in a csv that you wanted to upload to define the relations in an Infobox—i.e. the actual "data"? Do these have to be manually entered one row at a time on the table viewer or through the form that is generated with a Cargo table? I tried playing around with importing csv's into the wiki as well the External Data extension (which is mentioned elsewhere in this discussion thread) but I'm not sure how to take advantage of either to populate data into an existing Cargo table. Any help or a point toward the right resources would be great.

https://workingwithmediawiki.com/book/chapter16.html


 * The Data Transfer extension may be what you need in this case. (Although maybe you've used that already, because you talked about importing CSVs.) Yaron Koren (talk) 00:10, 15 October 2021 (UTC)


 * I don't understand how one is supposed to 'populate' the Cargo table once it has been declared and stored. The 'stored' part of this is the populating of the table, and is done by adding a Cargo-ified template to a page. When that page is saved, the data is added to the Cargo table. So, yes, the data does have to be 'manually' entered, and that's done on wiki pages and not via any Cargo-specific UI. Sam Wilson 10:35, 18 October 2021 (UTC)


 * Okay thank you everyone and thank you ! That was a big part of my confusion. I naively and erroneously assumed that the Cargo table was the original source of truth and, in turn, it was populating all the respective templates on every wiki page. Now I see that you can modify a pages template and the table updates, or in turn you can edit the row in the table viewer and the template changes in the article. Slowly but surely I'm getting it. Flyingratchet (talk) 21:50, 20 October 2021 (UTC)

How to find the empty value "where" in lua
how to find the empty value "where", just like in the Drilldown, you can select the empty value "nothing" _none.

example if frame.args ['class'] ~ = '' then args.where = args.where .. 'AND requires_class HOLDS LIKE "%' .. frame.args ['class'] .. '%"' else need to find all empty "class" values end

Thanks. Oleksii


 * Every list field gets a column in the main table named, so I think it's possible to query the absence of any value with e.g. Sam Wilson 05:34, 17 October 2021 (UTC)

How to not show a Template in Special:Drilldown
Hello, I understand that every Cargo table gets automatically added to the tabs in the Special:Drilldown page. Is there a way to not show some of them? Thank you. --Carloposo (talk) 15:47, 20 October 2021 (UTC)


 * I thought that there was a way to do that, but it looks like there isn't. Out of curiosity, is there any pattern to the tables you don't want shown, in terms of the set of fields, number of rows, etc.? Or is it just information that you don't want people to look at? Yaron Koren (talk) 16:05, 20 October 2021 (UTC)


 * You can get rid of all of them using CSS. Maybe you could similarly just make the nth one disappear? Or each one might even have its own id? Jonathan3 (talk) 22:15, 20 October 2021 (UTC)


 * Oh yeah, that's true. Or JavaScript, but CSS is easier. Yaron Koren (talk) 23:58, 20 October 2021 (UTC)


 * @Yaron: it's infos I want to store in Cargo tables but don't want people to look at. I'll probably query them via API, if it's possible, and make them available to other systems (e.g. to do some statistical analysis). @Jonathan: that's a good idea, thanks. --Carloposo (talk) 08:31, 21 October 2021 (UTC)
 * @Carloposo: sounds like the CSS or JS hiding method might work for you, but do remember that this doesn't actually make the data private at all, and that there are still a number of ways for users to see it. —Sam Wilson 08:35, 21 October 2021 (UTC)
 * @Sam: you are totally right. It would be very nice to be able to decide which table can be drilled and which not. Also @Yaron Koren for hiding bpmnData and ganttData from Drilldown, as I understand there are no customisation for them (e.g. hiding/renaming specific fields etc.)


 * This might not be ideal for you, but maybe you could mark every field of the relevant tables as "hidden" so that they don't become filters on Special:Drilldown and (I think) also don't show on Special:CargoTables. Jonathan3 (talk) 13:28, 22 October 2021 (UTC)

Can't save data into bpmnData and ganttData tables
Hello, I installed FlexDiagrams and run the maintenance scripts to create Gantt and BPMN tables. Then I created a page in each of the namespaces. No data is saved in the cargo tables (and no data pops up in the Drilldown tabs). Thanks. --Carloposo (talk) 10:12, 21 October 2021 (UTC)


 * What happens if you re-run those scripts - does the data show up then? Yaron Koren (talk) 14:08, 21 October 2021 (UTC)
 * It does, thanks @Yaron Koren --Carloposo (talk) 07:31, 22 October 2021 (UTC)


 * Okay, I just checked in a change to Cargo so that now, if a page is created or modified (or deleted), all the relevant "special tables", like _bpmnData and _ganttData, should get changed accordingly. Yaron Koren (talk) 17:17, 28 October 2021 (UTC)

SUM the output of COUNT(*) (SOLVED)
Hi Yaron,

I'm using the cargo_query to select a subset of a set of tables, with the goal to find out how many records satisfy that query. What I would like to do is using the SUM function to estimate this. So something like explained at: https://stackoverflow.com/questions/14880080/mysql-sum-of-a-count-with-group-by-clause

This is how far I am:

However, I would like to sum the count(*), so something like SUM ( COUNT(*) ). Any idea to do such? Now the result of the above is a string of ones. See: https://csdms.colorado.edu/wiki/Test_cargo1

I tried something like:

|fields=BibEntryCargo._pageName= _pageData._creationDate,COUNT(*),SUM(COUNT(*))

but that gives the following error: "Error 1111: Invalid use of group function (localhost)".

I'm using MySQL 8.0.26, MW 1.36.2, Cargo (3.0; latest downloaded 2 days ago) and ONLY_FULL_GROUP_BY is enabled.

Hope you can give a hint, --Kettner2 (talk) 17:08, 24 October 2021 (UTC)


 * @Kettner2: It's weird that that GROUP BY syntax is even working — I thought it had to be a separate parser function parameter. Anyway, it sounds like you want a list of years and page/record counts; something like this might do it: Note that I've joined on page ID to avoid issues with page titles. —Sam Wilson 01:42, 25 October 2021 (UTC)


 * Thank you @Sam! This is exactly what I was trying to get, record counts per year; and your formulation works, see: https://csdms.colorado.edu/wiki/Test_cargo2 (added a line to indicate the year I'm after, e.g.:|where= _pageData._creationDate LIKE '%2021%' ).


 * I'm glad this worked. And yes, I too am surprised that "GROUP BY" worked within "where" - that should probably be considered a bug. Yaron Koren (talk) 17:50, 25 October 2021 (UTC)

Empty variables missing in cargo_query template results (Work around)
Hi Yaron, all,

One more question using #cargo_query. I'm using something like the following query to parse publication information to a template:

Depending on the article type (book, journal paper, thesis, etc.) certain fields are empty. For example, a thesis might not have a DOI. In earlier versions of Cargo (or mysql), these empty fields would still be recognized and as such stayed as empty variables in my template that uses something like, , etc. However, when using the function above in mysql8+ and the latest version of Cargo, now empty fields are ignored, they are not passed on to the template. So for example, if the 3rd field is empty, so no "Title" is entered, then my variable becomes "Editors". So my question is: is there a way that empty fields are still recognized, such that this information is passed to a template? Thanks! --Albert Ke (talk) 19:57, 25 October 2021 (UTC)


 * I can't replicate this problem, although I'm not using MySQL 8 - I'm using the equivalent of an earlier version. So maybe the MySQL version is the issue - although I can't see how that would affect things... Yaron Koren (talk) 23:37, 25 October 2021 (UTC)


 * Thank you Yaron. What is somewhat odd is that the default print statement does list the empty fields, but when the results are passed through a template the empty fields are ignored. See for example:
 * Template: https://csdms.colorado.edu/wiki/Template:Sometemplate
 * Page with the 2 queries, 1st default print statement, 2nd using a template: https://csdms.colorado.edu/wiki/Test_1
 * Records that hold the data: https://csdms.colorado.edu/wiki/Reference:Reference-000002
 * In the above example, the 2nd query (using the template) has a value for the 4th parameter (Editors), while you see from the 1st query that doesn't use a defined print statement, the 4th parameter (Editors) is empty. Could it be that empty fields are not parsed correctly in the Carqo_query statement? --Kettner2 (talk) 02:03, 26 October 2021 (UTC)


 * OK, I found a work around that involves "named args=yes". Use this in the query. Then use named variables in your template and query. So the query would become something like:


 * And then in your template use, , etc. instead of , , etc. --Albert Ke (talk) 22:03, 26 October 2021 (UTC)


 * I wouldn't say this is "solved" yet, exactly - there's still a problem with the handling of unnamed/numbered arguments. If you're willing, it would be quite helpful if you could add the following line to the function displayRow within the file /includes/formats/CargoTemplateFormat.php, around line 18:

print_r($row);


 * ...and then see what is displayed on the page with the template query. I'm very curious to know if MediaWiki's DB-querying functions are not including fields that are blank in the results. Yaron Koren (talk) 00:10, 27 October 2021 (UTC)


 * Ok, the following is printed:

Array ( [_pageName] => Reference:Reference-000002 [BibType] => journalArticle [Title] => HYDROTREND: A CLIMATE-DRIVEN HYDROLOGIC-TRANSPORT MODEL FOR PREDICTING DISCHARGE AND SEDIMENT LOAD TO LAKES OR OCEANS [Journal] => Computers &amp; Geosciences [Volume] => 24 [Pages] => 51–68 [URL] => https://linkinghub.elsevier.com/retrieve/pii/S0098300497000836 [DOI] => 10.1016/S0098-3004(97)00083-6 [Note] => Auto downloaded ref at: 2019-12-31 [Firstname] => Syvitski, James P.; Morehead, Mark D.; Nicholson, Murray; )
 * Seems like data is presented for the right fields for this print statement.... --Kettner2 (talk) 00:25, 27 October 2021 (UTC)


 * Okay, thank you. No, that's bad, actually - there should be an 'Editors' parameter there with a blank value, for example, but instead it's just missing from the array entirely (as I suspected was the case). I'll have to see if there's some way to get around this problem. Yaron Koren (talk) 00:53, 27 October 2021 (UTC)


 * Thanks for looking into this Yaron! I have changed the status from 'solved' to 'work around'. --Kettner2 (talk) 01:16, 27 October 2021 (UTC)

Section with a variable name?
Is it possible to have a section that the user names?


 * Yes, if you have the section (and its header) as fields of a template. Yaron Koren (talk) 23:37, 25 October 2021 (UTC)

Links continue to be displayed as plaintext
I just want to remind this bug, see and. It has no relation to Extension:ConfirmEdit (I have never used it on my current testing mediawiki installation).

I tried many ways how to change code of my template but without any success. BTW, this code is extremely simple: * [ ]

Only workaround known to me is to purge cache of the page where the query is placed - which is very unsatisfactory. Any ideas how to solve this (no lua, please, as I use standard webhosting, not server hosting).

--Radouch (talk) 19:36, 30 October 2021 (UTC)


 * I get this too though purging the page hasn't bothered me. It's on a Cargo query using a template output format which displays images. The images appear as HTML instead, whenever the page is edited/saved, then as the actual images when the page is purged. I use ConfirmEdit but for account creation rather than page edits. I could try disabling it... Jonathan3 (talk) 20:25, 30 October 2021 (UTC)
 * As I mentioned, ConfirmEdit seems unrelated to this. For my case I just now "discovered" that I can use Concat, so now instead of template I use this query:


 * I hope thus I can replace most of possible "template-use" cases in the future. --Radouch (talk) 20:43, 30 October 2021 (UTC)


 * I created a simple query/template like yours above and get a similar result, even with ConfirmEdit disabled. I haven't quite worked it out (it's late here) but... The first time the page with the Cargo query is saved it shows HTML tags. This can be fixed by purging the page. The page will be "broken" again when the Cargo query is changed (e.g. changing the limit parameter). The page will not be "broken" again by a tiny edit to the page (e.g. adding a space character to the end of the page) but otherwise will break when edited (a few characters, or a space within the page). Sometimes just saving the page again fixes it but that seems unpredictable/rare. Jonathan3 (talk) 23:21, 30 October 2021 (UTC)


 * I think any query using the "template" output is affected in the same way, and the way to ensure it's parsed/displayed correctly is always to purge the page after editing/saving it. Jonathan3 (talk) 23:25, 6 November 2021 (UTC)

“This is not allowed in Cargo API queries”
''Exception caught: Error: Field alias "_pageID" starts with an underscore (_). This is not allowed in Cargo API queries.'' But why? --StasR (talk) 20:11, 8 November 2021 (UTC)


 * I don't remember now, and it doesn't look like I wrote it down anywhere. I think it's a technical issue with MediaWiki's API handling. Yaron Koren (talk) 14:57, 9 November 2021 (UTC)

How to retrieve _pageTitle in a template?
How on earth does one retrieve _pageTitle, inside of a template used by cargo to format a row? I could find no example of this... figured it out a long time ago by trial and error, but can't seem to figure it out again, and I no longer have a reference. Any help with this would be much appreciated. I'm used named parameters, so in a template I'm using things such as rather than. I can't figure out how in the word to retrieve this, tried, , , , etc., nothing works. I need to use named parameters, is that my issue??!? Thanks TiltedCerebellum (talk) 04:35, 10 November 2021 (UTC)


 * Nm, finally figured out what I was doing wrong. In the cargo statement  and in the surrogate template, . Presuambly the same can be done with   and in the surrogate template, . TiltedCerebellum (talk) 04:44, 10 November 2021 (UTC)

Nest cells for querying
This may not be a problem with with querying on-wiki specifically, but when querying externally, sometimes, having hundreds of columns may not be the best thing (on a JSON). With a JSON query specifically, is there a way I can concatenate some fields so they get "grouped" on a query via the API? What I mean is this: The  would be a column, consisting of other columns that have the results, so when I query the   column without specifying one language, it shows all the results in the languages I have added. Lakelimbo (talk) 14:19, 12 November 2021 (UTC)


 * Try using GROUP_CONCAT within the "fields" parameter (you will also have to do a "group by" - hopefully you know how to do that). Yaron Koren (talk) 01:32, 15 November 2021 (UTC)

Exclude images when storing table cell content
I'm trying to figure out if Cargo is suitable for my project - if I have an image (internal or external) in a table cell, how do I exclude it and only store/query the cell (link) text?
 * I'm not sure I understand - are you saying you have a value like "...some text here... [[File:SomeImage.jpg]] ...other text here..." in a template call on some page, but you want the value stored in the corresponding Cargo table to just be "...some text here... ...other text here..."? If so, you may have to create a custom Lua module for that text processing, with the Scribunto extension, and then put a call to that module inside the relevant field(s) in the relevant #cargo_store call(s). Yaron Koren (talk) 02:10, 15 November 2021 (UTC)
 * Yep, that's it. I have flag icons next to the country names (links) and want to store the text only, but also display the corresponding flag next to the country in tables creates with Cargo. Is there a built-in function for this in the extension? Since the filenames are always the same as the link text, perhaps it's easier to add the images with a script to keep the table cells clean for Cargo use?


 * How about storing the country name as plain text in a Cargo field (e.g. country=United Kingdom), and in the template that uses Cargo just display it as ?

Add category via Cargo query
I would like to "automatically" add category to the page via Cargo query. Lets have query like

This really adds category e.g., only that it also displays this in the page without parsing wiki code. So the result of the CONCAT is included twice, once as wikitext, once as nowiki text.

How can I get rid off the nowiki display?

--Radouch (talk) 12:51, 14 November 2021 (UTC)


 * Just found the solution: I have to add  to the query.
 * --Radouch (talk) 13:02, 14 November 2021 (UTC)

Length limit of CONCAT argument or template argument?
I try to use CONCAT as workaround for template bug, see above and. Unfortunately, it seems there is some limit (about 248 characters) for the length of CONCAT argument (e.g. the argument in  has 10 characters). Another possibility is that there is limit for template argument as whole: if counting with  it gives 255 characters (very suspicious number :-) ).

If this limit is reached, no data nor error is returned by the query.

Any ideas about the course of the problem and/or the solution of it?

--Radouch (talk) 14:13, 14 November 2021 (UTC)

Query of an object held on _pageName
This should be a simple query of attributes of an external object.

The _pageName is CPF00001 and the CP_Form_Variant is CPF00002. CPF00002 is the object that I would like to get additional field information. What am I doing wrong?

A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? Query: SELECT `_pageName` AS `Form Name`,`CP_Quality_Record` AS `Quality Record`,`CP_Process_Element` AS `Process Element`,`CP_Form_Applicability` AS `Applicability` FROM `cargo__CPF` WHERE _pageName HOLDS "CPF00002" ORDER BY `_pageName`,`CP_Quality_Record`,`CP_Process_Element`,`CP_Form_Applicability` LIMIT 100 Function: CargoSQLQuery::run Error: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'HOLDS "CPF00002" ORDER BY `_pageName`,`CP_Quality_Record`,`CP_Process_Element`,' at line 1 (localhost)


 * Try putting single quotes instead of double quotes around - that may make the difference. (Strings are meant to be enclosed in single quotes.) Yaron Koren (talk) 19:52, 15 November 2021 (UTC)


 * I think you want LIKE instead of HOLDS, as HOLDS is for lists and I don't think _pageName returns a list. Maybe Cargo isn't converting HOLDS into SQL seeing as the field isn't a list? Try . I wonder what that does for an empty CP_Form_Variant though.


 * Or, on second thoughts, if CP_Form_Variant will match the query exactly then.


 * I use double quotations for queries as a lot of my page names contain single quotation marks. Jonathan3 (talk) 09:51, 18 November 2021 (UTC)
 * Oh yeah, it should be "LIKE" (with the percentages) - I didn't even notice that part. Yaron Koren (talk) 15:24, 18 November 2021 (UTC)

#cargo_query: use "group by" and keep duplicates
Hello, I want to extract all the occurrences from a table, grouped by the values that a certain field has. Is that possible using 'group by'? Thanks. --Carloposo (talk) 09:19, 18 November 2021 (UTC)


 * I'm just an amateur at this but if you want to display every row then maybe it's "order by" that you want? Jonathan3 (talk) 09:52, 18 November 2021 (UTC)
 * Thanks Jonathan3. I didn't explain it properly, my bad. Suppose I have a table called "Fruits". It has these fields: Name, Color. I'd like to extract each row and print it like this:

etc. Using 'group by' results in:
 * Red
 * Apple
 * Strawberry
 * Yellow
 * Banana
 * Lemon
 * Red
 * Apple
 * Yellow
 * Banana

Thanks.


 * I think you could use templates to have queries within queries. Or Lua. Jonathan3 (talk) 11:45, 18 November 2021 (UTC)


 * Or the "table" output format can merge cells that have the same content, I think. I'm replying on mobile so the indentation isn't working - sorry. Jonathan3 (talk) 11:47, 18 November 2021 (UTC)
 * Thank you Jonathan3, they both worked. --Carloposo (talk) 13:01, 18 November 2021 (UTC)

Drilldown error
If a filter value in Drilldown contains the symbol ‘#’, then this leads to the error: [412c4661f00b7f08ae3ca0d8] /index.php/Special:Drilldown/redirects?selector=%3B%3B%3B%23 MWException from line 124 of <...>/html/extensions/Cargo/includes/CargoSQLQuery.php: Error in "where" parameter: the string "#" cannot be used within #cargo_query. This is especially strange when this filter option is listed before the input field among the options. It turns out that the error is caused by the link proposed by the Drilldown page.

I understand why this symbol is prohibited in arbitrary user queries, but in forms that explicitly provide for string input, it could be allowed (or automatically replaced with ). --StasR (talk) 17:56, 28 November 2021 (UTC)

Editing any page results in duplicate row
Hi, I just updated to MW1.37. I also updated all the extensions. Now, every time I edit a page, the page gets duplicated, even if I do not change anything. It happens using both PageForms and the Visual Editor. If I edit the page again all good, no other row is added. Thank you.


 * Does this happen with just one page type, or with all of them? (Assuming you have more than one Cargo-enabled template.) Yaron Koren (talk) 19:41, 30 November 2021 (UTC)
 * It happens with all the Cargo-enabled Templates.--Carloposo (talk) 08:31, 1 December 2021 (UTC)
 * Is this a public wiki? If not, would you be willing to pastebin the wikitext of one of the templates, and link to that? Yaron Koren (talk) 13:58, 1 December 2021 (UTC)
 * Sure https://pastebin.com/Qjsz7id2 --Carloposo (talk) 14:04, 1 December 2021 (UTC)


 * I can't reproduce this problem. Are you using the latest Cargo code? Yaron Koren (talk) 03:26, 2 December 2021 (UTC)
 * I am using the latest Cargo code. I think I found a pattern. If I edit a page and do not fill in all the fields, the record gets duplicated. If I re-edit the same page and fill in all the fields, the duplicated record gets deleted. --Carloposo (talk) 08:38, 2 December 2021 (UTC)
 * I think this issue is connected to the topic "Can't populate Cargo db fields with empty data" (see above). If you read there, I had issues populating the db with empty fields. I fixed it removing STRICT_TRANS_TABLES from mysql. In the meantime, you released a new version of Cargo with a fix, which I never had the chance to test because I was relying on the fix on mysql. I just put STRICT_TRANS_TABLES back and getting the same error when doing cargoRecreateData.--Carloposo (talk) 13:55, 2 December 2021 (UTC)
 * Should be linked to https://phabricator.wikimedia.org/T290165 and (somehow) https://phabricator.wikimedia.org/T285176 --Carloposo (talk) 09:03, 3 December 2021 (UTC)
 * Adding infos to give more context, sorry. I noticed PageForms (latest version 5.3) doesn't check mandatory fields anymore now. --Carloposo (talk) 10:52, 3 December 2021 (UTC)