Extension talk:Cargo

Using DataTables Styling Options
Currently, the "dynamic table" display option uses mostly default options for DataTables display format. It is highly preferrable to be able to customize DataTables' default styling options. For example: set "compact" mode for DataTables.

Is it possible to enable these options in any way?

Centering Map at Particular Location in Query
Is there a way to force a centering of a map to a particular point in a cargo query? Usually the map is zoomed and positioned such that all the "hits" of the query show and are centered. However, one can force a zoom level to override the computed scale. Is there a way to force a center point as well? That can be done using {{#display_map|center= ...} but I haven't found a way to incorporate that into a cargo query. Parma100 (talk) 13:55, 8 July 2021 (UTC)


 * Unfortunately, no; but a "center" parameter (or "starting bounds", which is what Page Forms has) would be a good idea. Yaron Koren (talk) 15:28, 9 July 2021 (UTC)


 * Okay. Weekend project. I'll see what I can come up with. Parma100 (talk) 16:01, 9 July 2021 (UTC)


 * I've got centering working for all three mapping services. Adds a  parameter and, if no   is specified, adds a zoom parameter to flesh out the services' requirements. Now I'm working on Leaflet clustering. I figure I'll wait until both tasks are working before I submit them for merging. Parma100 (talk) 13:26, 13 July 2021 (UTC)


 * Leaflet clustering now works. I'll be initiating a merge request directly. Parma100 (talk) 03:45, 15 July 2021 (UTC)


 * That's great - I'm looking forward to both patches! Yaron Koren (talk) 00:35, 16 July 2021 (UTC)

Query time span for calendar doesn't quite fit
Hi there! First of all: I love this extension. It is quite powerful.

Though, I have to admit that I only worked with this extension for roughly four hours and already hit what I would call a bug.

The situation given is like this:
 * I've got a simple table for events (Page, Start, End, Server (short name))
 * I would like to display the table in a calendar format
 * The calendar gets shown and some of the data displayed
 * The events in the table may have a time span of a single day or a whole month
 * Wiki is hosted by Miraheze so I don't have any file or database access

The problem is, some events are missing in the calendar. For example, I've got an event which takes place 2021-06-22 to 2021-07-20 and in the month July 2021 it doesn't get displayed. A similar problem occurs in weekly view, too.

The problem at hand (as far as I could analyze it) is with the way the data is being fetched. The GET looks like this:

As you can see, the initial query (and the previous&next buttons) set a limit (start=2021-06-27&end=2021-08-08) to the fetched data. In my eyes, my example event would partly take place in that limit. This might be a bug on fetching the dates within a given time frame. For me myself it would be enough if I could just set a higher threshold for start and end (is there a parameter for this?).

I hope I was able to describe my problem. Please don't hesitate to ask if you need further information or need to see our wiki. Thanks in advance!


 * You did in fact hit a bug! Sorry about that. I just checked in a fix for it - hopefully it will make its way to Miraheze before too long. Yaron Koren (talk) 17:39, 23 July 2021 (UTC)


 * Thanks for the quick fix! Miraheze merged the change into their productive system, too. I can confirm that the fix works fine. I'm looking forward to use Cargo more often in the future! Random visitor 12:48, 02 August 2021 (UTC)

Parsing order effect? Help welcome.
Using semantic mediawiki, I used to have a display programmed as :

I used link= in : http://dufal.fr/w/index.php?title=Mod%C3%A8le:ItemSuite&oldid=17785

Now replaced by link=

but I get a text version like

It seems to me that the parsing of [[Image: happens before the #cargo_query is extracted.

As a result, I cannot replicate with Cargo what used to work very well with SMW.

More specifically,
 * when the TOMEouLIVRE defines the field Image as a string, I see the name of the picture but not the image itself.
 * when the TOMEouLIVRE defines the field Image as a File, I see the image in the standard wikimedia format (framed, and with name under) which breaks my initial format.

Do I get something wrong ?

The first 4 highest level of page http://dufal.fr/wiki/Test use a series of formating of the field in TOMEouLIVRE table which I created to debug. The picture before the first title on that Test page is what I would expect to get (what I used to get with SMW's #show)

Many thanks.

Frederic.


 * Try adding "|no html" to the #cargo_query call. Yaron Koren (talk) 02:57, 8 August 2021 (UTC)

Also suffering from duplicate rows
I have recently forked a wiki from Fandom to my own host which makes use of extensive Cargo templates to store and query data. Unfortunately I'm finding that when I try save a page (via null edit) to create a new entry in a Cargo table the entry is being duplicated. An example can be seen on this page https://www.poewiki.net/wiki/Modifier:AbyssFireColdResistanceJewel1 When I run a null edit this entry gets duplicated and breaks subsequent queries.

Any idea why this might be happening?

Iamacyborg (talk) 21:15, 9 August 2021 (UTC)


 * I can't see this right now - right now there's no real data being stored for that page. Could you replicate the problem? Yaron Koren (talk) 03:14, 10 August 2021 (UTC)


 * You can see the dupes here https://www.poewiki.net/wiki/Special:CargoTables/mods Iamacyborg (talk) 07:18, 10 August 2021 (UTC)
 * Is there any data or additional information I could provide that might make debugging this easier? Iamacyborg (talk) 13:21, 10 August 2021 (UTC)


 * Okay, yes, I see the problem. That's very strange, and I don't know why it's happening. Does it only happen for null edits, or does it persist if you make some actual edit to a page? In any case, recreating the table will probably fix the issue, at least temporarily. Yaron Koren (talk) 14:01, 10 August 2021 (UTC)


 * I noticed in our templates we don't set any particular field as unique, might that be causing these dupes to happen? Iamacyborg (talk) 15:55, 10 August 2021 (UTC)


 * No, that shouldn't affect things - the duplicate check happens (or is supposed to happen) for all rows. Yaron Koren (talk) 21:40, 10 August 2021 (UTC)


 * Hmm, any idea how we might proceed? We have quite a few tables and having to rebuild them daily might get pretty slow, particularly if things break if someone inadvertently modifies or does a null edit of a page. Iamacyborg (talk) 07:22, 11 August 2021 (UTC)


 * I doubt you would have to rebuild them daily - you might only have to rebuild them once, unless null edits are planned to be a regular thing. Yaron Koren (talk) 19:42, 11 August 2021 (UTC)


 * Here's a weird one, we've found when editing pages that store data in our versions table that pages that end with a letter are not being duplicated https://www.poewiki.net/wiki/Special:CargoTables/versions You can see the duplicates in that table here https://www.poewiki.net/wiki/User:Iamacyborg/test Iamacyborg (talk) 07:17, 12 August 2021 (UTC)


 * After some further debugging, this seems related to Extension:PageImages, which when active, causes duplicate items to appear when a page is saved. Disabling the extension appears to have fixed the problem.


 * That's a great find - I just added a note about that to the "Known bugs" page. Yaron Koren (talk) 20:45, 19 August 2021 (UTC)


 * Is there any way to use both extensions? I started using PageImages fairly recently and have noticed more Cargo duplicates than usual. I don't know if it's related but it sounds likely. I like both extensions (though It's PageImages I'd give up if I had to choose one). Jonathan3 (talk) 22:35, 13 September 2021 (UTC)

We were having this issue over on Nookipedia and I found that performing a cargo_store without explicitly defining every field will lead to duplicates. I've created a task on Phabricator with more details. Thanks, ~ Super  Hamster  Talk Contribs 14:38, 1 September 2021 (UTC)

Calender colours don't fit for a compound query
Hi, I hit a bug when using the calendar output combined with a compound query.

The compound query is defined to do two queries. Each query defines a colour for the calendar like. The REST requests fits but the response doesn't fit as all results have the same colour.

Looking at the code I guess it's caused by the reuse of the variable  in CargoExport.php around line 122 and 148.

This bug might be lower priority as it is possible to walk around this problem by defining the colour as field:

--Random Miraheze user 20:38, 18 August 2021 (UTC)


 * That's strange; you can see the "color" parameter working correctly here. Maybe this is some bug that has since been fixed? Yaron Koren (talk) 19:05, 19 August 2021 (UTC)


 * Now that's interesting. Maybe it's a problem on our host's side. Sadly, I can't debug the code he's using. Well, we'll work with the workaround then. For documentation reasons, here is the request: https://counterside.miraheze.org/w/index.php?title=Special:CargoExport&tables[0]=EventPeriods&tables[1]=EventPeriods&join on[0]=&join on[1]=&fields[0]=Title,Start,End,Event=_pageName&fields[1]=Title,Start,End,Event=_pageName&where[0]=Server='SEA'&where[1]=Server='KR'&group by[0]=&group by[1]=&order by[0]=`Title`,`Start`,`End`,`Event`&order by[1]=`Title`,`Start`,`End`,`Event`&limit[0]=100&limit[1]=100&format=fullcalendar&color[0]=blue&color[1]=red&text color[0]=white&text color[1]=white&start=2021-05-30&end=2021-07-11&_=1629584478634 and the response: [{"title":"Circuit Link System","start":"2021-06-09 21:00:00","end":"2021-06-30 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Lee_Jisoo"},{"title":"Old Fear","start":"2021-07-07 21:00:00","end":"2021-07-28 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Old_Fear"},{"title":"Sweet Promotion","start":"2021-06-23 21:00:00","end":"2021-07-07 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Sweet_Promotion"},{"title":"Yoo Mina Recruitment Rate Up","start":"2021-06-30 21:00:00","end":"2021-07-21 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Yoo_Mina"},{"title":"Fenrir Yoo Mina Growth Limited Mission","start":"2021-06-30 21:00:00","end":"2021-07-21 21:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Yoo_Mina"},{"title":"Smell of the Game","start":"2021-06-21 19:00:00","end":"2021-07-19 19:00:00","color":"blue","textColor":"white","description":null,"url":"\/wiki\/Smell_of_the_Game"}] In this example, the "Smell of the Game" is the one which should be red, but it is not. In any case, thanks for your time! --Random Miraheze user22:38, 21 August 2021 (UTC)

Using dynamic data for the allowed values field parameters in #cargo_declare?
Hi, is it possible to use dynamic data, something like a Category: for example, to feed the  field parameter?

My goal here is that the end user could add new values to the  using a Form (created using PageForms) instead of manually editing the template where the   is called.

Igorabsorto (talk) 07:36, 26 August 2021 (UTC)


 * Hi! That's not recommended, because you would then have to recreate the table every time the set of values changed. It's better to not have "allowed values" in the Cargo declaration, and instead handle it in the form, with "value from category" or something like that. Yaron Koren (talk) 15:35, 27 August 2021 (UTC)


 * Great, that makes sense. Thank you, Yaron! Igorabsorto (talk) 09:17, 30 August 2021 (UTC)

Date display (again)
I have a similar query to that asked by Parma100 in Date Display - I have a date field, that I would like to change the format of the output. If the stored date has only year or month/year, the format is great (e.g. 1919 / March 1919) respectively), but if it also has day then the format is `Y/m/d` (e.g. 1919/03/14) which is not particularly 'friendly' and could be ambiguous - I would like the display to be the same as MW default i.e. 14 March 1919. I have not been able to decode what the solution was in that topic.

Playing with the code sample in topic linked above I can force the date to `j F Y` but that stops dates with less precision working as required - I've not been able to get a value from `Fieldname__precision` to use in a condition.

This is what I currently have in my template:

Many thanks, --Jaydublu (talk) 17:32, 28 August 2021 (UTC)


 * You could try something like this in a template:
 * Jonathan3 (talk) 22:43, 13 September 2021 (UTC)
 * Jonathan3 (talk) 22:43, 13 September 2021 (UTC)

One template that attach to multiple tables
Hi, is there a way that the recreate table function of "all" attached templates recognize them ? Actually only the last call to attach is recognized.

I know in "Storing data" it is written not to attach to more than one table. But nevertheless it seems that atleast the template itself gets everything right (i.e. it shows at the top that it writes to table A, writes to table B and so on).

But at the special page cargo-tables only the last is shown to write to the corresponding attached table. And only this is recreated if we redo the tables via web or php command.

In our case we want to have some templates that combine several cargo tables/templates (i.e. attach to) and write some fixed values at some fields to the tables and only some as template parameters are passed. Or is there a better way to do this? Hope the question was somehow clear ;)

Cheers, Brabi81 (talk) 13:24, 20 September 2021 (UTC)


 * In theory, there's nothing wrong with what you're doing; but in practice, I think there's a bug that prevents what you are trying to do from working. If that's true, then, until the bug is fixed, one thing you can do is have that template itself call other templates, each of which either declares or attaches to a single Cargo table. Yaron Koren (talk) 18:23, 20 September 2021 (UTC)


 * The thing with one template that call other templates works fine. Thanks for that idea! But I'll also try to keep one eye open to see if the bug is fixed in future patch notes. Thanks Brabi81 (talk) 05:43, 24 September 2021 (UTC)

Can't populate Cargo db fields with empty data
Hello, I've been running into this for weeks now. Basically, all Cargo fields appears to be mandatory. Running cargoRecreateData.php on a Table with a Page with some empty fields throws this:


 * What version of Cargo are you running? I think this problem has been fixed already. Yaron Koren (talk) 16:28, 22 September 2021 (UTC)


 * Hi, it's 2.8 --Carloposo (talk) 17:00, 22 September 2021 (UTC)
 * Hi Yaron, changing field types (Integer and Floats to String) fixes the issue. Does this ring a bell? BTW, the same issue is present with empty Date fields. --Carloposo (talk) 11:00, 23 September 2021 (UTC)


 * There are a few pages about this error and strict SQL mode, eg https://stackoverflow.com/questions/28606483/how-to-allow-empty-string-for-integer-in-mysql#39409037


 * I often save Cargo pages with empty date fields and it's ok, so maybe when Cargo creates tables it uses the database default which is different in your case. Jonathan3 (talk) 11:37, 23 September 2021 (UTC)


 * Thank you Jonathan, that made my day! --Carloposo (talk) 13:18, 23 September 2021 (UTC)


 * When I tried this out, all empty values were saved as "null", including for Integer fields. By coincidence, I just released version 3.0 of Cargo - I would try upgrading to this, because it could be that there was some recent change that is resulting in the different behavior. Of course, if you already modified your database settings, you might not able to know (and might not care) whether the code is acting differently. Yaron Koren (talk) 19:25, 23 September 2021 (UTC)


 * Thank you, Yaron. Is upgrading as easy as replacing the Cargo folder with the new one? --Carloposo (talk) 07:44, 24 September 2021 (UTC)
 * Yes. Yaron Koren (talk) 14:19, 24 September 2021 (UTC)

Quick way to switch in multiple replacement tables
Because of the duplicate row thing, I've set up a handy Windows shortcut using plink to log in and create replacement tables - but then it's a pain having to go to the website to click the ticks for several tables. It would be nice to be able to do them all at once.


 * Maybe there could be a button on Special:CargoTables to do that?
 * Or maybe a new command line script?
 * Or maybe the existing script could have an extra parameter to switch in the replacement tables at the end?

Thanks Jonathan3 (talk) 10:01, 24 September 2021 (UTC)


 * I've just saved all the pages as bookmarks that can be opened all at once in different tabs, then I can click "Switch" for each. It's quick enough, and uses replacement tables. (I found that when I recreated the tables nightly without a replacement table, they still would have duplicates, which maybe was because of the clashing with the job queue problem.) Jonathan3 (talk) 20:55, 5 October 2021 (UTC)

Mediawiki still showing data even though Cargo database deleted
I am making a catalog for my books and have hit the data duplication bug. I have mediawiki 1.36.1 and Cargo 3.0, on PstgreSQL 13. In the process if trying to debug the issue I erased all the cargo data related to the catalog. However the data still shows up in mediawiki even after purging. I am not seeing where it is getting the data from. Can you please help? Also, is there any progress on the data duplication bug?


 * What do you mean by the data showing up? And what data duplication are you seeing? I personally haven't seen this problem in a long time. Yaron Koren (talk) 01:40, 7 October 2021 (UTC)


 * When I go to Catalog:Books, the book shows up and I can still view its details.


 * As for data duplication, when I edit a book, a duplicate appears in the sql database and in the cargo table.


 * I am working on a smallest case example which may help clarify.


 * What's on the page Catalog:Books? And an example for the duplication problem would help a lot, yes. Yaron Koren (talk) 13:08, 7 October 2021 (UTC)

Example of duplicate data
As promised, here is an example of my book catalog.

The main Book template is the following:

This is the "Book" template.

The Book form is

This is the "Book" form. To create a page with this form, enter the page name below; if a page with that name already exists, you will be sent to a form to edit that page.

Title:

Subtitle:

Author(s):

Genre(s):

Publisher:

Year of publication:

ISBN:

Number of pages:

Owner:

I first create the tables which are created appropriately. I then create a very simple book with just a title, and everything else is blank.

In 'Category Books' only one book shows up as expected (sorry, cannot seem to be able to insert screenshot).

Category:Books Jump to navigation Jump to search Pages in category "Books"

This category contains only the following page. A

A Book

But in the Books Table I have two books listed: Table structure:

Title - String Subtitle - String Authors - List of Page Genres - List of Page Publishers - List of Page Year_of_publication - Date Number_of_pages - Integer ISBN - Integer Owners - List of Page

This table has 2 rows altogether.

Recreate data. Page 	Title 	Subtitle 	Authors 	Genres 	Publishers 	Year of publication 	Number of pages 	ISBN 	Owners A Book 	A Book A Book 	A Book

In the PostgreSQL database there are two entries with different _IDs:

_ID 	_pageName 	_pageTitle 	_pageNamespace 	_pageID 	Title 	Subtitle ...... 1       a book	     a book	                 0      201     A Book	         ...... 2       a book	     a book	                 0      201     A Book	         ...... This is the duplication problem I am trying to fix.


 * Your data structure looks fine. I didn't notice before that this was a PostgreSQL database - I'm guessing that is part of the issue; Cargo works with PostgreSQL, though it's much less well-tested than with MySQL, unfortunately. If you recreate the table, does the duplication go away? And if so, does it come back every time you resave the page? Yaron Koren (talk) 14:44, 11 October 2021 (UTC)


 * I went through the data structure line by line to simplify the problem to the bare minimum. The bug seems to be in the template. in #cargo_declare I declared a Subtitle, but in #cargo_store I did not use it. When I made that fix, everything works now. I tried it on a three line data structure and the same thing shows up. So I believe this is a bug and it should have raised an error or warning. --Lbundle (talk) 19:41, 11 October 2021 (UTC)

Critical performance drop after version 2.4
Yaron, hello! After a long break, I decided to update my wiki. Its peculiarity is very large pages that generate a lot of records. After deploying the update on the auxiliary test server, I decided to try to overwrite one page. The result was either a 504 error, or only partial saving of records (from 150 to 300, although the page generates about 1500 records). I tried rolling back from the Master version to previous versions. It worked (and significantly faster) only when returning to version 2.4.

The test server runs Debian 11, MariaDB 10.5.11 and MW 1.35.4. --StasR (talk) 22:23, 11 October 2021 (UTC)


 * That's very interesting; it might be due to the data duplication check, which Cargo right now runs before every insert. With the old code, does it really save all 1,500 records? Yaron Koren (talk) 23:33, 11 October 2021 (UTC)


 * Here are the PageValues of the largest page. It contains 1991 entries. Yes, sometimes there are errors when recording. I have scripts that check that the values are written to the database to the end and without duplicates. But in general, everything works. --StasR (talk) 08:04, 12 October 2021 (UTC)


 * That's very interesting. If you're willing, it would be great if you could try it again with the latest code, but making a small change to the code, to see if the duplication check is really the issue. In the file includes/parserfunctions/CargoStore.php, just change this line (line 328 or so):

$rowAlreadyExists = self::doesRowAlreadyExist( $cdb, $title, $tableName, $tableFieldValues, $tableSchema );


 * ...to this:

$rowAlreadyExists = false; // self::doesRowAlreadyExist( $cdb, $title, $tableName, $tableFieldValues, $tableSchema );


 * If you try that, please let me know if that fixes the performance problem. Yaron Koren (talk) 14:12, 12 October 2021 (UTC)


 * I checked the change on version 2.5. It didn't help. --StasR (talk) 14:51, 12 October 2021 (UTC)


 * As a result of these experiments, I lost the “Page Values” menu item. Do you have any ideas how to restore it? --StasR (talk) 15:21, 12 October 2021 (UTC)


 * It turns out that this happens after executing “git checkout 2.4”. --StasR (talk) 15:49, 12 October 2021 (UTC)


 * Okay, thank you for trying that. And thanks for clarifying that the problem came in version 2.5. Given that, I have a new theory as to what is causing the slowness: this change, which added the $cdb->lockForUpdate call to CargoStore.php. Could you try commenting out that line in later versions, and see if that speeds things up? Yaron Koren (talk) 16:27, 12 October 2021 (UTC)


 * YESSSS! ) Checked for REL1_35 (aka 2.6) and for ‘master’ aka 3.0 (but I am not ready for ‘master’ due to changes in the storing of empty values). Thanks. --StasR (talk) 17:04, 12 October 2021 (UTC)

Not show pages' namespaces on Special:Drilldown
Is this possible?

The problem (for me) currently is that they're all listed alphabetically according to namespace, which means that each column is like A, A cont'd, A cont'd...

Thanks. Jonathan3 (talk) 21:03, 12 October 2021 (UTC)

Help with conceptual knowledge gap with regards to implementing Cargo
Very naive question—I have have read many of the "getting started" type guides (like the link below) and watched several videos to wrap my mind around how Cargo works and even produced a simple working example on my wiki but I am still confused by one point that I can't quite seem to find information on in these guides (maybe only puzzling to me). I don't understand how one is supposed to 'populate' the Cargo table once it has been declared and stored. For example, with a simple example like an info box with authors and books) or something like that, what if you had 100s of entries in a csv that you wanted to upload to define the relations in an Infobox—i.e. the actual "data"? Do these have to be manually entered one row at a time on the table viewer or through the form that is generated with a Cargo table? I tried playing around with importing csv's into the wiki as well the External Data extension (which is mentioned elsewhere in this discussion thread) but I'm not sure how to take advantage of either to populate data into an existing Cargo table. Any help or a point toward the right resources would be great.

https://workingwithmediawiki.com/book/chapter16.html


 * The Data Transfer extension may be what you need in this case. (Although maybe you've used that already, because you talked about importing CSVs.) Yaron Koren (talk) 00:10, 15 October 2021 (UTC)

How to find the empty value "where" in lua
how to find the empty value "where", just like in the Drilldown, you can select the empty value "nothing" _none.

example if frame.args ['class'] ~ = '' then args.where = args.where .. 'AND requires_class HOLDS LIKE "%' .. frame.args ['class'] .. '%"' else need to find all empty "class" values end

Thanks. Oleksii