Extension talk:Cargo

Error: 1069 Too many keys specified; max 64 keys allowed
Recreating a CargoTable via the cargoRecreateTable.php script gave me the following error:  The table contains 61 fields.

The query that threw the error is:

Cargo always seems to create indexes for all table columns and also a sort of __precision index (I don't understand why) which eventually leads to a number of keys larger than 64 for this specific table.

A solution could be to disable indexing (apart from applying a leaner database design). Is there a way to disable indexing in Cargo? Would that make sense?


 * Indexing of all fields was add just a month ago - I'm glad you found this issue now, before the next version is released. I didn't know about that keys limit that MySQL has. I just checked in a fix for this, so if you get the latest code and recreate the table again, hopefully it will work. Yaron Koren (talk) 14:54, 7 August 2018 (UTC)


 * Thanks for the quick reply and the fix! We'll implement it this week and i'll let you know. I did not know of the key limit either. Apparently the key limit can only be changed by recompiling the code. I'd be interested in the reason for enabling indexing on all fields? Did you notice performance improvements? --Schtom (talk) 09:21, 8 August 2018 (UTC)


 * I misspoke before - not all fields are indexed; just fields that have a type other than "Text", "Wikitext" and "Searchtext". (And for date fields, there's an extra field, for the precision, that is also indexed.) The idea behind this is to get queries to run faster in general. For any specific table in any specific wiki, there may be only one or two fields where an index will make anything run faster - depending on the size of the table, etc. I could have made indexing a setting in #cargo_declare, to let admins decide which fields should get indexed and which should not. However, I figured it would be easier for admins, and more foolproof, to just index every field that could conceivably be used for sorting or joins. That's especially true given that I think most people running Cargo will not be DB experts. There's a tradeoff there in terms of slower writes to the DB, but I think it's worth it - on a wiki, read speed is much more important than write speed. Still, this is a very new feature, and I'm not sure it's the correct approach - I'd be glad to hear your thoughts on the matter. Yaron Koren (talk) 14:38, 8 August 2018 (UTC)


 * Thanks for the explanation and sorry for the late reply. We also did not implement your fix (yet), as our client decided to change his database design. Personally, I do think your approach and reasoning makes a lot of sense. However, I cannot assess the actual performance improvements. I wanted to let you know about another issue that occurred with the same client. He reported a database error when creating a new table. We then found that the identifier limit is restricted to 64 chars (Error: 1059). One of the identifiers was throwing the error was 'deadline_minigrid_people__precision_webmo_indicator_minigrid_people' (67 chars). The composed index keys may become an issue as well. It seems there could be a trade-off between usability/readability (users are forced to use shorter names) and performance. So far we have not rolled out our cargo-driven wiki and are merely beta-testing with this one client. These two issues (64 keys, 64 chars) were the only ones reported to us.--Schtom (talk) 13:53, 21 August 2018 (UTC)


 * Those are some pretty long table and field names! Anyway, thanks for letting me know about the problem, and sorry about that. I just checked in what I think is a fix. Yaron Koren (talk) 03:45, 22 August 2018 (UTC)


 * Kindly note that we do not consider these two issues as major problems. We find your extension incredibly useful. I agree that both the field and table names are pretty verbose. Still, I assume the identifier error was thrown by one of the index keys that is composed of both table and field name + underscores + the 'precision' label that you add. As a suggestion: You could maybe shorten the latter to leave a little more 'explanatory space' for the user e.g., 'pr' instead of 'precision'. --Schtom (talk) 10:38, 23 August 2018 (UTC)


 * That's great to hear! Yes, reducing the size of "precision" could make sense. I'll see how the current solution works out. Yaron Koren (talk) 15:27, 23 August 2018 (UTC)

Ad block (uBlock Origin) blocks Cargo script for displaying bar charts
On this page I noticed if I have uBlock Origin enabled the bar charts do not load. Disabled uBlock and the bar charts load normally.

Chrome dev console reveals this error:

Uncaught TypeError: Cannot read property 'disabled' of undefined at String. (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:458) at Array.d3_transitionPrototype.filter (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:221) at SVGSVGElement. (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:458) at load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24 at d3_selection_each (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24) at Array.d3_selectionPrototype.each (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24) at Array.chart (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:455) at Array.d3_selectionPrototype.call (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24) at Object.generate (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:562) at renderLoop (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:246)

Please advise. --pcj (talk) 19:47, 10 August 2018 (UTC)
 * Never mind, this was a Gamepedia-specific issue and has been resolved. --pcj (talk) 02:18, 2 September 2018 (UTC)


 * Good news! I had no idea what was causing the problem. Yaron Koren (talk) 13:03, 2 September 2018 (UTC)

SMW subobject transitioning to Cargo, table display issues
I'm trying to move a wiki from SMW to Cargo and one of the things we use is subobjects. E.g. A character may need two types of materials to ascend to the next stage -- 4 feathers and 5 fluid. Ideally this would be stored as a list with entries that have properties. i.e. Feathers (4);Fluid (5).

How do I do this in Cargo, or does anyone have a subobject-like store they could show an example of, please? Thanks!

Toriayl (talk) 14:21, 14 August 2018 (UTC)


 * One of the nice things about Cargo is that all data is stored in the same way, via #cargo_store - there's no distinction between regular storage and "subobjects". In other words, assuming you store all data via templates, it doesn't matter whether there's one template call on the page, or multiple calls. Does that make sense? Yaron Koren (talk) 17:27, 14 August 2018 (UTC)

Thanks for the reply, but sorry, still not really getting it. The point of sub objects is to link data to specific parameters entered. E. G. An address would have its post code and road number linked to it. Are you saying that my storage needs to be multi row within each page to accomplish similar functionality? I currently have a single template that loops through items and amounts entered and stores each as subobjects. I can't really use multiple templates instead, but maybe I need to figure out how to store them on different rows within the same template.. Was hoping there would be a single-row solution.

Alternatively, am I meant to store directly/is it possible to do so into a helper table for a field containing a list of values? Toriayl (talk) 22:25, 14 August 2018 (UTC)


 * Oh - I didn't realize you have a single big template that's storing an entire "table" of values. You can still use Cargo for this - you would just need to create a new template that holds one row of that table, then have the big template call that template in each place it previously called #subobject. Does that make sense? Yaron Koren (talk) 01:42, 15 August 2018 (UTC)

Thank you, I was hoping to have the info in one table but it sounds like another table would be a better idea for this. And rows would look like: Artemis - Feather - 1 Artemis - Feather - 5 Artemis - Fluid - 5 Hopefully that's on the right track but I'll worry about it later. I have more questions for now! (Sorry. XD) Not really subobject-related now, though.
 * 1) Is there a way to pull just the first result of a joined table? i.e. If Artemis matches with two entries on another table, and I want just something from the first entry matched. This one is not 100% needed; I'll probably manually input the values needed into each page (200 of them...), but I think I've read something about SQL being able to do this (though not very cleanly) so thought I might as well ask in case there's something for Cargo I've missed.
 * 2) I am having some trouble generating a table, particularly with the values in a list on it. This is the table: https://grandorder.wiki/Special:CargoTables/CEs Related template: https://grandorder.wiki/index.php?title=Template:CEInfoBox&action=edit Test page with issues: https://grandorder.wiki/User:Toriayl/sandwich2
 * The Effects parameter is stored in Effects=List of String, which appears to work fine on the Special:CargoTables table-browsing link above.
 * However, when I try to put it in a dynamic table query, I get a database error if I put it as the last item displayed.
 * The query works if I put it as the first or second item displayed, but lists with three items have the second item within pre tags. Something to do with the dot delimiter..?
 * Using template as an alternative, effects work but html tags (br, div style etc) are displayed.

Help would be really appreciated, thanks! Toriayl (talk) 14:20, 29 August 2018 (UTC)


 * Alright - 1) there may be a way to do with some hack, maybe using "group by=", depending on the exact structure of the data - I'm not sure. 2) I'm not entirely sure what's causing the strange behavior with the Effects field, but I think it's at least partly due to the somewhat unusual way in which that value is put together. The "cleaner" way to do it would be again with a multiple-instance template holding the effects and "LBs" - so that each effect is directly connected to its LB value, and so that you don't need to have an artificial limit on the number of effects. But barring that change, you may be able to get around this problem by replacing "Effects" in the query with "Effects__full=Effects". Yaron Koren (talk) 15:27, 29 August 2018 (UTC)

Thanks, but I don't get how it's unusual...isn't it a fairly normal list? Effects__full=Effects got around the database error, but it's still showing the 'second item pre' issue. Also pardon me if I'm wrong, but that's the same fix you were suggesting for subobjects, isn't it? i.e. Create another table with the effects, then join it with the main table.

But won't this cause the same problem as 1)? I want to only display one line with all the effects for an item, I don't want three lines for three effects, with all of the other info duplicated......

Toriayl (talk) 13:01, 30 August 2018 (UTC)


 * Yes, that's the same fix I suggested. In both cases, you have a table of data stored on the page, so it should be stored in a tabular fashion. When you display the data, you can display it all on one line, even if it represents three different "rows". What do you mean by "all of the other info duplicated"? Yaron Koren (talk) 19:18, 30 August 2018 (UTC)

How do I display it all on one line?! Have a look on my sandbox again, the first section. This is the only way I know how to query/display the table, and it shows multiple lines, one for each match in the join...and results in a duplication of the other information (rarity, ID, etc).

Toriayl (talk) 11:41, 1 September 2018 (UTC)


 * If you add "|format=list" to the query, it will display all results on one line. You can also do that with the "template" format, which gives you complete control over the display, although with that you have to create a template. The duplication thing is a problem, though it's a problem in the storage, not the querying. I wish I knew how to prevent duplicates in all cases. In any case, if you do a recreate data on the "ServantsNP" table again, it will probably fix the problem in that table. Yaron Koren (talk) 02:27, 2 September 2018 (UTC)

Thanks for trying to help, but it feels a little like we're on different wavelengths here. ^^" I have one page that has two rows associated with it. I ran a recreate data on the ServantsNP table and it gives me the same thing. E.g. Alexander B+ NP, Alexander B++ NP. This is correct. Alexander is linked to two different types of NP. But I want it to display, instead of on one line: It doesn't look like format=list or template can do this? Toriayl (talk) 15:41, 5 September 2018 (UTC)

Displaying a table on a page for searching by values
Maybe someone already asked such a question.

Is it possible to create a table on a page with search? or create such a table similar to Drilldown. imgur - (Oleksiy) 10:47, 22 August 2018 (UTC)


 * Do you know about the "dynamic table" format? Yaron Koren (talk) 16:09, 22 August 2018 (UTC)

Cargo not updating tables, nothing happening.
Hi, I am having an issue where cargo is not updating tables. Only some pages are being successfully stored while others aren't, despite there being no real difference. Existing pages also do not update despite the job queue being empty. Steps taken:


 * Recreated table using  - nothing happens but jobs are added, jobs will never progress (even with high run rate.) Also tried manually recreating data.
 * Manually ran jobs through the server, created only half of the data then stopped. No jobs left.
 * Recreated data on tables that had no population whatsoever - no jobs created.
 * Manually ran jobs through server, nothing happens.
 * If existing data on the table is updated, it does not update or create a job.

Semantic Mediawiki worked flawlessly beforehand, no changes were made between the two except for an update from MW 1.29 to 1.31, any idea to what the issue may be? Steps were taken from the FAQ and common problems as well, none seem to be the issue. -- 2607:FEA8:E2E0:602:CCB2:DD2C:F53E:4C48 08:17, 24 August 2018 (UTC)


 * I didn't understand all of this. What do you mean by "manually recreating data", and what happened when you tried it? What do you mean by tables with "no population whatsoever" - empty tables? If so, what should have happened that didn't happen? What do you mean by "existing data" being updated - a regular page save? And when you wrote "Semantic Mediawiki", did you mean Cargo? If not, I don't see the connection. Yaron Koren (talk) 13:22, 24 August 2018 (UTC)


 * Sorry, let me be more clear, I am not 100% on the terminology as of yet:
 * We used SMW before transferring over to Cargo, just updated templates using the migration guide. Only the first time we recreated data using the web browser then ran runJobs.php did it partially work. It did not grab every page's data and stopped midway. (SMW is completely disabled at the moment.)
 * "Manually created data" by running runJobs.php / php cargoRecreateData.php
 * The tables specifically aren't being updated, despite it should be picking up the changes, the jobs aren't being created/aren't functioning correctly.
 * I have a local copy of Cargo and confirmed the templates used work and will successfully create the table and fill them with data, its only on the server does it not, despite there being no errors.
 * I hope this helps, I am completely stumped. Is there any further debugging I can do? -- 2607:FEA8:E2E0:602:4D63:99F6:223C:D164 04:45, 26 August 2018 (UTC)


 * I just finally tried running Cargo on MW 1.32.0-alpha, and it seems to be working fine for me, including recreating data. So I don't know what's going wrong. Just so I understand - were you using Cargo with MW 1.29? And what version of Cargo are you running? Also, this isn't a public wiki by any chance, is it? Yaron Koren (talk) 03:44, 29 August 2018 (UTC)
 * Heres the link to the wiki, all the information should be on there. As an example, the machine table only has 4 entries, when 20-ish pages actually use the template and should be storing data. --2607:FEA8:E2E0:602:68B1:ACDA:6EA3:A27 22:49, 29 August 2018 (UTC)


 * Alright. Do you see any error messages when you run cargoRecreateData.php? Yaron Koren (talk) 18:15, 31 August 2018 (UTC)
 * No error messages, says its "done" but doesn't actually do anything. -- 2607:FEA8:E2E0:602:B01E:BB64:DD4F:C263 20:51, 1 September 2018 (UTC)
 * Additionally : I can get this to work on a local copy but not the actual server. That being said I have no personal access to the server, so it's difficult to diagnose. -- 2607:FEA8:E2E0:602:D9EB:B7E9:C8A0:6D6D 01:10, 3 September 2018 (UTC)
 * Is there any difference in versions between the local copy and the actual server? Also, if you don't have access to the server, how were you able to run cargoRecreateData.php? Yaron Koren (talk) 02:48, 3 September 2018 (UTC)
 * It's a weird administration system, I am able to ask the server admin to debug some stuff for me, but they are very slow at doing so and won't give me access - so the only information I get is basic. It makes this process much more difficult and frustrating. There is no difference in the versioning. -- 2607:FEA8:E2E0:602:1569:7DD9:88C1:6AD0 20:15, 3 September 2018 (UTC)
 * Update: I found out SMW was improperly uninstalled, after properly having it removed, cargorecreatedata.php started working and successfully filled tables. However, tables will still get stuck trying to populate via the web interface, and they will not update automatically if a page is edited. -- 2607:FEA8:E2E0:602:F5FA:7A1F:1CDF:59B3 19:03, 6 September 2018 (UTC)
 * Update 2: Debugging further found that jobs aren't being created by cargo to do the work, despite saying  The showjobs.php shows 0 jobs and none failed / abandoned. -- 2607:FEA8:E2E0:602:6134:8886:DF34:6CF7 21:23, 8 September 2018 (UTC)


 * This sounds a lot like the issues we have on Gamepedia. Does it recreate the data as expected when you blank edit the pages that store? --RheingoldRiver (talk) 22:25, 4 September 2018 (UTC)
 * No, it doesn't. -- 2607:FEA8:E2E0:602:A012:9E7D:B5D3:26CE 17:55, 5 September 2018 (UTC)

It's great that you were able to get cargoRecreateData.php to work, and that you diagnosed the issue with the web interface (it's not creating jobs). I don't know why jobs aren't being created, though. Do you know if other processes that create jobs are working on your wiki? For example, modifying a template should cause a job to get created for each page that calls that template, to refresh its contents - does that happen? Yaron Koren (talk) 01:42, 9 September 2018 (UTC)
 * Yes, other jobs are being created and work correctly, cargo does not. Cargo says it has created jobs at this page, but showjobs.php shows 0 jobs. Non-cargo jobs are also being processed correctly. -- 2607:FEA8:E2E0:602:C8E9:DA9F:601F:C5F 20:31, 10 September 2018 (UTC)

Hello, I'm having the same issue on https://dicoado.org/dico/Sp%C3%A9cial:CargoTables I was using SMW until now. First time I rewrote the template to use Cargo it worked fine but my variables weren't optimaly chosen. I deleted the tables and started fresh again. As you can see, when I used cargoRecreateData.php, only 16 articles were included in the table (it should be over 2k). Although it says other jobs are being created, cargo does not. DSwissK (talk) 05:36, 13 September 2018 (UTC)


 * At the moment, it looks like you have an even bigger problem - the cargo_tables DB table was deleted, but it shouldn't have been. You should call MediaWiki's update.php script again to restore it. Yaron Koren (talk) 15:09, 13 September 2018 (UTC)


 * Thanks for the fast answer. Yes, I was working on it. :) I found the issue. Somehow, I had only 20ish pages linked to my template when calling the special page so I used refreshLinks.php and now it's working like a charm ! :D Thanks ! DSwissK (talk) 19:35, 13 September 2018 (UTC)


 * Great! Yaron Koren (talk) 18:43, 14 September 2018 (UTC)

Cargo not updating tables
Hello Yaron, I have the same issue with mw1.31. : "Cargo not updating tables, nothing happening". The table is created with a form and a template (using #cargo_declare in the template) the data is store in the mw page (using #cargo_store in the template) when I save the form but the cargo table is empty. I used a separate cargo database. The only way I have found to populate the cargo table is to run "php cargoRecreateData.php" in the Cargo's /maintenance directory. (Jacques Masbou)


 * What version of Cargo are you running? Is this a public wiki? Did it work better with an earlier version of MW? What database are you running? Does everything work fine if you call cargoRecreateData.php? Yaron Koren (talk) 16:48, 29 August 2018 (UTC)

Cargo version 1.7 php 7.22 mw1.31 It is not a public wiki mysql 5.7.19 when I call cargoRecreateData.php sometimes it works sometime it does not work With cargo 1.5, php 5.6.31 mysql 5.7.18 and mw 1.29 it works fine

Update : with mw 1.29.2, php 7.1.21 mysql 5.7.19 cargo 1.7 it works fine


 * It's great that you isolated that the problem is due to the MediaWiki version, not the Cargo version. Still I don't know what's causing the problem - I'm running Cargo on MW 1.31 myself, and it seems to run fine. Do you have more than one Cargo table on your wiki? If so, is the problem happening in all of them? Yaron Koren (talk) 03:00, 2 September 2018 (UTC)


 * We have a similar issue, and a possible related issue which may give some clues. Cargo 1.7, PHP 7.0.31 (fpm-fcgi), Mediawiki 1.31.0.
 * When updating a template no job appears to be created to update the table data. When we run the maintenence/recreateData.php script it works as expected and creates the tables. However, when we run the "recreate data" command through the web ui on a table, it creates the table just fine but we are unable to successfully delete the tables, or switch to the new tables.
 * Recently we had an issue with the AWS extension which turned out to be due to the fact that our MediaWiki is configured to use a prefix on database tables. It is possible that this is a similar issue, since database prefixing is not standard wiki setup and usually not noted when it is? Ian T Small (Ian T Small|talk) 18:00, 18 September 2018 (UTC)


 * Just to be clear - calling cargoRecreateData.php and calling "Recreate data" from the wiki both work, but the latter leads to a situation where you can't delete the table, or switch in its replacement table? Yaron Koren (talk) 19:48, 18 September 2018 (UTC)

Error when creating a new data table
Hi,

I got this error when trying to create a new data table :

{   "error": { "code": "internal_api_error_DBUnexpectedError", "info": "[6377a1755353fc18463173ed] Exception caught: Wikimedia\\Rdbms\\Database::close: mass commit/rollback of peer transaction required (DBO_TRX set).", "*": "Wikimedia\\Rdbms\\DBUnexpectedError at /var/www/wiki/w/includes/libs/rdbms/database/Database.php(916)\n#0 /var/www/wiki/w/extensions/Cargo/includes/CargoUtils.php(818): Wikimedia\\Rdbms\\Database->close\n#1 /var/www/wiki/w/extensions/Cargo/includes/CargoUtils.php(534): CargoUtils::createCargoTableOrTables(Wikimedia\\Rdbms\\DatabasePostgres, Wikimedia\\Rdbms\\DatabasePostgres, string, CargoTableSchema, string, integer)\n#2 /var/www/wiki/w/extensions/Cargo/api/CargoRecreateTablesAPI.php(31): CargoUtils::recreateDBTablesForTemplate(integer, boolean)\n#3 /var/www/wiki/w/includes/api/ApiMain.php(1579): CargoRecreateTablesAPI->execute\n#4 /var/www/wiki/w/includes/api/ApiMain.php(535): ApiMain->executeAction\n#5 /var/www/wiki/w/includes/api/ApiMain.php(506): ApiMain->executeActionWithErrorHandling\n#6 /var/www/wiki/w/api.php(94): ApiMain->execute\n#7 {main}" } }

Looking at the SQL logs (I am using PostgreSQL), I can see the correct  statements, but no  :

2018-08-25 19:17:49 CEST [14818-22] wiki@wiki LOG: statement: CREATE /* Wikimedia\Rdbms\Database::query  */ TABLE "cargo__Artistes__Genres" ( "_rowID" Int, "_value" Varchar(300) ) 2018-08-25 19:17:49 CEST [14818-23] wiki@wiki LOG: statement: CREATE /* Wikimedia\Rdbms\Database::query  */ INDEX "row_id_Artistes__Genres" ON "cargo__Artistes__Genres" ("_rowID") 2018-08-25 19:17:49 CEST [14818-24] wiki@wiki LOG: statement: ROLLBACK /* Wikimedia\Rdbms\Database::close  */ 2018-08-25 19:17:49 CEST [14817-31] wiki@wiki LOG: statement: ROLLBACK /* MWExceptionHandler::rollbackMasterChangesAndLog  */ 2018-08-25 19:17:49 CEST [14817-32] wiki@wiki LOG: statement: BEGIN /* Wikimedia\Rdbms\Database::query (LCStoreDB::get)  */

Any help is welcome :)

--Erdnaxeli (talk) 17:25, 25 August 2018 (UTC)


 * Ok, I tried with the git master branch (instead of the latest zip) and it seems the bug was fixed. --Erdnaxeli (talk) 15:37, 31 August 2018 (UTC)


 * That's great to hear - I was planning to set up Cargo on Postgres to try to debug the issue, but I guess now I don't need to. Yaron Koren (talk) 18:14, 31 August 2018 (UTC)

File field : edit redlink instead of File:myFile
I am using
 * Special:CreateClass

To obtain Table structure:
 * bebe - String
 * myfile1 - File
 * bibi - String
 * myfile2 - File

then has uploadable for File fields.
 * Form:MyFileB1

Page creation and file uploading is working, But my page shows a redlink for File field
 * index.php?title=Blue16Test1A.jpg&action=edit&redlink=1

I think it should it be linking to
 * File:Blue16Test1A.jpg

Would you help me to link to uploaded file ?

Thanks in advance. --Wpe (talk) 15:15, 26 August 2018 (UTC)


 * Yes, the template should have the string "File:" before the template field holding the filename, so it should look like " File: ". Sorry, I realized that Special:CreateClass and Special:CreateTemplate don't put in that necessary prefix when creating the template; I just checked in a fix to Page Forms, so hopefully they will now. Yaron Koren (talk) 03:22, 27 August 2018 (UTC)

Missing _pageData
How to solve this problem? [9f30956ef6e5b7f44122218a] /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:CargoTables/_pageData MWException from line 196 of /home/hydra/public_html/extensions/Cargo/includes/CargoUtils.php: Error: Table _pageData not found.

Backtrace:


 * 1) 0 /home/hydra/public_html/extensions/Cargo/specials/CargoTables.php(86): CargoUtils::getTableSchemas(array)
 * 2) 1 /home/hydra/public_html/includes/specialpage/SpecialPage.php(522): CargoTables->execute(string)
 * 3) 2 /home/hydra/public_html/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(string)
 * 4) 3 /home/hydra/public_html/includes/MediaWiki.php(287): SpecialPageFactory::executePath(Title, RequestContext)
 * 5) 4 /home/hydra/public_html/includes/MediaWiki.php(862): MediaWiki->performRequest
 * 6) 5 /home/hydra/public_html/includes/MediaWiki.php(523): MediaWiki->main
 * 7) 6 /home/hydra/public_html/index.php(43): MediaWiki->run
 * 8) 7 {main} --Ruba159753 (talk) 13:54, 4 September 2018 (UTC)


 * Right, the _pageData table doesn't get created automatically - instead you have to create it by defining its columns, then calling a script. See here. Yaron Koren (talk) 19:19, 4 September 2018 (UTC)

Declaration Parameter to Index a Field
Would it be possible to add a flag in declaration of a field that makes the column indexed (but not unique)? --RheingoldRiver (talk) 18:49, 5 September 2018 (UTC)


 * Actually, in the latest Cargo code, every field now gets indexed - except for fields of type Text, Wikitext and Searchtext, and fields after #50 or so in tables with a large number of fields. Yaron Koren (talk) 23:16, 5 September 2018 (UTC)


 * Ah ok, we are still not on latest version until after MediaWiki upgrade. Good to know, thanks. --RheingoldRiver (talk) 00:57, 6 September 2018 (UTC)

Error: 1054 Unknown column '_pageData_alias._fullText'
Hi, I've been getting this error on a full text search. On investigation it looks like your more complex demonstrations of the MATCHES are suffering from the same issue. see error reports on http://discoursedb.org/wiki/Search_demo


 * Sorry about that - that's due to some recent changes that weren't fully tested. I just checked in what I think is a fix for this problem. Yaron Koren (talk) 14:01, 18 September 2018 (UTC)


 * Thanks for the quick fix. James Driscoll

Proposal for hook 'CargoBeforeStoreAllData'
Can I suggest the addition of a hook in CargoPageData.php ?

a simple call to

Hooks::run( 'CargoBeforeStoreAllData', array( &$title, &$pageDataValues, &$tableSchemas['_pageData'] ) );

before storing the data CargoStore::storeAllData( $title, '_pageData', $pageDataValues, $tableSchemas['_pageData'] );

this would enable other extensions to modify the fulltext data (and other params) before insertion, e.g. to sanitise text or add additional data to to a full text search

mediwiki wouldn't let me upload a patch file but here are the contents...

--- CargoPageData.php.original	2018-08-26 21:32:15.000000000 +0100 +++ CargoPageData.php	2018-09-19 16:03:24.115780698 +0100 @@ -137,6 +137,7 @@ 			$pageDataValues['_isRedirect'] = ( $title->isRedirect ? 1 : 0 ); } +		Hooks::run( 'CargoBeforeStoreAllData', array( &$title, &$pageDataValues, &$tableSchemas['_pageData'] ) ); CargoStore::storeAllData( $title, '_pageData', $pageDataValues, $tableSchemas['_pageData'] ); }

Thanks. James Driscoll


 * I'm always happy to add hooks into the code, and this one seems useful. One question, though: would it be better to put the hook inside the storeAllData function (right at the beginning of it)? storeAllData is called in three places: for regular data from #cargo_store, for "page data", and for "file data". The table name could be passed in to the hook, letting the outside code decide whether to modify the data for any one table. What do you think? Yaron Koren (talk) 16:14, 21 September 2018 (UTC)


 * that sounds like a better plan Yaron. Thanks ,James Driscoll


 * Okay - actually, now I'm not sure about this placement either. Are you already using such a hook locally? Because, looking through the code, it seems to make more sense to place it after the values have already been set by the code, but before they're stored in the database - maybe around line 331 of CargoStore.php, or halfway through the function. Otherwise, there's a good chance that the values set via the hook will get overridden, unless there are some serious hacks put in. On the other hand, if it's already working fine for you, then that's a different story. Yaron Koren (talk) 02:29, 24 September 2018 (UTC)


 * Hi Yaron, yes it's already working fine locally for me but I'm only modifying the '_fullText' field which I don't think you're changing? It makes sense for it to be as late as possible so values aren't overridden. James Driscoll


 * Ah, that explains it. Okay, I just added a hook, now called "CargoBeforeStoreData", which has the following structure:


 * It would be great if you could get the latest Cargo code, then adapt your own code to this new structure, and see if it still works. Yaron Koren (talk) 16:43, 24 September 2018 (UTC)


 * Works great thanks. I hope it's useful to others JamesDriscoll (talk) 19:38, 25 September 2018 (UTC)
 * Good to hear! And it's quite possible. Yaron Koren (talk) 19:51, 25 September 2018 (UTC)

Parser state cleared while parsing. Did you call Parser::parse recursively? on Special pages
We have a special page in an extension that relies heavily on cargo queries and are hitting this problem. "Parser state cleared while parsing. Did you call Parser::parse recursively? on Special page"

as a hack, in CargoUtils line 494

we are checking if the page is specifically our extensions special page and calling rather than your call of

Similarly, to avoid the same error, in CargoTemplateFormat line 73

we are checking if the page is our specialpage and just returning $text rather than calling CargoUtils::smartParse

are these checks meant to be specifically for landing on the cargo special pages rather than anyone else's special page?

Interestingly, on line 481 of CargoUtils, there is the comment that might be suggesting this is the case...

I hope this makes sense! JamesDriscoll (talk) 09:43, 27 September 2018 (UTC)


 * Sorry, I don't understand. What changes did you make to the code, and did they fix the problem? Yaron Koren (talk) 21:56, 27 September 2018 (UTC)


 * here are the patches. Although they work for my particular case I'm not suggesting these as the fix. I'm wondering if you should be explicitly checking in the code if you are landing on a cargo special page before calling smartparse and $parser->parse( $value, $title, $parserOptions, false );  JamesDriscoll (talk) 06:56, 28 September 2018 (UTC)




 * Sorry for the long delay. What is Special:Utilities/Search, if I may ask - is it part of a released extension? Is its source code public? Yaron Koren (talk) 16:47, 9 October 2018 (UTC)

Old vs new layout in drill-down (and forms)
Hi Yaron,

I have noticed that you have nice looking search-boxes and buttons and other values-boxes in drill-down at Discourse DB. The same goes for forms created in Page Forms. In my drill-down (Cargo 2.0-alpha) I have the old looking grey buttons and the same in forms (Page Form 4.4-alpha). I understand this has to do with a setting, but where and how do I change this to the newer looking layout? Pretor~nowiki (talk) 11:12, 28 September 2018 (UTC)


 * No, it's not a setting - it may just have to do with your MediaWiki version. What version of MW are you using? Yaron Koren (talk) 16:16, 28 September 2018 (UTC)


 * I'm using MediaWiki v1.31.1 and the issue has been there since the website was established with v1.30.0. It's only on Cargo and Page Forms related pages that this issue occurs, the rest of the website has the up-to-date layout with blue buttons etc. Pretor~nowiki (talk) 17:35, 28 September 2018 (UTC)


 * Oh, okay. The issue might be your version of Cargo - there actually is newer code than version 2.0-alpha. I would try upgrading your Cargo to the latest code. Yaron Koren (talk) 20:04, 28 September 2018 (UTC)


 * Upgraded Cargo from git (2.0-rc1) and that fixed the problem in drill-down, thank you :) Did the same with Page Forms (4.4-rc1), but that did'nt help for the forms. Should I post the same question in the Page Form talk page? Pretor~nowiki (talk) 20:53, 28 September 2018 (UTC)


 * I don't remember any big change in the display of Page Forms forms... was there? Yaron Koren (talk) 22:24, 28 September 2018 (UTC)


 * You have probably not updated Page Forms to use the "new" MediaWiki layout with blue buttons etc. No big deal and thanks for the tip that fixed this in drill-down! Pretor~nowiki (talk) 09:43, 29 September 2018 (UTC)

Cargo database shared between wikis
I've created a separate database for my cargo tables. I want multiple wikis to access the same content. However, each wiki has a different $wgDBprefix. Would this stop them from being able to access the same data? -- Prod (talk) 03:30, 4 October 2018 (UTC)


 * I don't think it's possible for a wiki to use another wiki's Cargo content anyway, at least not without some major hacks. If you want another wiki to get and display a wiki's Cargo data, you should use the External Data extension, and specifically the function #get_web_data, and have that access the main wiki's Special:CargoExport page. Yaron Koren (talk) 13:15, 4 October 2018 (UTC)

Special:CargoTables gives Fatal exception of type "Wikimedia\Rdbms\DBQueryError" on view table
I'm getting this error on all tables that contain list of datetimepicker Hopefully you can duplicate the error with this simple template....

JamesDriscoll (talk) 22:15, 4 October 2018 (UTC)


 * Could you add "$wgShowSQLErrors = true;" and "$wgShowExceptionDetails = true;" to your LocalSettings.php? That will hopefully show a more helpful error message. Yaron Koren (talk) 00:27, 5 October 2018 (UTC)


 * they were already on :-(  apache error log isn't showing anything either JamesDriscoll (talk) 06:41, 5 October 2018 (UTC)


 * Yes, I was just able to recreate this myself - there's a bug in the default querying of fields that hold a list of dates. Sorry about the problem; I'll look into it. Yaron Koren (talk) 18:59, 5 October 2018 (UTC)


 * I checked in a fix for this a few days ago - if you get the latest code, the problem should go away. Yaron Koren (talk) 16:12, 9 October 2018 (UTC)


 * Thanks, view is working but sadly the same error is appearing on drill down,sorry I didn't spot that on my first post!JamesDriscoll (talk) 20:30, 9 October 2018 (UTC)


 * Ah, yes. This was a tougher nut to crack, but I checked in some fixes that I think fix the issue for the display of date-list fields in Special:Drilldown. Yaron Koren (talk) 15:26, 16 October 2018 (UTC)

Automatic |no html added when query used in page forms?
Hi, I have a problem with the links not showing in the results of a cargo query when using it from Special/runquery. I think this is the same problem described in this post. I've found a workaround using instead of  to build the links, unfortunately it doesn't work with  and ends in things like ?'"`UNIQ--item-13--QINU`"'?. Any idea ?

Best regards, --Ludovic Strappazon (talk) 10:08, 5 October 2018 (UTC).


 * If you don't know Lua but have Scribunto extension, you can try using this wrapper to be able to effectively use a |format=template without actually using |format=template, and thus get around the bug altogether. --RheingoldRiver (talk) 18:25, 23 October 2018 (UTC)

i18n for dropdowns
Hi, I'm not sure if this is a Pageforms or Cargo issue, or just me misunderstanding :-)

Is there any way to have different label and values for dropdowns?

e.g. if, inside a #cargo_declare, I have something like...

the Pageform will end up producing something like this...

I'd like to be able to change the string displayed to the user without changing the value stored in the database

I'm imagining something on the lines of this in the #cargo_declare

Is there a way to do this already? Thanks JamesDriscoll (talk) 10:57, 12 October 2018 (UTC)


 * This sounds like a Page Forms issue - check out the mapping section, if you haven't seen it already. Yaron Koren (talk) 02:13, 14 October 2018 (UTC)


 * Thanks for the pointer Yaron - a quick experiment on a test page is looking promising. I was surprised the translation also happened on the page that gets created from the form but this may be a good thing in my case :-) JamesDriscoll (talk) 14:48, 14 October 2018 (UTC)
 * on further experimentation the edit form produces this...

editing a page results in the form not being able to pick the correct value - I assume because status = 'Live' rather than 'Label for Live' shouldn't the form look like this... and any display on a page created is a separate problem to what is displayed on the form? Perhaps I should be reporting a potential bug on a pageforms discussion page.... JamesDriscoll (talk) 15:18, 14 October 2018 (UTC)


 * That shouldn't happen. How are you doing the mapping? Yaron Koren (talk) 23:19, 14 October 2018 (UTC)
 * in the form Im going...


 * Okay, that explains it. The set of "values=" should be the actual values, not the labels. And in addition to "mapping cargo field", you also need to set "mapping cargo table" - otherwise it won't take effect. Yaron Koren (talk) 19:19, 15 October 2018 (UTC)
 * so how do I specify what the labels are? (I want the labels to be different to the vales) JamesDriscoll (talk) 20:29, 15 October 2018 (UTC)
 * That's what the mapping is for. It sounds like your best bet is to use a "mapping template" - create a template that calls #switch or some such to display the right label based on the value, which is represented by . Yaron Koren (talk) 20:37, 15 October 2018 (UTC)
 * sorry, I'm not grasping this. is there a way on the page form to have a different label to the value? I'm happy to do the switch on the actual document but I want the form to display one thing on the the drop down to the user but set a different value in the table.like this


 * Yes - have "values=" set to the real values (Prospective, etc.), then create a mapping template that displays the correct label depending on the value of, and set "mapping template=" to the name of that template. Yaron Koren (talk) 02:30, 16 October 2018 (UTC)



produces this in the form....

which would be great if the values were, Live, Retired and Prospective. I hope I've understood this time, thanks for your help. JamesDriscoll (talk) 07:19, 16 October 2018 (UTC)
 * the right bits end up in th database though, when I'm on a real computer I'll have a crack at updating the docs with an example,thanks again JamesDriscoll (talk) 09:47, 16 October 2018 (UTC)

No HTML in API queries
We are trying to generate some API feature with cargo but keep getting UNIQ-QINU elements in the response. From what I have read this need to be addressed by adding "no html" to queries but this option is not available in API query from what we have understood. What can we do about it? Thanks, Wess (talk) 07:01, 14 October 2018 (UTC)


 * I think it is available - try just adding "&no html" to the URL. Yaron Koren (talk) 23:37, 14 October 2018 (UTC)

CargoQuery cannot retrieve all the fields because of the default alias _pageName
Hi,

I was trying to run this query to get all the fields in the table https://dragalialost.gamepedia.com/api.php?action=cargoquery&format=json&tables=Adventurers

However I am met with this error:

It seems like the problem is with cargo table defaults to using _pageName if not provided any fields and API queries do not support underscore.

Is there a way so that if I don't provide any field names it would default to get ALL of the fields for that cargo table? something like fields=* would be fine as well --Pupper75 (talk) 21:17, 25 October 2018 (UTC)


 * Unfortunately, there's no way to automatically get all the fields in a table - that's unrelated to this underscore problem. You'll have to just manually specify each field. Yaron Koren (talk) 21:52, 25 October 2018 (UTC)


 * Are there work to add in this feature? Imagine if you have a SQL language that doesn't have  and you have to look through the table definition to get all the fields you need. --Pupper75 (talk) 02:08, 26 October 2018 (UTC)


 * I looked into supporting "*" before... there was some difficulty with doing that, though I can't remember what it was now. I admit that it's annoying. Yaron Koren (talk) 15:31, 26 October 2018 (UTC)

List table not created, HOLDS query error [FIXED in Cargo 2.0.1]
For test purposes, I created two templates (Book and Author) by copying from https://www.mediawiki.org/wiki/Extension:Cargo/Quick_start_guide. For reference, here is the declare portion:
 * 1) cargo_declare:_table=Books|Authors=List of Page|Genres=List  of String|Year_of_publication=Date|Number_of_pages=Integer

According to https://www.mediawiki.org/wiki/Extension:Cargo/Storing_data#Database_storage_details, there should be a separate table creates for the "Authors" field because it's a list, and there should also be a "Authors__full" column. But neither of those exist, just "Authors", which is a VARCHAR(300). I created the tables via the web interface, but for other tables I've used cargoRecreateData.php to the same effect.

The more important issue, possibly related - I tried to do an example query with "HOLDS":

But I get this error message:


 * Query: SELECT `_pageName` AS `Book`,`Authors` AS `Authors` FROM `cargo__Books` WHERE Authors HOLDS 'Foo' ORDER BY `_pageName`, `Authors` LIMIT 100 Function: CargoSQLQuery::run Error: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'HOLDS 'Foo' ORDER BY `_pageName`, `Authors` LIMIT 100' at line 1 (localhost:/var/lib/mysql/mysql102.sock)

Note: I'm using Page Schemas and Page Forms for my normal objects, but I tried to keep things simple for this example.

Here are my versions:
 * MediaWiki 1.31.1
 * PHP 7.0.27
 * MariaDB 10.2 - separate Cargo database (main wiki database is MariaDB 5.5.60, but I was getting Cargo errors with that)
 * Cargo 2.0
 * Page Forms 4.4.1
 * Page Schemas 0.4.8

Please let me know if there's any other information I can provide to help. --Samh78537 (talk) 20:47, 30 October 2018 (UTC)


 * Yikes, sorry about that - there turned out to be some pretty major bugs that snuck in to Cargo right before the release of version 2.0, a few weeks ago; clearly I didn't adequately test it. I just released a new version of Cargo, 2.0.1, that should work a lot better - please upgrade to that when you get the chance. Yaron Koren (talk) 03:40, 31 October 2018 (UTC)


 * Thanks! I was afraid I was doing something wrong. I've updated Cargo to 2.0.1 (dec2d3a) (also Page Forms to 4.4.2), but it still seems to be doing the same thing. Is there something else I need to do to make it take effect? I tried using the web link and cargoRecreateData.php, and I tried deleting the tables first, as well as the rows in cargo_tables. --Samh78537 (talk) 16:48, 31 October 2018 (UTC)


 * You just need to re-save the template, and recreate the table, and then everything else should work. Yaron Koren (talk) 16:50, 31 October 2018 (UTC)


 * Re-saving was the missing step; I didn't realize that saving the template did something before you recreate the table. --Samh78537 (talk) 13:52, 1 November 2018 (UTC)

In-page list of backlinks
Before I look into it myself I hope you don't mind me asking whether the following can be replicated with Cargo. "With the extension Semantic MediaWiki (SMW) links are categorized by specifying relations. For a given relation the backlinks of a page can be produced in-page. A series of queries, one for each relation (which seems cumbersome but can be put in a template like [1]), provides an in-page list of backlinks sorted by relation. Moreover, forward links and attributes of the resulting pages can also be provided, and also backlinks of backlinks." (from Help:What links here). Thanks. Jonathan3 (talk) 20:32, 3 November 2018 (UTC)
 * P.S. It turns out that this is simple in DPL3 - - but I still wonder about using Cargo. Jonathan3 (talk) 00:01, 4 November 2018 (UTC)

Transclude a page without its categories
If page A transcludes page B (which is is category B) then page A ends up in category B too... this happens whether I use normal transclusion, labelled section transclusion  or DPL3.

Would there be a way of using Cargo to transclude the page content up to but not including the first ? Thanks, Jonathan3 (talk) 23:58, 3 November 2018 (UTC)
 * There is Extension:NoCat which can prevent all categories on page A. That might be a bit too much for what you're after. I use it in conjunction with Cargo when listing e.g. a blog index page (which never needs its own categories). Sam Wilson 00:31, 4 November 2018 (UTC)


 * Thanks for that. Unfortunately I need the existing categories. As it happens, when you use DPL3 as a parser function and use reset=categories it has the same effect. When you use it as a tag with the same parameter it only removes the transcluded category (perfect) but then I can’t use it the way I need to in a template... Jonathan3 (talk) 10:28, 4 November 2018 (UTC)