Extension talk:Cargo

Error: 1069 Too many keys specified; max 64 keys allowed
Recreating a CargoTable via the cargoRecreateTable.php script gave me the following error:  The table contains 61 fields.

The query that threw the error is:

Cargo always seems to create indexes for all table columns and also a sort of __precision index (I don't understand why) which eventually leads to a number of keys larger than 64 for this specific table.

A solution could be to disable indexing (apart from applying a leaner database design). Is there a way to disable indexing in Cargo? Would that make sense?


 * Indexing of all fields was add just a month ago - I'm glad you found this issue now, before the next version is released. I didn't know about that keys limit that MySQL has. I just checked in a fix for this, so if you get the latest code and recreate the table again, hopefully it will work. Yaron Koren (talk) 14:54, 7 August 2018 (UTC)


 * Thanks for the quick reply and the fix! We'll implement it this week and i'll let you know. I did not know of the key limit either. Apparently the key limit can only be changed by recompiling the code. I'd be interested in the reason for enabling indexing on all fields? Did you notice performance improvements? --Schtom (talk) 09:21, 8 August 2018 (UTC)


 * I misspoke before - not all fields are indexed; just fields that have a type other than "Text", "Wikitext" and "Searchtext". (And for date fields, there's an extra field, for the precision, that is also indexed.) The idea behind this is to get queries to run faster in general. For any specific table in any specific wiki, there may be only one or two fields where an index will make anything run faster - depending on the size of the table, etc. I could have made indexing a setting in #cargo_declare, to let admins decide which fields should get indexed and which should not. However, I figured it would be easier for admins, and more foolproof, to just index every field that could conceivably be used for sorting or joins. That's especially true given that I think most people running Cargo will not be DB experts. There's a tradeoff there in terms of slower writes to the DB, but I think it's worth it - on a wiki, read speed is much more important than write speed. Still, this is a very new feature, and I'm not sure it's the correct approach - I'd be glad to hear your thoughts on the matter. Yaron Koren (talk) 14:38, 8 August 2018 (UTC)


 * Thanks for the explanation and sorry for the late reply. We also did not implement your fix (yet), as our client decided to change his database design. Personally, I do think your approach and reasoning makes a lot of sense. However, I cannot assess the actual performance improvements. I wanted to let you know about another issue that occurred with the same client. He reported a database error when creating a new table. We then found that the identifier limit is restricted to 64 chars (Error: 1059). One of the identifiers was throwing the error was 'deadline_minigrid_people__precision_webmo_indicator_minigrid_people' (67 chars). The composed index keys may become an issue as well. It seems there could be a trade-off between usability/readability (users are forced to use shorter names) and performance. So far we have not rolled out our cargo-driven wiki and are merely beta-testing with this one client. These two issues (64 keys, 64 chars) were the only ones reported to us.--Schtom (talk) 13:53, 21 August 2018 (UTC)


 * Those are some pretty long table and field names! Anyway, thanks for letting me know about the problem, and sorry about that. I just checked in what I think is a fix. Yaron Koren (talk) 03:45, 22 August 2018 (UTC)


 * Kindly note that we do not consider these two issues as major problems. We find your extension incredibly useful. I agree that both the field and table names are pretty verbose. Still, I assume the identifier error was thrown by one of the index keys that is composed of both table and field name + underscores + the 'precision' label that you add. As a suggestion: You could maybe shorten the latter to leave a little more 'explanatory space' for the user e.g., 'pr' instead of 'precision'. --Schtom (talk) 10:38, 23 August 2018 (UTC)


 * That's great to hear! Yes, reducing the size of "precision" could make sense. I'll see how the current solution works out. Yaron Koren (talk) 15:27, 23 August 2018 (UTC)

Ad block (uBlock Origin) blocks Cargo script for displaying bar charts
On this page I noticed if I have uBlock Origin enabled the bar charts do not load. Disabled uBlock and the bar charts load normally.

Chrome dev console reveals this error:

Uncaught TypeError: Cannot read property 'disabled' of undefined at String. (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:458) at Array.d3_transitionPrototype.filter (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:221) at SVGSVGElement. (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:458) at load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24 at d3_selection_each (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24) at Array.d3_selectionPrototype.each (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24) at Array.chart (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:455) at Array.d3_selectionPrototype.call (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24) at Object.generate (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:562) at renderLoop (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:246)

Please advise. --pcj (talk) 19:47, 10 August 2018 (UTC)
 * Never mind, this was a Gamepedia-specific issue and has been resolved. --pcj (talk) 02:18, 2 September 2018 (UTC)


 * Good news! I had no idea what was causing the problem. Yaron Koren (talk) 13:03, 2 September 2018 (UTC)

SMW subobject transitioning to Cargo, table display issues
I'm trying to move a wiki from SMW to Cargo and one of the things we use is subobjects. E.g. A character may need two types of materials to ascend to the next stage -- 4 feathers and 5 fluid. Ideally this would be stored as a list with entries that have properties. i.e. Feathers (4);Fluid (5).

How do I do this in Cargo, or does anyone have a subobject-like store they could show an example of, please? Thanks!

Toriayl (talk) 14:21, 14 August 2018 (UTC)


 * One of the nice things about Cargo is that all data is stored in the same way, via #cargo_store - there's no distinction between regular storage and "subobjects". In other words, assuming you store all data via templates, it doesn't matter whether there's one template call on the page, or multiple calls. Does that make sense? Yaron Koren (talk) 17:27, 14 August 2018 (UTC)

Thanks for the reply, but sorry, still not really getting it. The point of sub objects is to link data to specific parameters entered. E. G. An address would have its post code and road number linked to it. Are you saying that my storage needs to be multi row within each page to accomplish similar functionality? I currently have a single template that loops through items and amounts entered and stores each as subobjects. I can't really use multiple templates instead, but maybe I need to figure out how to store them on different rows within the same template.. Was hoping there would be a single-row solution.

Alternatively, am I meant to store directly/is it possible to do so into a helper table for a field containing a list of values? Toriayl (talk) 22:25, 14 August 2018 (UTC)


 * Oh - I didn't realize you have a single big template that's storing an entire "table" of values. You can still use Cargo for this - you would just need to create a new template that holds one row of that table, then have the big template call that template in each place it previously called #subobject. Does that make sense? Yaron Koren (talk) 01:42, 15 August 2018 (UTC)

Thank you, I was hoping to have the info in one table but it sounds like another table would be a better idea for this. And rows would look like: Artemis - Feather - 1 Artemis - Feather - 5 Artemis - Fluid - 5 Hopefully that's on the right track but I'll worry about it later. I have more questions for now! (Sorry. XD) Not really subobject-related now, though.
 * 1) Is there a way to pull just the first result of a joined table? i.e. If Artemis matches with two entries on another table, and I want just something from the first entry matched. This one is not 100% needed; I'll probably manually input the values needed into each page (200 of them...), but I think I've read something about SQL being able to do this (though not very cleanly) so thought I might as well ask in case there's something for Cargo I've missed.
 * 2) I am having some trouble generating a table, particularly with the values in a list on it. This is the table: https://grandorder.wiki/Special:CargoTables/CEs Related template: https://grandorder.wiki/index.php?title=Template:CEInfoBox&action=edit Test page with issues: https://grandorder.wiki/User:Toriayl/sandwich2
 * The Effects parameter is stored in Effects=List of String, which appears to work fine on the Special:CargoTables table-browsing link above.
 * However, when I try to put it in a dynamic table query, I get a database error if I put it as the last item displayed.
 * The query works if I put it as the first or second item displayed, but lists with three items have the second item within pre tags. Something to do with the dot delimiter..?
 * Using template as an alternative, effects work but html tags (br, div style etc) are displayed.

Help would be really appreciated, thanks! Toriayl (talk) 14:20, 29 August 2018 (UTC)


 * Alright - 1) there may be a way to do with some hack, maybe using "group by=", depending on the exact structure of the data - I'm not sure. 2) I'm not entirely sure what's causing the strange behavior with the Effects field, but I think it's at least partly due to the somewhat unusual way in which that value is put together. The "cleaner" way to do it would be again with a multiple-instance template holding the effects and "LBs" - so that each effect is directly connected to its LB value, and so that you don't need to have an artificial limit on the number of effects. But barring that change, you may be able to get around this problem by replacing "Effects" in the query with "Effects__full=Effects". Yaron Koren (talk) 15:27, 29 August 2018 (UTC)

Thanks, but I don't get how it's unusual...isn't it a fairly normal list? Effects__full=Effects got around the database error, but it's still showing the 'second item pre' issue. Also pardon me if I'm wrong, but that's the same fix you were suggesting for subobjects, isn't it? i.e. Create another table with the effects, then join it with the main table.

But won't this cause the same problem as 1)? I want to only display one line with all the effects for an item, I don't want three lines for three effects, with all of the other info duplicated......

Toriayl (talk) 13:01, 30 August 2018 (UTC)


 * Yes, that's the same fix I suggested. In both cases, you have a table of data stored on the page, so it should be stored in a tabular fashion. When you display the data, you can display it all on one line, even if it represents three different "rows". What do you mean by "all of the other info duplicated"? Yaron Koren (talk) 19:18, 30 August 2018 (UTC)

How do I display it all on one line?! Have a look on my sandbox again, the first section. This is the only way I know how to query/display the table, and it shows multiple lines, one for each match in the join...and results in a duplication of the other information (rarity, ID, etc).

Toriayl (talk) 11:41, 1 September 2018 (UTC)


 * If you add "|format=list" to the query, it will display all results on one line. You can also do that with the "template" format, which gives you complete control over the display, although with that you have to create a template. The duplication thing is a problem, though it's a problem in the storage, not the querying. I wish I knew how to prevent duplicates in all cases. In any case, if you do a recreate data on the "ServantsNP" table again, it will probably fix the problem in that table. Yaron Koren (talk) 02:27, 2 September 2018 (UTC)

Thanks for trying to help, but it feels a little like we're on different wavelengths here. ^^" I have one page that has two rows associated with it. I ran a recreate data on the ServantsNP table and it gives me the same thing. E.g. Alexander B+ NP, Alexander B++ NP. This is correct. Alexander is linked to two different types of NP. But I want it to display, instead of on one line: It doesn't look like format=list or template can do this? Toriayl (talk) 15:41, 5 September 2018 (UTC)

Displaying a table on a page for searching by values
Maybe someone already asked such a question.

Is it possible to create a table on a page with search? or create such a table similar to Drilldown. imgur - (Oleksiy) 10:47, 22 August 2018 (UTC)


 * Do you know about the "dynamic table" format? Yaron Koren (talk) 16:09, 22 August 2018 (UTC)

Cargo not updating tables, nothing happening.
Hi, I am having an issue where cargo is not updating tables. Only some pages are being successfully stored while others aren't, despite there being no real difference. Existing pages also do not update despite the job queue being empty. Steps taken:


 * Recreated table using  - nothing happens but jobs are added, jobs will never progress (even with high run rate.) Also tried manually recreating data.
 * Manually ran jobs through the server, created only half of the data then stopped. No jobs left.
 * Recreated data on tables that had no population whatsoever - no jobs created.
 * Manually ran jobs through server, nothing happens.
 * If existing data on the table is updated, it does not update or create a job.

Semantic Mediawiki worked flawlessly beforehand, no changes were made between the two except for an update from MW 1.29 to 1.31, any idea to what the issue may be? Steps were taken from the FAQ and common problems as well, none seem to be the issue. -- 2607:FEA8:E2E0:602:CCB2:DD2C:F53E:4C48 08:17, 24 August 2018 (UTC)


 * I didn't understand all of this. What do you mean by "manually recreating data", and what happened when you tried it? What do you mean by tables with "no population whatsoever" - empty tables? If so, what should have happened that didn't happen? What do you mean by "existing data" being updated - a regular page save? And when you wrote "Semantic Mediawiki", did you mean Cargo? If not, I don't see the connection. Yaron Koren (talk) 13:22, 24 August 2018 (UTC)


 * Sorry, let me be more clear, I am not 100% on the terminology as of yet:
 * We used SMW before transferring over to Cargo, just updated templates using the migration guide. Only the first time we recreated data using the web browser then ran runJobs.php did it partially work. It did not grab every page's data and stopped midway. (SMW is completely disabled at the moment.)
 * "Manually created data" by running runJobs.php / php cargoRecreateData.php
 * The tables specifically aren't being updated, despite it should be picking up the changes, the jobs aren't being created/aren't functioning correctly.
 * I have a local copy of Cargo and confirmed the templates used work and will successfully create the table and fill them with data, its only on the server does it not, despite there being no errors.
 * I hope this helps, I am completely stumped. Is there any further debugging I can do? -- 2607:FEA8:E2E0:602:4D63:99F6:223C:D164 04:45, 26 August 2018 (UTC)


 * I just finally tried running Cargo on MW 1.32.0-alpha, and it seems to be working fine for me, including recreating data. So I don't know what's going wrong. Just so I understand - were you using Cargo with MW 1.29? And what version of Cargo are you running? Also, this isn't a public wiki by any chance, is it? Yaron Koren (talk) 03:44, 29 August 2018 (UTC)
 * Heres the link to the wiki, all the information should be on there. As an example, the machine table only has 4 entries, when 20-ish pages actually use the template and should be storing data. --2607:FEA8:E2E0:602:68B1:ACDA:6EA3:A27 22:49, 29 August 2018 (UTC)


 * Alright. Do you see any error messages when you run cargoRecreateData.php? Yaron Koren (talk) 18:15, 31 August 2018 (UTC)
 * No error messages, says its "done" but doesn't actually do anything. -- 2607:FEA8:E2E0:602:B01E:BB64:DD4F:C263 20:51, 1 September 2018 (UTC)
 * Additionally : I can get this to work on a local copy but not the actual server. That being said I have no personal access to the server, so it's difficult to diagnose. -- 2607:FEA8:E2E0:602:D9EB:B7E9:C8A0:6D6D 01:10, 3 September 2018 (UTC)
 * Is there any difference in versions between the local copy and the actual server? Also, if you don't have access to the server, how were you able to run cargoRecreateData.php? Yaron Koren (talk) 02:48, 3 September 2018 (UTC)
 * It's a weird administration system, I am able to ask the server admin to debug some stuff for me, but they are very slow at doing so and won't give me access - so the only information I get is basic. It makes this process much more difficult and frustrating. There is no difference in the versioning. -- 2607:FEA8:E2E0:602:1569:7DD9:88C1:6AD0 20:15, 3 September 2018 (UTC)
 * Update: I found out SMW was improperly uninstalled, after properly having it removed, cargorecreatedata.php started working and successfully filled tables. However, tables will still get stuck trying to populate via the web interface, and they will not update automatically if a page is edited. -- 2607:FEA8:E2E0:602:F5FA:7A1F:1CDF:59B3 19:03, 6 September 2018 (UTC)
 * Update 2: Debugging further found that jobs aren't being created by cargo to do the work, despite saying  The showjobs.php shows 0 jobs and none failed / abandoned. -- 2607:FEA8:E2E0:602:6134:8886:DF34:6CF7 21:23, 8 September 2018 (UTC)


 * This sounds a lot like the issues we have on Gamepedia. Does it recreate the data as expected when you blank edit the pages that store? --RheingoldRiver (talk) 22:25, 4 September 2018 (UTC)
 * No, it doesn't. -- 2607:FEA8:E2E0:602:A012:9E7D:B5D3:26CE 17:55, 5 September 2018 (UTC)

It's great that you were able to get cargoRecreateData.php to work, and that you diagnosed the issue with the web interface (it's not creating jobs). I don't know why jobs aren't being created, though. Do you know if other processes that create jobs are working on your wiki? For example, modifying a template should cause a job to get created for each page that calls that template, to refresh its contents - does that happen? Yaron Koren (talk) 01:42, 9 September 2018 (UTC)
 * Yes, other jobs are being created and work correctly, cargo does not. Cargo says it has created jobs at this page, but showjobs.php shows 0 jobs. Jobs are also being processed correctly. -- 2607:FEA8:E2E0:602:C8E9:DA9F:601F:C5F 20:31, 10 September 2018 (UTC)

Cargo not updating tables
Hello Yaron, I have the same issue with mw1.31. : "Cargo not updating tables, nothing happening". The table is created with a form and a template (using #cargo_declare in the template) the data is store in the mw page (using #cargo_store in the template) when I save the form but the cargo table is empty. I used a separate cargo database. The only way I have found to populate the cargo table is to run "php cargoRecreateData.php" in the Cargo's /maintenance directory. (Jacques Masbou)


 * What version of Cargo are you running? Is this a public wiki? Did it work better with an earlier version of MW? What database are you running? Does everything work fine if you call cargoRecreateData.php? Yaron Koren (talk) 16:48, 29 August 2018 (UTC)

Cargo version 1.7 php 7.22 mw1.31 It is not a public wiki mysql 5.7.19 when I call cargoRecreateData.php sometimes it works sometime it does not work With cargo 1.5, php 5.6.31 mysql 5.7.18 and mw 1.29 it works fine

Update : with mw 1.29.2, php 7.1.21 mysql 5.7.19 cargo 1.7 it works fine


 * It's great that you isolated that the problem is due to the MediaWiki version, not the Cargo version. Still I don't know what's causing the problem - I'm running Cargo on MW 1.31 myself, and it seems to run fine. Do you have more than one Cargo table on your wiki? If so, is the problem happening in all of them? Yaron Koren (talk) 03:00, 2 September 2018 (UTC)

Error when creating a new data table
Hi,

I got this error when trying to create a new data table :

{   "error": { "code": "internal_api_error_DBUnexpectedError", "info": "[6377a1755353fc18463173ed] Exception caught: Wikimedia\\Rdbms\\Database::close: mass commit/rollback of peer transaction required (DBO_TRX set).", "*": "Wikimedia\\Rdbms\\DBUnexpectedError at /var/www/wiki/w/includes/libs/rdbms/database/Database.php(916)\n#0 /var/www/wiki/w/extensions/Cargo/includes/CargoUtils.php(818): Wikimedia\\Rdbms\\Database->close\n#1 /var/www/wiki/w/extensions/Cargo/includes/CargoUtils.php(534): CargoUtils::createCargoTableOrTables(Wikimedia\\Rdbms\\DatabasePostgres, Wikimedia\\Rdbms\\DatabasePostgres, string, CargoTableSchema, string, integer)\n#2 /var/www/wiki/w/extensions/Cargo/api/CargoRecreateTablesAPI.php(31): CargoUtils::recreateDBTablesForTemplate(integer, boolean)\n#3 /var/www/wiki/w/includes/api/ApiMain.php(1579): CargoRecreateTablesAPI->execute\n#4 /var/www/wiki/w/includes/api/ApiMain.php(535): ApiMain->executeAction\n#5 /var/www/wiki/w/includes/api/ApiMain.php(506): ApiMain->executeActionWithErrorHandling\n#6 /var/www/wiki/w/api.php(94): ApiMain->execute\n#7 {main}" } }

Looking at the SQL logs (I am using PostgreSQL), I can see the correct  statements, but no  :

2018-08-25 19:17:49 CEST [14818-22] wiki@wiki LOG: statement: CREATE /* Wikimedia\Rdbms\Database::query  */ TABLE "cargo__Artistes__Genres" ( "_rowID" Int, "_value" Varchar(300) ) 2018-08-25 19:17:49 CEST [14818-23] wiki@wiki LOG: statement: CREATE /* Wikimedia\Rdbms\Database::query  */ INDEX "row_id_Artistes__Genres" ON "cargo__Artistes__Genres" ("_rowID") 2018-08-25 19:17:49 CEST [14818-24] wiki@wiki LOG: statement: ROLLBACK /* Wikimedia\Rdbms\Database::close  */ 2018-08-25 19:17:49 CEST [14817-31] wiki@wiki LOG: statement: ROLLBACK /* MWExceptionHandler::rollbackMasterChangesAndLog  */ 2018-08-25 19:17:49 CEST [14817-32] wiki@wiki LOG: statement: BEGIN /* Wikimedia\Rdbms\Database::query (LCStoreDB::get)  */

Any help is welcome :)

--Erdnaxeli (talk) 17:25, 25 August 2018 (UTC)


 * Ok, I tried with the git master branch (instead of the latest zip) and it seems the bug was fixed. --Erdnaxeli (talk) 15:37, 31 August 2018 (UTC)


 * That's great to hear - I was planning to set up Cargo on Postgres to try to debug the issue, but I guess now I don't need to. Yaron Koren (talk) 18:14, 31 August 2018 (UTC)

File field : edit redlink instead of File:myFile
I am using
 * Special:CreateClass

To obtain Table structure:
 * bebe - String
 * myfile1 - File
 * bibi - String
 * myfile2 - File

then has uploadable for File fields.
 * Form:MyFileB1

Page creation and file uploading is working, But my page shows a redlink for File field
 * index.php?title=Blue16Test1A.jpg&action=edit&redlink=1

I think it should it be linking to
 * File:Blue16Test1A.jpg

Would you help me to link to uploaded file ?

Thanks in advance. --Wpe (talk) 15:15, 26 August 2018 (UTC)


 * Yes, the template should have the string "File:" before the template field holding the filename, so it should look like " File: ". Sorry, I realized that Special:CreateClass and Special:CreateTemplate don't put in that necessary prefix when creating the template; I just checked in a fix to Page Forms, so hopefully they will now. Yaron Koren (talk) 03:22, 27 August 2018 (UTC)

Missing _pageData
How to solve this problem? [9f30956ef6e5b7f44122218a] /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:CargoTables/_pageData MWException from line 196 of /home/hydra/public_html/extensions/Cargo/includes/CargoUtils.php: Error: Table _pageData not found.

Backtrace:


 * 1) 0 /home/hydra/public_html/extensions/Cargo/specials/CargoTables.php(86): CargoUtils::getTableSchemas(array)
 * 2) 1 /home/hydra/public_html/includes/specialpage/SpecialPage.php(522): CargoTables->execute(string)
 * 3) 2 /home/hydra/public_html/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(string)
 * 4) 3 /home/hydra/public_html/includes/MediaWiki.php(287): SpecialPageFactory::executePath(Title, RequestContext)
 * 5) 4 /home/hydra/public_html/includes/MediaWiki.php(862): MediaWiki->performRequest
 * 6) 5 /home/hydra/public_html/includes/MediaWiki.php(523): MediaWiki->main
 * 7) 6 /home/hydra/public_html/index.php(43): MediaWiki->run
 * 8) 7 {main} --Ruba159753 (talk) 13:54, 4 September 2018 (UTC)


 * Right, the _pageData table doesn't get created automatically - instead you have to create it by defining its columns, then calling a script. See here. Yaron Koren (talk) 19:19, 4 September 2018 (UTC)

Declaration Parameter to Index a Field
Would it be possible to add a flag in declaration of a field that makes the column indexed (but not unique)? --RheingoldRiver (talk) 18:49, 5 September 2018 (UTC)


 * Actually, in the latest Cargo code, every field now gets indexed - except for fields of type Text, Wikitext and Searchtext, and fields after #50 or so in tables with a large number of fields. Yaron Koren (talk) 23:16, 5 September 2018 (UTC)


 * Ah ok, we are still not on latest version until after MediaWiki upgrade. Good to know, thanks. --RheingoldRiver (talk) 00:57, 6 September 2018 (UTC)