Extension talk:Cargo/Archive March to May 2018

From MediaWiki.org
Jump to navigation Jump to search

timeline causing error

When using the following cargo query


I get the error on the page:

Caught exception: SyntaxError: expected expression, got '<'

and no timeline is displayed. Not sure what could be the problem. Using cargo version 1.6 (5a44fdc)

This doesn't seem to happen for me - I'd recommend upgrading to the latest version, 1.7 (released today!), to see if maybe that was a bug that's been fixed. Yaron Koren (talk) 19:52, 9 March 2018 (UTC)
Thanks for your reply, updated today but the issue persists. I noticed the error also occurs for me on this page from the Cargo example page: http://discoursedb.org/wiki/Virginia_United_States_Senate_election,_2006
Oh yeah, now I remember... this sometimes happens due to page caching, for reasons I can't explain. I need to look into, but I think you can get around the problem by installing the MagicNoCache extension, and adding "__NOCACHE__", on a line by itself, to any page (or better yet, template) that calls the "timeline" format. It's not an ideal solution, but it should work for now. Yaron Koren (talk) 14:04, 12 March 2018 (UTC)
I see, one of the example pages had __NOCACHE__ but I didn't know what it meant. I have updated Cargo, added the magicnocache extension and added __NOCACHE__ but now there is a different error "TypeError: v1 is NULL"

Table with more than 70 columns will not create

Hello, we're trying to work with a table that has 84 columns, but our tests are showing that if it has more than 70 columns it will not create. https://dota2.gamepedia.com/Template:User:Litzsch/Cargo_test is the test table currently. There isn't any data going in it yet, just trying to define the table. It creates super quick at 70 columns (up to buff4 in the table) but if I add just one more column to the declare it just gives a spinning wheel with no action. Any ideas? --pcj (talk) 13:22, 11 March 2018 (UTC)

I'm not sure why it's failing for you - these 70 rows appear to be all of type "String", which in Cargo translates to "VARCHAR(300)", which should be somewhere around 21,000 bytes, which is significantly less than MySQL's limit of 65,535 bytes per row.
However, it seems to me that what you're trying to do would better be accomplished with multiple-instance templates. Do you know about those, from Page Forms? Here, you're trying to store three different two-dimensional tables of data, within each row of this table. I think switching to multiple-instance templates would make for better storage, easier editing, and easier querying. Yaron Koren (talk) 14:17, 11 March 2018 (UTC)

how do I get plain numbers from aggregat functions


when I do a #cargo_query and select one integer field I get the integer back as a result – e. g. 5000 –, but when I query using MAX( integer field ) the result is a formatted number like 5,000.0. It's the same when using SUM(). I just wonder why this happens and if it's possible to suppress this formatting?

Doing the same thing using direct sql within pma produces plain numbers for all queries. So is it a cargo or mediawiki issue? Or am I just doing something wrong? --~o)sabine(o~ (talk) 15:10, 15 March 2018 (UTC)

It's a Cargo issue - some helpful formatting around Integer and Float fields. If you want to get rid of all the formatting, the easiest way to do it is to put "CONCAT()" around the whole thing, which will turn it (in Cargo's eyes) into a String field. If, on the other hand, you just want to switch around the the "," and ".", you can use the global settings $wgCargoDigitGroupingCharacter and $wgCargoDecimalMark for that. Yaron Koren (talk) 15:54, 15 March 2018 (UTC)
Thank you for the fast answer. I have to get plain numbers for calculations with #expr. All went well until the highest number reached 1000. After that my form didn't work any longer…
I apologize for bothering you, because when I read your answer I remembered reading this somewhere or at least something similar.
And well, I wrote this earlier – maybe a year ago –, but thanks again for this great extensions, especially Cargo and PageForms make my work a lot easier :) --~o)sabine(o~ (talk) 21:00, 15 March 2018 (UTC)
I'm glad to hear it! And no problem. Yaron Koren (talk) 14:50, 18 March 2018 (UTC)

Querying a table on the same page and writing the result to a field

I've known about this limitation for a while but I figured it was too specific to ask about, but it's come up a couple other times on other wikis since, that said I don't really expect this to be possible to fix but figured I'd bring it up anyway just in case - If you have row 1 of Table 1 stored on a page, and on the same page you make a query of Table 1 row 1 and attempt to save the value to Table 2, the order in which page content is saved makes it so that you don't actually get the queried value stored. Recreating the table will make everything show up as intended, but then further edits to the page break it again.

Also, you can't store a #var_final, so this isn't an option for saving a value that depends on a template lower on the page.

Any way to get around either of these, or is the current page save order impossible to touch? --RheingoldRiver (talk) 07:21, 18 March 2018 (UTC)

If you want to use a value earlier on the page than where it's set, that's going to be tricky. Out of curiosity, why do you need to store this value in two different tables? Maybe there's an easier way to structure this? Yaron Koren (talk) 14:48, 18 March 2018 (UTC)
Actually in the case today, he wanted to store in the same table (fighting moves evolve from each other, and he wanted to input only the "to" and compute the "from"), but he ended up just doing one single input for all instances on the page in order to do the computation without Cargo, and I don't remember what the second use case was. The reason I wanted to was: In PUBG, most games are played with teams of 4 players each, and that's how teams standardly are, but sometimes there's events with 2 players. In these 2-player events there's a mix of standard teams and pick-up teams. So in some cases I will be identifying a squad of two players by their proper team name, and sometimes by a generic "unaffiliated" tag with a page-specific unique ID to refer them together. (Unique ID is manually entered by the editor and I want it to be mandatory only in the case that it's actively disambiguating.) I have one table that associates players to teams, along with the team's seed etc, which is defined in the team rosters template, and a second one for tournament placements that does team + prize + placement etc (but no rosters). Results are higher on the page than rosters. Normally in queries I just join on the team field, but this case is more complicated, so in the results template I wanted to query the rosters table to get the participants and store directly in order to avoid more complicated queries when I use the data elsewhere, but still be able to enter players only one time. I ended up just giving up on this, and I've been doing the more complicated where clauses where it looks for team if unique id isn't defined. --RheingoldRiver (talk) 15:29, 18 March 2018 (UTC)
Now that I think about it I could probably have unique ID default to just be the team variable in this case since there will be an ID iff team *isn't* unique. So that's an even easier workaround. But it would still be nice if var_final could be made to work. --RheingoldRiver (talk) 15:31, 18 March 2018 (UTC)

Multiple HOLDS checks on a single field?

Does HOLDS not work for seeing if a list has multiple specific values? Looking at _pageData._categories specifically. I want to know if the page is in two different categories but it returns no results, contrary to expected behavior. Or am I doing something wrong?

Thanks --pcj (talk) 00:06, 22 March 2018 (UTC)

Can I get the hierarchical structure as a result?


I'm doing my first steps in using a hierarchy instead of a list of allowed values and I just wondered, if it's more of a visual help when having a long list of values or if it's possible to get a part of the hierarchical structure as a result.

I have a structure of categories


and a page that's part of category C. So it should automatically be part of A and B, too. But can I get all three categories as a result in this case or do I have to handle this in the template e. g. with a case trap? --~o)sabine(o~ (talk) 13:18, 22 March 2018 (UTC)

See 'The "WITHIN" command' - is this the sort of thing you're asking about? Though it doesn't work with MediaWiki categories, unlike SMW - you have to define the hierarchy in #cargo_declare. Yaron Koren (talk) 13:29, 22 March 2018 (UTC)
No, it's not the solution I was looking for. With the WITHIN command I can get the subcategories, but I need to get the supercategories of a subcategory. --~o)sabine(o~ (talk) 17:08, 22 March 2018 (UTC)
Oh... now I get it. No, it's not possible, but this is something I've actually wanted to add to Cargo for a while - the current plan is to have a parser function called #cargo_hierarchy_values, that takes in the parameters "table", "field", "root", "level" and maybe "direction", and displays a tree. It's good to know that there's an interest in it. Yaron Koren (talk) 20:23, 22 March 2018 (UTC)
Sounds great. I'm looking forward to it. But if I could vote for a new function, it would be child tables over this. I'm using a lot of them and it would be lots more, when handling them would be easier :D
But I don't want to push you in one direction, I know almost all programmers are short on time while having too much to work on. And as far I can see you're doing a really great job and good programmers have even more to do :) --~o)sabine(o~ (talk) 21:09, 22 March 2018 (UTC)
Thanks! And both of those are good to know. I'm trying to set the priorities now for future development, so it's good timing to be discussing this kind of thing. Yaron Koren (talk) 21:40, 22 March 2018 (UTC)
So I just stumbled in at the right time :D
I'm curious, will you start a poll for the most wanted new function or decide by yourself? --~o)sabine(o~ (talk) 11:31, 23 March 2018 (UTC)
A poll might make sense - and for Page Forms too. I'll have to think about it. Yaron Koren (talk) 21:53, 23 March 2018 (UTC)

Cargo Query showing escaped HTML tags when page is saved

The issue is similar with this one. It happens every time the page is saved.
You can test this by editing and saving this page. It should look like this. Purging will fix the page.
Seems to be a Cargo issue; any help on this appreciated :) Fbcpck (talk) 00:51, 24 March 2018 (UTC)

Related: Extension_talk:Cargo/Archive_October_to_December_2017#Links/images_sometimes_displaying_as_plaintext
Fbcpck (talk) 00:55, 24 March 2018 (UTC)
In that 2nd discussion you linked to, the people reporting the problem came to the conclusion that it was the presence of the ConfirmEdit extension that was somehow causing it to happen. It looks like you have ConfirmEdit installed - could you try temporarily disabling it and seeing if that makes the problem go away? Yaron Koren (talk) 00:40, 26 March 2018 (UTC)
Small update: a workaround for this issue is to run the cargo query within lua modules. Some examples here and here.
It's somehow faster than calling {{#cargo_query}} too!
Fbcpck (talk) 08:31, 19 April 2018 (UTC)

How to import existing data into Cargo tables?


maybe a naive question as a newcomer to MediaWiki and Cargo,

is it possible to import existing data e.g. in CSV format into a Cargo table? (in a user friendly way, i.e. without doing a SQL request or using the mediawiki API) Thanks in advance

Yes - the right approach depends on whether you want the data to "live" in the wiki or not. Do you want users to be able to modify the data in the wiki once it has been imported? If so, then the Data Transfer extension is the best approach. If you just want the wiki to display the data, but not edit it, you should use the External Data extension, and specifically the #get_web_data function, assuming the CSV comes from some outside API. In either case, you would then need to create a template that stores that data in Cargo, but the exact approach depends on which of these two approaches you take. Yaron Koren (talk) 14:27, 27 March 2018 (UTC)
Thanks a lot for the quick reply :-)

In my case it would be the Data Transfer extension, I will check it out! Patrick

Fields display as "UNIQ-item1-QINU" in queries

I have two template and tables working together, and when I try to run a query pulling data from both, the fields are displayed as 'UNIQ-item1'.

Does anybody know how to fix this? Here are my templates. The two #cargo_query are the ones that show up as 'UNIQ'.


_table = ClientBob
|writer = text
|campaign = text
|project = text

_table = ClientBob
|writer = {{{by}}}
|campaign = {{{campaign}}}
|project = {{PAGEID}}


_table = project
|invoice = text
|client = text
|date = text
|words = integer
|campaign = text
|writer = text

_table = project
|client = {{{client}}}
|date = {{{date}}}
|invoice = {{{invoice}}}
|words = {{{words}}}
|campaign = {{#cargo_query:tables=ClientBob |where=project = "{{PAGEID}}" |fields=campaign}}
|writer = {{#cargo_query:tables=ClientBob |where=project = "{{PAGEID}}" |fields=writer}}

Thanks for your help. • Supāsaru 20:28, 6 April 2018 (UTC)

Adding "|no html" to those queries may possibly fix the issue. Yaron Koren (talk) 22:27, 6 April 2018 (UTC)
That worked! Thanks! • Supāsaru 19:17, 9 April 2018 (UTC)

Find unused IDs


I'm trying to get a list of all IDs from an autoincremental field. The table holds products and in the beginning old products have been deleted leaving gaps in the sequence. Later old ones have just been replaced by other products. Now I want to do a query which shows all gaps and fill them by opportunity.

I found a working solution for this problem on this page. It works, when I do the query via phpMyAdmin, but I ran into trouble, when trying to translate it into a cargo query.

This is the example on the mentioned page:

SELECT a.id+1 AS start, MIN(b.id) - 1 AS end
    FROM testtable AS a, testtable AS b
    WHERE a.id < b.id
    GROUP BY a.id
    HAVING start < MIN(b.id)

And here's my cargo query:

{{#cargo_query:tables=Produkt=a, Produkt=b
 |fields=a.ID+1=start, MIN( b.ID ) - 1=end
 |where=a.ID < b.ID
 |group by=a.ID
 |having=start < MIN( b.ID )

When starting the query I get: Error: join conditions must be set for tables.

Did I miss something or does cargo require a join statement, when more than one table is involved – even if the two are in fact one with two different aliases? --~o)sabine(o~ (talk) 17:23, 9 April 2018 (UTC)

What a crazy query... anyway, I think in MySQL "JOIN ON" and "WHERE" are somewhat interchangeable, but in Cargo they're not, so if you just change "where=" to "join on=" in #cargo_query, the whole thing may work. I'm not sure I understand the query, though, so it might not work. Yaron Koren (talk) 17:59, 9 April 2018 (UTC)
So, if I get you right, a join on clause is required, when more than one table is involved?
I tried to convert the query, but it led to no results. I think, I will have to set up a new table to compare it with. Too bad, I found the above query very smart and elegant.
But anyways, thanks again for the fast reply --~o)sabine(o~ (talk) 08:20, 10 April 2018 (UTC)
Yes, "join on" is required if there are more than table - that's true in SQL, too, for the most part, although MySQL lets you get around it by putting the join conditions in "WHERE" instead. Looking more at the query, I think I understand it better, and actually now I'm surprised that it's not working in Cargo. Is this query on a publicly viewable wiki? Yaron Koren (talk) 14:43, 10 April 2018 (UTC)
No it isn't, atm it is a private wiki for testing purposes.
This is the query I used with pma:
SELECT a.id+1 AS start, MIN(b.id) - 1 AS END
    FROM `cargo__Produkt` AS a, `cargo__Produkt` AS b
    WHERE a.id < b.id
    GROUP BY a.id
    HAVING start < MIN(b.id)
The same results can be retrieved with:
SELECT a.id+1 AS start, MIN(b.id) - 1 AS END
    FROM `cargo__Produkt` AS a
    JOIN `cargo__Produkt` AS b ON a.id < b.id
    GROUP BY a.id
    HAVING start < MIN(b.id)
At last my translation for cargo:
{{#cargo_query:tables=Produkt=a, Produkt=b
 |join on=a.ID < b.ID
 |fields=CONCAT( a.ID + 1 )=start, CONCAT( MIN( b.ID ) - 1 )=end
 |group by=a.ID
 |having=start < MIN( b.ID )
When saving the page I get "Missing '=' in join condition (a.ID < b.ID).".
I assume the only allowed operator in cargo's ON clause is a =. --~o)sabine(o~ (talk) 09:45, 19 April 2018 (UTC)
Ah, yes - now I remember putting in that check, not that thinking that a join like that would ever be used. If possible, could you try replacing the line that throws that error (line 284 of includes/CargoSQLQuery.php) with the following line:
                                $joinParts = explode( '<', $joinString );
...and seeing if that works? Yaron Koren (talk) 12:51, 19 April 2018 (UTC)
I changed the line – after updating cargo from 1.5 to 1.7; lazy me :) – and the error vanished … or better changed into an "a.ID + 1" field unknown error when using the base version:
{{#cargo_query:tables=Produkt=a, Produkt=b
 |join on=a.ID < b.ID
 |fields=a.ID + 1=start, MIN( b.ID ) - 1=end
 |group by=a.ID
 |having=start < MIN( b.ID )
 |order by=a.ID
With a few little tweaks for plain numbers…
{{#cargo_query:tables=Produkt=a, Produkt=b
 |join on=a.ID < b.ID
 |fields=CONCAT( a.ID + 1 )=start, CONCAT( MIN( b.ID ) - 1 )=end
 |group by=a.ID
 |having=a.ID + 1 < MIN( b.ID )
 |order by=a.ID
…the query worked without throwing errors, but also without results :(
I did a counter-check of the query with pma again, but it worked as expected. --~o)sabine(o~ (talk) 16:47, 19 April 2018 (UTC)
Okay, I think I figured out the issue - adding support for '<' and '>' in joins wasn't quite as simple as that one-line change. I just checked in a change which I think actually fixes the issue; and also I think makes the handling of joins a little less hacky than it was before. Yaron Koren (talk) 13:27, 20 April 2018 (UTC)
Wow, thank you. Yes, it works!
Btw, did you think about the poll for the most wanted new features in Cargo and PageForms? --~o)sabine(o~ (talk) 06:57, 21 April 2018 (UTC)

Great. I have thought about it - the problem is just that I haven't had time to do anything more than fix bugs in all these extensions, so it might be overly optimistic to start talking about new features. Yaron Koren (talk) 01:54, 23 April 2018 (UTC)

Oh, I'm sad to read this :(
So I'll try to stay patient and bother you just in case I stumble over a world destroying bug :)
~o)sabine(o~ (talk) 12:53, 23 April 2018 (UTC)

'Create table' just stalls

I've recently added Cargo to my production server (my question above was to help me get it started on my dev server).

Now, when I create a template and click 'Create Data Table', the site just stalls. I get the 'wait' cursor, and nothing happens. I don't have root access on my server, so I used the web updater when I first installed Cargo. It appeared that the Cargo tables were added fine.

Does anybody know why the extension stalls?

Here are my server settings:

MediaWiki 1.28.2
PHP 7.0.29 (litespeed)
MySQL 5.6.38
ICU 57.1

Article path /$1
Script path /
index.php /index.php
api.php	/api.php
load.php /load.php

Current extensions:
InputBox 0.3
Cargo 1.3
A custom word-count Special page

Supāsaru 10 April 2018

You're using a very old version of Cargo - you should upgrade to the latest version. (Don't bother with branches like REL1_28.) Yaron Koren (talk) 21:36, 10 April 2018 (UTC)
I've just downloaded the latest version of Cargo, ran the web updator, and I still get the wait icon. The tables aren't created. Do you know what else could be causing this? • Supāsaru 00:32, 11 April 2018 (UTC)
Alright. If you know how, you should view the JavaScript console on that page - my guess is that there's a JS error that's preventing the code from running. Yaron Koren (talk) 02:25, 11 April 2018 (UTC)
Thanks, Yaron. I saw that Cargo was calling api.php, which I had disabled on my wiki. I enabled api.php, and that seemed to work. I took a moment to Cargo's installation instructions. • Supāsaru 15:29, 11 April 2018 (UTC)
Okay, great. I moved your note about disabling the API into the "Common problems" page, but it's helpful. Yaron Koren (talk) 16:56, 11 April 2018 (UTC)

Version compatibility policy

@Yaron Koren: Cargo doesn't use release branches does it? So I guess it has "master" compatibility policy? This should be set in the extension template. Although, I reckon we should have a 'tags' policy, for extensions that tag their releases (and maybe use semver). Sam Wilson 23:59, 10 April 2018 (UTC)

No, it doesn't use branches. I don't know what master compatibility is, but Cargo (as with all of my other extensions) tries to keep backwards compatibility with older MediaWiki versions - my general aim is to be compatible with around the last two years of versions, though most of my extensions currently support versions going back to 1.23, which is almost four years ago. If there's something that can be added in the extension templates to reflect that policy, that would be great. Yaron Koren (talk) 02:29, 11 April 2018 (UTC)
@Yaron Koren: Yeah, that fits the 'master' policy. It's only if you treat master as a development branch (and don't want people to use it in production) that it wouldn't be. But if you're not using REL branches, then it's not that. :) I've added it to the infobox. Sam Wilson 05:08, 11 April 2018 (UTC)
That's good to know. Thanks! Yaron Koren (talk) 15:04, 11 April 2018 (UTC)

Can I store an average of an array in another field?

I'm working on a wiki for a level of difficulty rating for music. My thought was to allow individual teachers to cast a "vote" in an array field of floats from 1-15, then use the average of those votes in another field. Is that doable? For example,

Field: Difficulty Vote (array, float)

Field: Average Difficulty (average of Difficulty Vote array)

In my heart of hearts, what I would really like to do is provide a Linear Analog Scale and capture the results as a one dimensional coordinate from 0-15 (click on this line your vote for level of difficulty). Then I could use proximity to coordinate in a query like map coordinates. (maybe I should just rate pieces on a level of difficulty from Los Angeles (easy) to New York (Transcendentally difficulty)

Yes, you can store it (just remember to add a "|no html" parameter to the #cargo_query call within #cargo_store), but it might be easier to just re-calculate the average every time. In either case, you'd need to use the MySQL AVERAGE() function. As for the linear scale - if you're using Page Forms, I suggest you check out the "rating" input type. It won't show cities, but maybe that's fine. :) Yaron Koren (talk) 00:51, 12 April 2018 (UTC)

Export as Excel not working

I try to export data in an Excel format, I get a fatal error. The exporting seems to work, but when I click on 'View XLS', that's when the fatal error appears. When I turned on debugging, I see \Cargo\specials\CargoExport.php: Class 'PHPExcel' not found.

Do you know how to resolve this? • Supāsaru 21:09, 12 April 2018 (UTC)

You need to install the PHPExcel PHP library for that feature to work. I know it's deprecated (in favor of the new PhpSpreadsheet), but I think it's still usable. Yaron Koren (talk) 21:43, 12 April 2018 (UTC)

PostgreSQL incompatibility in includes/parserfunctions/CargoStore.php

   $tableFieldValuesForCheck = array( '_pageID' => $pageID );

should be

   $tableFieldValuesForCheck = array( $cdb->addIdentifierQuotes( '_pageID' ) => $pageID );

otherwise you get

    ERROR:  column "_pageid" does not exist at character 93
    HINT:  Perhaps you meant to reference the column "<table>._pageID".
    STATEMENT:  SELECT /* Wikimedia\Rdbms\Database::select  */
      COUNT(*)  FROM "<table>"    WHERE _pageID = '8906' AND "Name" =

--IijimaYun (talk) 19:55, 15 April 2018 (UTC)

Thanks for pointing that out - I checked this change into the code, so hopefully now it works fully with Postgres. Yaron Koren (talk) 17:30, 16 April 2018 (UTC)

Problem with division operator

I'm having an issue with the query below. I have two tables, one with transaction history in different currencies and one with exchange rates. To create a column with USD_amounts, I join the tables and want to divide the transaction amount by an exchange rate and display this in a separate column. I haven't been able to get the syntax right. I modified Localsettings.php to include $wgCargoAllowedSQLFunctions[] = 'DIV'; Is there something I am missing? Your help is greatly appreciated.

ERROR MESSAGE:No field named "(t.Investment_amount/r.Rate)" found for any of the specified database tables.

{{#cargo_query: tables=transaction=t,fx_rates=r |join on=t.Currency=r.CurrencyID |fields=t.Investment_amount, t.Currency, r.Rate, (t.Investment_amount / r.Rate)=USD_amount |where=t.Investment_amount > 0 AND t.Investment_status='Committed' }}

Oh, that's an interesting use of Cargo. Adding to $wgCargoAllowedSQLFunctions won't have any effect because "/" is an operator, not a function. The easiest way I can think of to get this working is to change "(t.Investment_amount / r.Rate)" to "CONCAT(t.Investment_amount / r.Rate)" - CONCAT() with a single argument doesn't actually do anything, but it's useful as a hack because of the way Cargo's parsing currently works. I hope to improve the parsing code so that this sort of thing would not be necessary. Yaron Koren (talk) 13:43, 25 April 2018 (UTC)
That solved it! Thanks for the help and for your work developing all of these amazing Mediawiki extensions!! I now see what was the issue was. Another option is using ROUND(). -- 16:43, 25 April 2018 (UTC) Donovan
Great! Oh, yes - ROUND() is a much better choice, because then it would actually be serving a useful purpose. Yaron Koren (talk) 18:27, 25 April 2018 (UTC)

Creating a table with a unique parameter fails

Hello, we're trying to create a table with the new unique parameter, defined here. When trying to create the table through the wiki interface, the spinner just goes forever without creating the table. When trying to create through the API it returns an error:

{"error":{"code":"internal_api_error_DBQueryError","info":"[397d9d4f88c82b9f95abe588] Database query error."}}

Please advise. --pcj (talk) 18:42, 30 April 2018 (UTC)

I don't know why that's happening for you; I'm guessing that it's due to some database configuration, but I'm not sure. Could you convince the people running the server to add "$wgShowSQLErrors = true;" and "$wgShowExceptionDetails = true;" to LocalSettings.php, if only temporarily? That might show a more useful error message on the API page. Yaron Koren (talk) 13:21, 1 May 2018 (UTC)
I have an internal ticket in to try to address this. I suspect if they find anything they'll contact you. --pcj (talk) 12:40, 6 May 2018 (UTC)
["error"]=> string(96) "Specified key was too long; max key length is 767 bytes (****:3306)" ["errno"]=> int(1071) ["sql"]=> string(218) "CREATE TABLE `cargo__TestUnique` ( `_ID` Int NOT NULL UNIQUE, `_pageName` Varchar(300) NOT NULL, `_pageTitle` Varchar(300) NOT NULL, `_pageNamespace` Int NOT NULL, `_pageID` Int NOT NULL, `Field1` Varchar(300) UNIQUE )" Alexia E. Smith (talk) 12:53, 14 May 2018 (UTC)
Thanks, that's very helpful. This problem with keys being too long has come up quite a bit recently, due to a combination of the use of utf8mb4 encoding, and this new "unique" parameter. I'm still trying to figure out the best approach for it - what I would recommend in your case, until there's a true fix within Cargo, is to manually decrease the size of the field within #cargo_declare to 255, so that it's "Field1=String (size=255)". If that still doesn't work, you may be using utf8mb4 encoding, so try "Field1=String (size=191)". Yaron Koren (talk) 15:35, 14 May 2018 (UTC)
It looks like 255 worked here. Thanks all. --pcj (talk) 22:51, 14 May 2018 (UTC)

Checking for Cargo usage in templates

Is there any way of checking if a template is currently used from a #cargo_query and e.g. automatically adjust the templates margin? --

Only by doing a text search on the wiki. What does "adjust the templates margin" mean? Yaron Koren (talk) 04:49, 6 May 2018 (UTC)
An example of what I meant would be Cargo passing an implicit argument (e.g. cargousage) to the template. Then the template could check if that argument exists and adjust some values (increase a div margin, add a * in front to make it a list object instead of having a separate *{{Template1}} template, …).
Another usage would be during the SMW -> Cargo transition, where unnamed arguments often have to be lowered by one since Cargo doesn't automatically pass a link to the page as the first argument. Instead of creating a separate Cargo template, this would allow simply temporarily wrapping the parameters in an if statement like this: {{#if: {{{cargousage|}}}|{{{1}}}|{{{2}}} }} while the queries are updated.
Since it doesn't seem to exist I'd like to suggest adding it or something similar. The argument could even contain some other potentially useful information (e.g. the number of the current template). --
Oh, now I get it - you want the template to know whether it's being called as a formatting template from a Cargo query. Well, you can always do it manually, by adding something like 'yes' = cargoquery to the list of fields, if you're using named args. Or, if you're not using named args, you can put _pageName at the beginning of the list of fields, so that it matches SMW's output. Yaron Koren (talk) 13:56, 6 May 2018 (UTC)

Page record not deleted from table upon page delete?

Hello, we have an issue where Cargo is being affected by cache after table recreation, such that a purge doesn't always serve to fully update the table. I've found that deleting and restoring the page serves to work around this (if a bit drastic), however now we have an issue where the page's old ID is showing up in the table despite the old pageID (318205) no longer existing. Is there any way to clear out the erroneous data short of recreating the table? --pcj (talk) 21:14, 5 May 2018 (UTC)

That's not good; I wonder if this is related to the other problem you reported. But recreating the table seems like the easiest solution. I didn't understand the first problem you mentioned; what's the issue with the cache? Yaron Koren (talk) 04:52, 6 May 2018 (UTC)
Aye, it may be related to the other problem, though we have seen it on wikis where we didn't use the unique param at all (and before that was rolled out). This fix was actually after recreating the table from removing the unique parameter because it was causing DB query errors where it was used. What we were seeing in regards to the cache is recreating the table doesn't cause all associated rows to be re-added to the table. We sometimes have to null edit each affected page to ensure they show up, and in some cases delete/restore. --pcj (talk) 12:38, 6 May 2018 (UTC)
I just want to clarify that caching issue - it sounds like the problem is just that the table recreating doesn't fully populate the table, which wouldn't be related to caching or cache purging. Or is there something else going on? Yaron Koren (talk) 14:06, 6 May 2018 (UTC)
It doesn't immediately recreate the table, no. I believe GP uses a custom job runner that may be affecting it. --pcj (talk) 22:44, 14 May 2018 (UTC)

Using values stored on the same page

Hi. We're having some problems with Cargo when storing values that are based on values stored by templates on the same page (or even the same template). What happens is, that it will look fine on the page itself, but store <em>No results</em> in the Cargo table. Is there any way to make this work or to somehow work around it? --Litzsch (talk) 20:56, 11 May 2018 (UTC)

Yes, having a page query itself, then try to store the result of that query (or something based on that result), is tricky, if not impossible. If it's in the same template, can't you just use the template parameter value, instead of running a query? And in either case, do you really need to re-store this value? Can you just query the original value instead when you're running the main query, using a "join"? Or is this storage more complicated than that? Yaron Koren (talk) 22:11, 11 May 2018 (UTC)
We have a template for abilities that stores key value pairs in this table. However in addition to the normal value, for some values special value_aghs (an expression based on the corresponding value), value_tal (an expression based on the corresponding value and a value from a talents template on the same page), and a value_aghs_tal (value aghs + value tal). In the wikicode it looks like this (#show is still SMW because I stopped porting stuff in those templates for now, and Template:Show is a template emulating #shows function with Cargo):
| trait1 = Cast Range
| value1 = 750
| value1 tal = {{#expr:{{#show:#Chain Frost|?value1#}}+{{show|T|Lich|talent3right value}}}}
| value1 aghs = {{#expr:{{#show:#Chain Frost|?value1#}}+100}}
| value1 aghs tal = {{#expr:{{#show:#Chain Frost|?value1 aghs#}}+{{show|T|Lich|talent3right value}}}}
Taken from here
The expressions are different for every ability/trait though so we can't include them in the template itself. The stored values are then accessed on different pages around the wiki to create tables listing the values for each ability (e.g. the list of cast ranges here). The values are also used in notes and descriptions in the same template.
I've tried quite a bit, but failed, to come up with a solution that doesn't rely on re-storing and -using values.--Litzsch (talk) 22:56, 11 May 2018 (UTC)
Okay, I think your understand your basic data structure here. It's a complex one - normally this would be implemented either via a multiple-instance template (with one instance for each trait), or hard-coded - so that there would be explicitly-named fields for "Cast Range", etc. The latter option seems to make sense here - for instance, you're expecting that there will be one, and exactly one, trait named "Chain Frost", if I understand it correctly; so why not hardcode that in the template (and form)? That seems like it would make both data entry, and this kind of querying, easier. Or am I misunderstanding the situation? Yaron Koren (talk) 15:06, 14 May 2018 (UTC)

Page creation based on cargo data

Working on a complex project where I need to create pages from parsed data. The hairy parts:

  • Pages need to be initially created based on the data (After it's stored into cargo tables created via templates)
  • Certain data fields will be periodically updating from the external source
  • Other data fields will be open to editing via PageForms

Currently I'm looking at injecting the data directly via SQL into the cargo tables (data is too messy to be handled via ExternalData). Is there a way you know of where I can create the pages from the cargo tables? Any advice regarding the other parts will be greatly appreciated. Thank you.

Hi - I'm not sure I fully understand what you're trying to do, but I think I get the basic gist. You have two kinds of data - data that is meant to "live" in the wiki, as wikitext, and data that will remain external and be queried. These two kinds of data require different handling - and in neither case is directly modifying the Cargo tables the way to go, I don't think.
  • For data that lives in the wiki, it might be possible to use the Data Transfer extension to create the necessary template calls. If it's somehow more complex than that, it may require creating a custom script to populate the pages. One of those two options is probably the way to go, I would say.
  • For outside data, I do think External Data is the way to go. If it's too complex to do a direct SQL (or whatever it is) query, the best solution may be to create a wrapper script for the actual, providing a "mini-API" that External Data can query. Yaron Koren (talk) 14:34, 21 May 2018 (UTC)
Didn't know about the import capabilities of Data Transfer. I looked at AutoCreatePage and was thinking of using their parser function with cargo query and hoping that might work!
The data comes in JSON. I can work it into final CSV files that I assume External Data can then import regularly. Problem is how to ensure a behavior that's more or less like MySQL's ON DUPLICATE KEY UPDATE; I know a cargo field can be declared unique, but is that used like a key/index upon data importing with External Data?
There's nothing like "ON DUPLICATE KEY UPDATE" - although every Cargo table has an "_ID" field that auto-increments. Maybe that's good enough? Yaron Koren (talk) 03:44, 22 May 2018 (UTC)
A couple of issues with that. First, when I tried to define _ID as a cargo field I was hit with Error: Field name "_ID" begins with an underscore; this is not allowed. So, if I use External Data and have an _ID field in my CSV file, will that initiate an update instead of a new record?
The other issue is that to use _ID, I'll have to query each cargo table before hand to find each record's _ID then match each record with its counterpart in the JSON (using another mutual field). This actually is what got me to think of SQL-ing the whole thing since I'm getting myself into that mess anyway! Using SQL I can also easily create a unique index of the mutual field to then utilize ON DUPLICATE KEY UPDATE. My concern here of course is if any of this would conflict severely with Cargo's internal processes which I'm sensing from you. Is there something in particular that I should be weary of?

Right, the _ID field gets automatically created and populated by the code. I don't understand what you're trying to do, though - why not join based on the page name? Yaron Koren (talk) 00:26, 23 May 2018 (UTC)

So I went ahead and:

  • Used Data Transfer to import a set of data from a csv file, went pretty well:
    • Had to run touch.py afterwards on the pages for them to be parsed by cargo
    • Each row was duplicated in the cargo table!
  • Used External Data on a CSV file on the server with slightly altered data (just like proposed case above with CSV file being updated from an external site then)
    • Instead of updating existing rows as desired, more rows were created
    • Yet again, rows were duplicated

You can see the resulting table here: https://californiavalues.cf/Special:CargoTables/Data_City_California

You're using a rather old version of Cargo, which is why I think you're seeing that duplicates problem - there have been a number of fixes since then to try to avoid duplication. (Though in your case, calling touch.py on each page may have contributed to the issue - I don't think you needed to call it.) Anyway, I'm guessing you have that old version because you used the "Extension Distributor" to get the version to corresponding to your MediaWiki version. For Cargo (and my other extensions), you shouldn't do that, because they have backward compatibility - instead you should just get the latest version. Yaron Koren (talk) 13:31, 29 May 2018 (UTC)
I actually use a script that clones the extension with the branch matching the wiki's version as per MW recommendation. That was REL1_30 so I guess Cargo's branch of it wasn't updated. Switched to master so now it's updated and I recreated the table. Most duplicates are gone as you predicted but still each row was duplicated instead of being updated by External Data. The question is if there's a way to force External Data to update instead of create new records as I understood is possible from your suggestion early on.
I'm not sure I understand what you're trying to do - maybe in part because I can't view the wikitext on your site. But are you asking about having External Data modify Cargo data that's already being stored via wikitext on the page? If so, that can't be done - though I don't really understand how it could work even in theory. If the outside data being queried by External Data changes, and the template calls on the wiki change as well, which one should the Cargo data reflect? Though again, maybe I'm misunderstanding your issue. Yaron Koren (talk) 03:19, 30 May 2018 (UTC)
You got it! In theory the latest call should be saved. For my case, externally queried fields don't and shouldn't be manually edited, i.e. I won't even show them in the PageForm.
Weirdly, it looks like Data Transfer is better suited for my case since I saw it has an "Overwrite only fields contained in the file" option and so it won't affect the other fields of the template (which will be editable). Problem is that this can't be done via a call like External Data. I guess this is a feature request for External Data to save directly into template based pages just like Data Transfer does!
Well, I suppose it's possible to have two copies of a data set, and for each row show the one that was edited most recently, but that strikes me as a bad idea; and I'm not planning to modify any of the software to support it. Yaron Koren (talk) 13:14, 30 May 2018 (UTC)
That does sound like a bad idea and it's my bad for not being clearer. Scratch all above. It seems Data Transfer's import functionality is the way forward. What I'm looking for now is a way to easily interface with it. One way is to have a wiki function, like External Data's #display_external_table, but instead of displaying/saving to Cargo, it'd import the data directly using 'Data Transfer'. Something like:

{{#import_external_data: template=template name |data=set of parameters, separated by commas |overwrite='all' OR 'append' OR 'contained' OR 'skip' }}

The other, possibly simpler way would be to expose Data Transfer import functionality with a maintenance script. So it'd be like extensions/DataTransfer/maintenance/importCSV.php --overwrite=contained path/to/file.csv

What would "overwrite=contained" do? That may be the key to what I don't understand here. Yaron Koren (talk) 02:35, 31 May 2018 (UTC)

It's straight from the options at Data Transfer's Special:ImportCSV page. Illustration here: https://i.imgur.com/jAQLY3X.png
Oh, I get it - in the Data Transfer code, that's called the "merge" option. Yes, a maintenance script for "import CSV" sounds like a good idea, since that's by far the most used of the Data Transfer operations. Yaron Koren (talk) 15:00, 1 June 2018 (UTC)

Store Many Records in Table, One At a Time?

I can't store more than one record in a table, if added singly.

When i call `cargo_store`, it writes that one record to the table, as expected.

However, the existing record in the table gets deleted. I can't store more than one record.

Here's my template code:

! TagName
| {{{TagName|}}}
! Description
| {{{Description|}}}

Here's my storage code:

_table = TopicTags
|TagName = MyTag3
|Description = Description three.

! TagName
| {{{TagName|}}}
! Description
| {{{Description|}}}

What am i doing wrong?

(Btw, i have no idea what is the purpose of the code which begins {|. I can't find any explanation in the extension help-doc, which makes me think it's core syntax, or code for another extension.)

Johnywhy (talk) 07:08, 24 May 2018 (UTC)

Progress: i realized this is caused by the build-in fields, such as "Page".
The reason is that when i try to add more records from same page, it appears that your extension only allows one record per page-- each record is associated with one page only. I cannot add more records from same page.

Johnywhy (talk) 07:10, 24 May 2018 (UTC)

How to Remove Built-In Fields?

How to remove "Page" field, and other build-in fields?

Update: I removed the built-in fields using phpMyAdmin. That breaks the extension. So, i cannot have a table without those built-in fields?
What is the "Page" field, and why do you want to remove it? Yaron Koren (talk) 15:48, 24 May 2018 (UTC)
"table will additionally hold the following columns:
  • _pageName - holds the name of the page from which this row of values was stored.
  • _pageTitle - similar to _pageName, but leaves out the namespace, if there is one.
  • _pageNamespace - holds the numerical ID of the namespace of the page from which this row of values was stored.
  • _pageID - holds the internal MediaWiki ID for that page.
  • _ID - holds a unique ID for this row."


i need ability to create tables with ONLY the columns of my choice. when i try to add more records from same page, it appears that your extension only allows one record per page-- each record is associated with one page only. Your extra fields prevent adding more records from same page.
i need ability to create records which are NOT associated with particular pages. I need ability to create tables with ONLY the columns of my own choice.
i created a table with the following declare:
But when i opened the table in phpMyAdmin i found the extra columns which i did not specify. When i deleted the built-in columns, then i received this error when tried to view the table in Special:CargoTables/MyTable

> Error: 1054 Unknown column '_pageName' in 'field list' (localhost) Johnywhy (talk) 05:16, 25 May 2018 (UTC)

Alright. No, you can't get rid of those "helper" fields - the code relies on them. To be clear, you can have more than one record/row per page - there's one record for each template call, and you can have more than one call to a template on a page. Yaron Koren (talk) 13:36, 25 May 2018 (UTC)
"you can have more than one record/row per page"
How? Can you show an example of that? All your built-in fields are primary keys, preventing a repeat of the same page.
Ok, i tested this, and i see how more than one record can be added by the same page. While all page-associated fields are the same for all page-records, the _ID field is different for each one. That allows multiple records added by same page.
But apparently only works if both records are added at the same time by the same page. I need ability to add multiple records at different times, by the same page. That would require forcing the _ID field to auto-increment. How can that be done?
How am i adding different records at different times from the same Template? My template contains parameters-- each time the template is rendered, the parameters passed into the template determine the data to be added to the table. I don't see how that can work with this extension. Please let me know if there's a way.
Apparently, it's your code that's deleting all existing page-records before adding new ones to the same page. Sure, you might mod your extension to allow my use-case, but it's not really what your extension is intended to do. This is the wrong extension for my use-case.
Extension:External_Data would be the right extension, if it could do writes.
For now, i still haven't found a way to do structured reads/writes both, with either a db-table, or xml/json source, or wikipage (i would be happy with any of those formats).
Cargo section of documentation updated.

How can I run a cargo_compound_query from a separate extension's php code?

I have an extension that I'm trying to move to an external server while still keeping the data layer managed and accessible by wiki. Because of this I'm writing a little backend extension for packaging together queries for different application page load states. I'd like to use the handler for cargo_compound_query directly as if I was using the parser function in a page.

It sounds like the real problem is not that this is from a separate extension, but rather from a separate wiki. If this were just #cargo_query it would be pretty easy to do, via the API; but given that this is #cargo_compound_query, I don't think there's a direct API call that you can make, unfortunately. What about calling each sub-query separately via the regular query API, though, then combining all the results together in your code? Yaron Koren (talk) 15:54, 24 May 2018 (UTC)
It's not a separate wiki, it would be just a standalone server for hosting an application. The only wiki interaction is at the data layer. We like all the benefits of having the data in the wiki itself but have frontend needs that surpass the native abilities of MW and extensions. I currently have an api module roundtable that I've made which sits on the wiki that contains the Cargo data. So I just need to be able to query the cargo databases using the #cargo_query or #cargo_compound_query parser function BUT from the context of the roundtable api module.
Alright - well, you may be able to do it by calling CargoCompoundQuery::run(), in a format like "CargoCompoundQuery::run( $dummyVar, 'tables=A;fields=B,C;', 'tables=D;fields=E;where=" ... and so on. Yaron Koren (talk) 16:12, 25 May 2018 (UTC)

Issue with special characters (sometimes)?

I have a page that's not behaving how I expect it to, and I think it has to do with how Cargo is handling special characters. In this drilldown, you can see that there are only 2 options available for the field "OtherName," however here everything appears as it should. Here is the page in question, the issue is that not all of his names are bolded in the roster. This is the module used to generate that display.

Of the above, I think only the first two links are relevant to show the issue, but if you need more info I can give more explanation/examples of what I'm doing. I'm not 100% sure this is a bug with Cargo, but I can't figure out any reason that this would break only when there are special characters. Thanks --RheingoldRiver (talk) 07:01, 25 May 2018 (UTC)

This does seem to be a bug in Cargo with the handling of special characters. However, maybe it's not that important in this case? I'm wondering why you even have Cargo storage for the "PlayerRedirects" info - it seems to be just a copy of information already found in the "InfoboxPlayer" template/table. Maybe you can just get rid of the table...? Yaron Koren (talk) 13:45, 25 May 2018 (UTC)
The table is 100% needed to be used in a join condition because any of the player's names could be used in the TournamentRosters table, so that's the only way I have to make sure that PlayerResults actually pulls all relevant events. Also that field in Infobox Player is entered manually, and the PlayerRedirects table is generated automatically based on redirects to the page using DPL (eventually the manual data entry will be gotten rid of completely but there are a few complications with that). --RheingoldRiver (talk) 20:15, 25 May 2018 (UTC)
Alright. Well, in any case, the "PlayerRedirects" drilldown interface doesn't seem that useful. (I would think.) So, this still seems to be a bug, but maybe it's not a major one. Yaron Koren (talk) 23:49, 25 May 2018 (UTC)
I tried doing another query here and it does seem to be just ignoring the special characters, which is weird because it deals with them properly in a join condition. So hopefully this is just a small thing and easy to fix. --RheingoldRiver (talk) 00:18, 26 May 2018 (UTC)

Does line 136 in CargoSQLQuery.php imply that wildcards (*) are not allowed even in context of fields/SELECT?

$regexps = array(
			'/\bselect\b/i' => 'SELECT',
			'/\binto\b/i' => 'INTO',
			'/\bfrom\b/i' => 'FROM',
			'/\bunion\b/i' => 'UNION',
			'/;/' => ';',
			'/@/' => '@',
			'/\<\?/' => '<?',
			'/\-\-/' => '--',
			'/\/\*/' => '/*', # Line 136
			'/#/' => '#',
		foreach ( $regexps as $regexp => $displayString ) {
			if ( preg_match( $regexp, $tablesStr ) ||
				preg_match( $regexp, $noQuotesFieldsStr ) ||
				preg_match( $regexp, $noQuotesWhereStr ) ||
				preg_match( $regexp, $noQuotesJoinOnStr ) ||
				preg_match( $regexp, $noQuotesGroupByStr ) ||
				preg_match( $regexp, $noQuotesHavingStr ) ||
				preg_match( $regexp, $noQuotesOrderByStr ) ||
				preg_match( $regexp, $limitStr ) ||
				preg_match( $regexp, $offsetStr ) ) {
				throw new MWException( "Error: the string \"$displayString\" cannot be used within #cargo_query." );

If you're asking about having something like "fields=*" in the #cargo_query call, then yes, that's unfortunately not possible. Yaron Koren (talk) 23:51, 25 May 2018 (UTC)
Where is the best place to create proposals for changes? I'd like to propose a way to implement * for fields =
I guess phabricator.wikimedia.org. People have made that particular request before (I don't know if on Phabricator). I remember looking into before and deciding not to implement it, but I don't remember why, or how long I actually thought about it. Yaron Koren (talk) 18:34, 29 May 2018 (UTC)

How to Delete a Table?

How to Delete a Table?

Go to Special:CargoTables and then click "Delete." --RheingoldRiver (talk) 12:36, 26 May 2018 (UTC)

Remove page associations?

Cargo is designed to associate data with pages. I need to read/write a table without any page-associations. I just want to maintain a list-- not semantic or page-specific features. Is there any extension or method for reading/writing a table with wikitext, without page-associations?

I have a namespace on my wiki called "CargoConcept" (name borrowed from SMW's concept) that I use just to save cargo tables that don't have page associations. You can see pages here. --RheingoldRiver (talk) 16:06, 27 May 2018 (UTC)

Datetime and Templates

I can't seem to get datetimes working to accept templates. The current template I use has a template that displays UTC time with a title/caption with EST and PST, but cargo stores that as Null/Jan 1, 1970. Is this possible or do I have to go about it a different way? Thanks.

— Preceding unsigned comment added by UnderscoreNorth (talkcontribs) 12:07, 29 May 2018

Does the template just display text, or is there some formatting/HTML involved? If there's any formatting, Cargo can't read it - you may need to put something else in #cargo_store. Yaron Koren (talk) 13:33, 29 May 2018 (UTC)

Cargo query with template display format does not expand <code> tags from wikitext field

I have a pretty straight forward query which returns it's result into a template via named args. When being displayed it doesn't parse the <code> tags. Is this a limitation of cargo currently?

My query:

|named args=yes
|order by=Title ASC

My Template:

{{#arraymap:{{{Title}}}|,|x|<b>[[{{{PageName}}}|x]]</b>|&nbsp;or&nbsp;}}: {{{ShortDesc}}}

--Wuestenarchitekten (talk) 04:05, 30 May 2018 (UTC)

It might be a bug. What version of Cargo are you running? Yaron Koren (talk) 11:50, 30 May 2018 (UTC)
Currently on 1.4 from last Novmeber 15:34, 30 May 2018 (UTC)
Updating to 1.7 fixed that - thanks! Wuestenarchitekten (talk) 16:00, 30 May 2018 (UTC)
Great! Yaron Koren (talk) 18:06, 31 May 2018 (UTC)

small qol request: link to the declaring table when attaching

Could this message also say "The table is declared by xyz template" with a link to the template? It would be a nice QOL change to make it easier to find declaration templates without having to add to the documentation manually. Thanks --RheingoldRiver (talk) 09:02, 30 May 2018 (UTC)

That's a good idea. I just checked in a change that does this - hopefully it'll get to you before too long. Yaron Koren (talk) 18:05, 31 May 2018 (UTC)