Extension talk:Cargo

Delimiter for |format=template
Is it possible to set some delimiter when |format=template? It was possible in SMW, and there's some queries I'm trying to switch over to Cargo that were set up like that. --RheingoldRiver (talk) 16:18, 7 January 2018 (UTC)


 * No - I haven't seen a good reason for it yet. What do you want the delimiter to be? Yaron Koren (talk) 21:28, 7 January 2018 (UTC)


 * In some cases &amp;ensp; (although this one is generally fine to just append to the template, sometimes the template is used in multiple places and I don't always want the leading/trailing space), often &#8226;. Sometimes a break between table cells, this is the template. There were probably other cases that I used a separator in SMW with template format but these are the ones that come to mind. --RheingoldRiver (talk) 23:06, 7 January 2018 (UTC)


 * Is the current behaviour to insert a space between result item templates, or is it a newline? Or nothing? (I know I could check, but the couple of places I'm using it it seems to be a newline, but I'm not sure.) Sam Wilson 01:40, 11 January 2018 (UTC)


 * Okay, that makes sense. I checked in a "delimiter" parameter for the "template" format earlier today - hopefully it will get to you before too long. Sam - by default it doesn't put in anything. Yaron Koren (talk) 03:13, 11 January 2018 (UTC)
 * Thanks! --RheingoldRiver (talk) 12:19, 11 January 2018 (UTC)

Jobs issue
MW 1.30, Cargo 1.30.

If $wgJobRunRate > 0, records are sometimes stored to DB twice when you save a page. --StasR (talk) 21:21, 12 January 2018 (UTC)


 * Sorry, what version of Cargo are you using? Yaron Koren (talk) 22:45, 12 January 2018 (UTC)


 * Git branch REL1_30 — v. 1.4 (1798b60) 13.09.2017 --StasR (talk) 23:03, 12 January 2018 (UTC)


 * Oh - you should upgrade to version 1.5, which is when the duplicates problem was fixed. Or just use the very latest code. In general, the "REL" branches are not useful for any of my extensions, and possibly not for any non-WMF extensions. Yaron Koren (talk) 23:31, 12 January 2018 (UTC)


 * Thanks, it helped! --StasR (talk) 00:03, 13 January 2018 (UTC)

Underscore inconsistency Part II
Previously I [https://www.mediawiki.org/wiki/Extension_talk:Cargo/Archive_August_to_September_2017#Underscore_inconsistency? reported] there is inconsistency with use of underscores in queries and templates. For some reason underscores must be replaced with spaces when referring to fields in templates. (Of course, if this is changed, many templates must be redone, as they now use the workaround of space instead of underscore).

It looks like I came across another issue related to this one (unless "type" is another banned SQL keyword?). In for instance Enchantment_poe1, try to click on one of the "next 250" and such links, and an error message pops up (notice the lack of underscore): Table "Enchantment poe1" not found in Cargo database. –Pangaearocks (talk) 09:08, 13 January 2018 (UTC)


 * Wow - it's surprising that no one saw, or at least reported, that error until now. My guess is that it's in part because the "view next pages" functionality in Special:Drilldown is rarely used, since the page is meant to winnow down options to a manageable size. Anyway, I just checked in a fix for the problem; sorry about that. Yaron Koren (talk) 14:01, 15 January 2018 (UTC)

Missing tail end of record set when calling same #cargo_store template multiple times from the same article
I have certain pages for which I want to store multiple rows of data in the same table for a single page, I'm doing this by calling the same #cargo_declare/#cargo_store template multiple times from the page, with different variable values. While this does work, I have noticed that random pages often experience missing rows. Upon examining this, I've noticed that it's always the tail end of the records that go missing. Consider a page trying to store 5 rows with the values 1-5 to some table, using a template, and five separate calls like this:

When this works, five records are created like

When rows are missing, it's always an uninterrupted set from the end, leaving the table in a state like for example

Different rebuilds of the same data results in different pages missing a different amount of records (from the end), so I'm ruling out malformed input, as a template call that sometimes works does not always work. I have experimented with recreating the data into a replacement table, checking for missing rows, null-editing all pages calling the template, checking again for missing rows, null-editing again, etc. and each set of null-edits has a chance to add some of the missing rows, but not always, and not necessarily all of them (causing a page to still be missing the tail end of records, but a lower amount of them).

Do you have any insight into what might be causing this behavior? Worth mentioning is that I'm using a Scribunto module to calculate the values which are to be set, and using the Lua  for each call, which may or may not prevent the Cargo data from being properly stored on page re-cache, as pages with missing records do not get them added after the incomplete replacement table has been switched in, even through null-edits or scheduled page cache refreshes. SaschB (talk) 05:09, 14 January 2018 (UTC)


 * My guess is that the jobs are timing out before completion, possibly due in part to the complexity of these storage calls. Is your "job run rate" set to something higher than 1? If so, I would try bringing it down to 1 (or even less). Or, if you have command-line access, you could try running the cargoRecreateData.php script, instead of recreating from the web interface - that should work better. Yaron Koren (talk) 14:04, 15 January 2018 (UTC)


 * This is on Gamepedia, so the admin/config is handled by other people, I asked them about the job run rate, and it's apparently set to 1. A separate issue is that, while newly created pages (with less complex call structures) are picked up fine, any edits to pages which already have some data stored do not seem to update the live tables. This means I have to use the web interface to recreate tables (and subsequently null-edit all category members, as it were) after any edit I want immediately usable. I believe these recreates might not be strictly necessary, but in many cases I can't wait 24 hours for the tables to refresh, and null-edits or purges of pages do not update live tables - only replacement tables (additionally, null-edits to pages which store data in replacement tables delete that page's rows from the live "read-only" version of that table, which is yet another problem).


 * All of this could probably be solved if a null-edit or purge of a single page would properly update the live table, since this would eliminate the need for recreating tables after every edit. Is this something you recognize as an issue, and might know what it could be caused by? (Under SMW, purging a page did properly update the values) SaschB (talk) 19:24, 15 January 2018 (UTC)


 * While a replacement table exists, a "live" table is read-only, as you note. (Or at least, it's intended to be that way.) Given that, there is (or is supposed to be) no way to modify the "live" tables that still have a replacement table - it has nothing to do with caching or purging. Maybe that's an error in design - I wasn't taking into account the possibility that recreating a table could become a very lengthy, complex process, as it is for you, due to (I think) your combination of complex template calls and lack of command-line access. Yaron Koren (talk) 13:31, 16 January 2018 (UTC)


 * They're not strictly read-only though, as any edits to pages storing data in them deletes rows from them (I consider deletion a write-operation). Even so, the live tables are effectively read-only even while there are no active replacement tables, as page edits aren't stored in them unless the table is recreated. SaschB (talk) 16:32, 17 January 2018 (UTC)


 * Oh... if the "live" tables are getting modified, that sounds like a bug. Yaron Koren (talk) 20:50, 17 January 2018 (UTC)


 * This was indeed a bug, and I believe it's fixed now. Yaron Koren (talk) 22:02, 21 January 2018 (UTC)

Query continuation
Hi !

I've started to use the Cargo API, but I'm facing an issue. I want to make a request that would yield massively over 500 results, but the server limits the results to the first 500 first elements. Is there a way to "continue" a request, to get the next results ?


 * For some reason, I never added an "offset" parameter to #cargo_query, or to the "cargoquery" API action. That was probably a mistake. There might be another way to do this, though: you can use the page Special:ViewData as an API (its "csv" and "json" formats basically function like an API), and it does allow an "offset=" query string parameter. Yaron Koren (talk) 02:18, 21 January 2018 (UTC)

Option to store integers without commas
Is there any way to do this, or if not could it be added? We would like to be able to use #expr: on results queried that have type Integer, and also store years (2017, 2018, etc) as type Integer and have them print without commas. Right now we just #replace all of the commas with nothing but it would be nice to be able to skip this step. --RheingoldRiver (talk) 22:44, 20 January 2018 (UTC)


 * To be clear: the values are stored without commas; the commas are added when the values are displayed. I didn't know that #expr failed if there were commas in the numbers - that's too bad. You may be able to get rid of them just by putting "CONCAT" around the field name within the #cargo_query "fields=" parameter, so it's "CONCAT(MyIntField)" instead of just "MyIntField". But as far as I know, there's no reason to store years as type Integer - I think it's always better to store them as type Date, which will get rid of the commas problem there. Yaron Koren (talk) 02:24, 21 January 2018 (UTC)


 * We have a very weird reason to store years as Integer type, which is that our 2011 and 2012 seasons in competitive League of Legends esports were called Season 1 and Season 2, and then after that it switched to "2013 Season" etc. So seasons are 1, 2, 2013, 2014, etc. That said we could probably just store as String. I'll check if CONCAT works to get around the other problem, thanks. --RheingoldRiver (talk) 22:57, 21 January 2018 (UTC)

Case Sensitivity in HOLDS
Sorry, one more question - when we move our game stats from SMW to Cargo in the future, I want to join a table that has a column listing all the different names player has played as to the table of game data and use HOLDS to get all of the games that the player played in, regardless of name at the time. Is there any way to make this query case-sensitive? We don't always have disambiguation links for players that had the same name but with different casing (for example ILLusion and illuSioN), especially when the player with unusual casing is very obscure. --RheingoldRiver (talk) 22:51, 23 January 2018 (UTC)


 * No problem, keep the questions coming. I don't think you can do that with "HOLDS" directly. Here you can see the longer query that a "HOLDS" query corresponds to - I think you'll have to go with the longer query, and put something like "LOWER" around both sides of the comparison. Yaron Koren (talk) 14:54, 24 January 2018 (UTC)
 * Sorry, that's the opposite of what I want - ILLusion and illuSioN should be distinct entries so that if I query one I only get his games and none from the other. Is that how it works by default? --RheingoldRiver (talk) 19:44, 24 January 2018 (UTC)
 * Oh... never mind. Yes, HOLDS is case-sensitive. Yaron Koren (talk) 19:53, 24 January 2018 (UTC)
 * Are you sure? Illviljan tried here and it seems to be insensitive. - https://pathofexile.gamepedia.com/User:Illviljan/cargo#Case_sensitivity_in_HOLDS --RheingoldRiver (talk) 20:42, 24 January 2018 (UTC)
 * Oops again! Somehow I forgot that the '=' operator in MySQL is case-insensitive. I just looked it up, and thankfully there's an easy solution: change "HOLDS" to "HOLDS BINARY" (or "=" to "= BINARY"). Yaron Koren (talk) 22:10, 24 January 2018 (UTC)
 * Yep that works! Thanks --RheingoldRiver (talk) 23:58, 24 January 2018 (UTC)

Feature Request: Parameter for adding styles/classes to display formats
Would it be possible to add a parameter for styling the results of a cargo query? I understand that this can be done with templates but since there are already a bunch of nice display formats available it would be great to be able to specify |class=red to make colored versions of the tables, for instance.


 * Right, there's no "class" parameter for any of the formats. You can accomplish the same thing, though, by wrapping the query in a "div" or "span" tag, like '&lt;div class="red"&gt;'. Then the CSS needs to look like "div.red table ...". Hopefully this is only a little bit of a hack. Yaron Koren (talk) 15:21, 25 January 2018 (UTC)

Attaching to multiple tables
Hi! Is there a specific reason why you can only ATTACH to exactly ONE table within a template? While having multiple instances of #cargo_attach (on different tables each), only the last one is computed and applied during data recreation... --149.243.232.3 14:44, 26 January 2018 (UTC)


 * That sounds like a bug - I'll have to look into it. Sorry about that. Yaron Koren (talk) 17:17, 26 January 2018 (UTC)

False positives from HOLDS queries
Using a query like this sometimes produces results where in fact isn't part of FocusHeroes. (For example returning ) Am I doing something wrong? SaschB (talk) 03:56, 28 January 2018 (UTC)


 * The query looks correct - something strange is happening with the storage. I would recommend recreating that table - if you still get errors like that even after recreating it, this may be a bug that still exists in Cargo. Yaron Koren (talk) 05:00, 28 January 2018 (UTC)


 * Recreating the table seems to have worked, but if whatever caused it to happen is still in effect, it's bound to reappear in the future, in which case I'll come back to this post. SaschB (talk) 12:39, 28 January 2018 (UTC)


 * That's good. Yes, please let me know if you see this problem again. Now I'm wondering about a different problem, though, which is that the table you just recreated, SummoningFocuses, has a lot of duplicates - and it shouldn't have any, given that the Cargo code now includes a check to avoid duplication. I can't think of any reason why that's happening, and I couldn't replicate the problem on my system (using some of your data). Do you know if the Cargo code on your wiki exactly matches what's in the main Cargo code (at least, at some relatively recent point in time)? Or are there any custom changes? Yaron Koren (talk) 17:03, 28 January 2018 (UTC)


 * Couldn't tell you if there are custom changes, but you have some contact at Gamepedia, right? You should be able to ask them about it. I did recreate this table in my usual fashion: recreate with replacement table -> bot-null-edit all category members calling the cargo_store template -> switch in replacement table. This usually causes duplicates for the ~50% of rows the "recreate table" command manages to pick up straight away, since they are also null-edited a minute later, causing the duplicates. SaschB (talk) 17:46, 28 January 2018 (UTC)


 * Aha - the fact that these are bot edits may have something to do with it, since bot edits seem to cause problems for Cargo, at least in some systems. Why are you doing these null bot edits? If it's to help with recreating the data, they may not be necessary. Yaron Koren (talk) 23:05, 28 January 2018 (UTC)


 * Yeah, the bot null-edits are to ensure all pages are present in the cargo table "immediately", so that the replacement table can be safely switched in without causing missing data for an unknown period of time. The bot null-edits do cause duplicates, but at least they seem to totally prevent missing items. A "Recreate using replacement table" I performed just now seems to have stopped at 58 records while the current live table contains 90 unique _pageName records. SaschB (talk) 15:50, 29 January 2018 (UTC)


 * Well, it's at 81 rows now, although it looks like the jobs have all run (you can always check at this link). I don't know why 9 of the pages didn't get their data stored. In any case, I think you can safely call that null-edit bot now if you want, without fear of leading to duplicate data. Yaron Koren (talk) 17:42, 29 January 2018 (UTC)


 * That's a useful link I didn't know about, I'll keep that in mind when tinkering/troubleshooting SaschB (talk) 20:17, 30 January 2018 (UTC)


 * Haven't done any null-editing on the pages in this table/category yet, a couple of new pages have been added, so the total should now be 91 or 92. Table is up to 89 records, 72 hours after being recreated. SaschB (talk) 18:45, 31 January 2018 (UTC)

I guess there were remaining jobs after all, then, even though that statistics API link said there were none. Well, it's great that the table is getting fully populated, although it's too bad that it's taking so long. This is the kind of thing that makes me glad that the "replacement table" option exists, though. Yaron Koren (talk) 20:05, 31 January 2018 (UTC)

CSV output only exports 100 rows
The CSV output format only outputs the first 100 rows. Is there any way to get it to output more? --pcj (talk) 18:56, 29 January 2018 (UTC)


 * You can set the number with "limit=" - hopefully you can set it high enough to include all the rows. Yaron Koren (talk) 19:12, 29 January 2018 (UTC)