Extension talk:Cargo

Delimiter for |format=template
Is it possible to set some delimiter when |format=template? It was possible in SMW, and there's some queries I'm trying to switch over to Cargo that were set up like that. --RheingoldRiver (talk) 16:18, 7 January 2018 (UTC)


 * No - I haven't seen a good reason for it yet. What do you want the delimiter to be? Yaron Koren (talk) 21:28, 7 January 2018 (UTC)


 * In some cases &amp;ensp; (although this one is generally fine to just append to the template, sometimes the template is used in multiple places and I don't always want the leading/trailing space), often &#8226;. Sometimes a break between table cells, this is the template. There were probably other cases that I used a separator in SMW with template format but these are the ones that come to mind. --RheingoldRiver (talk) 23:06, 7 January 2018 (UTC)


 * Is the current behaviour to insert a space between result item templates, or is it a newline? Or nothing? (I know I could check, but the couple of places I'm using it it seems to be a newline, but I'm not sure.) Sam Wilson 01:40, 11 January 2018 (UTC)


 * Okay, that makes sense. I checked in a "delimiter" parameter for the "template" format earlier today - hopefully it will get to you before too long. Sam - by default it doesn't put in anything. Yaron Koren (talk) 03:13, 11 January 2018 (UTC)
 * Thanks! --RheingoldRiver (talk) 12:19, 11 January 2018 (UTC)

Jobs issue
MW 1.30, Cargo 1.30.

If $wgJobRunRate > 0, records are sometimes stored to DB twice when you save a page. --StasR (talk) 21:21, 12 January 2018 (UTC)


 * Sorry, what version of Cargo are you using? Yaron Koren (talk) 22:45, 12 January 2018 (UTC)


 * Git branch REL1_30 — v. 1.4 (1798b60) 13.09.2017 --StasR (talk) 23:03, 12 January 2018 (UTC)


 * Oh - you should upgrade to version 1.5, which is when the duplicates problem was fixed. Or just use the very latest code. In general, the "REL" branches are not useful for any of my extensions, and possibly not for any non-WMF extensions. Yaron Koren (talk) 23:31, 12 January 2018 (UTC)


 * Thanks, it helped! --StasR (talk) 00:03, 13 January 2018 (UTC)

Underscore inconsistency Part II
Previously I [https://www.mediawiki.org/wiki/Extension_talk:Cargo/Archive_August_to_September_2017#Underscore_inconsistency? reported] there is inconsistency with use of underscores in queries and templates. For some reason underscores must be replaced with spaces when referring to fields in templates. (Of course, if this is changed, many templates must be redone, as they now use the workaround of space instead of underscore).

It looks like I came across another issue related to this one (unless "type" is another banned SQL keyword?). In for instance Enchantment_poe1, try to click on one of the "next 250" and such links, and an error message pops up (notice the lack of underscore): Table "Enchantment poe1" not found in Cargo database. –Pangaearocks (talk) 09:08, 13 January 2018 (UTC)


 * Wow - it's surprising that no one saw, or at least reported, that error until now. My guess is that it's in part because the "view next pages" functionality in Special:Drilldown is rarely used, since the page is meant to winnow down options to a manageable size. Anyway, I just checked in a fix for the problem; sorry about that. Yaron Koren (talk) 14:01, 15 January 2018 (UTC)

Missing tail end of record set when calling same #cargo_store template multiple times from the same article
I have certain pages for which I want to store multiple rows of data in the same table for a single page, I'm doing this by calling the same #cargo_declare/#cargo_store template multiple times from the page, with different variable values. While this does work, I have noticed that random pages often experience missing rows. Upon examining this, I've noticed that it's always the tail end of the records that go missing. Consider a page trying to store 5 rows with the values 1-5 to some table, using a template, and five separate calls like this:

When this works, five records are created like

When rows are missing, it's always an uninterrupted set from the end, leaving the table in a state like for example

Different rebuilds of the same data results in different pages missing a different amount of records (from the end), so I'm ruling out malformed input, as a template call that sometimes works does not always work. I have experimented with recreating the data into a replacement table, checking for missing rows, null-editing all pages calling the template, checking again for missing rows, null-editing again, etc. and each set of null-edits has a chance to add some of the missing rows, but not always, and not necessarily all of them (causing a page to still be missing the tail end of records, but a lower amount of them).

Do you have any insight into what might be causing this behavior? Worth mentioning is that I'm using a Scribunto module to calculate the values which are to be set, and using the Lua  for each call, which may or may not prevent the Cargo data from being properly stored on page re-cache, as pages with missing records do not get them added after the incomplete replacement table has been switched in, even through null-edits or scheduled page cache refreshes. SaschB (talk) 05:09, 14 January 2018 (UTC)


 * My guess is that the jobs are timing out before completion, possibly due in part to the complexity of these storage calls. Is your "job run rate" set to something higher than 1? If so, I would try bringing it down to 1 (or even less). Or, if you have command-line access, you could try running the cargoRecreateData.php script, instead of recreating from the web interface - that should work better. Yaron Koren (talk) 14:04, 15 January 2018 (UTC)


 * This is on Gamepedia, so the admin/config is handled by other people, I asked them about the job run rate, and it's apparently set to 1. A separate issue is that, while newly created pages (with less complex call structures) are picked up fine, any edits to pages which already have some data stored do not seem to update the live tables. This means I have to use the web interface to recreate tables (and subsequently null-edit all category members, as it were) after any edit I want immediately usable. I believe these recreates might not be strictly necessary, but in many cases I can't wait 24 hours for the tables to refresh, and null-edits or purges of pages do not update live tables - only replacement tables (additionally, null-edits to pages which store data in replacement tables delete that page's rows from the live "read-only" version of that table, which is yet another problem).


 * All of this could probably be solved if a null-edit or purge of a single page would properly update the live table, since this would eliminate the need for recreating tables after every edit. Is this something you recognize as an issue, and might know what it could be caused by? (Under SMW, purging a page did properly update the values) SaschB (talk) 19:24, 15 January 2018 (UTC)


 * While a replacement table exists, a "live" table is read-only, as you note. (Or at least, it's intended to be that way.) Given that, there is (or is supposed to be) no way to modify the "live" tables that still have a replacement table - it has nothing to do with caching or purging. Maybe that's an error in design - I wasn't taking into account the possibility that recreating a table could become a very lengthy, complex process, as it is for you, due to (I think) your combination of complex template calls and lack of command-line access. Yaron Koren (talk) 13:31, 16 January 2018 (UTC)


 * They're not strictly read-only though, as any edits to pages storing data in them deletes rows from them (I consider deletion a write-operation). Even so, the live tables are effectively read-only even while there are no active replacement tables, as page edits aren't stored in them unless the table is recreated. SaschB (talk) 16:32, 17 January 2018 (UTC)


 * Oh... if the "live" tables are getting modified, that sounds like a bug. Yaron Koren (talk) 20:50, 17 January 2018 (UTC)


 * This was indeed a bug, and I believe it's fixed now. Yaron Koren (talk) 22:02, 21 January 2018 (UTC)

Query continuation
Hi !

I've started to use the Cargo API, but I'm facing an issue. I want to make a request that would yield massively over 500 results, but the server limits the results to the first 500 first elements. Is there a way to "continue" a request, to get the next results ?


 * For some reason, I never added an "offset" parameter to #cargo_query, or to the "cargoquery" API action. That was probably a mistake. There might be another way to do this, though: you can use the page Special:ViewData as an API (its "csv" and "json" formats basically function like an API), and it does allow an "offset=" query string parameter. Yaron Koren (talk) 02:18, 21 January 2018 (UTC)

Option to store integers without commas
Is there any way to do this, or if not could it be added? We would like to be able to use #expr: on results queried that have type Integer, and also store years (2017, 2018, etc) as type Integer and have them print without commas. Right now we just #replace all of the commas with nothing but it would be nice to be able to skip this step. --RheingoldRiver (talk) 22:44, 20 January 2018 (UTC)


 * To be clear: the values are stored without commas; the commas are added when the values are displayed. I didn't know that #expr failed if there were commas in the numbers - that's too bad. You may be able to get rid of them just by putting "CONCAT" around the field name within the #cargo_query "fields=" parameter, so it's "CONCAT(MyIntField)" instead of just "MyIntField". But as far as I know, there's no reason to store years as type Integer - I think it's always better to store them as type Date, which will get rid of the commas problem there. Yaron Koren (talk) 02:24, 21 January 2018 (UTC)


 * We have a very weird reason to store years as Integer type, which is that our 2011 and 2012 seasons in competitive League of Legends esports were called Season 1 and Season 2, and then after that it switched to "2013 Season" etc. So seasons are 1, 2, 2013, 2014, etc. That said we could probably just store as String. I'll check if CONCAT works to get around the other problem, thanks. --RheingoldRiver (talk) 22:57, 21 January 2018 (UTC)

Case Sensitivity in HOLDS
Sorry, one more question - when we move our game stats from SMW to Cargo in the future, I want to join a table that has a column listing all the different names player has played as to the table of game data and use HOLDS to get all of the games that the player played in, regardless of name at the time. Is there any way to make this query case-sensitive? We don't always have disambiguation links for players that had the same name but with different casing (for example ILLusion and illuSioN), especially when the player with unusual casing is very obscure. --RheingoldRiver (talk) 22:51, 23 January 2018 (UTC)


 * No problem, keep the questions coming. I don't think you can do that with "HOLDS" directly. Here you can see the longer query that a "HOLDS" query corresponds to - I think you'll have to go with the longer query, and put something like "LOWER" around both sides of the comparison. Yaron Koren (talk) 14:54, 24 January 2018 (UTC)
 * Sorry, that's the opposite of what I want - ILLusion and illuSioN should be distinct entries so that if I query one I only get his games and none from the other. Is that how it works by default? --RheingoldRiver (talk) 19:44, 24 January 2018 (UTC)
 * Oh... never mind. Yes, HOLDS is case-sensitive. Yaron Koren (talk) 19:53, 24 January 2018 (UTC)
 * Are you sure? Illviljan tried here and it seems to be insensitive. - https://pathofexile.gamepedia.com/User:Illviljan/cargo#Case_sensitivity_in_HOLDS --RheingoldRiver (talk) 20:42, 24 January 2018 (UTC)
 * Oops again! Somehow I forgot that the '=' operator in MySQL is case-insensitive. I just looked it up, and thankfully there's an easy solution: change "HOLDS" to "HOLDS BINARY" (or "=" to "= BINARY"). Yaron Koren (talk) 22:10, 24 January 2018 (UTC)
 * Yep that works! Thanks --RheingoldRiver (talk) 23:58, 24 January 2018 (UTC)

Feature Request: Parameter for adding styles/classes to display formats
Would it be possible to add a parameter for styling the results of a cargo query? I understand that this can be done with templates but since there are already a bunch of nice display formats available it would be great to be able to specify |class=red to make colored versions of the tables, for instance.


 * Right, there's no "class" parameter for any of the formats. You can accomplish the same thing, though, by wrapping the query in a "div" or "span" tag, like '&lt;div class="red"&gt;'. Then the CSS needs to look like "div.red table ...". Hopefully this is only a little bit of a hack. Yaron Koren (talk) 15:21, 25 January 2018 (UTC)

Attaching to multiple tables
Hi! Is there a specific reason why you can only ATTACH to exactly ONE table within a template? While having multiple instances of #cargo_attach (on different tables each), only the last one is computed and applied during data recreation... --149.243.232.3 14:44, 26 January 2018 (UTC)


 * That sounds like a bug - I'll have to look into it. Sorry about that. Yaron Koren (talk) 17:17, 26 January 2018 (UTC)

False positives from HOLDS queries
Using a query like this sometimes produces results where in fact isn't part of FocusHeroes. (For example returning ) Am I doing something wrong? SaschB (talk) 03:56, 28 January 2018 (UTC)


 * The query looks correct - something strange is happening with the storage. I would recommend recreating that table - if you still get errors like that even after recreating it, this may be a bug that still exists in Cargo. Yaron Koren (talk) 05:00, 28 January 2018 (UTC)


 * Recreating the table seems to have worked, but if whatever caused it to happen is still in effect, it's bound to reappear in the future, in which case I'll come back to this post. SaschB (talk) 12:39, 28 January 2018 (UTC)


 * That's good. Yes, please let me know if you see this problem again. Now I'm wondering about a different problem, though, which is that the table you just recreated, SummoningFocuses, has a lot of duplicates - and it shouldn't have any, given that the Cargo code now includes a check to avoid duplication. I can't think of any reason why that's happening, and I couldn't replicate the problem on my system (using some of your data). Do you know if the Cargo code on your wiki exactly matches what's in the main Cargo code (at least, at some relatively recent point in time)? Or are there any custom changes? Yaron Koren (talk) 17:03, 28 January 2018 (UTC)


 * Couldn't tell you if there are custom changes, but you have some contact at Gamepedia, right? You should be able to ask them about it. I did recreate this table in my usual fashion: recreate with replacement table -> bot-null-edit all category members calling the cargo_store template -> switch in replacement table. This usually causes duplicates for the ~50% of rows the "recreate table" command manages to pick up straight away, since they are also null-edited a minute later, causing the duplicates. SaschB (talk) 17:46, 28 January 2018 (UTC)


 * Aha - the fact that these are bot edits may have something to do with it, since bot edits seem to cause problems for Cargo, at least in some systems. Why are you doing these null bot edits? If it's to help with recreating the data, they may not be necessary. Yaron Koren (talk) 23:05, 28 January 2018 (UTC)


 * Yeah, the bot null-edits are to ensure all pages are present in the cargo table "immediately", so that the replacement table can be safely switched in without causing missing data for an unknown period of time. The bot null-edits do cause duplicates, but at least they seem to totally prevent missing items. A "Recreate using replacement table" I performed just now seems to have stopped at 58 records while the current live table contains 90 unique _pageName records. SaschB (talk) 15:50, 29 January 2018 (UTC)


 * Well, it's at 81 rows now, although it looks like the jobs have all run (you can always check at this link). I don't know why 9 of the pages didn't get their data stored. In any case, I think you can safely call that null-edit bot now if you want, without fear of leading to duplicate data. Yaron Koren (talk) 17:42, 29 January 2018 (UTC)


 * That's a useful link I didn't know about, I'll keep that in mind when tinkering/troubleshooting SaschB (talk) 20:17, 30 January 2018 (UTC)


 * Haven't done any null-editing on the pages in this table/category yet, a couple of new pages have been added, so the total should now be 91 or 92. Table is up to 89 records, 72 hours after being recreated. SaschB (talk) 18:45, 31 January 2018 (UTC)

I guess there were remaining jobs after all, then, even though that statistics API link said there were none. Well, it's great that the table is getting fully populated, although it's too bad that it's taking so long. This is the kind of thing that makes me glad that the "replacement table" option exists, though. Yaron Koren (talk) 20:05, 31 January 2018 (UTC)


 * Most of the time, when adding new data, we can't wait several days, as information is often announced/made available mere hours before being needed (by users), so I guess I'll keep batch-null-editing to help it along, even if it causes duplicates. SaschB (talk) 08:28, 1 February 2018 (UTC)


 * Alright. In any case, you can always create a replacement table the "right way" afterwards, as you did here. Yaron Koren (talk) 15:41, 1 February 2018 (UTC)


 * I'll probably skip that step, as there's no simple way of knowing when the replacement table is finished (the one in this example wasn't finished after 72 hours, and I don't know when/if it would have been finished without manual intervention). I'd rather have a complete table full of duplicates than an incomplete table without duplicates. SaschB (talk) 17:21, 1 February 2018 (UTC)


 * A replacement table lets you have both, in a sense, until it gets filled completely and can be switched in... Yaron Koren (talk) 18:24, 1 February 2018 (UTC)


 * Well, yes, I see your point, but with new data being added with our frequency, we often have to rebuild the tables once every two weeks - so keeping track of a table for one week in order for it to lack duplicates (but work the same) for one week doesn't make sense to me. I know we're not supposed to have to recreate the tables when data is added/edited, and some of these issues have been solved, but I'm not sure if all of them have been (yet). SaschB (talk) 21:57, 1 February 2018 (UTC)


 * Yeah, on that note - sorry to belabor the discussion, but I didn't understand what you meant about new data being added. Are you just talking about more data, without any changes to the data structure? If so, what's the benefit of recreating the table? Yaron Koren (talk) 22:08, 1 February 2018 (UTC)


 * I am just talking about new data (INSERT or UPDATE), but with previous issues of pages not updating the live cargo tables correctly/immediately upon page edits, my workaround to ensure all new data was stored was the previous procedure of recreate -> replacement -> null-edit -> switch in. As of recently, it seems the storage is a bit more robust, so hopefully I won't have to in the future. If I have to, however, that's what I'll do. SaschB (talk) 12:12, 3 February 2018 (UTC)

It happened again: shows  as a result, even though the value isn't there. Also /+ and / SaschB (talk) 16:59, 3 February 2018 (UTC)


 * That's too bad that this storage problem is still there in the latest code, but of course it's good to know that it's there. I looked into this, and I think I figured out what was causing it. I just checked in a fix - hopefully it'll get to you soon, and hopefully it really does fix the problem. Yaron Koren (talk) 05:26, 5 February 2018 (UTC)

CSV output only exports 100 rows
The CSV output format only outputs the first 100 rows. Is there any way to get it to output more? --pcj (talk) 18:56, 29 January 2018 (UTC)


 * You can set the number with "limit=" - hopefully you can set it high enough to include all the rows. Yaron Koren (talk) 19:12, 29 January 2018 (UTC)

Error with Special:CargoTables after deleting attached templates
Hey there. I recently deleted a bunch of templates that were attached to a table and subsequently visiting the Special:CargoTables page gives me this: [94d2634e1606751386c8fc50] /a3/index.php?title=Special:CargoTables TypeError from line 145 of /home/sites/1b/e/e73edcb322/public_html/a3/includes/linker/LinkRenderer.php: Argument 1 passed to MediaWiki\Linker\LinkRenderer::makeLink must implement interface MediaWiki\Linker\LinkTarget, null given, called in /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/includes/CargoUtils.php on line 971

Backtrace:


 * 1) 0 /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/includes/CargoUtils.php(971): MediaWiki\Linker\LinkRenderer->makeLink(NULL, NULL, array, array, array)
 * 2) 1 /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/specials/CargoTables.php(298): CargoUtils::makeLink(MediaWiki\Linker\LinkRenderer, NULL)
 * 3) 2 /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/specials/CargoTables.php(353): CargoTables->tableTemplatesText(string, array, array)
 * 4) 3 /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/specials/CargoTables.php(28): CargoTables->displayListOfTables
 * 5) 4 /home/sites/1b/e/e73edcb322/public_html/a3/includes/specialpage/SpecialPage.php(522): CargoTables->execute(NULL)
 * 6) 5 /home/sites/1b/e/e73edcb322/public_html/a3/includes/specialpage/SpecialPageFactory.php(576): SpecialPage->run(NULL)
 * 7) 6 /home/sites/1b/e/e73edcb322/public_html/a3/includes/MediaWiki.php(283): SpecialPageFactory::executePath(Title, RequestContext)
 * 8) 7 /home/sites/1b/e/e73edcb322/public_html/a3/includes/MediaWiki.php(851): MediaWiki->performRequest
 * 9) 8 /home/sites/1b/e/e73edcb322/public_html/a3/includes/MediaWiki.php(512): MediaWiki->main
 * 10) 9 /home/sites/1b/e/e73edcb322/public_html/a3/index.php(43): MediaWiki->run
 * 11) 10 {main}

I'm not sure if the deletion of the templates was actually the cause but I'm at a loss re: how to fix it. Viewing the table pages themselves seems to be fine and nothing else is borked as far as I can tell.

Rurikawa (talk) 13:46, 1 February 2018 (UTC)


 * Very odd - looking through the code, and your backtrace, it looks like the problem is that one of the "attaching" templates that you deleted did not get its "attached to a table" page property deleted from the page_props table - so the Cargo code sees the property, goes to display a link to that template, then crashes because the template doesn't exist. I didn't know that could happen. Anyway, I just checked in what is hopefully a fix, so that the code checks whether a page is null before trying to link to it. Yaron Koren (talk) 15:29, 1 February 2018 (UTC)


 * That fixed it! Thank you for the quick response. Rurikawa (talk) 19:14, 1 February 2018 (UTC)

Data not updating.
Hello,

I have installed Cargo, and added a couple of templates (http://crewsgenealogy.com/wiki/index.php?title=Template:Infobox_person). I belive I have added the code correctly, cargo doens't produce any errors when I set up the tables, but it doesn't seem to be saving any of the information. I have tried the maintenance script cargoRecreateData.php which ran without errors, but still nothing. I have several pages with the template, so I would think something would be saved. If it makes any difference it seems to be stuck with the message "Note: One or more of these tables are currently being populated, via the job queue." even after I ran the script.

Any suggestions?


 * I see the #cargo_declare which defines the table itself, but I can't see a #cargo_store, which defines which data to store in the table. Might be as simple as that? SaschB (talk) 12:18, 3 February 2018 (UTC)
 * Example:

Automatic |no html added when query used in page forms?
I have this query in a form, and the query works but it seems like it's adding a |no html to it. This is how the same query looks embedded in a page. Is this a Cargo issue, and if so any way to fix it? Thanks! --RheingoldRiver (talk) 17:35, 3 February 2018 (UTC)


 * There are a number of complicated aspects to that query you have - maybe the most relevant one is that the query is itself calling a template, in order to make the table display more customized. If it's not too much work, could you try testing to see if a simple table display of the query, instead of that template display, results in that same error? If not, there may be some workarounds (though ideally your current query should work as well). Yaron Koren (talk) 14:36, 4 February 2018 (UTC)


 * That table actually has no fields that directly have markup stored into them, but I made this instead and the markup is showing properly. We also have a number of query forms in SMW with relatively complicated "line" templates such as this one that I was hoping to be able to switch over to Cargo in the next couple months. --RheingoldRiver (talk) 21:56, 4 February 2018 (UTC)


 * Something that may be related is that on occasion we will see similar behavior on an actual wikipage, i.e. it looks like this after saving. As soon as another user opens the page, it displays normally [edit - actually this may not always be the case, after a comment from someone on another Gamepedia wiki that they had the issue persist on another user viewing the page]. This is the page that screencap is from, though it can happen on a number of pages, and this problem isn't unique to the LoL wiki, a bunch of Gamepedia wikis have experienced the same thing. In the case of that page I changed "Smeb" to "Smeb2" and back, repeatedly. Whether the bug happens or not is relatively inconsistent, but I was playing around with user rights on my bot account, and it seemed to happen more often when I had Admin than when I didn't. --RheingoldRiver (talk) 22:49, 4 February 2018 (UTC)


 * It sounds like you're saying that the Special:RunQuery bug only happens when the "template" format is used in the query. Which is still bad, but not as bad as if happened for every query. Until this bug is actually fixed, I can think of two possible workarounds: (1) switch to the "table" format, and replicate as much of the logic as possible using MySQL functions, and (2) do the whole query display in a Lua module. (I suppose there's a third option, which is to use SMW for all these queries, but I'm leaving that aside for now. :) ) Would either of these be an option?
 * As for the second issue - this is the first I've heard of this. My only thought is that this is related to whether the page is being cached or not; although I can't see the error on that page in (I think) either state. If you can figure out some way to reproduce it consistently, that would be very helpful - though that's probably obvious. Yaron Koren (talk) 01:20, 5 February 2018 (UTC)


 * Yeah that should probably be an option for this template, though we definitely have some that I wanted to convert to Cargo where it wouldn't be possible. I'll let you know if I see any (Cargo-related) bugs with it when trying to change it to just a table query.


 * I tried for about an hour to find a combination of user rights that led to the problem happening consistently, and sadly I couldn't. This error has been around for a while, but it's always been super intermittent; and it's never happened when I'm logged in as myself (in Firefox fwiw, while my bot is in Chrome). And since our last issue with saving pages was a conflict with another Gamepedia extension it's also possible that that's what this is. AFAIK this is also only happening when |format=template (maybe suggesting the two are related?), but nearly all of my queries are |format=template so that's not necessarily the case. I'll ask around and see if anyone had the same issue occur with another format. --RheingoldRiver (talk) 07:57, 5 February 2018 (UTC)


 * Oh also, let me know if you want Editor on our wiki ever since we have FlaggedRevs enabled. --RheingoldRiver (talk) 07:59, 5 February 2018 (UTC)