Extension talk:Cargo

Problems with list fields
Hi

I'm using cargo for some time now and I accumulated some problems with list fields:
 * 1) Recreating cargo tables produces php notices Undefined index in Cargo/parserfunctions/CargoStore.php on line 293 and 315 (not really a problem, since error reporting is only activated on my dev system where I noticed this)
 * 2) Having a field definition in Semantic Forms with cargo table and cargo field set to point to a list field produces a "Table does not exist" when accessing the form for the first time (aka: the table is emtpy). One has to remove cargo table and cargo field, create at least one entry and then they can be added again. Methinks that has to do something with type ahead suggestions (propagation of the autocomplete field).
 * 3) Most anoying: I have some list fields of type page whose *__full field exceeds the 300 characters leaving me with one of two options: I can cargo query normally for the field in the table, knowing very well that the result will be cut [0] or I can create a join and then loose the linking to the pages having to reintroduce it manully in the query [1]. Note: size in the #cargo_store definition does neither apply to page fields in generell nor to the *__full field of the list. Furthermore I have a bot gathering some data using a generic function which has no way of knowing, whether it queries a list field or a normal field making it impossible to use the join workaround.

I would appreciate any input on this...

[0] Normal query

[1] Join query

On system (amongst others): MW 1.23.14 Semantic Forms 3.7 Cargo 1.0.1

Tobias (talk) 12:18, 1 July 2016 (UTC)
 * Just noticed, calling index.php?title=Group:ToManyMembers&action=pagevalues yields the same unwanted trimmed result, indicating this also accesses the *__full field. :(
 * Tobias (talk) 10:00, 5 July 2016 (UTC)


 * Hi, sorry for the delay. For #2 - what version of SF are you using? (It's not 3.7, because that hasn't come out yet.) For #3, I just checked in a small change that I think fixes this issue - I didn't fully test it. Yaron Koren (talk) 04:38, 7 July 2016 (UTC)


 * I'm sorry, you are right. I'm on SF 3.6 (0dae7d5) 16:28, 4. Mai 2016. Thanks for responding. I'll try your fix asap. Tobias (talk) 05:56, 7 July 2016 (UTC)

Not getting anything but page name
I've got queries that were working before I updated to mediawiki 1.27 and the latest version of Cargo via git, but now they only return the page name. None of the other fields in the query return anything, but when I go the page and look at the page values the values are there, and stay there after a purge.

--Cody3647 (talk) 17:00, 14 July 2016 (UTC)
 * Not working
 * Working


 * Very strange! I haven't seen this before. It looks like the issue is with the display of any field that has an alias - whether it's "Page" instead of "_pageName" or "npc permission" instead of "npc_permission", etc. Beyond that, I don't know. Could you try to narrow down whether it's due to the difference in the MediaWiki vs. the Cargo versions? Yaron Koren (talk) 18:08, 14 July 2016 (UTC)


 * I've updated Cargo on the mediawiki 1.26 installation via git and everything seems to be working, though the version listed in history hasn't updated. --Cody3647 (talk) 18:54, 14 July 2016 (UTC)


 * Okay - it must be the MediaWiki version, then. My guess is that the issue is with extension.json, which is only called for MW 1.27 and higher. Could you try commenting out the extension.json stuff, at the top of Cargo.php, in the non-working wiki, and see if that fixes the problem? Yaron Koren (talk) 20:28, 14 July 2016 (UTC)


 * I changed the version compare to 1.28, and the 1.27 install is still not working. --Cody3647 (talk) 22:11, 14 July 2016 (UTC)


 * Oh well. I guess there are three options: you can try to figure out the problem yourself, you can give me login access on your server and I can try to figure it out, or I can upgrade my own wiki to 1.27 and see what happens. (I do plan to upgrade my wiki anyway, but I didn't have any specific timeframe for that.) Yaron Koren (talk) 00:14, 15 July 2016 (UTC)


 * Looking at the sql query that is being passed in the $res object in CargoSQLQuery::run, its getting extra ` around aliases.  .  I commented out   in CargoGalleryFormat::display fixes it, but instead of adding it to the stylesheet like a normal wiki gallery does, it adds it in a style block in the head.  Only a problem because it then overwrites some changes I'd made to the gallery css through the skin. --Cody3647 (talk) 00:09, 16 July 2016 (UTC)


 * Did you really report this problem before? I don't remember hearing about it before. But yes, you're right - the 'gallery' format was only displaying correctly when there was no page caching. I just checked in a change that disables the page cache when the gallery format is called, which is my go-to fix for cases like these. Hopefully that solves this problem. Yaron Koren (talk) 01:30, 18 July 2016 (UTC)


 * I took a look again, and I'm not sure the cache needs to be disabled. The following two lines, before I was using only one, seem to work and adds it into the external stylesheet loaded on page load rather than the dynamic style block added after page load, so no jumping from list to gallery anymore whether just refreshed or purged.  Also, just FYI: Missing Gallery Module for Packed-hover gallery
 * --Cody3647 (talk) 02:27, 18 July 2016 (UTC)


 * Ah! That sounds familiar; although I didn't think it was the same issue. Anyway - isn't my solution superior, in that it doesn't overwrite any CSS? Or does it not really matter? Or does my solution not work for you? Yaron Koren (talk) 02:33, 18 July 2016 (UTC)


 * I don't think it overwrites anything, just adds the module into the stylesheet like it would if you just used the gallery tag. I'm not opposed to turning off the cache, I just know that its generally useful?  I know I have a number of pages with multiple queries, several galleries and others, and just figure that its better to have those cached rather than not.  --Cody3647 (talk) 02:37, 18 July 2016 (UTC)


 * Alright - if there's no overwriting, then yours is indeed the better solution. I just checked it in. Thanks! Yaron Koren (talk) 14:52, 18 July 2016 (UTC)

List of URL data type leads to SQL error
I have a template with a Cargo declaration of the form:

When trying to view the resulting table, I get an SQL syntax error (1064) reported in CargoSQLQuery::run. The reported query is as follows: SELECT `_pageName` AS `Page`,CONCAT('[', Field1, ' URL]') AS `Field1`,CONCAT('[', Field2, ' URL]')__full AS `Field2`,`Field3` AS `Field3`  FROM `cargo__Tests`    ORDER BY `_pageName` LIMIT 100 The problem seems to be with the concatenation of the parts of Field2 call. This is with Cargo 1.0.1 (5ed9912) on MediaWiki 1.27.0 and MariaDB 10.0.25. -- Paul T (talk) 20:28, 1 August 2016 (UTC)


 * Sorry for the delay. Yes, this was a bug - I just checked in a fix for it. Yaron Koren (talk) 17:00, 4 August 2016 (UTC)


 * Thank you. That works! Paul T (talk) 08:52, 5 August 2016 (UTC)

Undefined constants when specifying additional fields for _pageData table
Following the instructions in on the "Storing page data" part of the Storing data page, I have added $wgCargoPageDataColumns[] = CARGO_STORE_CREATION_DATE; to LocalSettings.php, below Cargo's require_once call. When trying to create the table, I get the following: $ php setCargoPageData.php PHP Notice: Use of undefined constant CARGO_STORE_CREATION_DATE - assumed 'CARGO_STORE_CREATION_DATE' in /var/www/html/w/LocalSettings.php on line NNN The same happens for the other constants. The _pageData table is successfully created, but it doesn't include any of the optional fields.

Have I missed some subtlety, or is this a bug? Many thanks! Paul T (talk) 10:52, 10 August 2016 (UTC)


 * Ah, it looks like the handling for the page data constants with extension.json - which is used with MediaWiki 1.27 and higher - doesn't work. I just changed it to a setup that's probably better in all cases. The constant names have changed - you should replace "CARGO_STORE_" with "CargoPageData::" in each of the names, so it would look like:

$wgCargoPageDataColumns[] = CargoPageData::CREATION_DATE;
 * If you get the latest code, please try this out and let me know if it works for you. Yaron Koren (talk) 13:52, 10 August 2016 (UTC)
 * Thanks. With the latest code and that change, I'm getting this: PHP Fatal error: Class 'CargoPageData' not found in /var/www/html/w/LocalSettings.php on line NNN
 * Paul T (talk) 15:16, 10 August 2016 (UTC)
 * That's not good. Are you calling Cargo in LocalSettings.php with wfLoadExtension, by any chance? Also, what version of MediaWiki are you running? Yaron Koren (talk) 02:01, 11 August 2016 (UTC)
 * No, it's: require_once( "$IP/extensions/Cargo/Cargo.php" );
 * ... which has been working fine in all other respects. The attempt to set $wgCargoPageDataColumns[] is immediately below that. This is Cargo 1.0.1 (e0dbafe) with MediaWiki 1.27.0 on PHP 5.6.24 on Debian. Paul T (talk) 08:29, 11 August 2016 (UTC)


 * Okay, I finally tested this out directly. It turns out that constants unfortunately can't be used with extension.json at all; so I had to change the settings to be strings instead. So now the relevant call is instead: $wgCargoPageDataColumns[] = 'creationDate'; ...and the other possible values are 'modificationDate', 'creator', 'fullText', 'categories' and 'numRevisions'. Sorry about all the problems; hopefully this works for you now. Yaron Koren (talk) 18:00, 12 August 2016 (UTC)


 * That works -- thank you. Paul T (talk) 18:43, 15 August 2016 (UTC)

Hiding fields from drilldown but not from table view
I have a minor feature suggestion. The "hidden" parameter in a field definition allows a field to be hidden from drilldown. It would be good to be able to do that without also hiding it from the Special:CargoTables view. The particular use case I have in mind is for fields containing filenames: it's good to be able to see that they're there in the table view, but they're of no use in a drilldown (and just add clutter) because they're unique. This could be done by adding another keyword, or perhaps it might just make sense to exclude file fields from drilldowns by default? Paul T (talk) 19:02, 15 August 2016 (UTC)


 * Good point - the latter makes the most sense. I just added "File" to the list of types not handled by Special:Drilldown. Yaron Koren (talk) 15:52, 16 August 2016 (UTC)


 * Thank you - that's much neater! Paul T (talk) 21:57, 16 August 2016 (UTC)

Duplicated Records
I'm having a problem with mysteriously duplicated records in a couple of tables. I've read in the archives that a few people probably had the same problem. It might be due to me having used the recreate table functionality, but I thought I did it for only one of the two tables. What's the current wisdom to eliminate or minimize the issue? -- manu3d (talk) 22:02, 15 August 2016 (UTC)


 * The most foolproof way seems to be to call the "cargoRecreateData.php" script - once you're sure that there are no more "recreate data" jobs still waiting to be run. (You can call MediaWiki's own runJobs.php script to be sure.) Yaron Koren (talk) 15:54, 16 August 2016 (UTC)


 * We are having a similar problem: ending up with duplicated records after editing pages using forms. This particularly seems to happen when there are list fields involved. I think I may have tracked down the cause. When making edits that cause these duplicates, these errors appear in Apache's error.log:

PHP Notice: unserialize: Error at offset 193 of 200 bytes in /var/www/html/w/extensions/Cargo/Cargo.hooks.php on line 128, referer: [site url]/w/index.php?title=[page title]]&action=formedit PHP Warning: Invalid argument supplied for foreach in /var/www/html/w/extensions/Cargo/Cargo.hooks.php on line 129, referer: [site url]/w/index.php?title=[page title]&action=formedit
 * These are in the deletePageFromSystem function that is called as the first step in the onPageContentSaveComplete hook, so I think they are causing (some of) the original records for the page not to be deleted, such that the new version then becomes a duplicate. The unserialize error seems to be caused by the contents of the field_tables field in the cargo_tables table being truncated to 200 characters by the database schema, so I suppose this problem will only show up on tables with several list fields. Paul T (talk) 07:03, 17 August 2016 (UTC)


 * Changing the type of field_tables from varchar(200) to text (and recreating everything) seems to fix the problem. The table on our site that was accumulating duplicates now has a field_tables value that is 337 characters long. This issue seems to be specific to MySQL/MariaDB, as /sql/Cargo.pg.sql already used a text type with unlimited length. Paul T (talk) 10:48, 17 August 2016 (UTC)

Page moves do not seem to work
I noticed that moving a page (i.e. changing its title) did not seem to lead to the Cargo tables being updated: the table still showed the old title. I can see that it is meant to be handled, using the onTitleMoveComplete hook. I think the problem may be related to this, which appeared in Apache's error.log: PHP Warning: Parameter 3 to CargoHooks::onTitleMoveComplete expected to be a reference, value given in /var/www/html/w/includes/Hooks.php on line 195, referer: [site url]/wiki/Special:MovePage/[original page name] This is with Cargo 1.1 (3e1b9f3) on MediaWiki 1.27.0. Paul T (talk) 06:23, 17 August 2016 (UTC)


 * Yikes! It looks like you're seeing this issue, which is a bug in either MediaWiki or HHVM. Are you using HHVM/HipHop by any chance? If not, what version of PHP are you using? Yaron Koren (talk) 12:29, 17 August 2016 (UTC)


 * No, it's nothing exotic: PHP 5.6.24-0+deb8u1 (apache2handler), which is the default for Debian. From the command line, it looks like this:

$ php -v PHP 5.6.24-0+deb8u1 (cli) (built: Jul 26 2016 08:17:07) Copyright (c) 1997-2016 The PHP Group Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
 * Paul T (talk) 13:28, 17 August 2016 (UTC)


 * Alright. This seems to be a bug in MW 1.27; hopefully it'll get fixed soon. Yaron Koren (talk) 18:24, 23 August 2016 (UTC)


 * It looks like this bug might be with us for a while. I just checked in a workaround to it in the Cargo code, that seems like it will work with all MediaWiki versions. Thankfully it was a pretty easy fix. Yaron Koren (talk) 18:47, 13 September 2016 (UTC)

Unnecessary error when a field value in a where clause contains the name of a virtual field
When one of the values passed in a where clause of cargo_query call happens to contain the name of a list field, an error is thrown incorrectly.

Here's a minimal test case that shows the problem. If you define a table as follows:

and run this query:

It results in this error: Error: operator for the virtual field 'Test.Field2' must be 'HOLDS', 'HOLDS NOT', 'HOLDS LIKE' or 'HOLDS NOT LIKE'.

Obviously, this is a contrived example. It causes a problem in practice, however, when a page title that's used as part of a query happens to contain the name of a list field.

This seems to be caused by the preg_match call on line 676 of CargoSQLQuery.php matching on the entire clause, rather than just the field names. Disabling that error by commenting out the test seems to make the query work. However, clearly there must be a better fix. This is with Cargo 1.0.1 (3f84330) or 1.1 (3e1b9f3). Paul T (talk) 13:58, 23 August 2016 (UTC)


 * That does seem like a bug... until there's a real solution, you could probably get around it with a hack like replacing that string with something like "CONCAT( 'A string that contains the text Fie', 'ld2 somewhere within it ')". Yaron Koren (talk) 18:10, 23 August 2016 (UTC)


 * Thanks. Alas, that doesn't work in our case, as the where clause in question is actually of the form " Field HOLDS ". The other workaround would be to change the field names to be less likely to crop up in the data, but for now I've just commented out that line in CargoSQLQuery.php, and we'll live without that error message. Paul T (talk) 18:38, 23 August 2016 (UTC)

Updating a field
Hi, I'd like to update a single field in a database table but when I do this it creates a complete table row with just the updated fields in it. How can I just update a single entry?


 * Sorry, I don't understand what exactly you want to do, or what happened when you tried it. Yaron Koren (talk) 22:09, 5 September 2016 (UTC)

Apologies, what I'd like to do is similar to this,

If I try to do it with;

Then it adds a second row into the table rather than updating the existing field on the existing row.


 * I'm still not quite sure I understand, but you should generally have only one #cargo_declare, and one #cargo_store, call within a template, except for special cases. Do you have more than one? If so, you should combine them into one, then recreate the table. Yaron Koren (talk) 15:30, 6 September 2016 (UTC)

Missing filter breadcrumb in drilldown when selecting from a combobox
We have noticed what looks like a bug in the Special:Drilldown. Normally, there is a trail of breadcrumbs at the top of the page, giving the fields and values on which you have filtered, each with a cross icon (filter-x.png) that you can click on to remove that filter.

However, this doesn't seem to work when you filter on a field that has a large number of values, triggering the use of a combobox, rather than an in-line list of values. The relevant fieldname still appears in the header, but no value and no cross icon. Looking at the URL, I can see that the problem arises when the filter filter appears in the form _search_FieldName[0]=Value, rather than simply FieldName=Value or FieldName[0]=Value.

This is problematic because (a) it's not clear to the user what they've already filtered on, and (b) there's no easy way for them to remove the filter. Increasing $wgCargoDrilldownMinValuesForComboBox to avoid the use of comboboxes is a temporary workaround, but will hurt usability, as some of the lists are very long.

This is on Cargo 1.1 (3e1b9f3). Paul T (talk) 16:41, 7 September 2016 (UTC)


 * Oops! I believe that's due to some typos entered into the code three months ago. Sorry about that. I just checked in what I think is a fix. Yaron Koren (talk) 18:06, 7 September 2016 (UTC)


 * Thank you -- yes, that fixes it. Paul T (talk) 18:48, 7 September 2016 (UTC)

Special:ViewData returning blank tables
The Special:ViewData page does not seem to be working for us. It seems to run the query correctly, in that it throws appropriate SQL errors if you get the field names wrong, and the numbers in the header are plausible, e.g. "Showing below up to 68 results in range #9 to #76." However, it doesn't actually display any data. If you ask for table format, you get a table with the appropriate rows and columns, but with all value cells blank. For other formats, you get a string of commas, or empty bullet points. This occurs both for a call resulting from the "More..." link below a properly working cargo_query call, and for simple, manually constructed queries on a test table. This is on Cargo versions 1.1 (f3599cf) or (3e1b9f3), on MediaWiki 1.27.0 on PHP 5.6.24 (on Apache) and MariaDB 10.0.26 or 5.5.47. Paul T (talk) 19:23, 7 September 2016 (UTC)


 * That's very strange. Does this always happen for one table, or can you reproduce the problem with any table? Yaron Koren (talk) 22:02, 7 September 2016 (UTC)


 * It seems to be any table, including a trivial test table. Paul T (talk) 09:19, 8 September 2016 (UTC)


 * I have no idea. This seems unlikely, but maybe check the HTML to see if the values are showing up but are somehow being rendered invisible? Yaron Koren (talk) 13:46, 8 September 2016 (UTC)


 * To continue publicly an offline discussion: this seems to be a compatibility problem with MediaWiki 1.27: From that version onwards, MediaWiki automatically quotes fieldnames; Cargo's quoting (which was necessary with earlier versions) leads to them being double-quoted and failing to match the expected names when formatting the result. Cargo code for most other queries already checks the MediaWiki version and only adds quotes on older versions, but the Special:ViewData page uses separate code that didn't have this test. I've sent Yaron a short patch that fixes it for me. Paul T (talk) 10:35, 12 September 2016 (UTC)


 * To confirm, the current git version of the Cargo code contains a patch from Yaron that fixes this. Thank you! Paul T (talk) 14:32, 15 September 2016 (UTC)


 * Thank you also! Yaron Koren (talk) 18:16, 15 September 2016 (UTC)

Change formatting of cells in dynamic table
I'd like the vertical align of these to be "top" rather than "middle". What should I change to make this work? Thanks in advance. Jonathan3 (talk) 19:46, 23 September 2016 (UTC)


 * Just answered my own question... add .cargoDynamicTable td {vertical-align:top !important} to MediaWiki:Common.css :-) Jonathan3 (talk) 19:54, 23 September 2016 (UTC)

Error: the SQL function "SUBSTRING_INDEX" is not allowed.
This would be useful for me. I've tried fields=SUBSTRING_INDEX(Description, '==', 1) to take the first part of the Description, up to the first heading, but get the error above. Is there a way round this?

I have to confess I'm still using version 0.10 (c930560). Thanks in advance. Jonathan3 (talk) 20:08, 23 September 2016 (UTC)


 * You just need to add it to the array $wgCargoAllowedSQLFunctions - see here. Yaron Koren (talk) 20:58, 23 September 2016 (UTC)


 * I've added $wgCargoAllowedSQLFunctions[] = 'SUBSTRING_INDEX'; in the line after require_once( "$IP/extensions/Cargo/Cargo.php" ); but still get the error message. I get a similar message for LEFT even if I add it similarly. What am I doing wrong? Thanks Jonathan3 (talk) 13:37, 26 September 2016 (UTC)


 * I don't know, that's very strange... maybe try upgrading Cargo? Also, what version of MediaWiki are you using? Yaron Koren (talk) 14:14, 26 September 2016 (UTC)


 * I'm afraid I'm still using MW 1.25.3. I'll upgrade it as soon as I can, and will update this page to let you know if the problem disappears. Thanks. Jonathan3 (talk) 14:33, 26 September 2016 (UTC)

Changing column names
I have a table with a "Description" column, but would like to (a) change "Description" into "Summary", and (b) add a "Detail" column next to it. How do I change this? Incidentally, I have other tables with "Description" columns too. Thanks in advance. Jonathan3 (talk) 13:42, 26 September 2016 (UTC)


 * Change the template (including the #cargo_declare and #cargo_store calls it), then go to the "Recreate data" tab at that template page and hit the button. Yaron Koren (talk) 14:09, 26 September 2016 (UTC)


 * Cheers. What will this do with those existing pages with text under the "Description" heading? Will that text be kept and assigned to one of the new columns? Jonathan3 (talk) 14:42, 26 September 2016 (UTC)


 * No, those will all have to be modified as well. If there are a lot of such pages, the Replace Text extension may be useful. Yaron Koren (talk) 19:31, 26 September 2016 (UTC)

Form for user to search data
Would it be possible to add a form for the user to search data? I guess I mean something like a customised version of Special:ViewData. Thanks. Jonathan3 (talk) 22:50, 30 September 2016 (UTC)


 * You can use the Semantic Forms Special:RunQuery feature for that. Yaron Koren (talk) 17:54, 2 October 2016 (UTC)

Cargo & SMW, 1.27.1
Mediawiki 1.27.1, SMW 2.4.1, Cargo master.

When Cargo is on, on Special:AllPages lost all links related to SMW :-( --StasR (talk) 14:37, 8 October 2016 (UTC)


 * Yes, that's a new SMW "feature" - see here. Yaron Koren (talk) 13:07, 9 October 2016 (UTC)


 * Thanks! --StasR (talk) 18:53, 9 October 2016 (UTC)

setCargoPageData.php terminating with error 1214
Hi Yaron

I was adding some of the optional fields to the _pageData table following the documented instructions when it terminated with the following error:

Error: 1214 The used table type doesn't support FULLTEXT indexes (localhost)

This occurs because I am using MySQL v 5.5 which needs "ENGINE=MyISAM" on the create table instruction to allow FULLTEXT indices.

I had a look at CargoUtils.php which has code to trap for this situation but the code wasn't executing because the global $wgCargoDBtype is null. If I explicitly set $wgCargoDBtype in the public static function getDB to be $dbType after that variable is set then the create table function works as expected.

I wondered if I have misunderstood something which has resulted in $wgCargoDBtype being null when running setCargoPageData.php?

Many thanks, Duncan 14 Oct 2016


 * Sorry about that! That was a bug in Cargo; I think I just fixed it. Yaron Koren (talk) 14:45, 14 October 2016 (UTC)