Extension talk:External Data/Archive 2017 to 2018

Is the usage of SQL group by possible?
I would consider "group by" extremly useful since it would be possible to create statistics for database content really easily. --WolfgangFahl (talk) 12:11, 10 January 2017 (UTC)
 * 1) Is the usage of SQL "group by" possible?
 * 2) if yes how?
 * 3) if no as it was pointed out in Extension_talk:External_Data/Archive_2013 where would a CR be posted?


 * No, it's still not possible, though I can see how it would be useful. What's a CR? Yaron Koren (talk) 14:35, 10 January 2017 (UTC)

How to pass header value to request?
With some API we need to pass in the headers some value (for example: https://docs.gitlab.com/ee/api/README.html#authentication) I tried url=--header "PRIVATE-TOKEN: xxxxxxxxxxxxxx" https://myserver/api/v1/list/all or url=https://myserver/api/v1/list/all,array('PRIVATE-TOKEN'=>'xxxxxxxxxxxxxxxx') But not chance. How can we achieve this? if this is possible, can we have an example of how?


 * The #get_web_data function only works with RESTful APIs; #get_soap_data might work in this case. Yaron Koren (talk) 15:52, 17 February 2017 (UTC)

Dash character in data
Hello

We found when displaying csv file data with a) cells that contain just a dash character, or b) cells that contain text and start with a dash character that any attempt to render this data into a table will cause rows to break at that cell. This is presumably because the parser inserts | for a column break and then the - from the string, resulting in |- which is a special string which means 'new row'.

To resolve this I edited the function GetCSVData in the file ED_Utils.php. In the foreach loop just before the return statement, I added a check for the dash character which replaces it with a dash wrapped in tag. This resolves the issue. Note that this issue likely still exists when other parsers like get_db_data are used, since I only resolved it in the GetCSVData function.

Note that I have had to replace the word nowiki with n0wiki below, because it was impossible to display this code since it was closing nowiki brackets.


 * Great, thanks for this patch! Is that first "if" check necessary? Maybe everything can be handled with just the "if substr" call? Yaron Koren (talk) 12:19, 14 April 2017 (UTC)

PHP Notice in error logs
Hi I can see theses messages in the apache server error log

PHP Notice: Second parameter should not be a timeout. [Called from EDUtils::fetchURL in /opt/mediawiki/extensions/ExternalData/ED_Utils.php at line 882]

ExternalData 1.8.3 Mediawiki 1.27.3

Cheers

Ubibene

Daisy Chaining
Hey, is there a way to daisy-chain queries? As far as i can see one can only enter one URL to get data from. My usecase: get some data, create a table with it and then use coordinates from the first query in a secondary query for reverse geocoding (for each individual entry). Is this possible somehow or is further explainantion required?


 * If I understand the question correctly, you can just cycle through the data twice, by calling #for_external_table twice - the data won't get cleared after the first call. Or am I misunderstanding something? Yaron Koren (talk) 03:20, 31 May 2017 (UTC)
 * In my specific usecase i'm building a table from an overpass query which returns xml. I then want to geocode every coordinate of that table via nominatim (returns xml too) to a specific country.

Error: Did not find alias for special page 'GetData'.
Hi, i use Version 1.8.3 (6705c56) from 2017-07-17T22:03:53. This Error is already marked as fixed, but i still get these message. see commit on Phabricator/"Add file for special page aliases". My wiki installation is Version 1.29.0 in german language

--TomyLee (talk) 09:28, 28 July 2017 (UTC)


 * Yes, an "alias" file was created, but no aliases have been defined yet. Where are you seeing that error? I didn't know that languages for which an alias has not been defined produced an error message. Yaron Koren (talk) 15:21, 28 July 2017 (UTC)


 * I see it only on the page "Special pages", theis message is printet 2 times. In LocalSettings i have set "$wgDevelopmentWarnings = true" for my upgrade process. TomyLee (talk) 06:10, 31 July 2017 (UTC)

detect_encoding expects parameter 1 to be string (solved)
Hi, if i execute runjobs.php i get lots of notices for this extension. It says:

--TomyLee (talk) 09:55, 28 July 2017 (UTC)


 * That's very unexpected. If possible, could you add a line above that line that looks like "if (is_object( $dbField ) ) print_r( $dbField );", and let me know what it prints out? Yaron Koren (talk) 15:18, 28 July 2017 (UTC)


 * i Insert your code and also a "Debug" message that prints without condition. Here is one example output:

The error comes from a Query to Microsoft SQL Server with column datatype "date" with value "2011-10-13" --TomyLee (talk) 06:52, 31 July 2017 (UTC)
 * I have solved this error by insert a new check bevor the convert line. my ED_Utils.php is now:


 * Great find! Sorry, I think I missed your previous message here. I just checked in a patch based on this code - hopefully this problem is now solved. Yaron Koren (talk) 15:36, 11 August 2017 (UTC)

Undefined index: UseWindowsAuth and port (solved)
as a third error message if i execute runphp.php i get the notice:

i know, this is not part of ExternalData but i use Microsoft Server only with External Data.

Maybe you need thes two new(?) parameters for mssql. In "MWLBFactory.php" i found these part of code:


 * This sounds like something recently added to MediaWiki. What's MWLBFactory.php? Maybe I can copy the approach from there. Yaron Koren (talk) 15:22, 28 July 2017 (UTC)
 * I don't know the file. i search for these variables to see if there is maybe a new config variable for my LocalSettings i missed. But the only appereance i found was in classMWLBFactory
 * Did you set and ? --wargo (talk) 09:05, 31 July 2017 (UTC)
 * Yes --TomyLee (talk) 09:50, 31 July 2017 (UTC)
 * I have solved the problem for me. I had to added little bit of code. It seems that the parameters not used by MediaWiki, maybe in a later release? But the warning messages never comes up again:You have to add these two new Parameters for the connection definition. To get this: In Localsettings add these new variables (DbName ist the Name/Id of your connection): in ExternalData.php at Line 60 add these two lines:   in ED_Utils.php the static function getDBData has to be modified at 3 areas: Line 180 add   Line 193 add   Line 244,260 and 273 add to the array of parameters   --TomyLee (talk) 13:03, 11 August 2017 (UTC)

can't get to xml field from api query call
A query i added to a template looks like this:

on the final page though this returns the error: Undefined Index title

running the query in the sandbox returns this xml though with the fields title.

What am I missing?

Thanks Markus


 * That's strange - I was able to get essentially the same thing working, though with a hardcoded category name. Does this happen with all pages that call this template? Could you try calling it with a category whose name doesn't contain spaces or non-ASCII letters? Yaron Koren (talk) 01:12, 1 August 2017 (UTC)


 * Thanks Yaron! I should have tried this... Using a hardcoded Category name makes it work.

Addressing an Array
Is it possible to pull information out of an array? I need to pull information out of the statuses array.

Is is something like?

I understand that this isn't in any way scalable, but the vast majority of the time the array does not contain more than one member, and at this stage I am just trying to learn about ExternalData's capabilities.


 * Unfortunately, no - if this were stored in XML, you could do it with the "use xpath" parameter, but there's no such parameter for JSON. (There is a JSONpath technology, but it's poorly supported, from what I understand, and it's not supported by External Data.) Yaron Koren (talk) 01:10, 4 September 2017 (UTC)
 * Thanks, good to know

JSON GET header
Is it possible to send a JSON GET header with ExternalData?


 * You mean, have an API query string composed of JSON? I don't see why not... Yaron Koren (talk) 17:12, 13 September 2017 (UTC)


 * Sorry, I am rather new to JSON. The GET query I have in Postman requires the header section with a key value pair. I am not sure how define a header using ExternalData. Does that make sense?


 * Can you define this query as a single URL, with all the JSON in the query string? If so, you can just pass that in to #get_web_data. Yaron Koren (talk) 13:33, 14 September 2017 (UTC)