Extension talk:External Data/Archive 2017 to 2018

JSON issue
Hi - I'm having an issue with json data. I'm using External Data v1.3.5. Mediawiki version is 1.16.2

--Skenaja 21:40, 30 January 2012 (UTC)
 * Json:
 * My SMW page:
 * The "tags" array in the json doesn't seem to be parsed...


 * Hi - my guess is that the issue is the double brackets around the "tags" value, which I think makes it an array of an array. Why are there two brackets there - do you know? Yaron Koren 13:14, 31 January 2012 (UTC)
 * I'm not too sure, I've raised it with that site's developers. The json does parse OK @ JSLint.--Skenaja 16:54, 31 January 2012 (UTC)


 * Actually, the brackets might not be the issue - the problem might just be External Data's parsing of arrays in JSON. Someone just sent me a patch, I think coincidentally, that might fix this very issue. Hopefully I can add it in sometime in the next few days. Yaron Koren 17:05, 31 January 2012 (UTC)


 * I've had it confirmed to me that the double brackets are indeed arrays of arrays - here's another sample json file which shows it better: --Skenaja 22:56, 31 January 2012 (UTC)

Get data from Sqlite Db
Hi,

I would like to know if it's possible to retrieve data from sqlite db ? I've tested a few conbinations in Localsettings but not luck... I use Mediawiki 1.16.5.

Thanks in adavnce. Vincent.


 * There was no support for SQLite, for some reason - I don't know why I never added it in. I just added it now, I think, along with support for DB2, which was also missing. If you use SVN, you should re-get the code - if not, you can just duplicate this change in your code. Then, you should be able to set "sqlite" as the DB server type. Please let me know if that worked... Yaron Koren 14:36, 31 January 2012 (UTC)


 * Thanks for the fast reply, I'll give you feedback shortly.
 * Vincent.
 * Could you tell if my Localsettings.php is good ?

$edgDBServer['SQLITE'] = "/mnt/sqlite/infos_db"; ==> folder of sqlite db $edgDBServerType['SQLITE'] = "sqlite"; $edgDBName['SQLITE'] = "infos_db"; $edgDBUser['SQLITE'] = ""; $edgDBPass['SQLITE'] = "";
 * Error: ( SQLite database not accessible)
 * Thanks in advance


 * I assume you're setting the username and password in the original - beyond that, I don't know. I'm used to the DB server being a name or a URL of some sort, and not a directory, but I don't know how SQLite works. Yaron Koren 15:45, 31 January 2012 (UTC)


 * HI, After a few search around compatibility between php and sqlite. It seems that sqlite version i use (sqlite3) uses PDO function (PHP) to connect to that db. Connection and query syntax seem to be different from ED_Utils.php code. I'm not skilled enough to review ED_Utils.php code.
 * Thanks for the help. Great job, i use it for Mysql access without any problem. Vincent
 * Hi, I'm gone through sqlite connection. it works great. For information you have to :
 * Add $wgSQLiteDataDir = "/root/forlder/of/sqlite"; in your Localsettings.php
 * Vérify you have RW rights on sqlite db.

$edgDBServer['SQLITE'] = ""; ==> Not used $edgDBServerType['SQLITE'] = "sqlite"; $edgDBName['SQLITE'] = "infos_db"; ==> Name of sqlite db without extension (Be sure to have .sqlite extension because DatabaseSsqlite.php script will add it automatically) $edgDBUser['SQLITE'] = ""; ==> Not used $edgDBPass['SQLITE'] = ""; ==> Not used
 * Thanks Yaron, it also works great for Oracle Database (10G R2). Vincent G

Thanks for letting me know about that, and for diagnosing the issue. I just checked in some code that I think handles this issue in a nicer way - you can see the new documentation for how to run it. If you get the new code and try it out, please let me know if it works for you. Yaron Koren 20:26, 3 February 2012 (UTC)

Issue
I am trying to get CSV data from a page in the same wiki. I have duplicated the example from: http://discoursedb.org/wiki/Fruits_table

I have tried both the explicit URL and the fullurl and various combinations of raw data and Special:GetData in the URL. No matter what, the tables are blank. When I follow the links, the following test appears in the browser window: Name,Color,Shape Apple,Red,Round Banana,Yellow,Oblong Orange,Orange,Round Pear,Yellow,"Pear-shaped"

The tables are completed when I pull directly from the example external URL: http://discoursedb.org/wiki/Special:GetData/Fruits_data

I am must be doing something bone-headed, but can't see my error. Did I miss some setting? Here is the abbreviated content of my Fruits table page:

Thanks in advance, Rodger

From my local wiki using explicit URL (Fails)
Using: https://10.34.161.20/index.php?title=Special:GetData/Fruits_data&action=raw

From my local Wiki using fullrul (Fails)
Using fullurl:

From external Wiki (Works)
Using http://discoursedb.org/wiki/Special:GetData/Fruits_data

I have tried on a couple of my wiki VMs with the same result., including updated version, but same behavior regardless. Seems like it wouldn't work for the external URL if there was a version conflict.

MediaWiki	1.18.1, Semantic Bundle, ..External Data (Version 1.3.5), ...

Discussion

 * Hi - that's quite a bit more information than was necessary. :) But I admire your thoroughness. Anyway, I assume the issue is that the server, strange as it may sound, can't access itself. It's trying to reach its data via the IP address, and that one, I assume, is closed off. You'll need to modify something for the server to be able to access itself via HTTP. Yaron Koren 04:36, 15 February 2012 (UTC)
 * That makes sense. Thanks for the quick response! Rodger
 * I don't think I can overcome the server access. I don't have the expertise to suggest a change to the admins, and don't know if would be feasible. Is there a way that something like the following could be made to work?

#get_web_data:url= or #get_web_data:url=
 * Or a parallel #get_local_data? I am familiar with the internal objects, but would really like to use CVS or XML when I have it, rather than translate to internal object syntax.


 * I can't think of a way to do that, unfortunately... though, what about using "localhost" instead of the IP address? That might work. Yaron Koren (talk) 02:50, 17 February 2012 (UTC)


 * Anybody found a solution with this issue? I have the same issue using a Wiki in an secure environment (https://wiki.xxx) (Christoph, 9. May 2012)

get_db_data Parsing
This is related to the discussion from about a year ago: http://www.mediawiki.org/wiki/Extension_talk:External_Data/Archive_2011#get_db_data_-_Data_parsing

This data retrieval DOES NOT work when the field with a convert is in the 2nd position

This data retrieval DOES work when the field with a convert is in the 1st position

--Wolcott 20:10, 16 February 2012 (UTC)

Have table display results horizontally rather than vertically
I have this query against the MySQL db that goes something like this:

From that I am able to pull in the results into a table like so:

And it displays the results just fine. However, I was wondering if there is a simple way that I am missing where you could have the extension display the results horizontally instead of vertically.

So instead of seeing a list of names like this:

I want it to look like this:

--Stanimalz (talk) 16:04, 22 February 2012 (UTC)


 * Hi - I would try removing the "|-" line under the #for_external_table line. Yaron Koren (talk) 16:08, 22 February 2012 (UTC)


 * That worked perfectly. Thanks for the quick response.--Stanimalz (talk) 17:47, 22 February 2012 (UTC)

Oracle usage
Hi,

can somebody tell me how to get the extension working with oracle? I can't figure out which settings i have to set.

And how does the Localsettings.php look like if i use the extension to connect an Oracle DB.

My Attempt:

$edgDBServer['bla'] = "blabla.bla.de"; $edgDBServerType['bla'] = "oracle"; $edgDBName['bla'] = "bladb"; $edgDBUser['bla'] = "blauser"; $edgDBPass['bla'] = "blapw";

And my Query looks like:


 * This should probably be more obvious in the documentation, but you should have "db=bla" instead. Yaron Koren (talk) 13:30, 13 March 2012 (UTC)


 * If i type in "db=bla" i receive the following error in my wiki:


 * Database error
 * A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was:
 * ALTER SESSION SET CURRENT_SCHEMA=BLA
 * from within function "DatabaseOracle::selectDB". Database returned error ": ".
 * Any idea?

These is a Systable which is readable by my user.

-jan


 * This could be an error in MediaWiki's handling of Oracle, or maybe just a permission error. I would try getting some more basic data from that database, like the contents of a regular table, to see where the problem is coming from. Yaron Koren (talk) 17:29, 13 March 2012 (UTC)


 * I'm also having trouble setting up an Oracle connection in Mediawiki 1.18.1 and ExternalData 1.3.5, with a configuration that works fine in Mediawiki 1.16.0 with the same ExternalData version. By trouble I mean I am getting the Database error with ALTER SESSION SET CURRENT_SCHEMA=BLA as the last attempted query when attempting a very basic query from a normal table (same query works fine if I write a simple php script with oci calls). I noticed that between these versions of MW, the DatabaseOracle.php include file has grown twofold. Any thoughts on what change may be breaking External Data? Thanks for your help in advance. --Omer


 * My problem was solved by making the following change in LocalSettings.php:

old: $edgDBServer['somedb'] = "the.db.server.com"; new: $edgDBServer['somedb'] = "the.db.server.com/somedb";
 * where "somedb" is the instance name. I hope this helps others with similar issues when upgrading MediaWiki past 1.16. --Omer

Microsoft Sql-Server Error [solved]
Hi,

can somebody tell me how to get the extension working with Microsoft Sql Server and MediaWiki 1.18? I have a working MediaWiki 1.15 installation with External Data connecting to Microsoft Sql Server 2008R2

Now i want to Upgrade to MediaWiki 1.18 and all works fine, but the ExternalData Extension fires some Warnings and Ends up with an Fatal error. I don't think that it is a configuration error, because of the running MediaWiki 1.15 installation. The Messages i get:

Details of this Installation are on my UserPage, the Servertype in LocalSettings is set to mssql:

--TomyLee (talk) 06:08, 4 April 2012 (UTC)


 * Hi - if possible, could you include here the #get_db_data call you're making? Feel free to change around any table or field names, if it's private. Yaron Koren (talk) 15:15, 4 April 2012 (UTC)


 * the get_db_data i have testet are:


 * and


 * --TomyLee (talk) 05:44, 5 April 2012 (UTC)


 * Okay - if the same error happens for both of those calls, then the issue isn't with the #get_db_data call. I should have asked this before, but - did you see the note in the documentation about MSSQL? It could be that you just need to use the Extension:MSSQLBackCompat extension, if you're not using it already. Yaron Koren (talk) 16:14, 5 April 2012 (UTC)
 * I have installed Microsoft drivers for PHP for SQL Server and the External Extension works with MW 1.15, but not with MW 1.18. (Within the MW 1.15 installation i need to replace mssql with sqlsrv in DatabaseMssql.php to get it working). The DatabaseMssql.php within the MW 1.18 was unchanged. Now I have changed the Code a little bit to print out the variable around the error and can see some of the Data from the Sql-Server. The connecting to SQL Server works and MW gets the data but can't processing it. The MSSQLBackCompatExtension is only to query MSSQL from Linux, we are running MW on Windows. So I think the Microsoft drivers for PHP are the right choice (also we don't have a /etc/freetds.conf configuration file).
 * Maybe this helps; I have insert  at line 978 and can see that the result is the first Datarow of my Datatable:


 * Then I have print out the two parts of Line 1028 and gets: $this->mRows[$this->mCursor] : -1, $arrNum : empty Array (with print_r($arrNum) --TomyLee (talk) 05:50, 10 April 2012 (UTC)


 * Unfortunately, I know very little about SQLServer, or about MediaWiki's support for it. And I don't know much about running MediaWiki on Windows, for that matter. If you can, I would write about this to one of the general MediaWiki venues, like the #mediawiki IRC channel or the mediawiki-l mailing list. Yaron Koren (talk) 13:41, 10 April 2012 (UTC)


 * I had a very similar problem. All I did was comment out lines 974 and 976 and added a while loop into DatabaseMssql.php it seems that sqlsrv_fetch_array only gets one row? And mine works fine now


 * to

Craig Purchase (talk) 16:11, 21 June 2012 (UTC)


 * This works for me too! Thanks very much for your help. --TomyLee (talk) 06:36, 22 June 2012 (UTC)


 * That's great news! I just added this tip to the documentation. Yaron Koren (talk) 15:51, 22 June 2012 (UTC)

POST instad of GET
In order to connect to a local webservice I needed to post some data.

Only very few code modifications were sufficient to achieve that.

Should I share that changes. If yes where and how?

--Schubi87 (talk) 21:43, 11 April 2012 (UTC)


 * Yes, that would be great! You can either create a patch in Bugzilla, or send me an email at yaron57 at gmail.com. Yaron Koren (talk) 19:23, 12 April 2012 (UTC)


 * Are you going to release the post option I send you soon, or should I create a new branch?

--Schubi87 (talk) 10:12, 28 June 2012 (UTC)

Extension:External Data fails without notice
Hello,

We've been trying to implement External Data extension to the company's wiki, but it seems to behave a little unstable. We have got it working already, but from times to times it fails again. I could not say the causes, but I suspect it has to do with some sort of updates.

Some months ago, we had External Data 1.3.5 working on Mediawiki 1.7.1 and Semantic Mediawiki 1.5.5 for testing. We created two or three Sandbox pages todo test it. Recently, we updated Mediawiki to version 1.7.3. I could not say that update is the cause bacause we have not accessed those Sandbox pages to check it. No other extension/database/PHP was updated during this time. Last week I started working to bring the extension to production environment, and tried to use the documentation examples before using my own files. Nothing worked. Pages simply doesn't retrieve any information from SCV files either inside or outside the wiki. Then I've checked the older pages that were working in the earlier tests. They were failing too.

I tried to update External Data to version 1.3.6, but the problem remains.

Well, I know the information I am giving above is not enough to make a diagnosis. Never mind about that. But I'm not sure where to get the information to do that.

Is there any checklist we have to observe in order to avoid External Data stops working after Mediawiki or third party extensions updates. Any other idea on where to search in order to find the cause of the problem?

Thanks a lot in advance.


 * I would try copying the text from this page to a page on your wiki, and seeing if that works. By the way, when you wrote "MediaWiki 1.7.3", I assume you meant 1.17.3? Yaron Koren (talk) 13:39, 19 April 2012 (UTC)

Hi.

I did. But, it didn't work, also. Tks Pslima (talk) 16:57, 19 April 2012 (UTC)


 * Okay. What operating system is the wiki running on, and what version of PHP are you using? Yaron Koren (talk) 17:18, 19 April 2012 (UTC)

Hi. Tks Pslima (talk) 15:57, 26 April 2012 (UTC)
 * Debian 6.0 - kernel 2.6.32-5-openvz-amd64
 * PHP	5.3.3-7+squeeze8 (apache2handler)
 * MySQL	5.1.61-0+squeeze1

Query Returns no values?
Hi

I am attempting to use this Extension on my Wiki to retrieve data from a CSV file stored on a dropbox (which is available to the Public), the problem i get is that is runs, but returns blank values. I have looked for help but cant seem to find the answer anywhere.

The code i am using is is a sample which is avaible on the Extensions page:

.
 * Germany borders the following countries and bodies of water:
 * Germany has population.

I get no error messages when save the page either.

Its important you know that this is being run from an internal server from work, so i am wondering, could that have anything to do with the fact that i cant retrieve any values? as in, would the Firewalls from the server be preventing this extension from extracting the data? I say this because i ran the same piece of code on "Discourse DB" Wiki (as they have the extension installed) and it worked... Thanks

Claudio


 * Yes, my guess is that the firewall is preventing your server from hitting outside URLs. If you have command-line access on your server, you can try calling something like "curl http://google.com " to see if that works. Yaron Koren (talk) 15:33, 29 June 2012 (UTC)

External data still flawed after SMW refreshData
Referring to External_data_flawed_after_SMW_refreshData, we are getting the same error with SMW 1.6.1. In other words, we observe that after rebuilding the SMW database a number of user pages that fetch name/mail/phone data from LDAP are showing flawed data. There are groups of user pages that show the same data, even though they have different LDAP usernames. The only way to correct it is by clicking "edit" and then save. The database tables show that the wrong values for name/mail/.. are stored in e.g. smw_atts2 together with the s_id of the flawed page. This is not fun when there are hundreds of pages.... What to do? Bernadette (talk) 19:08, 28 June 2012 (UTC)


 * The extension SemanticUpdateOnPurge might be able to help in this situation. Yaron Koren (talk) 15:35, 29 June 2012 (UTC)


 * I think I found the bug: in doGetLDAPData (and all other doGet... functions) the global $wgTitle->getTitle is used to gather the name of the current page. However when running the jobqueue, this call always returns "runJobs.php". The use of $wgTitle is deprecated anyway, and using the preferred $parser->getTitle cures the problem in my case. - LosWochos (talk) 16:01, 17 July 2012 (UTC)


 * Thanks for that suggestion! I just checked your suggested changes into the Git repository - hopefully that fixes things. Yaron Koren (talk) 21:10, 18 July 2012 (UTC)

Issues getting xml to work using xpath
I'm trying to get external data to work using xml and xpath and was hoping someone could help me out. So far I can access an xml to populate a table or substitute a word but once I try and use xpath nothing is being returned. I have looked at the example and followed them to get csv and xml(with out xpath) working.

test:

{| class="wikitable" ! NAME


 * Did you try actually using XPath in the 'data' parameter, like "data=name=/abc/name" or something? Yaron Koren (talk) 12:35, 6 July 2012 (UTC)

Thanks for getting back to me. Yes, I tried using "data=name=/abc/name" I also copied the examples from http://www.discoursedb.org/wiki/External_Data_XPath_demo to see if I could get it to work. are there any other extensions that I should also install to get this to work?

example from http://www.discoursedb.org/wiki/External_Data_XPath_demo after being copied locallly, sample.xml which is the same as the one from the example is access from the address show below

Salutation:

Max length:

Address:

URL: Thank you for your help Mw49 (talk) 04:39, 9 July 2012 (UTC)


 * Just to make sure: are you using the latest version of External Data, 1.4? This feature was only added in very recently. Yaron Koren (talk) 12:54, 9 July 2012 (UTC)

Yep that looks to be the problem, sorry for wasting your time. --Mw49 (talk) 05:14, 10 July 2012 (UTC)


 * That's great. No problem - it didn't take much time on my side. Yaron Koren (talk) 12:48, 10 July 2012 (UTC)

#for_external_table support for #get_ldap_data
The repository looked ad-hoc so I had no idea where to contribute this...

diff --git a/ED_ParserFunctions.php b/ED_ParserFunctions.php index a5e8e8d..19c9bb9 100644 --- a/ED_ParserFunctions.php +++ b/ED_ParserFunctions.php @@ -201,10 +201,13 @@ class EDParserFunctions { $mappings = EDUtils::paramToArray( $args['data'] ); // parse the data arg into mappings

$external_values = EDUtils::getLDAPData( $args['filter'], $args['domain'], array_values( $mappings ) ); - -              // Build $edgValues -              foreach ( $mappings as $local_var => $external_var ) { -                      $edgValues[$local_var][] = $external_values[0][$external_var][0]; + +              foreach( $external_values as $i => $row ) { +                      foreach ( $mappings as $local_var => $external_var ) { +                              if ( array_key_exists( $external_var, $row ) ) { +                                      $edgValues[$local_var][] = $row[$external_var][0]; +                              } +                       }                }                return; }

As a side-note I also had to add "ldaps://" in my $edgLDAPServer, too.


 * Hi - this looks good, but what bug does it fix? (I know very little about the LDAP handling.) Yaron Koren (talk)


 * On the extension page it says "Note that #get_ldap_data will only retrieve one result." - even though a full result was being fetched by LDAP the array format being constructed was not kosher. User:Memeyou