Extension talk:External Data

From MediaWiki.org

Contents 1 External data showing non-cached data but semantic data incorrect 2 Allowing non-REST APIs? 3 Error when running SMW_refreshData.php 4 Limit #get_external_data to specific URLs 5 Internationalization 6 Nothing Displays

[edit] External data showing non-cached data but semantic data incorrect

I'm pulling data from twitter and marking the data using semantic mediawiki, and the external data is updating, but the semantic properties aren't updating. They aren't updating even when I hit the refresh tab. See:

http://ec2-174-129-9-164.compute-1.amazonaws.com/wiki/Twitter_searches

http://ec2-174-129-9-164.compute-1.amazonaws.com/wiki/Adeletiblier

http://ec2-174-129-9-164.compute-1.amazonaws.com/wiki/Special:Browse/Adeletiblier

Any idea why this is going on?
--Ryan lane 03:11, 17 July 2009 (UTC)

Btw, if I actually modify the page, it updates the semantic data. Simple purges don't work though.
--Ryan lane 03:15, 17 July 2009 (UTC)

Yes, that's a definite problem when using External Data in conjunction with SMW - the data, once retrieved, is stored in the SMW tables, and after that it becomes hard to update it. Refreshing pages won't work, because that doesn't lead to any database updates. At the moment, I think the best solution (and the one that some people already use) is to create a cron job to run the SMW script "SMW_refreshData.php" (which does update the database) at a regular interval, like once a day. I should add something to the extension page about that...

Ah, this solves my problem, thanks!
--Ryan lane 13:51, 17 July 2009 (UTC)

By the way, your site's a great demonstration of the possibilities of External Data and SMW together - I might end up using it for demo purposes, assuming it stays up... Yaron Koren 13:44, 17 July 2009 (UTC)

Amazon EC2 is expensive ;). I put it up for demo purposes for a talk I'm giving at BarCampNOLA on community software development where SMW is the backend, SF is the frontend, templates are business logic, and Plotters and Gadgets are user writable javascript. It is unlikely I'll keep the site up unless I find some people who want to co-host (and co-pay). At some point I'll probably get a VPS. I have the database and wiki files saved, so I'll let you know the new site when I bring it up.
--Ryan lane 13:51, 17 July 2009 (UTC)

Ah, okay. I'm glad to hear that you're spreading the word about the "extended SMW" system. By the way, you can host a wiki that uses all those extensions for free on Referata. Yaron Koren 14:19, 17 July 2009 (UTC)

[edit] Allowing non-REST APIs?

I'm sure I know the answer to this, but would it be possible for this to support non-REST APIs in the future? Something like SOAP or XML-RPC?
--Ryan lane 13:57, 17 July 2009 (UTC)

I don't see why it wouldn't be possible... it just needs to be implemented, including coming up with a good way to express all the data being sent, via a parser function call (presumably something like #get_soap_data, etc.) Yaron Koren 14:21, 17 July 2009 (UTC)

Awesome. I may take a crack at this at some point in the future...
--Ryan lane 17:37, 17 July 2009 (UTC)

[edit] Error when running SMW_refreshData.php

I'm getting the following error when I run SMW_refreshData.php:

Fatal error: Call to a member function getText() on a non-object in /var/www/w/extensions/ExternalData/ED_ParserFunctions.php on line 20

Which is: $cur_page_name = $wgTitle->getText();

Do I have something misconfigured?
--Ryan lane 20:39, 17 July 2009 (UTC)

Oh yes, sorry about that - this has come up as an issue before. There's a fix that needs to be applied to Semantic MediaWiki to get this working. In the file "/includes/storage/SMW_SQLStore2.php", under line 1278 ("foreach ($titles as $title) {"), you should add the following two lines:

                        global $wgTitle;
                        $wgTitle = $title;

I should add that to the documentation as well; and of course add the patch to SMW. Yaron Koren 15:30, 18 July 2009 (UTC)

Cool. That works thanks!
--Ryan lane 03:10, 19 July 2009 (UTC)

[edit] Limit #get_external_data to specific URLs

For security reasons, it would be nice to limit the external data to trusted sites only. Would this be possible? If I gave a patch for this, would you accept?
--Ryan lane 19:34, 27 July 2009 (UTC)

Hm, that's a good idea - how about an "$edgAllowExternalDataFrom" variable, that holds an array of domain names, akin to MediaWiki's $wgAllowExternalImagesFrom setting? I'd definitely accept that. Presumably, if the value were null, all URLs would be accepted. Yaron Koren 19:42, 27 July 2009 (UTC)

That sounds perfect to me.
--Ryan lane 20:05, 27 July 2009 (UTC)

[edit] Internationalization

Looks like the error messages are echoed directly in english without i18n support. Can I send you a patch adding support?

Btw, I have commit access, do you prefer that I send patches, or commit and let you revert things that don't fit properly? Either way I'll ask first about changes I'd like to make.
--Ryan lane 20:55, 27 July 2009 (UTC)

Yeah, that's true, I never translated all the error messages; which would be a good idea. If you have commit access, I think it's definitely better if you check in changes directly; although it's good of you to ask first (most people don't bother :) ). Yaron Koren 22:39, 27 July 2009 (UTC)

[edit] Nothing Displays

When I use the example on your site I get nothing displaying.

I tried running the SMW_refreshdata.php and had to apply the above patch, but I now get the following error:

PHP Notice:  Undefined offset:  1 in /opt/apps/itwiki/itwiki/extensions/ExternalData/ED_ParserFunctions.php on line 56

I have also tried accessing a json source and I get the following string displayed on the page:

UNIQb8b7293751eb2cb-pre-00000000-QINU

My guess is that there's an error in your #get_external_data call - all parameters after the first two need to contain either '=' or '=='. You've definitely uncovered a bug, though - the error-handling should be more graceful. The second problem sounds bad, too - maybe it's caused by the same error? I hope so. Yaron Koren 13:42, 28 August 2009 (UTC)

I upgraded the version of SemanticMediaWiki to 1.4.3 from 1.4.2 (mediawiki 1.14), it works with json type external data but not with the csv germany example. Is there a page with tested versions on dependent apps i can check? -JS 12:42, 31 August 2009 (UTC)

Sorry, what do you mean by tested versions and dependent apps? The Discourse DB Germany page is, of course, a demo page... Yaron Koren 19:57, 31 August 2009 (UTC)

eg. for version x of external data, the following versions of mediawiki, semanticmediawiki etc are needed/have been tested. I also had to add JSON support for php on my server. -JS 20:38, 31 August 2009 (UTC)

I see. No, there isn't... the CSV format is a special case, since in SMW 1.4.3 that format began to be printed out with a header row; so the format in #get_external_data should be 'csv with header' instead of 'csv'. Yaron Koren 21:53, 31 August 2009 (UTC)

HI back with another error, trying to use the LDAP source:

[Thu Oct 15 22:56:59 2009] [error] [client 10.0.3.213] PHP Notice:  
Undefined offset:  1 in /wiki/extensions/ExternalData/ED_Utils.php on line 46, 
referer: http://wiki/index.php/External_Data_Testing

[Thu Oct 15 22:56:59 2009] [error] [client 10.0.3.213] PHP Warning:  
ldap_search() [<a href='function.ldap-search'>function.ldap-search</a>]: 
Search: No such object in /wiki/extensions/ExternalData/ED_Utils.php on line 
103, referer: http://wiki/index.php/External_Data_Testing

[Thu Oct 15 22:56:59 2009] [error] [client 10.0.3.213] PHP Warning:  
ldap_get_entries(): supplied argument is not a valid ldap result resource in 
/wiki/extensions/ExternalData/ED_Utils.php on line 104, referer: 
http://wiki/index.php/External_Data_Testing

thanks JS

Hi.... can you paste your #get_ldap_data call here? - Borofkin 06:03, 26 October 2009 (UTC)

{{#get_ldap_data:
 |domain=domain_def
 |filter=(sAMAccountName=username)
 |data=title=title
}}

{{#external_value:title}}

thanks -JS