Extension talk:External Data

Invalid JSON on #get_web_data
I plan on doing some upgrades to mediawiki and possibly php as a shot in the dark, but before doing that I figured I would ask here.

Here is the exact json of the table I'm trying to display. The data itself is junk just to play with.

[{"IPAddress":"192.168.000.001","Type":"Server","Device":null,"Location":null,"Comment":null,"HostnameDNS":null,"OSFirmware":null,"LastVerified":"1\/7\/2021","PlantNetwork":"999Office"},{"IPAddress":"192.168.000.002","Type":"Server","Device":"Virtual Server","Location":"ERA - Server Room","Comment":"File and Print Server","HostnameDNS":"fileserver","OSFirmware":"Server 2016","LastVerified":"1\/7\/2021","PlantNetwork":"999Office"},{"IPAddress":"192.168.000.003","Type":"Server","Device":"Virtual Server","Location":"ERA - Server Room","Comment":"Video Server","HostnameDNS":"videoserver","OSFirmware":"Linux","LastVerified":"1\/7\/2021","PlantNetwork":"999Office"},{"IPAddress":"192.168.000.004","Type":"Server","Device":"Physical Server","Location":"ERA - Server Room","Comment":"Web Server","HostnameDNS":"webserver","OSFirmware":"IBM OS\/400 7.2","LastVerified":"1\/7\/2021","PlantNetwork":"999Office"},{"IPAddress":"192.168.000.005","Type":"Server","Device":"Physical Server","Location":"ERA - Server Room","Comment":"Mainframe","HostnameDNS":"mainframe","OSFirmware":"IBM OS\/400 7.2","LastVerified":"1\/7\/2021","PlantNetwork":"999Office"}]

I'm getting this from a really generic json_encode( $data ); function in select_ipam.php, which queries a Microsoft SQL server database. I have already verified that my header information is set correctly as another user pointed out.

header("Content-Type: application/json; charset=utf-8"); echo( json_encode( $data ) );

I also tried using the "use jsonpath" approach but it just gives me the same thing.


 * I'm not sure what the exact problem you're seeing is, but I would recommend switching to the latest version of External Data - the version you're running is rather old. Yaron Koren (talk) 00:46, 8 January 2021 (UTC)


 * I was able to get it working with some different syntax not using a template and a version update. Now if I can just figure out how to get it to update more frequently. When I click edit and save the page, without making any other changes it updates the table, but otherwise it doesn't want to grab new data from the php post.  Are there any plans to integrate an "ajax mode" that would allow the table data to be refreshed say every 10 seconds without a full page reload?


 * Okay, that's good. No... that sounds like it would require a whole different approach - JavaScript-based instead of PHP-based. Yaron Koren (talk) 20:18, 8 January 2021 (UTC)

get_web_data error after updating mediawiki
updating to mediawiki 1.35.1 and using the latest external data  results in my template: erroring with:

Could not get URL http://localhost/api.php?action=query&list=categorymembers&cmtitle=Category:CHOPs&format=xml&cmlimit=500 after 3 tries.

Clicking the url though returns the expected XML..

--Wuestenarchitekten (talk) 21:47, 13 January 2021 (UTC)


 * Silly question, but did this work before? Yaron Koren (talk) 14:00, 14 January 2021 (UTC)

#get_db_data with 'join on' statement returning blank table
Apologies in advance, I am somewhat new to SQL but I have some experience. I am attempting to use #get_db_data to pull data from a local server DB to display sports statistics. One of the tables has a field (Position) that I need for the wikitable, so I was hoping to join these two tables using the 'PlayerID' field. I can't do this server-side because it would add considerable time to processing and, again, I don't have enough experience.

Using the aliases throughout and executing the query returns no syntax issues or DB errors, but #for_external_table returns an empty table. I get the appropriate number of rows in the table, so I assume it's returning nulls or default values. Without the ability to debug there's no way to track down where in the process it's losing the values.

Here's the code I wrote:

Documentation is limited for the 'join on' statement for this extension, and there aren't even any reported issues from the past (I've checked the archives). Should I create a template and use #display_external_table instead? Any help would be appreciated.

2603:8001:4948:DF00:7D69:EB88:902:704 09:50, 26 January 2021 (UTC)gplehner


 * I don't know. My guess is that the problem is in the "data" parameter, somehow - that External Data is failing to find the external variables listed. I would try taking out those table aliases, i.e. keeping the original table names... there's a chance that that will fix the problem. Yaron Koren (talk) 15:11, 26 January 2021 (UTC)


 * Thanks, I had time to come back to this today. Removing the table aliases from the "data" parameters returned Error 1052. I incrementally removed each alias until I narrowed the issue to two parameters, team and playerID. Both tables have a field with the name 'Team' and 'PlayerID'. It doesn't seem to matter which one I request in the "data" parameter, both print out the default/'else' values I set. The 'data' parameter certainly is understanding the alias names.


 * I should also mention that I switched to using '#display_external_table' in an attempt to pass data parameters that way but it didn't work either. Thanks for your response & suggestions. 45.51.157.21 08:21, 28 January 2021 (UTC) -gplehner


 * If you have database server access you should create a VIEW on the database and then do the select on that view.
 * In the view you can join whatever table you want without touching the tables.
 * See: https://www.w3schools.com/sql/sql_view.asp
 * In your case you create a view called PlayerSeasonStats. --Felipe (talk) 13:14, 28 January 2021 (UTC)


 * I meant removing table aliases from the entire query, by the way, not just from "data". Yaron Koren (talk) 16:26, 28 January 2021 (UTC)

Alex Mashin (talk) 06:30, 3 August 2021 (UTC)
 * I'm seeing the same issue in the current version. "data" doesn't seem to be able to access fields when they are prefixed with their table alias. Seems this behavior has introduced with REL1_31
 * Can you reproduce undesired behaviour on this database?

function to check whether any line of external data matches a string
My use case for External data is reading a single csv file, and then displaying different tables for lines that contain video/pdf/image etc, indicated by a type column. While I can use {{#ifeq:{{#external_value:type}} to check the first line of data, the header, has been read, I can't decide to only display the video table if videos are available. How hard would it be to have a function that checked all entries in a particular column against a search string, returning true if any matched? At the moment I have lots of tables with headers and no content. Vicarage (talk) 16:48, 7 March 2021 (UTC)


 * If I understand this question correctly, you may be able to accomplish this with the Variables extension - instead of displaying the table directly, set it in a variable, and then only display that variable (and its header) if it's non-empty. Yaron Koren (talk) 02:12, 8 March 2021 (UTC)

Thanks, I'll look at that. My initial workaround was to create a counts.csv file, with no header, and check against {{#ifexpr:{{#external_value:imagecount}} > 0, where imagecount=column 4 of the file. Vicarage (talk) 11:57, 8 March 2021 (UTC) If you are OK with mixing rows with different media types in one table, not grouped by media type, you can use only one  with a. Note the pipe after the semicolon: without it  containing variables will not work. Alexander Mashin{{sup| talk}} 08:31, 17 September 2022 (UTC)
 * Now you can use  or   in standalone mode, i.e. with the parametres previously passed to   and without a previous call to it. Use   to filter the output. That will formally mean several queries to external data sources, but web requests and program outputs are cached.

retrieving a template reference from external_data.
I'd like to be able to retrieve a json object that contains one key ('data') and a string that contains mediawiki formatted text. Is there a way to parse that retrieved string, so that it displays on the page as formatted data, rather than plain text?

Example: I'd like to retrieve  from an external source, and have it render the template

173.16.242.134 04:23, 8 March 2021 (UTC) Alex Mashin (talk) 14:54, 24 July 2021 (UTC)
 * Try using Lua bindings and.

display_external_table should do nothing if its template function returns nothing
In http://wiki.johnbray.org.uk/Problem1 I read a 2 column file, and want to put items from the second column in a table ONLY when the first column has value "good". The template function has a switch so nothing is returned, but display_external_table adds a blank line to the last table box, which spoils the format. Using delimiter= just makes the table wider.

As this is a key use case for display_external_table, it would seem to make sense to just do nothing if there was nothing to display. Vicarage (talk) 09:52, 10 March 2021 (UTC)
 * You can achieve this, if you handle delimitation yourself in the template, and pass  as the  :

Template:

Query: Alexander Mashintalk 05:44, 17 August 2022 (UTC)

Fatal exception of type "InvalidArgumentException" When trying to access mysql database.
Recently updated to MediaWiki 1.35 from 1.31, and updated the extension to the latest version for 1.35. Previously the get_db_data queries were working fine, but now they are throwing an Invalid Argument Exception.

When I run a debug log I get the following output, related to one of the exceptions.

[exception] [YGRqrMgm-gbpiKaSuVS3ewABaww] /index.php?title=Template:CementKiln&action=submit InvalidArgumentException from line 68 of /home1/wikiwast/public_html/includes/libs/rdbms/database/domain/DatabaseDomain.php: Prefix must be a string. I have re-loaded the extension several times, and tried a couple of different databases.
 * 1) 0 /home1/wikiwast/public_html/includes/libs/rdbms/database/Database.php(291): Wikimedia\Rdbms\DatabaseDomain->__construct(string, NULL, NULL)
 * 2) 1 /home1/wikiwast/public_html/includes/libs/rdbms/database/DatabaseMysqlBase.php(111): Wikimedia\Rdbms\Database->__construct(array)
 * 3) 2 /home1/wikiwast/public_html/includes/libs/rdbms/database/Database.php(433): Wikimedia\Rdbms\DatabaseMysqlBase->__construct(array)
 * 4) 3 /home1/wikiwast/public_html/extensions/ExternalData/includes/EDUtils.php(238): Wikimedia\Rdbms\Database::factory(string, array)
 * 5) 4 /home1/wikiwast/public_html/extensions/ExternalData/includes/EDParserFunctions.php(377): EDUtils::getDBData(string, string, array, string, array, NULL, array)
 * 6) 5 /home1/wikiwast/public_html/includes/parser/Parser.php(3340): EDParserFunctions::doGetDBData(Parser, string, string, string, string)
 * 7) 6 /home1/wikiwast/public_html/includes/parser/Parser.php(3047): Parser->callParserFunction(PPFrame_Hash, string, array)
 * 8) 7 /home1/wikiwast/public_html/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(array, PPFrame_Hash)
 * 9) 8 /home1/wikiwast/public_html/includes/parser/Parser.php(2887): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
 * 10) 9 /home1/wikiwast/public_html/includes/parser/Parser.php(1556): Parser->replaceVariables(string)
 * 11) 10 /home1/wikiwast/public_html/includes/parser/Parser.php(651): Parser->internalParse(string)
 * 12) 11 /home1/wikiwast/public_html/includes/content/WikitextContent.php(374): Parser->parse(string, Title, ParserOptions, boolean, boolean, NULL)
 * 13) 12 /home1/wikiwast/public_html/includes/content/AbstractContent.php(590): WikitextContent->fillParserOutput(Title, NULL, ParserOptions, boolean, ParserOutput)
 * 14) 13 /home1/wikiwast/public_html/includes/EditPage.php(4282): AbstractContent->getParserOutput(Title, NULL, ParserOptions)
 * 15) 14 /home1/wikiwast/public_html/includes/EditPage.php(4187): EditPage->doPreviewParse(WikitextContent)
 * 16) 15 /home1/wikiwast/public_html/includes/EditPage.php(2965): EditPage->getPreviewText
 * 17) 16 /home1/wikiwast/public_html/includes/EditPage.php(701): EditPage->showEditForm
 * 18) 17 /home1/wikiwast/public_html/includes/actions/EditAction.php(71): EditPage->edit
 * 19) 18 /home1/wikiwast/public_html/includes/actions/SubmitAction.php(38): EditAction->show
 * 20) 19 /home1/wikiwast/public_html/includes/MediaWiki.php(527): SubmitAction->show
 * 21) 20 /home1/wikiwast/public_html/includes/MediaWiki.php(313): MediaWiki->performAction(Article, Title)
 * 22) 21 /home1/wikiwast/public_html/includes/MediaWiki.php(940): MediaWiki->performRequest
 * 23) 22 /home1/wikiwast/public_html/includes/MediaWiki.php(543): MediaWiki->main
 * 24) 23 /home1/wikiwast/public_html/index.php(53): MediaWiki->run
 * 25) 24 /home1/wikiwast/public_html/index.php(46): wfIndexMain
 * 26) 25 {main}

the LocalConfig.php entries are..

$edgDBServer['engy']="localhost"; $edgDBServerType['engy']="mysql"; $edgDBName['engy']="wikiwast_energy"; $edgDBUser['engy']="wikiwast_energy"; $edgDBPass['engy']=" "; The query I used was, being called from a template. Where the value of EPR is passed to it through the template call. All the data values are correct and contained within the mysql database.

I am at a loss as to what the problem is, I hope that someone might see this and be able to help.


 * For External Data - and really any extension whose compatibility policy is "Master maintains backwards compatibility", which includes all of my extensions - you should not use a REL_ branch, but instead just use the latest code - either the truly latest code, or the latest released version. Using either one may fix this specific problem. Yaron Koren (talk) 13:12, 31 March 2021 (UTC)
 * Thanks a Million Yaron that has fixed the issue. Will remember that in future.

Usage in DE:WP
Hy! This extension seems not to be installed on any Wikipedia, even on Mediawiki not! Tried a usecase on DE:WP, without success. Any possibility to get it enabled on a local Wikipedia? Regards, Uwe Martens (talk) 20:44, 25 April 2021 (UTC)


 * That's outside of my domain. But it would be great if this extension could be installed on a Wikimedia site! Yaron Koren (talk) 12:26, 26 April 2021 (UTC)


 * Thanks for your response! I'll discuss it with a responsible admin. I hoped one of them is active here, but anyway, best regards, Uwe Martens (talk) 13:07, 26 April 2021 (UTC)

No data with XML namespace
Hi Yaron, I bumped into something weird. It seems that External Data is not always able to fetch data when the XML file contains a URL for xmlns. I guess the reason could be, in some cases, that the URL is not accessible (you'd be surprised how often that happens) or in others, that the URL redirects to a new one. Lots of sites still use xmlns="http://www.w3.org/1999/xhtml", with that URL redirecting to the more secure https://www.w3.org/1999/xhtml/. I've created a snippet that you can try out for yourself:

   Jo Nesbø 

Cavila 11:13, 19 August 2021 (UTC)


 * Use  and add it to the XPath query:
 * res: Alex Mashin (talk) 11:43, 19 August 2021 (UTC)
 * Alternative:
 * res: Alex Mashin (talk) 11:52, 19 August 2021 (UTC)
 * res: Alex Mashin (talk) 11:52, 19 August 2021 (UTC)
 * res: Alex Mashin (talk) 11:52, 19 August 2021 (UTC)


 * Thanks, that seems to work! And thanks also for the patch. Cavila 17:39, 19 August 2021 (UTC)

Extension Page
So, the great dismemberment of this page is afoot. But I don't think that merely moving its sections to subpages is enough. The structure of the documentation needs refactoring, whether it is stored in one page or split between several. Alexander Mashintalk 14:05, 30 September 2021 (UTC)

Alexander Mashintalk 16:35, 30 September 2021 (UTC)
 * What kind of things specifically do you recommend? Yaron Koren (talk) 16:17, 30 September 2021 (UTC)
 * I think, a more logical structure of the manual, regarding text formats, would be: fetching - preparsing - parsing - mapping - displaying data. At this point, everything about parsing text is told within the section about, although it also applies to and . The  is also unbalanced. The Lua part should be better integrated. Anyway, refactoring the manual will not be an easy task, due to the complexity and "non-linearity" of the subject, but I think it is necessary.


 * Oh, I forgot that #get_program_data also uses those same formats. Yes, it does seem to make sense to move all the information about the different formats into a new page, including related information like using XPath and so on. I don't know if that's what you refer to as preparsing or parsing, or both (I'm guessing both). Similarly, it probably makes sense to move the information about caching data into a new page - right now it's in the documentation for #get_web_data, for no strong reason. And it probably makes sense to create a page for mapping as well, i.e. the handling of the "data" parameter - I thought at first that this was too simple to justify a separate page, but I see now that quite a few features have been added to it. What do you mean by the #get_db_data part being unbalanced? Was it the incorrect header sizes? (I just fixed that.) I disagree with integrating Lua better - I think, at least for now, that the use of Lua is rare enough that it's fine to require anyone who wants to use Lua for External Data to learn about the parser functions first, even if they're not planning to use the parser functions. Yaron Koren (talk) 17:58, 30 September 2021 (UTC)


 * Well, I've now made a few changes to the documentation, based on this discussion; let me know what you think. Yaron Koren (talk) 21:49, 1 October 2021 (UTC)
 * The structure makes more sense now. Alexander Mashintalk 17:53, 2 October 2021 (UTC)

Call database function
It is possible or will ever be implemented a way to call a database function? I'm actually working on PostgreSQL. Alexander Mashintralk 12:11, 18 December 2021 (UTC) Alexander Mashintalk 15:11, 19 December 2021 (UTC)
 * The MediaWiki RDBMS library quotes all table names; the parentheses get inside double quotes and are treated by the database server as a part of the identifiers; so, currently, no.
 * If you update the extension now, you can define a prepared statement for your connection to the PostgreSQL database, as described here. It does not get quoted.

One missing XML element means no data is retrieved at all
I'm using something along these lines:

If the XML contains elements xxx, yyy and zzz, all works fine.

If the XML is missing anything, for example xxx, it'll say:
 * For the variables:  etc.
 * For the variables:  etc.

Is there any way to ensure that yyy and zzz are set even when xxx is missing from the XML source? gets rid of the first error message but has no effect on whether yyy and zzz are set. Thanks. Alexander Mashintalk 04:11, 15 August 2022 (UTC)
 * Try the solution described here.

Substitution of Output?
Hello! Is there a way to get the display parser functions to work with substitution?. Normally if I wanted to write the output of a parser function to an article I would include  or something. When I do this with #display_external_table or #for_external_table there is no output. I figure I'm either doing it wrong or it's not possible on account of how things naturally operate. Thanks - Lbillett (talk) 13:00, 10 February 2022 (UTC) Alexander Mashintalk 04:21, 15 August 2022 (UTC)
 * The documentation page you referred to says:
 * Substitution is a separate process that is performed before expansion of any non-substituted templates, parser functions, variables or parameters.
 * Therefore, your substitution of  will be carried out before   is executed, that is, when there is no data. You can try to safe substitute a template that includes both   and.
 * Ah. Thank you for the insight Alex. I'll give that a try. Lbillett (talk) 11:11, 11 October 2022 (UTC)

File path works fine, Directory not working
My issue seems exactly the same as this one mentioned in 2014. Using the file path works fine, but switching to the directory mode (for CSV files) fails completely with "Directory does not have file".

Screenshot: https://i.imgur.com/nRLRWxr.png

My page source:

LocalSettings.php:

$wgExternalDataSources['CSV']['path'] = 'CSV/';

(I've tried with and without the / after CSV)

weapons.csv, which is inside my CSV folder:

Name,Type,Ammunition,Capacity "Revolver","Weapon","Revolver Round","6"

Thoughts:

Alexander Mashintalk 02:31, 17 August 2022 (UTC)
 * Could XAMPP be the culprit? Do I need some obscure PHP extension enabled to make this work? I had to enable "extension=intl" in my php.ini just to get Mediawiki installed. GregariousJB (talk) 03:39, 4 March 2022 (UTC)
 * Upgrade the extension and try again.

Database configuration style changed from 2.x to 3.x
After much digging, I noticed that of Extension:External Data/Databases removed the information about how to configure External Data 2.x. There's no mention of this in the revision history (edit summaries), or the article body itself. It was a long shot that I found any mention of the old style at all, in this Git commit.

Was the removal of the old-style config from /Databases just accidental, or is External Data 2.x now non grata and you don't want to clutter up the documentation with any mention of it?

I'd be happy to restore the missing examples, something like

"Note: Version 2.x of External Data used a configuration style like $edgDBName['ID'] = '…' …"

unless there's a reason not to. I'm stuck at 2.x for now (for reasons), and there must be other users in the same predicament. --Ernstkm (talk) 21:05, 9 March 2022 (UTC)


 * Sorry for not responding before. It's tricky to try handle different versions in the same documentation, especially as the documentation gets more complex. Yes, do feel free to add notes about the old ways of doing things - I think that would be helpful. Yaron Koren (talk) 15:56, 28 March 2022 (UTC)

data from mysql not get refreshed on a page
Hi,

Mediawiki 1.35.2

ExternalData 3.0 (70116a9)

Mariadb / mysql database source

After i modify the content in the database it does not get refreshed on the wiki page. new content is presented ONLY after i re-save the page (without any changes on it). Is it expected?

I would imagine the current content of the database is whown on a page every time it is opened or refreshed.

2A01:110F:E4E:6A00:482A:A928:A8EF:E396 16:06, 21 March 2022 (UTC)


 * My experience with MediaWiki so far is that it's fairly aggressive about caching (that is, it is conservative about re-rendering pages if it doesn't have to). I assume this stems from its primary duty as a worldwide encyclopedia, where page rendering performance is a big concern. So what you're describing sounds like normal MediaWiki behavior to me. It will eventually update the document, just not when you were expecting.


 * As a workaround
 * if you are already using Extension:Cargo on your wiki, you should see a "Purge cache" option in the "More" menu/tab at the top
 * it may be called something else in your language
 * Semantic MediaWiki provides the same, but it's called "Refresh" in the menu
 * you can also add  to the end of the URL in the address bar and press Enter.
 * In any case, this "purge" action should re-render the view(s) of your data, updating the page for all visitors to your wiki. See Manual:Purge for more information. Hope that helps! —Ernstkm (talk) 17:44, 21 March 2022 (UTC)

Getting data from oracle db
Hi,

My Setup:


 * MediaWiki 1.37.1
 * PHP 7.4.28 (fpm-fcgi)
 * MariaDB 10.3.34-MariaDB-0ubuntu0.20.04.1
 * External Data 2.4.1

I try to get data from an oracle database with the get_db_data function. Now I'm very confused by the explanation in the configuration description.

There it's described to put all the values in one array (Extension:External Data/Databases) but if I do so there are errors that values like $edgDBServer, $edgDBServerType and $edgDBName are missing. So I configured them as singles with $edgDBServerType['mydb'] = 'oracle', ... But now I always get the error

Error: Unknown database type oracle

My question is now if there is the type oracle supported and how the configuration must look like?

I have also installed the oracle instaclient_21_5 which is working with my oracleDB.

Another question would be if there is planned to provide some mariaDB support?

Best regards Role-end (talk) 15:18, 24 March 2022 (UTC)


 * I'm pretty sure that querying Oracle no longer works with MediaWiki 1.34 and higher - sorry that the documentation is not clearer on this. You can connect to MariaDB, though - just use "mysql" as the DB type. Yaron Koren (talk) 19:44, 24 March 2022 (UTC)
 * Hi,
 * Thank you very much for your quick help.
 * I've tried with mariaDB and it works perfect. This is even a better solution than oracle because we already have the mariaDB so there is no exta effort.
 * Best Regards Role-end (talk) 08:07, 28 March 2022 (UTC)

syntax error, unexpected '?', expecting variable (T_VARIABLE) in ...bootstrap.php on line 29
When I enable the extension, trying to access any page gives me just this error:

Parse error: syntax error, unexpected '?', expecting variable (T_VARIABLE) in /var/lib/mediawiki/extensions/ExternalData/vendor/symfony/polyfill-php80/bootstrap.php on line 29

This is line 29 in that bootstrap.php file:

I'm running Mediawiki 1.31.10 on Debian 10 with Apache and PostgreSQL, and with PHP version 7.4.3. Albert25 (talk) 14:52, 1 April 2022 (UTC)


 * Are you sure you're running PHP 7.4.3? That "nullable types" feature seems to have been added to PHP in version 7.1. Yaron Koren (talk) 15:15, 1 April 2022 (UTC)
 * Right! Thank you! While php on the command line gave me version 7.4.3, Apache was actually still using 7.0! So the solution was:
 * Albert25 (talk) 15:30, 1 April 2022 (UTC)
 * Albert25 (talk) 15:30, 1 April 2022 (UTC)
 * Albert25 (talk) 15:30, 1 April 2022 (UTC)
 * Albert25 (talk) 15:30, 1 April 2022 (UTC)


 * Ah! That makes sense. Yaron Koren (talk) 15:36, 1 April 2022 (UTC)

#get_file_data : examples? Cannot get it to work.
I'm trying to use, but cannot make it work. Are there some working examples somewhere?

In "LocalSettings.php" I have:

wfLoadExtension( 'ExternalData' ); $wgExternalDataSources['mypath']['path'] = '/docs/www-wiki/';

In a page, I have

If I try, it is ignored, I just get it back as if it were normal text.

If I try, I get an error saying "no local variable "mycnt" has been set".

Before that, I had tried, in the hope of getting just the raw content of the file, but got an error saying "No "data" parameter specified".

I don't know what else to try. The documentation in https://www.mediawiki.org/wiki/Extension:External_Data/Local_files seems very incomplete for someone new to this extension. Albert25 (talk) 18:09, 1 April 2022 (UTC)


 * I finally got it working with a change in "LocalSettings.php":
 * And in the page, to display the full file content as text:
 * Or as CSV in a table:
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * And in the page, to display the full file content as text:
 * Or as CSV in a table:
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Or as CSV in a table:
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Or as CSV in a table:
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Or as CSV in a table:
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)
 * Albert25 (talk) 09:23, 4 April 2022 (UTC)

No luck with start line
Thanks for adding in the start line/end line feature! This would be really helpful for relatively bulky datasets (say, 50,000+ records) that cannot be loaded in their entirety. I tried it out with #get_web_page and csv with header, but while I can set end line to limit the number of results, any value for start line higher than 1 results in no data at all. Is there something I overlooked? Cavila 07:18, 21 April 2022 (UTC)


 * Cannot reproduce. Can you give an example? Alexander Mashintalk 03:42, 16 August 2022 (UTC)

Creating a page per entry in a DB table
I'm new to the External Data extension. It looks like it's basically what I want for a certain use case. I am wondering, though: As part of what I'm doing, I want a Wiki page per entry in a certain DB table. New entries can be added to the DB table at any time.

Am I correct in thinking that the best (perhaps only) way to do this would be externally (to External Data)? For example, having an external program that periodically runs, checks the DB for new entries, and creates new Wiki pages for any that are found? Or is there some way to do this from within External Data?

Thanks. Rwv37 (talk) 18:27, 18 July 2022 (UTC)


 * You might be able to do it with a combination of External Data and the #formredlink parser function from Page Forms, with the "create page" parameter set - in other words, have #display_external_table generate one big page of links, and then Page Forms would create new page(s) every time a user (or script) went to that big page and new links were there. How workable this would be, though, I'm not sure, especially if there were thousands or more entries in the DB table. A custom script may be the safer option. Yaron Koren (talk) 19:11, 18 July 2022 (UTC)

Fix for @ in node names with JSONPath
I recently ran into an issue similar to https://github.com/json-path/JsonPath/issues/798. If you select a node beginning with an at-sign, Eternal Data will not exit gracefully. From that bug rapport, it seems like a fix is available in that it now lets you escape that character (Java only I guess), although no commit is referenced. Do you think this is fixable? Cavila 11:48, 13 August 2022 (UTC)

Alexander Mashintalk 04:18, 16 August 2022 (UTC)
 * This works as expected, returning "@@@AAA":
 * Other variants display an error message about either invalid JSONpath or absent value, also as expected. I was unable to cause an unhandled exception or error 502. Please give an example, if you can.
 * Other variants display an error message about either invalid JSONpath or absent value, also as expected. I was unable to cause an unhandled exception or error 502. Please give an example, if you can.

Alexander Mashintalk 03:13, 17 August 2022 (UTC)
 * Dear Alex, thanks for replying. In your example, you're using the bracket notation, whereas I was using the other convention, the dot notation. Converting to bracket notation helped to overcome this particular limitation with the at-sign and fix the issue for me. Cavila 14:04, 16 August 2022 (UTC)
 * I don't think that the dot notation should be expected to work with '@'. This character has a special meaning in JSONpath and is not an allowed character in JavaScript variable names, although, I presume, can be used in property names, but only using brackets and quotes.
 * It is actually rather common in JSON-LD, but of course, dot notation is just an alternative style that is a little faster to write but comes with some limitations when you're trying to query the document. Cavila 18:03, 17 August 2022 (UTC)

Error with MSSQL
Hello, few days ago I configured ExternalData extension and since then I'm constantly trying to connect to database. I have odbc and driver successfully installed. The error emerges after connecting to database when I try to write a clauses.

Can anyone point me right direction to resolve this problem? ChrisOlx (talk) 09:58, 22 August 2022 (UTC)

Alexander Mashintalk 13:15, 22 August 2022 (UTC)
 * ok I found out solution: I had to add |join on= clause and it worked ChrisOlx (talk) 12:07, 22 August 2022 (UTC)
 * Please post the wikicode, even though your database may be not publically available. The error has to be handled gracefully.

Security of mw.ext.externalData.getDbData Lua function
Is the Lua call "mw.ext.externalData.getDbData" secured in the same manner as the corresponding "get_db_data" call? Specifically, with respect to prepared statements:

If prepared statements are defined in the LocalSettings.php file, the documentation for get_db_data says that it will not allow arbitrary queries to the DB, so only the specifically defined prepared statements will be allowed. Is the same true for mw.ext.externalData.getDbData within Lua code? I don't see anything in the docs explicitly mentioning it one way or the other.

I know I could just try it and see, but I don't want to rely on the assumption that if my test shows that this sort of protection seems to exist, then this sort of protection really does intentionally exist (rather than just happening to coincidentally exist in this particular version of code, with the particular setup that I used, and so on).

Thanks, Rwv37 (talk) 07:13, 25 September 2022 (UTC) Alexander Mashintalk 09:27, 25 September 2022 (UTC)
 * Yes. It is handled by the same PHP code and is going to stay so.
 * Great. Thank you! -Rwv37 (talk) 09:40, 25 September 2022 (UTC)

Issue fetching data from other wiki via Special:Ask page
We've been using ED for many years now, and the newest update is causing fetching issues. In trying to switch to the new Standalone mode, only the first paragraph of the semantic value of a property of type text is being rendered. Here's the deets:

Fetching wiki: External Data: 3.2

Host wiki:

The data on the host wiki that is stored in Property:Description via the page "A Comparison of the Concepts of Buddha-Nature and Dao-Nature of Medieval China" is: Abstract

This thesis, a comparison of the concepts of buddha-nature and dao-nature in the medieval period (from the 5th to the 10th centuries) of China, presents a historical investigation of the formation of the idea that insentient things are able to possess buddha-nature in medieval Chinese Mahāyāna Buddhism. In Chinese Mahāyāna Buddhism, the concept of buddha-nature was originally defined as a potential possessed by sentient beings that enabled them to achieve buddhahood. From the 6th century, the concept was reinterpreted within the Chinese Buddhist tradition so that insentient things were also able to possess buddha-nature. Recent scholarship has pointed out that the idea of insentient things having buddha-nature is a combination of Buddhist and Daoist ideas based on the concept of the all-pervading Dao found in the Zhuangzi 莊子. In this sense, buddha-nature seems to be interpreted as equivalent with the Dao of Daoism. My project suggests that the reinterpretation of buddha-nature in association with the insentient realm should be elucidated in a more nuanced way than the idea of all-pervasiveness of the Dao. A historical, doctrinal investigation of the intellectual formation of the concept of buddha-nature in Chinese Mahāyāna Buddhism demonstrates a new interpretation of buddha-nature in the context of insentient things having buddha-nature. Further, through a historical investigation of intellectual exchange between Buddhism and Daoism, some evidence provided in this project illustrates that the idea of insentient things having dao-nature in Daoism was not inherited from Buddhism, but drawn from Daoist tradition. This new perspective is different from that of some contemporary scholars who have claimed that the idea of insentient things having dao-nature was borrowed from Chinese Buddhism. A chronological investigation of the discussion of nature in Chinese thought demonstrates that the idea of insentient things having buddha-nature incorporates earlier Daoist traditions found in Arcane Study. When using the following on the fetching wiki:

I can only get to display:

Abstract

This is consistent with other values. Only the first paragraph is actually fetched. The csv contains the right data. The whole thing is there, in the csv, or JSON which i've also tried. But the function doesn't pull the whole thing.

Thanks for any suggestions or help! Jeremi Plazas (talk) 17:27, 24 October 2022 (UTC) Alexander Mashintalk 02:00, 9 January 2023 (UTC) Alexander Mashintalk 02:05, 10 January 2023 (UTC)
 * Please give the date and commit of External Data from Special:Version of the client wiki. If there is a regression in External Data, it was introduced after the 5th of October, 2022. See this example of correctly working External Data (3.2 (d77e75a) 00:06, October 5th 2022).
 * Update the extension and try again.

New 3.2 version causing issues when fetching complex values from other wiki
Hi there,

We've been using the External Data extension for years now and managed to make it work really well for us to fetch data from our private wikis onto more public wiki sites.

The new version 3.2 is causing some issues for us. Here is the gist:

Simple values (Semantic property of type text) come through fine with the new method:

Result:

Authors of English Works

More complex values (Semantic property of type text) however do not (they used to just fine), the function doesn't seem to fire at all. For the value:

* "The Guhyagarbhatantra and its XIVth Century Commentary Phyogs-bcu mun-sel." PhD Thesis, SOAS, University of London (3 vols), 1987. See [1]
 * Dudjom Rinpoche's The Nyingma School of Tibetan Buddhism: Its Fundamentals and History. Wisdom Publications, Boston. 1st edition (2 vols), 1991; 2nd edition (1 vol), 2002; ISBN 0861711998. See Wisdom Books
 * Tibetan Medical Paintings. Serindia Publications, London (2 Vols); ISBN 0-906026-26-1. 1992. See Serindia
 * Tibet Handbook. Footprint Handbooks, Bath. 1st edition, 1996; 2nd edition 1999; 3rd edition 2004; ISBN 1900949334 See Footprintbooks
 * Bhutan Handbook. Footprint Handbooks, Bath. 1st edition, 2004. See Footprintbooks
 * Tibetan Elemental Divination Paintings, Eskenazi & Fogg, London. 2001. See Whiteberyl
 * Tibetan Elemental Divination Paintings: Illuminated Manuscript from the White Beryl of Sangs-Rgyas Rgya-Mtsho. Holberton, Paul Publishing, 2008. ISBN 9780953994106 ISBN 0953994104* An Encyclopaedic Tibetan-English Dictionary, (Nationalities Publishing House/ SOAS, Beijing. Vol. 1, 2001). Enquiries: [mailto:tedic@hotmail.com tedic@hotmail.com]
 * "A Rare Series of Tibetan Banners." In Pearls of the Orient, Serindia, 2003. See Serindia* The Tibetan Book of the Dead: First Complete English Translation, Penguin Classics, 2005.
 * Jokhang: Tibet's Most Sacred Buddhist Temple, Thames & Hudson, 2010
 * The Guhyagarbha Tantra: Dispelling the Darkness of the Ten Directions. (Snowlion)
 * An Encyclopaedic Tibetan-English Dictionary. (Nationalities Publishing House/ SOAS, Beijing. Vols. 2-3.

Using the function:

We get an error as a result:

Error: no local variable "publications" has been set.

Multi-paragraph text values also don't come through properly, only the first paragraph is displayed. For the value:

Gyurme Dorje (1950 – 5 February 2020) was a Scottish Tibetologist and writer. He was born in Edinburgh, where he studied classics (Latin and Greek) at George Watson's College and developed an early interest in Buddhist philosophy. He held a PhD in Tibetan Literature (SOAS) and an MA in Sanskrit with Oriental Studies (Edinburgh). In the 1970s he spent a decade living in Tibetan communities in India and Nepal where he received extensive teachings from Kangyur Rinpoche, Dudjom Rinpoche, Chatral Rinpoche, and Dilgo Khyentse Rinpoche. In 1971 Dudjom Rinpoche encouraged him to begin translating his recently completed History of the Nyingma School (རྙིང་མའི་སྟན་པའི་ཆོས་འབྱུང་) and in 1980 his Fundamentals of the Nyingma School (བསྟན་པའི་རྣམ་གཞག) - together this was an undertaking that was to take twenty years, only reaching completion in 1991. In the 1980s Gyurme returned to the UK and in 1987 completed his 3 volume doctoral dissertation on the Guhyagarbhatantra and Longchenpa's commentary on this text at the School of Oriental and African Studies (SOAS) at the University of London.

From 1991 to 1996 Gyurme held research fellowships at London University, where he worked with Alak Zenkar Rinpoche on translating (with corrections) the content of the Great Sanskrit Tibetan Chinese Dictionary to create the three volume Encyclopaedic Tibetan-English Dictionary. From 2007 until his death he worked on many translation projects, primarily as a Tsadra Foundation grantee. He has written, edited, translated and contributed to numerous important books on Tibetan religion and culture including The Nyingma School of Tibetan Buddhism: Its Fundamentals and History (2 vols.) (Wisdom, 1991), Tibetan Medical Paintings ( 2 vols.) (Serindia, 1992), The Tibet Handbook (Footprint, 1996), the first complete translation of the Tibetan Book of the Dead, and A Handbook of Tibetan Culture (Shambhala, 1994). (Source Accessed Jul 14, 2020)

Using:

We get:

Gyurme Dorje (1950 – 5 February 2020) was a Scottish Tibetologist and writer. He was born in Edinburgh, where he studied classics (Latin and Greek) at George Watson's College and developed an early interest in Buddhist philosophy. He held a PhD in Tibetan Literature (SOAS) and an MA in Sanskrit with Oriental Studies (Edinburgh). In the 1970s he spent a decade living in Tibetan communities in India and Nepal where he received extensive teachings from Kangyur Rinpoche, Dudjom Rinpoche, Chatral Rinpoche, and Dilgo Khyentse Rinpoche. In 1971 Dudjom Rinpoche encouraged him to begin translating his recently completed History of the Nyingma School (རྙིང་མའི་སྟན་པའི་ཆོས་འབྱུང་) and in 1980 his Fundamentals of the Nyingma School (བསྟན་པའི་རྣམ་གཞག) - together this was an undertaking that was to take twenty years, only reaching completion in 1991. In the 1980s Gyurme returned to the UK and in 1987 completed his 3 volume doctoral dissertation on the Guhyagarbhatantra and Longchenpa's commentary on this text at the School of Oriental and African Studies (SOAS) at the University of London.

For the full list of values you can fetch from our commons site, go [https://commons-new.tsadra.org/index.php/Special:Browse/:Dorje,-5FG. here].

We have even tried using curl to fetch the data and it all comes through fine:


 * curl 'https://commons.tsadra.org/index.php?title=Special%3AAsk&q=%5B%5BDorje%2C+G.%5D%5D&po=PersonTypeRaw&p%5Bformat%5D=csv'
 * curl 'https://commons.tsadra.org/index.php?title=Special%3AAsk&q=%5B%5BDorje%2C+G.%5D%5D&po=Bio&p%5Bformat%5D=csv'
 * curl 'https://commons.tsadra.org/index.php?title=Special%3AAsk&q=%5B%5BDorje%2C+G.%5D%5D&po=Publications&p%5Bformat%5D=csv'

Could it be an issue with the extension's code?

The wiki from which we are fetching data is:

with

The wiki onto which we are bringing the data is:

with

For a recap with live functions, see: https://buddhanature-new.tsadra.org/index.php/User:JeremiP/ExternalData_New_Method

Thanks for any suggestions. Would really appreciate the help. Jeremi Plazas (talk) 18:46, 9 December 2022 (UTC) Alexander Mashintalk 03:12, 9 January 2023 (UTC) Alexander Mashintalk 02:05, 10 January 2023 (UTC) Also, you don't need to use, if you work in standalone mode. Alexander Mashintalk 05:40, 18 January 2023 (UTC)
 * The problem seems to be the same as in previous issue: the fact that currently, the extension is unable to parse CSV with line breaks in quoted fields. This is a regression introduced in August 2022. I have filed the bug T326514 and will try to fix it soon
 * Update the extension and try again.
 * Thank you so much for looking at this, and the updates are working for us with the function that i shared! Thank you again. In the spirit of troubleshooting, there is one last function we were using in the past that doesn't seem to work anymore still. That is using the new method with a wiki's api and in JSON format as follows:
 * We just get:
 * Error: no local variable "publications" has been set.
 * Thank you again so much for your help on this, it's making a huge difference for us.
 * Jeremi Plazas (talk) 17:33, 17 January 2023 (UTC)
 * This should work: . Without JsonPath,   cannot be reached, as it is nested rather deeply in the JSON.
 * This should work: . Without JsonPath,   cannot be reached, as it is nested rather deeply in the JSON.
 * Thank you so much! that worked.
 * Jeremi Plazas (talk) 17:53, 19 January 2023 (UTC)

Prepared statements not working
Good evening! I'm trying to set up prepared statements for a database query, but I'm struggling to make it work. While querying data from the external database does work without prepared statements (and using #external_value), queries with the prepared statement return an error message. Is someone able to help me make this work? This is the software and configurations that I've tried with:

I've also tried the query parameter without quotes as per the instructions. But that returns: "TypeError: mysqli::prepare: Argument #1 ($query) must be of type string, array given".

What am I doing wrong?

Cheers, Lucas Devaroo (talk) 22:00, 13 December 2022 (UTC)
 * Please make sure that your External Data is as per commit 9ac9342 of November 22nd, which introduced parameter types per prepared statement. Or, since the prepared statement has no parametres, change the data source settings to:

Do not use quotes in the  parametre anyway. Alexander Mashintalk 03:39, 9 January 2023 (UTC)

accessing the internal variable (__json)
I need some help. Background: I'm working on an extension that uses a webcall to retrieve a JSON object and converts it into concise html code. This extension now is working properly, but lacks caching. This JSON object is complex: containing multiple nested arrays with varying number of elements and it took me a while to program it in php.

My current line of thought is to transfer the entire json object (that is stored in the internal variable __json) to my extension.

Question 1) how does one access the internal variables?? The code below doesn't work.

Question 2) do you have any comments / suggestions on my approach?


 * first bullit:
 * second bullit:

Harro Kremer (talk) 21:20, 3 January 2023 (UTC) Alexander Mashintalk 03:46, 9 January 2023 (UTC)
 * Since  is not a string but a Lua table, it is only accessible from Lua (Scribunto is needed) and not from wikitext:

Using External Data in a Template
I want to fill a custom Infobox Template with data from ExternalData. The data will reside in a CSV file matching the non-namespace part of the page importing.

However, this tries to fetch data from a CSV file named "myinfobox" instead What is a better way to achieve this?


 * Is the template called "infobox myinfobox", or just "myinfobox"? And do you see the problem on the "mypage" page, or right on the template page? Yaron Koren (talk) 00:34, 10 January 2023 (UTC)
 * my template page is called "Vorlage:Infobox_VM". The problem occurs on "mypage" as the template page doesn’t really have any visible content nor has it the CSV providing information. "mypage" tries to read from a csv called Vorlage:Infobox_VM.csv instead if mypage.csv.

Alexander Mashintalk 02:11, 10 January 2023 (UTC)
 * This looks like a strange way to invoke a template: . Why not just  ? And, as said above, fetching data from   is what to be expected on the template page itself, unless the code is wrapped with.


 * yes you’re right, i tried being verbose for clarity. in my page i have only and that seems to work well enough. I'm not really concerned what csv is being read on the template page itself, as that page isn’t really supposed to be looked at. I want "mypage" to look at mypage.csv through the template. that’s what doesn’t work.


 * What are the settings for the data source  in   ?
 * that path and all the files in it are world readable. that stuff works in principle. i am able to pull info from csv. just not via the page name in the template.
 * that path and all the files in it are world readable. that stuff works in principle. i am able to pull info from csv. just not via the page name in the template.

Alexander Mashintalk 10:59, 14 January 2023 (UTC)
 * Could not reproduce the issue at my MediaWiki installation. Looks like there you have some sophisticated wiki code that substituted  too early on.
 * I solved the problem this way:  on "mypage" i call  and on the template page i use that variable to find the right file. 24.134.95.253 15:51, 17 January 2023 (UTC)

Slightly different csv data files, only one works
I have 2 48 line csv files, < 10k in size with a minor difference, but only one can be read in ExternalData in 1.39

Simplified case https://johnbray.org.uk/expounder/Extdataproblem1 uses

get_web_data: url=https://files.johnbray.org.uk/Documents/Expounder/Q/532/9928/datagood and then get_web_data: url=https://files.johnbray.org.uk/Documents/Expounder/Q/532/9928/databad

checking that has something from a line of the csv file. datagood works, but databad does not. The difference between the files is a few characters on one line, and the good file is actually longer than the bad

< "item",+1996-04-05T00:00:00Z,+1996-04-08T00:00:00Z,"","in Heathrow wit h, , , , , ,","","","","","","",51.4673,-0.4529

> "item",+1996-04-05T00:00:00Z,+1996-04-08T00:00:00Z,"","in Heathrow wit h, , , , AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBA,","","","","","","",51.4673,-0.4529

Both files are < 10k in size, 48 lines, and the good file is actually longer than the bad

wc datagood databad 48  338  9245 datagood 48  341  9180 databad Vicarage (talk) 14:05, 13 January 2023 (UTC) Alexander Mashintalk 10:45, 14 January 2023 (UTC)
 * Neither of the CSV files is formed perfectly from PHP's point of view. The problems start at the line 7 (one-based). For  this causes the automatically recognised delimiter to be   rather than  . You can overcome this by adding.
 * Thanks for the quick response. That cured the problem, both for my trivial and full cases. Vicarage (talk) 11:00, 16 January 2023 (UTC)

What is the best practice to fetch many values (>300) from the same place?
Is it better to use the legacy method, like #get_web_data, and fetch all the values at once, then display using #external_value; or is it better to use the new method, #external_value with source parameter, 300 times? What performance consideration might there be?

Jeremi Plazas (talk) 17:51, 17 January 2023 (UTC) Alexander Mashintalk 03:22, 18 January 2023 (UTC)
 * If you use caching, the difference is not that big. Using  will save cache lookups, but the legacy mode will stop working once MediaWiki is upgraded to use Parsoid, since it does not guarantee parsing order. The optimal solution is to handle data fetching and display with one Lua function, where you can save the fetched data into a variable and later display it.
 * Thanks, we'll look into Lua. We do have caching setup so the standalone method might be fine, now that you've helped us iron out the kinks. Thanks again for the help! Jeremi Plazas (talk) 17:54, 19 January 2023 (UTC)

Some JSONPATH doesn’t retrieve results
I have a JSON file with Information about network interfaces in it. I am retrieving this information with get_file_data and two jsonpath instructions. However, only one of them seems to be executed/filled with data.

This is my json: and this is the template code i use to retrieve the data: i call this template in the following fashion: where netnames is just an array, usually only one entry like  and filename points to the correct file

You could try a regular expression:. Also, you can get the bulleted list without a template: Alexander Mashintalk 06:23, 19 January 2023 (UTC)
 * If  is an array, I don't know how transcluding it within single quotes in a JsonPath query could work. Neither , nor   is a working JsonPath.   would be. I would suggest replacing   with  , but this does not seem to be implemented.

Strange bug parsing CSV with pipe character
I have a field called  which is called in. All of the values have at least one pipe character, and the page cuts off everything before and including the first pipe character. But the strange thing is that it only happens if is located in a specific place.

The page can be seen at https://comprehensibleinputwiki.org/wiki/Mandarin_Chinese/Videos and the external data is at https://comprehensibleinputwiki.org/wiki/Data:Mandarin_Chinese/Videos. If you look at the wiki source of the first link, I have twice, one of them in a hidden div. If you view the source of the page, the first one is missing part of the value, while the hidden one is complete. Dimpizzy (talk) 20:08, 22 January 2023 (UTC) Alexander Mashintalk 02:01, 23 January 2023 (UTC)
 * Add  to.