Extension:External Data/Web pages

The External Data extension can be used to extract data from pages or documents on the web in a variety of formats, including CSV, JSON, XML, HTML, GFF and INI. This retrieval can either be done directly, or, if necessary, using the SOAP protocol.

#get_web_data - CSV, JSON, XML, etc.
To get data from a web page that holds structured data, call the parser function #get_web_data. It can take the following syntax:

The parameters format, delimiter, regex, start line, end line, header lines, footer lines, use xpath, default xmlns prefix, use jsonpath, json offset and allow trailing commas all relate to parsing specific data formats; for information on these parameters, see.

An explanation of the other parameters:


 * url - sets the full URL of the file being retrieved.
 * data - holds the "mappings" that connect local variable names to external variable names. Each mapping (of the form  ) is separated by a comma. External variable names are the names of the values in the file (in the case of a header-less CSV file, the names are simply the indexes of the values: 1, 2, 3, etc.), and local variable names are the names that are later passed in to.
 * means that all existing external variables are to be mapped to internal ones. Not applicable to formats and parser functions in which external data is defined based on data values.
 * Several special external variables can be set:
 * containing complete XML structure, for  format; can be used in Lua,
 * containing complete JSON structure, for  format; can be used in Lua,
 * ,,   and   containing information about text cutout with start line, etc.,
 * containg the complete text, for  format,
 * containg the time (Unix timestamp) that data was fetched on,
 * -, if data could not be fetched, ans stale cache was used;   otherwise,
 * - number of attempts needed to fetch the data.
 * filters - sets filtering on the set of rows being returned. You can set any number of filters, separated by commas; each filter sets a specific value for a specific external variable. It is not necessary to use any filters; most APIs, it is expected, will provide their own filtering ability through the URL's query string.
 * post data - an optional parameter that lets you send some set of data to the URL via POST, instead of via the query string.
 * cache seconds - an optional parameter that sets the number of seconds that the values from this call should be cached; if it is less than, if there is any, the latter will apply; and if the effective cache expiration time is zero, caching is forbidden.
 * use stale cache - an optional parameter that allows this function to use an expired cache entry if it cannot retrieve the real data.
 * suppress error - an optional parameter that prevents any error message from getting displayed if there is a problem retrieving the data.

More than one call can be used in a page. If this happens, though, make sure that every local variable name is unique.

For data from XML sources, the variable names are determined by both tag and attribute names. For example, given the following XML text:

the variable type would have the value Apple, and the variable color would have the value red.

Similarly, the following XML text would be interpreted as a table of values defining two variables named type and color :

A CSV file must be literally a CSV file, i.e., delimited by commas. A call for a headerless CSV file might look:



while a call to CSV with a header row might look like:



where the header contains, which is retrieved as   in the wiki.

You can also set caching to be done on the data retrieved, and string replacement to hide API keys; see the "Usage" section, below, for how to do both of those.

Getting data from a non-API text file
If the data you wish to access is on a MediaWiki page or in an uploaded file, you can use the above methods to retrieve the data assuming the page or file only contains data in one of the supported formats:


 * for data on a wiki page, use " " as part of the URL;
 * for data in an uploaded file, use the full path.

If the MediaWiki page with the data is on the same wiki, it is best to use the fullurl: parser function, e.g.



Similarly, for uploaded files, you can use the filepath: function, e.g.

For wiki pages that have additional information, the External Data extension provides a way to create an API of your own, at least for CSV data. To get this working, first place the data you want accessed in its own wiki page, in CSV format, with the headers as the top row of data (see here for an example). Then, the special page 'GetData' will provide an "instant API" for accessing either certain rows of that data, or the entire table. By adding "field-name=value" to the URL, you can limit the set of rows returned.

A URL for the 'GetData' page can then be used in a call to #get_web_data, just as any other data URL would be; the data will be returned as a CSV file with a header row, so the 'format' parameter of #get_web_data should be set to 'CSV with header'. See here for an example of such data being retrieved and displayed using #get_web_data and #for_external_table. In this way, you can use any table-based data within your wiki without the need for custom programming.

Data caching
You can configure External Data to cache the data contained in the URLs that it accesses, both to speed up retrieval of values and to reduce the load on the system whose data is being accessed. To do this, you can run the SQL contained in the extension file 'ExternalData.sql' in your database, which will create the table 'ed_url_cache', then add the following to your LocalSettings.php file, after the inclusion of External Data:

You should also add a line like the following, to set the expiration time of the cache, in seconds; this example line will cache the data for a week:

By default, if data cannot be retrieved, and a cache table exists, #get_web_data will use the cached value for this data even if the cache has already expired. To disallow this, add the following to LocalSettings.php:

String replacement in URLs
One or more of the URLs you use may contain a string that you would prefer to keep secret, like an API key. If that's the case, you can use the array $edgStringReplacements to specify a dummy string you can use in its place. For instance, let's say you want to access the URL "http://worlddata.com/api?country=Guatemala&key=123abcd", but you don't want anyone to know your API key. You can add the following to your LocalSettings.php file, after the inclusion of External Data:

Then, in your call to #get_web_data, you can replace the real URL with: "http://worlddata.com/api?country=Guatemala&key=WORLDDATA_KEY".

Whitelist for URLs
You can create a "whitelist" for URLs accessed by #get_web_data : in other words, a list of domains, that only URLs from those domains can be accessed. If you are using string replacements in order to hide secret keys, it is highly recommended that you create such a whitelist, in order to prevent users from finding out those keys by including them in a URL within a domain that they control.

To create a whitelist with one domain, add the following to LocalSettings.php:

To create a whitelist with multiple domains, add something like the following instead:

HTTP options
By default, #get_web_data allows for HTTPS-based wikis to access plain HTTP URLs, and vice versa, without the need for certificates (see Transport Layer Security on Wikipedia for a full explanation). If you want to require the presence of a certificate, add the following to LocalSettings.php :

Additionally, the global variable $edgHTTPOptions lets you set a number of other HTTP-related settings. It is an array that can take in any of the following keys:


 * - how many seconds to wait for a response from the server (default is 'default', which corresponds to the value of $wgHTTPTimeout, which by default is 25)
 * - whether to verify the SSL certificate, if retrieving an HTTPS URL (default is false)
 * - whether to retrieve another URL if the specified URL redirects to it (default is false)

So, for instance, if you want to verify the SSL certificate of any URL being accessed by #get_web_data, you would add the following to LocalSettings.php :

#get_soap_data - web data via SOAP
The parser function #get_soap_data, similarly to #get_web_data, lets you get data from a URL, but here using the SOAP protocol. It is called in the following way:

All of the LocalSettings.php settings that can be applied for #get_web_data can also be applied for #get_soap_data: $edgCacheTable, $edgCacheExpireTime, $edgStringReplacements, $edgAllowExternalDataFrom and $edgAllowSSL.

All of the parsing-related parameters that #get_web_data supports (format, delimiter, use xpath, etc.) can be used for #get_soap_data as well; see.