Extension talk:Data Transfer

Error while uploading

Hi, My wiki uploads a file through the datatransfer extension. However, it always gives me the same error:

Error: the column 0 header, 'Gemeente[GemeenteCode]', must be either 'Title', 'Free Text' or of the form 'template_name[field_name]'

I do have a template Template:Gemeente that has a field GemeenteCode. (see code). So where can it go wrong?

<> <noinclude> This is the "Gemeente" template. It should be called in the following format: <pre> {{Gemeente |Naam= |ProvincieNaam= |GemeenteCode= |Inwoners= |InwonerKlasse= |WGRNaam= |Twitteradres= |URL= }} </pre> Edit the page to see the template text. </noinclude><includeonly> {| class="wikitable" ! align="left" |Naam Gemeente |align="left" | [[Naam::{{{Naam|}}}]] |- ! align="left" |Naam Provincie | align="left" |[[ProvincieNaam::{{{ProvincieNaam|}}}]] |- ! align="left" |GemeenteCode | align="left" |[[GemeenteCode::{{{GemeenteCode|}}}]] |- ! align="left" |Aantal Inwoners | align="left" |[[Inwoneraantal::{{{Inwoners|}}}]] |- ! align="left" |Inwoner Klasse | align="left" |[[GemeenteGrootteKlasse::{{{InwonerKlasse|}}}]] |- ! align="left" |Regionaam | align="left" |[[WGRNaam::{{{WGRNaam|}}}]] |} ==Nieuws== {{#ask: [[Category:Nieuws]] [[Gemeente::{{SUBPAGENAME}}]] | sort=date | order=descending | format=broadtable }} {{#widget:Twitter Search |query={{SUBPAGENAME}} |title={{SUBPAGENAME}} |caption=What people say about {{SUBPAGENAME}} }} {{#widget:Twitter |user={{Twitteradres}} |count=5 }} [[Category:Gemeente]] </includeonly> <>

Error after installation

I have attempted to install the Data Transfer Extension, but when trying to go to the View XML page I get the following error:
"Warning: Invalid argument supplied for foreach() in C:\mediawiki\extensions\DataTransfer\specials\DT_ViewXML.php on line 189". This error is printed 7 times. The page looks fine below the errors. Is there an easy fix? Jascraig 12:17, 23 April 2008 (UTC)

Hi, thanks for the bug report. I think I know the problem - you have some pages that don't belong to any category, and Data Transfer wasn't expecting those. There's an easy fix, which I'll add to the next version of the extension. Yaron Koren 14:28, 23 April 2008 (UTC)

Yaron, Thanks, that fixed the problem. Now I'm having problems when I click Submit on the View XML page, instead of getting the XML I expect, I'm redirected back to my Wiki Main Page. I am going to try and diagnose this locally, but if you have any ideas what could be causing this I would appreciate the information. Thanks for the quick response. Jascraig 16:21, 23 April 2008 (UTC)

About dt_xml_* messages

I'm a translater in translatewiki.net. dt_xml_* software messages such as dt_xml_namespace are used in xml outputs as names of elements or attributes. They are, however, translatable. When you export xml files in some wiki, they can't be imported in different language wikis because of lacking portability of xml outputs. Is this intentional? --fryed-peach 15:01, 16 April 2009 (UTC)

Well, Data Transfer was never meant to help with transferring pages from one wiki to another, since Special:Export and Special:Import already handle that; it's meant only for transferring data with non-wiki systems; so the tags can say anything. Yaron Koren 16:19, 16 April 2009 (UTC)

Thank you for your quick response. The current documentation is not clear to me around interwiki-portability. But now I understand its design. --fryed-peach 06:03, 17 April 2009 (UTC)

CSV example needed

I am unclear on how to properly format a CSV file to have it accepted by this extention. I am attempting to import a CSV file exported from Excel or Open Office.

Without the proper headers, the import generates this error: Error: the column 0 header, 'Revised: June 23, 2009', must be either 'Title', 'Free Text' or of the form 'template_name[field_name]'

As the documentation says, actually it's pretty easy. All you have to do is preparing the header, in one line. IE, with the following CSV:

George, 22, 1977, He is a good guy

Julian, 24, 1979, He is a bad guy

The proper header would be:

Title, People[date], People[year], Free Text

You also have to create the Template with the proper variable to take it. In this case, would be Template:People, and in there you should have something like this "He is {{{date}}} year old, was born in {{{year}}}". And that's it.

#ask queries treated as templates

Hi... the XML output seems to treat #ask queries as templates, so I get:

<Template Name="#ask:_[[Category:Owned_entities]]_[[owner::School]]_[[status::Active]]"><Field Name="?is a">Type</Field><Field Name="?user">User</Field><Field Name="?description">Description</Field></Template>

Is that the expected behaviour? - Borofkin 04:39, 14 January 2010 (UTC)

Call to a member function getNamespace()

I have a CSV-file with 5400 rows of information created in Openoffice Calc and saved as CSV. Just titles and a short free text on each row. When I try to import the CSV I get an error saying:
Fatal error: Call to a member function getNamespace() on a non-object in /home/christoffer/public_html/wiki/includes/JobQueue.php on line 277
This error seems to have something to do with non-UTF-8 encoded characters, but as far as I know I don't have any "special" character and my files are always saved with UTF-8 encoding. If I take some random rows from the file it works, but the whole file won't work. I have also tried to import a file with several non-UTF-8 characters like åäö, lots of quotation marks, commas and so on, and it also works. Is it the size of the file that is the problem (to big) or is it something else? --Squall Leonhart 22:37, 8 April 2010 (UTC)

Hi - if possible, could you try splitting this up into smaller files, and try importing those? At the very least, it'll help determine whether the problem is the file size, or some specific row(s); and maybe help isolate the offending rows, if any. Yaron Koren 00:47, 10 April 2010 (UTC)

I tried that now. I think I found the error. Some of the rows did contain characters like [], {} and ~ in the title column. When I removed them everything worked like a charm! --Squall Leonhart 13:16, 10 April 2010 (UTC)

Okay, that's good to know, thanks. I'll try to fix the code so that it handles those characters more gracefully. Yaron Koren 21:20, 11 April 2010 (UTC)

This has been fixed now in SVN, and it'll be fixed in the next version of Data Transfer as well, FYI. Yaron Koren 05:32, 13 April 2010 (UTC)

Multiple templates & ImportCSV

One thing I've noticed - if you attempt to import a CSV file to create a page which has multiple instances of the same template, it silently just uploads the last one specified. eg in CSV:

Title, Template1[field1], Template1[field1]

Testpage, foo1, foo2

The workaround was to use ImportXML which does let you specify multiple versions of the same template. --92.236.50.225 17:05, 2 May 2010 (UTC)

EDIT: hmm... have also noticed that "&amp;" in field values don't get re-imported by ImportXML, even if correctly escaped... --92.236.50.225 20:38, 2 May 2010 (UTC)

The hack I've used is to create various templates, then Renaming them by ReplaceText extension, like this:

Title, Template1[field1], Template2[field1]

Testpage, foo1, foo2

Template2 is exactly the same as Template1, and once you imported it you go to Special:ReplaceText and replace Template2 for Template1 (yeah, its a tiny dirty hack XD)

Ooh, that's not bad! Yeah, I can't think of another solution for it. Yaron Koren 16:44, 12 May 2010 (UTC)

Templates Inside of Templates

I run into a lot of XML parsing difficulties when it runs across a page with a template inside of a template. For instance, if the wikitext is:

{{Box|Home Demo|{{Mp3|Aisumasen Home.mp3}}}}

==Harmony==

So we have {{Bridge|key=I|start=ii}}, 

then the xml file gives

<Template Name="Box"><Field Name="1">Home Demo</Field><Field Name="2">Mp3</Field><Field Name="3">Aisumasen Home.mp3}}}</Field><Field Name="">=Harmony==

So we have Bridge</Field><Field Name="key">I</Field><Field Name="start">ii</Field><Field Name="4"></Field><Field Name="5">, 

It then seems to interpret main body text as various fields, which runs into all sorts of weird problems from then on.

Douglas (May 25 2010)

That looks like a bug in the code. Though Data Transfer isn't really geared for templates-within-templates, so I'm not that surprised that it doesn't work. Yaron Koren 13:20, 26 May 2010 (UTC)

Template Calls with Special Characters

For instance, I sometimes have the line

<br>

as a parameter input. Example:

{{Template|param=I love you<br>...or do I?!}}

The XML then looks for a matching

</br>

Dunno if this is a bug in the code or a poor convention of mine.

Okay, I fixed this, so in case anyone wants to know: the convention between HTML and XHTML is that HTML permits the unclosed line break while XHTML requires that it be closed like

<br />

One more problem, however, arises with the use of

<source lang=text enclose=div>...</source>

I use this in my template input all the time when I want a box, for instance, to display some terminal text. I'll write something to ignore equal signs when its inside angular brackets.

-Douglas (May 26 2010)

overwrites existing pages

hi,

how can I prevent overwriting existing pages with this import-mechanism?

Thx. --Rolze 08:27, 15 June 2010 (UTC)

Unfortunately, you can't - but that would be a nice feature to have. Yaron Koren 12:21, 15 June 2010 (UTC)

Ok, so I - as a faithful semantic mediawiki fan and user - request that this feature gets on the roadmap of this fine extension! And that it won't be forgotten on the nice2have-wishlist! ;) --Rolze 21:29, 15 June 2010 (UTC)

I think this would be a nice feature, too. Cheers --kgh 22:42, 15 July 2010 (UTC)

still needed! Is there a possibility to get this on the roadmap? --Rolze 10:14, 22 April 2011 (UTC)

Hi - that feature's already in there, as of version 0.3.7 - you can "skip" pages that already exist. Yaron Koren 15:46, 22 April 2011 (UTC)

Bot flag is being ignored

Hi Yaron, so far the bot flag is being ignored by this extension. All the log actions of this extension are displayed on recent changes even though the importing user has a bot flag. This is not really a problem, but it would be nice if the extension accepts the bot flag in the future. Cheers --kgh 22:41, 15 July 2010 (UTC)

semicolons as separators

Hi, it is me again. :-) I would be very useful if semicolons are the separators of choice with this extension, since they are by far more rare in use within texts. So if texts containing a comma get imported, they are destined to ruin the respective imported page. Hmm, is it possible to use text markers like ' or " for texts. This should be a workaround. Cheers --kgh 22:48, 15 July 2010 (UTC)

Hi - well, CSV is a very standard format, unlike, say, "SSV". In CSV, values that contain commas are meant to be enclosed within quotes, which solves the problem. Yaron Koren 23:04, 15 July 2010 (UTC)

That is true and every user should stick to this to keep the format exchangeable without having to prepare the file for further use. Thank you and cheers --kgh 07:16, 16 July 2010 (UTC)

Enhancement to DataTransfer

Hi, a nice enhancement would be, if one is able to insert an individual description for the upload instead of the standard "CSV-Import". Another wish would be, that the name of the file used for import is piked up and displayed, too. To me the second request seems to be more important. Cheers --kgh 22:20, 19 July 2010 (UTC)

That's a good idea, and you asked at just the right time - I was in the middle of getting a new version ready for a release. This feature is now available (though only for CSV import right now, not for XML import) in the latest version, 0.3.6. Yaron Koren 16:31, 26 July 2010 (UTC)

Cool, thank you! Great work! I am very happy about this. I am mostly doing CSV imports, thus my urgent needs a fulfilled, which does not mean... :-) Cheers --kgh 08:51, 27 July 2010 (UTC)

SVN tag for 0.3.6 missing

The last SVN tag is 0.3.5 --RScheiber 22:40, 14 September 2010 (UTC)

Why does this extension exist?

RE: It should be noted that Data Transfer is not an ideal solution for backing up one's wiki, or transferring wiki pages from one MediaWiki site to another; for that, the much better solution is to use MediaWiki's built-in "Special:Export" and "Special:Import" pages.

So why have this extension at all if "Special:Export" and "Special:Import" is better? Adamtheclown 06:05, 24 November 2010 (UTC)

What do you do in case your data is not stored in xml? That is the reason for this extension. Cheers --kgh 11:28, 24 November 2010 (UTC)

Expected pages got mediawiki error

My error message: [1] Since I don't see why this extension exists, since "Special:Export" and "Special:Import" already moves pages better, I will not try to get assistance to fix this problem. Adamtheclown 06:17, 24 November 2010 (UTC)

I think you've misunderstood the point of Data Transfer - it's for getting in data from any system other than MediaWiki. Here you seem to be using MediaWiki's XML, but Data Transfer has its own XML format that's different. Yaron Koren 15:03, 24 November 2010 (UTC)

mediawiki error

My error message: Fatal error: Call to undefined function wfloadextensionmessages() in /home/zasve/public_html/oglasi/extensions/DataTransfer/specials/DT_ViewXML.php on line 18

Just when I approach http...www...zasve.net/oglasi/index.php?title=Posebno:Specialpages

wgLanguageCode = "sr-el";

Bonzo 12:45, 16 December 2010 (UTC)

You must be using the very latest MediaWiki version. I'll fix that soon; for now, you can just delete that line in the code. Yaron Koren 17:50, 16 December 2010 (UTC)

• • Yes. version 1.9.2 . Thanks :).

Bonzo 18:00, 16 December 2010 (UTC)

Oh, you're using version 1.9.2? Never mind, then. :) That version's not supported by Data Transfer. Yaron Koren 18:36, 16 December 2010 (UTC)

File size

Yaron, hi! First, I love this extension! Really makes my life easier! I´m importing 48k articles from Excel. Everything is ok, but I´m doing some tests and dividing the articles into multiple csv files everytime is painfull. How can I make your extension accept bigger csv files? Is that possible? Thanks, Filipe.

Hi - what's the error message you get when you try to upload a bigger file? Yaron Koren 04:59, 27 December 2010 (UTC)

Doesn´t give a message. Returns to the same page like nothing has happened.

Hi - that's odd. What versions of MediaWiki and Data Transfer are you using? Yaron Koren 04:08, 5 January 2011 (UTC)

0.3.4. If my file is TOO big, it does that. I got a message once, when I removed some lines from the CSV file, that there were too many items. And if I remove even more, leaving like 1 thousand items, it works perfectly. Filipe, 7 January 2011 (UTC)

Okay, but what MediaWiki version? Yaron Koren 16:00, 7 January 2011 (UTC)

1.16 (http://www.wikireceitas.com.br/Especial:Vers%C3%A3o) Filipe, 8 January 2011 (UTC)

Yaron can you help me woth the default tables I have to populate? I´m trying to make and excel macro to do that for me. Filipe, 8 January 2011 (UTC)

Sorry, I don't know what you mean by that. Yaron Koren 15:00, 10 January 2011 (UTC)

Enhancement to DataTransfer

Hi, since the last version of this extension it is possible to add data to an existing page. However, it is not possible to add data to an existing template.

E. g. in case you want to add "World" to

{{Template1
|Property1=Hello
}}

you will get

{{Template1
|Property1=Hello
}}
{{Template1
|Property2=World
}}

instead of

{{Template1
|Property1=Hello
|Property2=World
}}

since this extension does not identify, that the same extension is already on the page.

There is a quite dirty workaround to do it Remove }} from the respective pages with extension ReplaceText and then import

|Property2=World}}

The disadvantage of this is, that you have two new revisions per page and that it is only possible once.

Perhaps it is possible to enhance this extension to avoid this. Hopefully this is not a daydream. ;)

Cheers --[[kgh]] 14:34, 9 February 2011 (UTC)

Importing Category pages

How do I import Category pages? I have a large number of these externally generated. If I try to generate my pages with a block beginning:

<Namespace Name="Category">
<Page Title="catname">
<Free_Text>
.... etc

DataTransfer strips the Namespace information and creates the page within Main, not within Category.

If instead I try to reverse engineer it by using ViewXML on an existing Category page, it gives me a block beginning <Category Name="catname"> which includes the contents of every page in the category, which is not what I want. Marinheiro 10:00, 9 March 2011 (UTC)

This was easier than I thought (though I did get a bit misled by the Help page): just use Category:category name as the page title, eg.

<Page Title="Category:Inactive Resources">
<Free_Text>
[[Category:Resources]]
</Free_Text>
</Page>

217.169.30.146 22:53, 12 March 2011 (UTC)

Version to download for MediaWiki 1.17

This is a just a tip for anybody like me who was having trouble with this extension after upgrading to Mediawiki 1.17. Any files that I tried to upload just gave the "The file you uploaded seems to be empty. This might be due to a typo in the file name. Please check whether you really want to upload this file." error.

If you follow the "Download snapshot" link on the Data Transfer Extension page, and select Mediawiki 1.17.x, you will be given version 0.3.6 of this extension which is not compatible with Mediawiki 1.17. You have to either select "trunk" on the Download Snapshot page, or download the code directly from the main Extension page. The version of this extension in the Semantic Bundle is also not compatible with Mediawiki 1.17.

Ability to add a single property to many pages in an automated fashion

Hello -- I know that this question was already asked to a certain extent and I wanted to ask if any progress had been made or to ask if there was another work-around.

I also would like to be able to add a property and value to a given page after the page is already created. I was wondering if you could use the export function on each page and then modify each file to have the additional property/field and then to re-import all of those same pages.

Thanks in advance, Dc321 14:54, 18 September 2011 (UTC)

There's now what might be a better solution to that problem: the "autoedit" functionality in Semantic Forms, which is available as both a parser function and an API action (the latter might be more helpful if you have a lot of pages, and if you know at least a minimal amount of programming.) I would look into that one. Yaron Koren 21:01, 19 September 2011 (UTC)

Order of templates

Hello, did anyone find out what in what order the templates are printed in the article text? To me it seem quite randomly, but maybe there is a system? rotsee 20:50, 12 October 2011 (UTC)