Extension talk:Data Transfer

MWException
Hi,

I have previously tried to solve this problem here and it has been suggested that I contact yourselves to see if I can get it resolved, Many thanks,

--Budstellabecks (talk) 15:03, 3 January 2014 (UTC)

My wiki is currently bugged. I think it may have something to do with the same info being submitted twice as a csv import.

As a side effect I'm unable to save pages as well as a few other things.

MediaWiki 	1.20.4 PHP 	5.3.3 (cgi-fcgi) MySQL 	5.0.92-50-log

The error I get using $wgShowExceptionDetails = true; bring up the following,

Thanks in advance for any info

Internal error Jump to: navigation, search

The Title object did not provide an article ID. Perhaps the page doesn't exist?

Backtrace:


 * 1) 0 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/includes/parser/ParserOutput.php(390): LinksUpdate->__construct(Object(Title), Object(ParserOutput), true)
 * 2) 1 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/includes/WikiPage.php(1749): ParserOutput->getSecondaryDataUpdates(Object(Title))
 * 3) 2 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/includes/WikiPage.php(1617): WikiPage->doEditUpdates(Object(Revision), Object(User), Array)
 * 4) 3 [internal function]: WikiPage->doEdit('{{Individual Mo...', 'Add more months...')
 * 5) 4 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/includes/Article.php(1820): call_user_func_array(Array, Array)
 * 6) 5 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/extensions/DataTransfer/includes/DT_ImportJob.php(77): Article->__call('doEdit', Array)
 * 7) 6 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/extensions/DataTransfer/includes/DT_ImportJob.php(77): Article->doEdit('{{Individual Mo...', 'Add more months...')
 * 8) 7 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/includes/Wiki.php(594): DTImportJob->run
 * 9) 8 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/includes/Wiki.php(556): MediaWiki->doJobs
 * 10) 9 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/includes/Wiki.php(447): MediaWiki->restInPeace
 * 11) 10 /websites/123reg/LinuxPackage23/yo/ur/si/yoursitehost.co.uk/public_html/partymixers/manager/index.php(59): MediaWiki->run
 * 12) 11 {main}


 * Hi - I would delete the contents of your "job" database table - that should at least take care of these current problems. Then, if you're ready to do the import thing again, it might work better now that you've updated all the software... but if you get the same problems, please let me know. Yaron Koren (talk) 16:07, 3 January 2014 (UTC)

Import CSV error, line break SOLVED
In case you use Mac OS X, make sure you save your CSV file with Windows line break format. I have Excel and TextWrangler (just another text editor). I used Excel to edit the CSV file. Then I reopened the file in TextWrangler and save as, specifying that it uses Windows line breaks (CRLF) (as per the CSV specification linked to on the extension page), rather than Classic Mac (CR).

--Tommyheyser (talk) 22:19, 9 March 2014 (UTC)

That's very helpful! It's odd, a linux open source system using the DOS convention, but I suppose they do it deliberately.

In vi you need to type:


 * %s/^M/^M^J/

Incidentally, this isn't the right place to ask, but, is there a way to have vi instead of the editor mediawiki uses as a standard?

Fustbariclation (talk) 05:13, 10 March 2014 (UTC)


 * Tommyheyser - thanks for diagnosing that issue; I just added a note about that to the documentation. Fustbariclation - I believe CSV files saved with both Windows and Linux are handled fine; it's only Mac (apparently) that requires special treatment. (It could well be that the Data Transfer code could be modified to handle the Mac style as well; I haven't looked into it.) Yaron Koren (talk) 21:54, 10 March 2014 (UTC)

How to add text at the top the page while keeping current page content, ?
Thanks,

Nicolas NALLET (talk)


 * If you choose the "append" option, it will add content to the bottom of the page; there's no way to add content to the top of the page, as far as I know, but you could probably easily change the "append" behavior in the code. Yaron Koren (talk) 14:40, 25 March 2014 (UTC)


 * Ok Thanks Yaron it works, in the file, line 61

I have replaced

by

Nicolas NALLET (talk)

Extension breaking /mw-config/ [SOLVED]
Hi Yaron,

Firstly, I must say that I really admire your work and use several of your extensions. Bravo!

Two years ago I used DataTransfer successfully on an earlier incarnation of my website. I was then using it with the Semantic bundle which is impressive, but the bundle was unsuited to my needs and I no longer use it.

Today I upgraded my site to Mediawiki 1.24 and coincidentally installed the DataTransfer extension soon afterwards. I have been trying to get Special:ImportCSV and Special:ViewXML to work but without success so far. Both just seem to hang. Import CSV responds with "Importing... 2 pages will be created from the CSV file" but it does not create either of the new pages. Two years ago Special:ImportCSV worked okay.

Later, when I visited the config page - eg. http://www.example.com/mw/mw-config/index.php - I found that DataTransfer did cause an error:

Warning: include_once(/languages/DT_LanguageEn.php) [function.include-once]: failed to open stream: No such file or directory in /home/mysite/public_html/mw/extensions/DataTransfer/DataTransfer.php on line 98

Warning: include_once [function.include]: Failed opening '/languages/DT_LanguageEn.php' for inclusion (include_path='.:/usr/lib/php:/usr/local/lib/php') in /home/mysite/public_html/mw/extensions/DataTransfer/DataTransfer.php on line 98

Fatal error: Class 'DT_LanguageEn' not found in /home/mysite/public_html/mw/extensions/DataTransfer/DataTransfer.php on line 102

I am hopeful that the /mw-config/ related error may in someway be symptomatic of the reason that I cannot get the extension to work.

Thanks in advance for any advice.

Regards, Cruickshanks (talk) 09:20, 26 September 2014 (UTC)


 * Are you using the latest version of Data Transfer, 0.6? The line numbers make it seem like you're using an older version. Yaron Koren (talk) 12:45, 26 September 2014 (UTC)


 * Yaron, I have just found that the Download Snapshot, using the Special:ExtensionDistributor for Mediawiki Release 1.23, was downloading Version 0.5. I have now downloaded Version 0.6 using the adjacent (Git master) link in the Infobox. It is still breaking Config however, but with different line numbers now:


 * Warning: include_once(/languages/DT_LanguageEn.php) [function.include-once]: failed to open stream: No such file or directory in /home/mysite/public_html/mw/extensions/DataTransfer/DataTransfer.php on line 106


 * Warning: include_once [function.include]: Failed opening '/languages/DT_LanguageEn.php' for inclusion (include_path='.:/usr/lib/php:/usr/local/lib/php') in /home/mysite/public_html/mw/extensions/DataTransfer/DataTransfer.php on line 106


 * Fatal error: Class 'DT_LanguageEn' not found in /home/mysite/public_html/mw/extensions/DataTransfer/DataTransfer.php on line 110


 * Special:ViewXML is now working with Version 0.6 installed. I have not tried ImportCSV as I have a lengthy BOT running at present, but my 'spider senses' guess that ImportCSV will probably work now.


 * By the way, I incorrectly said in my first post I had upgraded to 1.24; I meant to say I had upgraded to 1.23.4.


 * Regards, Cruickshanks (talk) 15:43, 26 September 2014 (UTC)


 * Oh, okay. I looked through my emails and realized that I had a an email discussion with someone about this very same problem a few weeks ago. We weren't able to figure out the underlying cause, but were able to fix it by just commenting out line 77 of DataTransfer.php, which reads "dtfInitContentLanguage($wgLanguageCode);". It's fine to do because the language stuff is only used for a few minor features, like XML grouping, that you're probably not using. Yaron Koren (talk) 16:40, 26 September 2014 (UTC)


 * Thanks Yaron, that fixed it. Regards, Cruickshanks (talk) 18:49, 26 September 2014 (UTC)

Multi-instance templates not preserved in output
If you import a file and update fields in the base template of a page it removes any multi instance templates embedded within and replaces them with the text "Array" this can be solved quite easily in DT_PageStructure.php public function toWikitext { if ( $this->mIsTemplate ) { $wikitext = '";      return $wikitext;     } else {	return $this->mFreeText;     } }

XML Import Only Imports 2 Pages
Hello:

I have an XML file coded to create 80 new pages.

When I try to do the import, I get a message saying:


 * Importing...


 * 2 pages will be created from the XML file.

How do I create all 80 pages at once?

Thank you,

--Patricia Barden (talk) 20:37, 6 January 2015 (UTC)


 * I'm guessing that there's some issue with the XML. Could you pastebin it or something? Yaron Koren (talk) 22:47, 6 January 2015 (UTC)


 * Here is a link to the XML file on Google Drive: https://drive.google.com/file/d/0BxAhvHwTqRzPTVRjbEFaVTVkRTg/view?pli=1
 * By the way, I'm using MediaWiki 1.23.6 and Data Transfer 0.6
 * Thank you so much for taking a look.--Patricia Barden (talk) 02:03, 7 January 2015 (UTC)


 * Ah, great. I'm guessing the problem is the quotes in the page title of the 3rd page - those need to be escaped, like with \" or something. Yaron Koren (talk) 02:10, 7 January 2015 (UTC)


 * I escaped the quote marks in the Page Title attribute by using
 * After doing that for all Page Title attributes, the pages were created in a flash.
 * It was not necessary, however, to escape the quote marks in the Field Name attribute values. This worked just fine:
 * Thought you'd like to know. Thank you again! --Patricia Barden (talk) 16:49, 7 January 2015 (UTC)
 * Thought you'd like to know. Thank you again! --Patricia Barden (talk) 16:49, 7 January 2015 (UTC)
 * Thought you'd like to know. Thank you again! --Patricia Barden (talk) 16:49, 7 January 2015 (UTC)

Nothing exported
I just get the page with:

This XML file does not appear to have any style information associated with it. The document tree is shown below.

And nothing else:

Product	Version MediaWiki	1.24.2 PHP	5.5.22 (cgi-fcgi) MySQL	5.5.42-cll

I'm actually trying to move properties from one SMW to another - 'export pages' doesn't seem to export properties, even when mentioned specifically.

Fustbariclation (talk) 09:25, 11 June 2015 (UTC)


 * Are you trying create a "copy" of property pages, pages with properties on them or maybe both? If so you should be fine when you use . This will export the actual page content, it does not matter what is on them. You have to make sure you export everything you need for it to work on the other wiki, including categories and templates. This is why categorising pages is important :) it makes duplicating pages easier. When you import the resulting XML on your destination Wiki it will covert the page content back to properties and values. Regards, --Felipe (talk) 09:50, 11 June 2015 (UTC)


 * When testing SMW MW's Import/Export XML is used to import property/value declarations from another wiki. See also swm@org Export content --MWJames (talk) 11:05, 11 June 2015 (UTC)

Nothing imported
Hello,

each time i try to import pages it simply does not work. It tells "0 pages will be created from the XML file". Mediawiki version 1.24.1.

Here is a pastebin in order to see if you can help me.

http://pastebin.com/kdS3K2Ci

Thanks.

P.D: Could it be because of language? Perhaps it does not simply understand spanish, but that would be really extrange, as extension itself gave that XML to me.

--00Maiser00 (talk)


 * I think I made a bad mistake by having the XML field names be translatable, instead of just always being in English. For the case Spanish, I think the issue is that the translation for "ID" is "Id." - I think the "." is causing the problem. It could be that the accents in "Páginas", etc. are also a problem. If it's not too much trouble, I would suggest replacing all the Spanish field names with their English-language equivalents (you can find them here), which will hopefully work. And I probably need to change that internationalization thing. Yaron Koren (talk) 17:11, 3 August 2015 (UTC)

I just made a program to translate all these tags from spanish to english, still not working. It may be because i have putted some tag incorrectly. Could you provide me with a viewxml in english where i can see all elements please?

Moreover, this is what is actually not working for me (as an example, trying to import forms).

http://pastebin.com/ZFFULXdi

Could it be because it is trying to read in spanish, but does not understand it? I have tried with other pieces of xml (simple ones) and is still not importing, no matter the language i use.

--00Maiser00 (talk)


 * It should be "Free_Text", not "Free Text" - maybe that's the issue. Yaron Koren (talk) 00:21, 17 August 2015 (UTC)

I changed that and it is still not working. Could you send me some text which can be confirmed is working so i can test if it is because of the syntax or simply because a problem in the import when the wiki is in another language?

--00Maiser00 (talk)

Any change yet for the language? If it is related to vocabulary, i guess it is regarding something which can not be changed in the xml generated document. Perhaps it is because of "user hidden" vocabulary in the import or something? I am pretty surprise i am the only non english user which is talking about this issue :S.

EDIT: I can confirm the problem is not solved changing vocabulary to english, i tried importing elements from your example page: http://discoursedb.org/wiki/Special:ViewXML And it still did a 0 import.--00Maiser00 (talk)

Error when trying to import categories
Hi,

this time, when i choose categories in viewXML i get an error telling:

This page contains the following errors:

error on line 2 at column 1: Extra content at the end of the document Below is a rendering of the page up to the first error.

Nothing more is showed, even with $wgShowExceptionDetails=true;

What could be the cause? --00Maiser00 (talk)


 * Does that happen for any category? Yaron Koren (talk) 13:32, 2 September 2015 (UTC)

Deprecated : Methods with the same name as their class when used with PHP 7
Hello. During testing under PHP 7 you get a deprecated warning because of 2 methods with the same name as their class. Filed a  bug report  for that with a patch. I hope it is done correctly. Regards, --Felipe (talk) 13:04, 25 September 2015 (UTC)

Special:ImportCSV No such special page
Hi. Thanks in advance for any suggestions.

I do a 1) Fresh Mediawiki 1.26 install. With composer I install 2) mediawiki/semantic-media-wiki "2.3.1" 3) ediawiki/semantic-forms "3.5" 4) mediawiki/semantic-result-formats "2.3" 5) mediawiki/data-transfer "dev-master" 6) Add    include_once( "$IP/extensions/DataTransfer/DataTransfer.php" );      at end of LocalSettings 7) Run php update.php 8) resolve an Error in DataTransfer.php namely "undefined variable wgScriptPath" by copying $wgScriptPath = "/mw126"; from LocalSettings.

But the extension does not appear in Special:Version which is confirmed by Special:ImportCSV "No such page". Tried Mediawiki 1.23 but same problem. Thanks.

Setup Details XAMPP for Windows 7.0.3

Product 	Version MediaWiki 	1.26.2 PHP 	7.0.3 (apache2handler) MariaDB 	10.1.10-MariaDB

Composer.json "require": { "composer/semver": "1.0.0", "cssjanus/cssjanus": "1.1.1", "ext-iconv": "*", "liuggio/statsd-php-client": "1.0.16", "oyejorge/less.php": "1.7.0.9", "mediawiki/at-ease": "1.1.0", "oojs/oojs-ui": "0.12.12", "php": ">=5.3.3", "psr/log": "1.0.0", "wikimedia/assert": "0.2.2", "wikimedia/cdb": "1.3.0", "wikimedia/composer-merge-plugin": "1.3.0", "wikimedia/ip-set": "1.0.1", "wikimedia/utfnormal": "1.0.3", "wikimedia/wrappedstring": "2.0.0", "zordius/lightncandy": "0.21", "mediawiki/semantic-media-wiki": "2.3.1", "mediawiki/semantic-forms": "3.5", "mediawiki/semantic-result-formats": "2.3", "mediawiki/data-transfer": "dev-master" },


 * Can Data Transfer be installed/included via Composer? I'm not sure it can - maybe that's the issue. I would try removing it from composer.json, and installing/including it manually. Well, you're already (also) including it manually. Yaron Koren (talk) 04:55, 10 March 2016 (UTC)

Yes it does install via Composer. 

Yes removing it from composer.json solved the problem.

Thank you.

Possible to edit only parts
I want to use Data transfer to upload an csv-file to write into the my SemanticFormular. Could I create the Page with an frist CSV upload and the most important information and fill the articles with more information later with an secound upload, withour blanking/overwritting the existing information? --Tester of Datatransfer (talk) 15:06, 27 April 2016 (UTC)


 * You have a great username! Unfortunately, no - the Data Transfer import is not "smart". You would probably have to do that kind of import with a bot/script. Yaron Koren (talk) 19:34, 27 April 2016 (UTC)

Feature request evaluation level (nested templates problem)
Hi,

using Data Transfer 0.6.1 the problem arises that in the 1st template level the value of a parameter can have further nested templates (2nd level). The 2nd level is also transformed to xml but embedded in partially Wiki-parsed HTML from the 1st level value, that is: plain text becomes &lt;p> (or &lt;ul>, &lt;dl> etc.), the value is then regarded as Wiki code to be parsed. See the XML output examples. Without the nesting templates it would transform to: But with nesting templates it parses each 1st-level-value partially that is split by a 2nd-level-template:

Why is the value in the 1st level (that has nested templates in it) partially wiki parsed? Could there be a feature to not evaluate that deep? Do you know what I mean? I did not expect that the 1st-level value, that embeds 2nd-level-templates, is parsed partially. Any solution to that?

Kind regards --Andreas P. 11:36, 18 May 2016 (UTC)


 * That sounds like a bug. Yaron Koren (talk)
 * Well, possibly not: I saw just now that there are  and   that prevent parsing. The default setting   causes this (for me unexpected) behaviour. --Andreas P. Icon_External_Link_E-Mail.png 14:22, 18 May 2016 (UTC)
 * Ah -so if you set it to false, it works fine? Yaron Koren (talk) 16:02, 18 May 2016 (UTC)
 * Works almost fine, but a bug occurs when there are nested templates. See the issue I just filed on https://phabricator.wikimedia.org/T135770 --Andreas P. Icon_External_Link_E-Mail.png 21:03, 19 May 2016 (UTC)
 * And could there be a feature to set the transformation level of templates? --Andreas P. Icon_External_Link_E-Mail.png 08:06, 20 May 2016 (UTC)
 * That sounds like a good idea. Yaron Koren (talk) 13:03, 20 May 2016 (UTC)

Schema file for ViewXML import?
I have hundreds of non-wiki XML files, each containing thousands of entries which I'm trying to import as separate pages in my wiki. This extension seems like the right way to go. I'm trying to create an XSLT (Using the 30 day free trial of Altova MapForce because I have no idea what I'm doing) to convert the XML from the schema it's currently distributed in to the format generated by the Data Transfer ViewXML page. Is there an .xsd schema file available for the import? Thanks. Tcrimsonk (talk) 19:09, 29 June 2016 (UTC)


 * No, unfortunately. Yaron Koren (talk) 02:25, 30 June 2016 (UTC)

Best way to escape commas and double quotes in creating CSV to import?
Related to my question above, it would probably be easier for me to convert my existing non-wiki XML files into CSV for import rather than trying to match your XML schema. However, the existing XML contains fields with text including both commas and double quotes. How should I indicate which of these are deliberate delimiters in a way that Data Transfer will recognize? Thanks again. Tcrimsonk (talk) 19:09, 29 June 2016 (UTC)


 * I agree that CSV would probably easier than XML. You should just look up how to handle commas and quotes in CSV; I think they both require extra double quotes. Yaron Koren (talk) 02:33, 30 June 2016 (UTC)


 * Thanks; from what I've found, there are various ways to handle commas and quotes in CSV, and the choice depends on what program is going to be interpreting/processing that CSV. So I would still need to know how Data Transfer (or perhaps just MediaWiki in general) handles them.  I'll try a CSV import with all fields encased in double quotes and see how that works.  I'll reply back here once I know.  (Unless someone else has a suggestion.) Tcrimsonk (talk) 03:25, 30 June 2016 (UTC)

Automatic/Bot/Script based Importing?
Is there any way to monitor a given directory, and automatically import any XML/CSV files that appear there? Or is there a command line way to invoke the import? Tcrimsonk (talk) 03:25, 30 June 2016 (UTC)


 * Unfortunately, no - Data Transfer doesn't provide an API. You would need to create a custom bot/script to do it, I think. Yaron Koren (talk) 14:47, 30 June 2016 (UTC)

If the extension should be extended with automatic importing, what would be the way to go? Would a maintenance script that reads a file and does the import be a simple way to do that?--AdSvS (talk) 12:03, 26 October 2016 (UTC)


 * A built-in import script does sound like the easiest approach, yes - I hadn't thought of that. Yaron Koren (talk) 17:14, 26 October 2016 (UTC)

Version 0.6.2 doesn't display properly special characters like é
with Utf-8 é is displayed Ã© after csv importing.

Previous versions like 0.6.1 displays special characters (like é) properly. -- Nicolas NALLET 24. Feb. 2017, 13:13‎


 * What if you use UTF-16 instead? Yaron Koren (talk) 14:30, 24 February 2017 (UTC)


 * Sounds like you ran exactly into the same situation as I did as described in . So yes, saving as UTF-16 and importing as UTF-16 should work. Also saving as UTF-16, then re-saving as UTF-8 and importing as UTF-8 should work too. The only thing seemingly not to work is saving as UTF-8 and importing as UTF-8 directly without doing the UTF-16 loop. Cheers --&#91;&#91;kgh&#93;&#93; (talk) 14:57, 24 February 2017 (UTC)


 * Yes it's working with UTF-16, thansk guys

Nicolas NALLET Wiki-Valley.com, Semantiki.fr (talk) 13:55, 27 February 2017 (UTC)

CSV / Spreadsheet - How???
This might be cause by a very stupid mistake, but I am unable to get DataImport work.

After creating a csv-file: Title,Cheese[Country],Cheese[Texture],Free Text Mozarella,Italy,Semi-soft,It's good on pizzas! Cheddar,England,Hard/semi-hard,"Often sharp, but not always." Gorgonzola,Italy,"buttery or firm, crumbly","salty, with a ""bite"" from its blue veining" Stilton,,"",needs more data and a Template:Cheese: This cheese is from and the texture is generally

... all I see is: "Importing... 4 pages will be created from the CSV file."

My logfile shows these Exceptions and Errors: (shortened ) [exception] [93065002f4979874bc4e15f3] /Special:ImportCSV  DBUnexpectedError from line 2852 of /var/www/html/mediawiki/includes/libs/rdbms/database/Database.php: MWCallableUpdate::doUpdate: Cannot flush snapshot because writes are pending (WikiPage::insertOn, Revision::insertOn, Revision::insertOn, WikiPage::updateRevisionOn, WikiPage::updateRedirectOn, RecentChange::notifyNew). salty, with a "bite" from its blue veining requestId=93065002f4979874bc4e15f3 (id=224,timestamp=20170225134852) t=121 error=DBUnexpectedError: Invalid atomic section ended (got WikiPage::doCreate). [exception] [6add1bd85ff9e614981c4c0e] /index.php?action=emailtowiki&prefix=EmailToWiki  DBUnexpectedError from line 2852 of /var/www/html/mediawiki/includes/libs/rdbms/database/Database.php: MWCallableUpdate::doUpdate: Cannot flush snapshot because writes are pending (WikiPage::insertOn, Revision::insertOn, Revision::insertOn, WikiPage::updateRevisionOn, WikiPage::updateRedirectOn, RecentChange::notifyNew). It's good on pizzas! requestId=6add1bd85ff9e614981c4c0e (id=287,timestamp=20170226025908) t=189 error=DBUnexpectedError: Invalid atomic section ended (got WikiPage::doCreate).

But where are the 4 pages??? No pages have been created. And why is it 4 pages? I would expect 1 page ...

What am I missing? Any help is highly appreciated! TieMichael (talk) 07:49, 26 February 2017 (UTC)


 * I hadn't seen the error "Cannot flush snapshot because writes are pending" before, so I did a web search on it, and I found some bug reports like this. This looks like a bug in MW 1.28.0, which I assume is what you're using. Hopefully it will get fixed soon.
 * It's four pages because each row gets turned into a page. Yaron Koren (talk) 14:08, 26 February 2017 (UTC)