Extension talk:SimpleBatchUpload

Jump to navigation Jump to search

About this board

FrugalTPH (talkcontribs)

I keep experiencing an ERROR: Server communication failed, for any file I try to upload. I can upload the files in question successfully using the default mw file uploader, though. Any idea what could be causing this?

I have the following set in my Localsettings.php: $wgEnableUploads = true; $wgEnableWriteAPI = true;

Reply to "Server communication failed"
Postprefix (talkcontribs)

[05-Jun-2019 15:19:54 America/New_York] PHP Fatal error:  Uncaught UnexpectedValueException: callback 'SimpleBatchUpload\SimpleBatchUpload::initCallback' is not callable in /home/riotawfb/nosubject.com/includes/registration/ExtensionRegistry.php:360

possible conflict with other extensions...

Revansx (talkcontribs)

*follows* I'm setting up a new MW 1.31 system and have had problems with this as well. Haven't had time to troubleshoot it yet. All I know is that it fails to load for some reason.

Mitchell316 (talkcontribs)

I'm also having the same issue with the SimpleBatchUpload extension. Any word on a fix or new version?

F.trott (talkcontribs)
F.trott (talkcontribs)

Can you all confirm that you installed either the SimpleBatchUpload.1.4.0.with.dependencies.tar.gz or the SimpleBatchUpload.1.4.0.with.dependencies.zip file or used Composer for installation.

In particular, can you confirm that you did NOT

  • do a git clone
  • use the "Download ZIP" link on GitHub.
Mitchell316 (talkcontribs)

Yes, I downloaded from the .tar.gz file.

Revansx (talkcontribs)

I'm running this successfully on 1.31.1 fwiw

F.trott (talkcontribs)

I updated the extension over the summer to simplify installation and do CI testing (incl. on MW 1.31). I intend to release v1.5 in the next days.

Revansx (talkcontribs)
Reply to "breaks on Mediawiki 1.31.0"

New feature: Rename the filename with a regular-expression

Ankostis (talkcontribs)

Are there any thoughts for providing a method to modify the filename of the uploaded file with some kind of user-defined callback function (in javascript, lua, template) accepting the original file and converting it into the final one?

F.trott (talkcontribs)

It's not currently planned, but you're welcome to submit a patch.

Ankostis (talkcontribs)

But i wouldn't know how to go about doing it. Maybe allow a new "special" parameter `@fnametemplate` in {{#batchupload}}? Would it be possible for javascrript code to call back to server to discover the filename? Or better allow a parameter named `@fnamefun` and let the user attach such a global javascript function? Or to the prototype of the upload??

F.trott (talkcontribs)

There's probably a number of ways to go about it. One of the main points to consider is security. Allowing batch uploading dozens of files is already borderline dangerous and probably not a good idea to activate for public wikis.You have to carefully consider how to implement your functionality without opening additional attack vectors.

Ankostis (talkcontribs)

FYI, you may review my take on this problem in this commit: https://github.com/ankostis/SimpleBatchUpload/commit/2bd2c7c It is not for the wider public (e.g. it uses ES template literals) but you can get what i wanted to achieve:

Add a new parameter that specifies a regular expression for renaming the uploaded file-names:

{{#batchupload: ...|+rename= #pattern#flags -->replace-string}}
  • Patterns can be surrounded with #/@! chars, and
  • can be followed by regex-flags 'idmuy', while
  • all spaces in the replace-string after the --> are significant (and respected).


Here it is the ''regex'' i use to embed the page-name of the page holding the upload-control into the start of the uploaded-filename, erasing any extra spaces or punctuation-chars supplied by the user:

{{#batchupload:ItemDoc|owner={{PAGENAME}}|+rename= !^({{PAGENAME}}[ -_./+]*)?!iu -->{{PAGENAME}}-}}
Longphile (talkcontribs)

@Ankostis The rename with regex feature is a great addition; unfortunately, it is not working for me when I tried your fork. I am running MW 1.31.1 with php 7.2.11. Using drag and drop or browsing to select a file to upload, I get "ERROR: The filename is not allowed." I would like to be able to rename uploaded files to ensure unique filenames in our wiki.

Ankostis (talkcontribs)

Please take the very latest from rename-files branch. I had indeed a bug when you didn't specify a rename.

Revansx (talkcontribs)

Hi @Ankostis, Does your "rename-files" branch work with MW 1.30?

...also, how hard would it be for you incorporate a new switch that won't let it upload a file that already exist?

Is that something I could tempt you to add? :-)

Ankostis (talkcontribs)

With my basic knowledge of mw and how to debug it, it should be a day's work, which i don't have for the foreseeable future. But you can give it a try: first locate the JS-API that searches if a page exists (eg exists()), and then use chrome's dev-tools in debug-mode to craft the javascipt code for the following actions:

  • Specify a regular expression to extract the new switch-parameter (i would call it `+skip if exists` or `+new files only`)
  • call the API discovered above,
  • avoid the upload and just set a message in the returned result explaining why the upload was denied.
Revansx (talkcontribs)

sounds easy. I really need to figure this out.

Would I be making this change to your 'rename-files' branch or would I be making a branch of your branch? This is where I get confused about git.

Revansx (talkcontribs)

@Ankostis .. so.. I just downloaded and tested your "rename-files" branch... and I love it! .. thank you!

Quick question .. what would happen if the PAGENAME contains "/" characters? (I.e. subpages) would that break anything?

Revansx (talkcontribs)

I just tested it and see that it converts the "/" to "-". Perfect!

Revansx (talkcontribs)

Can you tell us how to write the' 'regex'' part so that all files dropped on the button are set to the same name? (I don't envision using it for multiple files, but I would like to create an upload button that exists to upload one and one one file of a specific name. Thank you!

Revansx (talkcontribs)

I'm really proud of myself. I did it :-)

Here is the 'regex' I used to upload files to a fixed filename regardless of what the supplied filename is:

{{#batchupload:ItemDoc|owner={{PAGENAME}}|+rename= !([a-z0-9_ ()-.&]*)?!iu -->myfilename.xyz|}}
Ankostis (talkcontribs)

> Would I be making this change to your 'rename-files' branch or would I be making a branch of your branch? This is where I get confused about git.

You got to clone my repo to your own github accont and push there a new branch.

> I'm really proud of myself. I did it :-)

Great! Is that enough for you? I mean, don't you want to deny uploads if files exist?

Revansx (talkcontribs)

I do. I do.. but baby steps.. for now site's needs are met by checking for the expected file using the following:

| [[Media:somefile.xyz]]
| {{#batchupload:ItemDoc|owner={{PAGENAME}}|+rename= !([a-z0-9_ ()-.&]*)?!iu -->myfilename.xyz|}}

But it's on my list of things to do to learn to clone the repo and push a new branch.. and then.. make the commits and test it.. I'm eager to try. would you be willing to review my commits when I have them?

Ankostis (talkcontribs)

Sure. But better ask the original author to do that - i have as much experience in mw/smw as you do :-)

Revansx (talkcontribs)

aye. fair enough. Nonetheless. I'll let you know when I do so.

F.trott (talkcontribs)

I am always happy to review pull requests.

Regarding checking and rejecting uploads I am not sure if the proposed approach would be the best solution and if SimpleBatchUpload would be the best place to put it:

  1. Any such functionality that is implemented client-side in JavaScript is easy to circumvent, so is at best a cosmetical measure.
  2. Each file check would require an additional request to the server, making the whole thing inefficient.
  3. The functionality would only affect uploads by SimpleBatchUpload. Standard uploads would still be possible.

A better way might be to find a hook in the upload process that checks an upload and blocks it if necessary. Manual:Hooks/UploadVerifyUpload and Manual:Hooks/UploadVerifyFile look both promising.

It might also be worth to look for an extension that already provides this or a similar functionality, to use it as is or take it as a template for a dedicated extension providing this functionality. What about Extension:UploadBlacklist?

Revansx (talkcontribs)

@F.trott, Thanks for weighing in.

[Security] - My problem is not with security as much as it is with cosmetics/logistics so this would be fine for me.

[Efficiency] - The time it takes for a user to upload the file is likely much longer than the time it takes the script to do the check. This is not a concern for me.

[only works for this SimpleBatchUpload] - yes, exactly. That's a feature not a bug :-)

That said, I've researched and tested so many upload extensions and SimpleBatchUpload is really the best for my application. None of my users want to touch the Upload Wizard or the Standard Upload.. I can add your SimpleBatchUpload buttons to templates and SMW page classes and build amazing tools to solit file from people. With @Ankostis's branch I can force filenames and with the expensive file checking in the template i can disable the button with page logic. That said I'd *really* like to have this feature.. So I hope to add it as discussed.. and I do hope you'll be willing to include these features into your master branch even if it does come with some caveats on security and efficiency and completeness.

F.trott (talkcontribs)
Revansx (talkcontribs)

F.trott, you rock! ... I'll test it soon enough, but what kind of error message would SimpleBatchUpload show in such a case? ... that and (since I haven't read everything on the topic yet) do you know if there is any way of enabling the re-upload permission for the creator (aka original uploader) of each file? i.e. UserX can re-upload UserX's files, and UserY can re-upload UserY's files but neither can re-upload the others files. Is that possible?

Revansx (talkcontribs)

@Ankostis, I'm using your version of SimpleBatchUpload that allows a name change and specifically I'm using your example of adding the pagename as a prefix per:

{{#batchupload:ItemDoc|owner={{PAGENAME}}|+rename= !^({{PAGENAME}}[ -_./+]*)?!iu -->{{PAGENAME}}-}}

The problem I'm having now is that when someone "re-uploads" a file that already has the pagename as a prefix, the regular expression does something strange and produces a file that has no filename at all.

Am I doing something wrong? .. Have you considered this case where a file that has been uploaded with the "add pagename prefix" is downloaded (with the prefixed name) and re-uplaoded with the same tool?

any thoughts on how to handle this?

Ankostis (talkcontribs)

Yes i have, and should be working. Just make sure you have latest commit of the branch, since there was a bug in the intermediate commit i mentioned in my initial post.

Revansx (talkcontribs)

looks like this works well:

|+rename= !^({{PAGENAME}}-)?!iu -->{{PAGENAME}}-}}
Revansx (talkcontribs)

is there any progress on getting this feature included in the main extension?

Revansx (talkcontribs)
Reply to "New feature: Rename the filename with a regular-expression"

need to alter the CSS for VisualEditor

Summary by Revansx

I fixed this on my installation by making edits to ../rev/ext.SimpleBatchUpload.css

Revansx (talkcontribs)

Hi. This extension has become an important part of my site. Continued thanks for providing it :-) .. One way I use it is to allow uploads that "pre-tag" images with semantic properties based on the page that the #batchupload is used on.

This works wonderfully, however, the number of characters involved in the additional properties can be quite long. This is of course transparent in use, but when a user edits the page using VisualEditor (VE) the template for #batchupload renders with the css settings white-space:nowrap; on the span.uploadzone element.

I can verify this by using the browser developer mode and altering this css setting manually, and I have tried to implement it in Mediawiki:Common.css without success. I'm not sure what the expected behavior is for Common.css in VE edit mode, but I'm wondering if this could be most easily solved in the extension itself?

Here is a screenshot of the issue as it appears on my site in VE edit mode:

Notice how the horizontal scroll bar is about 5% of the screen and will scroll waaaay more to the right.. this is the problem. Anyone have any advice on how to fix this with CSS?

F.trott (talkcontribs)

Not sure that this is really a problem of the SimpleBatchUpload extension. I don't have too much experience with VE, but as I see it it should not spell out the input tag at all. If VE would actually show the file-upload button, then the CSS might have a chance to catch and hide it. As it is, this is just more text, so the CSS is not seeing it.

Revansx (talkcontribs)

well.. I know that there is a special Print.css page for CSS that is only used when printing a page. I'm not sure if Common.css is used when in VE edit mode. If it is, then I think I just need someone to help me determine how to refer to the SimpleBatchUpload element better. I'm not super savvy with CSS and this one is hard to identify for me.

Summary by Revansx

I don't understand it, but i am clearly mistaken. The code works great. Thanks!

Revansx (talkcontribs)

Clean installation on MW 1.33, upload button not rendered

Tommyheyser (talkcontribs)

I did a clean install of MediaWiki 1.33 on Windows Server 2016. Installed the extension via composer.local.json, ran composer update --no-dev and added the wfLoadExtension( 'SimpleBatchUpload' ); line to LocalSettings.php.

Extension is shown on the Version page, but Special:BatchUpload only shows the text "Select files (or drop them here)...", a button with text "Choose Files" and "No file chosen".

The error log shows the following:

[resourceloader] Unexpected general module "ext.SimpleBatchUpload" in styles queue.
[resourceloader] Unexpected general module "ext.SimpleBatchUpload.jquery-file-upload" in styles queue.

I didn't have this problem in MW 1.31. Has there been any breaking changes since?

F.trott (talkcontribs)

If you install dev-master instead of ~1.4, does that help?

Tommyheyser (talkcontribs)

Yup, that fixed it. I'm guessing it has something to do with the change in 1.32 related to api that was fixed with Fix #21. Thanks. I hadn't thought of switching to dev-master, should've read the commits.

Reply to "Clean installation on MW 1.33, upload button not rendered"

Latest version does not work on MediaWiki 1.33

Summary by F.trott

Bug is fixed. Available in master now or in the next release when it comes out.

Ti infotrad (talkcontribs)

Dragging and dropping a file does not do anything. Upon debugging, find the following error in the console

load.php?lang=en&modules=startup&only=scripts&skin=vector:2 Error: Unknown module: mediawiki.api.edit

   at sortDependencies (load.php?lang=en&modules=startup&only=scripts&skin=vector:8)
   at sortDependencies (load.php?lang=en&modules=startup&only=scripts&skin=vector:9)
   at resolveStubbornly (load.php?lang=en&modules=startup&only=scripts&skin=vector:9)
   at Object.load (load.php?lang=en&modules=startup&only=scripts&skin=vector:21)
   at index.php?title=Special:BatchUpload:9
   at Array.RLQ.push (load.php?lang=en&modules=startup&only=scripts&skin=vector:48)
   at load.php?lang=en&modules=startup&only=scripts&skin=vector:48
   at load.php?lang=en&modules=startup&only=scripts&skin=vector:48

Find this module and others are deprecated (https://phabricator.wikimedia.org/T196802)

Fix by modifying as follows:

Change the line in src/SimpleBatchUpload.php

'dependencies' => [ 'ext.SimpleBatchUpload.jquery-file-upload', 'mediawiki.Title', 'mediawiki.api.edit', 'mediawiki.jqueryMsg' ],


'dependencies' => [ 'ext.SimpleBatchUpload.jquery-file-upload', 'mediawiki.Title', 'mediawiki.api', 'mediawiki.jqueryMsg' ],

Revansx (talkcontribs)
Ti infotrad (talkcontribs)
F.trott (talkcontribs)

test for existing file

Summary by Revansx

author invites anyone interested in this feature to develop it and they will likely incorporate it.

Revansx (talkcontribs)

is there a way to disable SimpleBatchUpload for existing files?

For example: if I try to upload 12 files at once and 3 already exist, those 3 would fail and the other 9 would succeed. Is that possible? (talkcontribs)

I am also interested in this or at least if the uploader could warn the user that the are overwriting an existing file.

Revansx (talkcontribs)

@F.trott, Has there been any updates that address this question?

F.trott (talkcontribs)

No, sorry. If you need this functionality you'll have to submit a patch.

Revansx (talkcontribs)

Ok. I'll be happy to try my hand at providing a patch that does this, but would you be willing to say a few words and provide a little guidance about where in the code this feature would be most efficiently added? All I'm asking for is a little help getting oriented under the hood.

F.trott (talkcontribs)

The whole extension is rather simplistic. It uses two elements: the MW Upload API and a jQuery File Upload Plugin. To implement your feature you would need to

  • introduce a switch on the server side to enable the feature. This could be a simple config variable if you want to always turn the feature on. You might also think of a case by case solution for the #batchupload parser function.
  • transfer the setting to the client. Either using a "data-" attribute in the generated HTML or a JS variable (see MakeGlobalVariablesScript and ResourceLoaderGetConfigVars hook).
  • use the setting on the client side. You'd have to make "ignore warnings" depend on the setting.

Depending on how sophisticated you want this to be you might add an option (button, link) to force uploading/overwriting after the first fail.

Revansx (talkcontribs)

I'll try to tinker with it. Thanks for your help. I'm still novice.

Revansx (talkcontribs)

I'd be happy to just hard-code it as the default behavior so it fails for files that already exist.

i'm using SimpleBatchUpload (SBU) to set the template and template properties upon upload.. the problem is that if the file already exists, then SBU ignores the template and template data.. such that after the batch upload the user doesn't know which files do and do not have the template data. I would also be happy if SBU could be made to overwrite the existing template data.. that would be fine too, but the current behavior of uploading the file and ignoring the template data if the file already exists ruins the usefulness of the template call feature. (unless i'm missing something)


1.3.1 Suddenly stopped working on MW 1.30 site

Summary by Revansx

permissions error

Revansx (talkcontribs)

I recently had this extention working wonderfully well on my MW 1.30 site. I recently installed the "Semantic Extra Special Properties" extension (not sure if this is related) and then some time later discovered that the SimpleBatchUplaod extension no longered worked.

  1. I uploaded a file using the normal Special:Upload page to confirm that uploads were still working.
    • Standard upload page works fine.
  2. Then I went to Special:BatchUpload and tried to upload the file.
    • The page renders with the following error in the Browser Dev tools:
[Wz5GTIhOg9IHdMpVK-mH@wAAAAM] /mediawiki/demo/load.php?debug=false&lang=en&modules=ext.SimpleBatchUpload%7Cext.SimpleBatchUpload.jquery-file-upload%7Cmediawiki.api.edit&skin=vector&version=1z0k7yf   MWException from line 810 of /opt/htdocs/mediawiki/includes/resourceloader/ResourceLoaderFileModule.php: ResourceLoaderFileModule::readScriptFiles: script file not found: "/opt/htdocs/mediawiki//vendor/blueimp/jquery-file-upload/js/jquery.fileupload.js"

#0 /opt/htdocs/mediawiki/includes/resourceloader/ResourceLoaderFileModule.php(356): ResourceLoaderFileModule->readScriptFiles(array)
#1 /opt/htdocs/mediawiki/includes/resourceloader/ResourceLoaderModule.php(719): ResourceLoaderFileModule->getScript(ResourceLoaderContext)
#2 /opt/htdocs/mediawiki/includes/resourceloader/ResourceLoaderModule.php(687): ResourceLoaderModule->buildContent(ResourceLoaderContext)
#3 /opt/htdocs/mediawiki/includes/resourceloader/ResourceLoader.php(1069): ResourceLoaderModule->getModuleContent(ResourceLoaderContext)
#4 /opt/htdocs/mediawiki/includes/resourceloader/ResourceLoader.php(783): ResourceLoader->makeModuleResponse(ResourceLoaderContext, array, array)
#5 /opt/htdocs/mediawiki/load.php(53): ResourceLoader->respond(ResourceLoaderContext)
#6 {main}

I have confirmed that the file: /opt/htdocs/mediawiki/vendor/blueimp/jquery-file-upload/js/jquery.fileupload.js does exists.

any thoughts on what could be the cause of the sudden failure?

F.trott (talkcontribs)

Did you by any chance also upgrade MW recently?

I think you are seeing this issue: https://phabricator.wikimedia.org/T180994

If so, then you could apply the workaround mentioned on that ticket, i.e. modify /vendor/.htaccess

It should also work to run composer update from the /extensions/SimpleBatchUpload directory

Revansx (talkcontribs)

When I attempted to run composer in the /extensions/SimpleBatchUpload directory I recieved the following response:

[root@mysystem SimpleBatchUpload]# sudo -u meza-ansible /usr/local/bin/composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package mediawiki/mediawiki could not be found in any version, there may be a typo in the package name.

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.
 - It's a private package and you forgot to add a custom repository to find it

Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.
[root@mysystem SimpleBatchUpload]#

and advice?

F.trott (talkcontribs)

Remove the requirement for mediawiki/mediawiki from the composer.json. It normally makes sure that you have a compatible MW, but since you run composer directly from the extension directory that mechanism does not work. I guess I should just include that JS lib directly in the extension at some point, but that means that it will not be updated automatically anymore.

Revansx (talkcontribs)

composer did the update, but still no joy in the functionality.

[root@mysystem SimpleBatchUpload]# sudo -u meza-ansible /usr/local/bin/composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing blueimp/jquery-file-upload (v9.22.0): Downloading (100%)
Writing lock file
Generating autoload files
[root@mysystem SimpleBatchUpload]#

Upload Page produces the exact same error as initially reported above. hmm.

Revansx (talkcontribs)

It's just a warning, but could this: JQMIGRATE: jQuery.fn.bind() is deprecated be an issue?

Revansx (talkcontribs)

I changed /mediawiki/vendor from:

Deny from all


<FilesMatch "\.(gif|jpe?g|png|css|js|json|woff|woff2|svg|eot|ttf|ico)$">
  Order allow,deny
  Deny from all

.. no joy

F.trott (talkcontribs)

For the moment I have no idea why it still does not work after the composer update. If you want to fix .htaccess you have to Allow (not Deny) access to .js.

F.trott (talkcontribs)

Oh, maybe check that the file permissions are correct. I see that you are using the root account for maintenance. It could well be, that the webuser does not have access to your files.

Revansx (talkcontribs)

It was a permissions issue due to the account composer was running on.

Revansx (talkcontribs)

we can mark this as resolved. thanks! (talkcontribs)

The extension is very good. But how to integrate with PageForm?

Ankostis (talkcontribs)

You may assign semantic-properties to uploaded files by giving a template as the 2nd argument to in the #batchupload parser-function. This template may contain annotations like that:

[[Has some property::{{PAGENAME}}]]

That way the uploaded file will be linked to the page that was created from (the page hosting the #batchupload function).

Ankostis (talkcontribs)

For future reference: i used mediawiki-javascript to have a listbox next to the uploader giving the property(ies) to set on the uploaded file page.

This is the markup on the "owner" page for the files, \where the uploaded controls is to be presented:

<!-- $wgRawHtml = true; is needed -->
  <legend>Set doc topic</legend>
  <select id="doc-type" title="Document type">
    <option value="--undefined--">--generic--</option>
    <option value="Insurance">Insurance</option>
    <option value="Manual">Manual</option>
    <option value="Maintenance record">Maintenance record</option>
    <option value="Consignment note">Consignment note</option>
{{#batchupload:File template|owner={{PAGENAME}}|doctopic=}}

This is the template for the file pages:

{| class="wikitable"
! Document of:
| [[Doc of::{{{owner|Orphaned documents}}}]] 
! Document topic:
| [[Doc topic::{{{doctopic|--undefined--}}}]] 

And this is the javascript to modify the template given in #batchupload parser-function:

$(function () {
    $('.filer-prop-selector').each(function() {
        var topic = $('select#doc-type', this);
        var uploader = $('input.fileupload', this);
        function update_templates() {
            var topicval = topic.val();
            // eg: "{{File template|owner=A page|doctopic=}}"
            var old_templates = uploader.data('text')
            if ( !old_templates ) {
                console.warn(`OLD: ${old_templates}, OFFICIAL: ${offval}, TOPIC: ${topicval}`);
            } else {
                var new_templates = old_templates.replace(/doctopic=([^|}]*)/, `doctopic=${topicval}`);
                // Set attr or dataset don't work! 
                // See: https://github.com/blueimp/jQuery-File-Upload/wiki/API
                uploader.fileupload('option', 'text', new_templates);
                //Update also 'data-text' to be visible in chrome's devtools.
                uploader.attr('data-text', new_templates);
        update_templates();  // ensure set even if not fired.

// [[Category:Filer machinery]]

F.trott (talkcontribs)

Currently no integration is foreseen.

The SpecialPage could probably made includable, so you could display it on a Form and upload files, but that would be it. There is no way to transfer the names of the uploaded files to the final page.

If you a feature like this, I think you should ask for it on the PageForms discussion page (or open a Phabricator task).

Reply to "Integrate with PageForm"