Topic on Extension talk:SimpleBatchUpload

New feature: Rename the filename with a regular-expression

27
Ankostis (talkcontribs)

Are there any thoughts for providing a method to modify the filename of the uploaded file with some kind of user-defined callback function (in javascript, lua, template) accepting the original file and converting it into the final one?

F.trott (talkcontribs)

It's not currently planned, but you're welcome to submit a patch.

Ankostis (talkcontribs)

But i wouldn't know how to go about doing it. Maybe allow a new "special" parameter `@fnametemplate` in {{#batchupload}}? Would it be possible for javascrript code to call back to server to discover the filename? Or better allow a parameter named `@fnamefun` and let the user attach such a global javascript function? Or to the prototype of the upload??

F.trott (talkcontribs)

There's probably a number of ways to go about it. One of the main points to consider is security. Allowing batch uploading dozens of files is already borderline dangerous and probably not a good idea to activate for public wikis.You have to carefully consider how to implement your functionality without opening additional attack vectors.

Ankostis (talkcontribs)

FYI, you may review my take on this problem in this commit: https://github.com/ankostis/SimpleBatchUpload/commit/2bd2c7c It is not for the wider public (e.g. it uses ES template literals) but you can get what i wanted to achieve:

Add a new parameter that specifies a regular expression for renaming the uploaded file-names:


{{#batchupload: ...|+rename= #pattern#flags -->replace-string}}
  • Patterns can be surrounded with #/@! chars, and
  • can be followed by regex-flags 'idmuy', while
  • all spaces in the replace-string after the --> are significant (and respected).

Example

Here it is the ''regex'' i use to embed the page-name of the page holding the upload-control into the start of the uploaded-filename, erasing any extra spaces or punctuation-chars supplied by the user:

{{#batchupload:ItemDoc|owner={{PAGENAME}}|+rename= !^({{PAGENAME}}[ -_./+]*)?!iu -->{{PAGENAME}}-}}
Longphile (talkcontribs)

@Ankostis The rename with regex feature is a great addition; unfortunately, it is not working for me when I tried your fork. I am running MW 1.31.1 with php 7.2.11. Using drag and drop or browsing to select a file to upload, I get "ERROR: The filename is not allowed." I would like to be able to rename uploaded files to ensure unique filenames in our wiki.

Ankostis (talkcontribs)

Please take the very latest from rename-files branch. I had indeed a bug when you didn't specify a rename.

Revansx (talkcontribs)

Hi @Ankostis, Does your "rename-files" branch work with MW 1.30?

...also, how hard would it be for you incorporate a new switch that won't let it upload a file that already exist?

Is that something I could tempt you to add? :-)

Ankostis (talkcontribs)

With my basic knowledge of mw and how to debug it, it should be a day's work, which i don't have for the foreseeable future. But you can give it a try: first locate the JS-API that searches if a page exists (eg exists()), and then use chrome's dev-tools in debug-mode to craft the javascipt code for the following actions:

  • Specify a regular expression to extract the new switch-parameter (i would call it `+skip if exists` or `+new files only`)
  • call the API discovered above,
  • avoid the upload and just set a message in the returned result explaining why the upload was denied.
Revansx (talkcontribs)

sounds easy. I really need to figure this out.

Would I be making this change to your 'rename-files' branch or would I be making a branch of your branch? This is where I get confused about git.

Revansx (talkcontribs)

@Ankostis .. so.. I just downloaded and tested your "rename-files" branch... and I love it! .. thank you!

Quick question .. what would happen if the PAGENAME contains "/" characters? (I.e. subpages) would that break anything?

Revansx (talkcontribs)

I just tested it and see that it converts the "/" to "-". Perfect!

Revansx (talkcontribs)

Can you tell us how to write the' 'regex'' part so that all files dropped on the button are set to the same name? (I don't envision using it for multiple files, but I would like to create an upload button that exists to upload one and one one file of a specific name. Thank you!

Revansx (talkcontribs)

I'm really proud of myself. I did it :-)

Here is the 'regex' I used to upload files to a fixed filename regardless of what the supplied filename is:

{{#batchupload:ItemDoc|owner={{PAGENAME}}|+rename= !([a-z0-9_ ()-.&]*)?!iu -->myfilename.xyz|}}
Ankostis (talkcontribs)

> Would I be making this change to your 'rename-files' branch or would I be making a branch of your branch? This is where I get confused about git.

You got to clone my repo to your own github accont and push there a new branch.

> I'm really proud of myself. I did it :-)

Great! Is that enough for you? I mean, don't you want to deny uploads if files exist?

Revansx (talkcontribs)

I do. I do.. but baby steps.. for now site's needs are met by checking for the expected file using the following:

{{#ifexist:File:somefile.xyz
| [[Media:somefile.xyz]]
| {{#batchupload:ItemDoc|owner={{PAGENAME}}|+rename= !([a-z0-9_ ()-.&]*)?!iu -->myfilename.xyz|}}
}}

But it's on my list of things to do to learn to clone the repo and push a new branch.. and then.. make the commits and test it.. I'm eager to try. would you be willing to review my commits when I have them?

Ankostis (talkcontribs)

Sure. But better ask the original author to do that - i have as much experience in mw/smw as you do :-)

Revansx (talkcontribs)

aye. fair enough. Nonetheless. I'll let you know when I do so.

F.trott (talkcontribs)

I am always happy to review pull requests.

Regarding checking and rejecting uploads I am not sure if the proposed approach would be the best solution and if SimpleBatchUpload would be the best place to put it:

  1. Any such functionality that is implemented client-side in JavaScript is easy to circumvent, so is at best a cosmetical measure.
  2. Each file check would require an additional request to the server, making the whole thing inefficient.
  3. The functionality would only affect uploads by SimpleBatchUpload. Standard uploads would still be possible.

A better way might be to find a hook in the upload process that checks an upload and blocks it if necessary. Manual:Hooks/UploadVerifyUpload and Manual:Hooks/UploadVerifyFile look both promising.

It might also be worth to look for an extension that already provides this or a similar functionality, to use it as is or take it as a template for a dedicated extension providing this functionality. What about Extension:UploadBlacklist?

Revansx (talkcontribs)

@F.trott, Thanks for weighing in.

[Security] - My problem is not with security as much as it is with cosmetics/logistics so this would be fine for me.

[Efficiency] - The time it takes for a user to upload the file is likely much longer than the time it takes the script to do the check. This is not a concern for me.

[only works for this SimpleBatchUpload] - yes, exactly. That's a feature not a bug :-)


That said, I've researched and tested so many upload extensions and SimpleBatchUpload is really the best for my application. None of my users want to touch the Upload Wizard or the Standard Upload.. I can add your SimpleBatchUpload buttons to templates and SMW page classes and build amazing tools to solit file from people. With @Ankostis's branch I can force filenames and with the expensive file checking in the template i can disable the button with page logic. That said I'd *really* like to have this feature.. So I hope to add it as discussed.. and I do hope you'll be willing to include these features into your master branch even if it does come with some caveats on security and efficiency and completeness.

F.trott (talkcontribs)
Revansx (talkcontribs)

F.trott, you rock! ... I'll test it soon enough, but what kind of error message would SimpleBatchUpload show in such a case? ... that and (since I haven't read everything on the topic yet) do you know if there is any way of enabling the re-upload permission for the creator (aka original uploader) of each file? i.e. UserX can re-upload UserX's files, and UserY can re-upload UserY's files but neither can re-upload the others files. Is that possible?

Revansx (talkcontribs)

@Ankostis, I'm using your version of SimpleBatchUpload that allows a name change and specifically I'm using your example of adding the pagename as a prefix per:

{{#batchupload:ItemDoc|owner={{PAGENAME}}|+rename= !^({{PAGENAME}}[ -_./+]*)?!iu -->{{PAGENAME}}-}}

The problem I'm having now is that when someone "re-uploads" a file that already has the pagename as a prefix, the regular expression does something strange and produces a file that has no filename at all.


Am I doing something wrong? .. Have you considered this case where a file that has been uploaded with the "add pagename prefix" is downloaded (with the prefixed name) and re-uplaoded with the same tool?


any thoughts on how to handle this?

Ankostis (talkcontribs)

Yes i have, and should be working. Just make sure you have latest commit of the branch, since there was a bug in the intermediate commit i mentioned in my initial post.

Revansx (talkcontribs)

looks like this works well:

|+rename= !^({{PAGENAME}}-)?!iu -->{{PAGENAME}}-}}
Revansx (talkcontribs)

is there any progress on getting this feature included in the main extension?

Revansx (talkcontribs)
Reply to "New feature: Rename the filename with a regular-expression"