Requests for comment/Future of magic links

Background
Magic links are a feature of MediaWiki core that create automatic links for 3 hardcoded external identifiers:


 * -> ISBN 0-7475-3269-9
 * -> PMID 1234
 * -> RFC 1234

For the purposes of this RfC, we are not considering free external links (e.g. typing just https://www.example.org ) to be a magic link.

Problem
These are hardcoded, inflexible, un-localizable, and generally unexpected. If this feature were proposed today, it would be rejected in favor of using templates or interwiki links. There have been long standing requests to make them disable-able, move them to an extension, or remove them outright.

In many cases, local templates are preferable and more advanced than magic links. For example, on the English Wikipedia, Template:ISBN checks for invalid ISBNs and adds them to a tracking category for editors to fix up.

Proposal
As of 309528 it is now possible to disable magic link functionality using. Our eventual goal should be to remove all magic link functionality from MediaWiki core. While moving them to an extension would allow us to remove them from core, it would probably entrench magic links even deeper in the parser as we would need a mechanism and hook system to support an extension to add magic links.

Instead, we should:
 * ✅ Add a tracking category (via categories, one for each link type)
 * Add a parser function that replicates the functionality of the magic word (validation and linking to Special:Booksources).
 * It has also been suggested that we could use a "ISBN" virtual namespace that redirects to Special:Booksources
 * Discussion on phab:T148274 is still ongoing.
 * VisualEditor and other editing tools would continue to support ISBN/RFC/PMID "magic" links as they do, but convert it to a proper link client-side.
 * ✅ RFC and PMID should be added to the Wikimedia interwiki map and the MediaWiki default one if they aren't already.
 * ✅ In the MediaWiki 1.28 release, default magic links to being disabled. Wikimedia wikis would still have it enabled for now.
 * Encourage users to migrate to using the parser function and interwiki links or local templates (e.g. w:Template:ISBN), probably using bots and other assisted tools.
 * Update: this part was controversial, and will be re-evaluated based on the response to disabling by default.
 * We would retain and co. but they will always return false, in order to signal to extensions that the magic links are disabled.
 * Old revisions would no longer have autolinks, but that should not reduce the readability of the content as they are well known identifiers.
 * Move Special:Booksources and the ISBN parser function to an extension.

Comments
What for would you do this? It will clutter all the projects with extra template calls and other extraneous crap. Rich Farmbrough 18:03, 27 December 2016 (UTC).


 * I'd say this is summarised in the section. —Th e DJ (Not WMF) (talk • contribs) 15:44, 2 January 2017 (UTC)

There's concern on enwiki (here, for example) about the loss of the ISBN and PMID magic links. There's also concern about the lack of discussion. The first most of us knew about this was when a bot operator submitted a request to remove the links. Can the disabling of the links be halted so that a wider discussion can be held first? SarahSV (talk) 19:05, 29 December 2016 (UTC)


 * Lets look at one "reason" un-localizable,  that issue is just tagged as "won't fix" for "the absence of objections."  I would have said that localisation is totally trivial, certainly as easy as localising a set of templates.  Rich Farmbrough 23:16, 2 January 2017 (UTC).


 * Has there been any real discussion on the actual sites? This will have a huge impact. [It seems to me that this change goes against consensus? See Removal of ISBN magic links.] Jeblad (talk) 13:12, 15 April 2017 (UTC)
 * Yes, discussions are on-going. The English Wikipedia had an RfC and agreed to deprecated and eventually remove magic links. Legoktm (talk) 00:31, 12 July 2017 (UTC)
 * Have you asked the German speaking community? Please provide a link. --Eingangskontrolle (talk) 15:06, 27 March 2018 (UTC)


 * At first I thought this cleanup might be a good idea, then I started to wonder if magic links is just the kind of feature that might solve the currency converter problem. Now we write into articles what a specific currency might be, and how it might be converted to a local currency, but by using magic links for the short three-letter codes we could simply parse out the currency and add a currency converter without cluttering the wikitext. This is not that common, so perhaps it is acceptable to use a parser function in those cases. It would probably not go long before someone have created templates for all currencies, with no real additional functionality, if it is added as a parser function. I would prefer silent parsing of an inline magic link. [Just to be clear; I do not support this RfC.] Jeblad (talk) 13:18, 15 April 2017 (UTC)


 * I think this is a really, really bad idea. It will result in many, many, many future ISBN citings for which editors won't use the template and thus won't be linked anymore. I am strongly opposing this RfC. --Matthiasb (talk) 05:20, 23 May 2017 (UTC)


 * Please keep the magic links, why increase work load for editors? And why clutter the source text, so that it becomes less readable to the vast majority of non-coders (writing an encyclopedia) and potential new authors? Cheers, --Ghilt (talk) 11:00, 23 May 2017 (UTC)


 * I find the magic links to be useful too. They should not be disabled. I'm even proposing more magic links, e.g. for DOIs and URNs! --Gretarsson (talk) 15:30, 23 May 2017 (UTC)


 * Magic links are useful, templates are geeky. So the more templates, the less attractive the wikiverse will get for non-geeks. So like Matthiasb, Ghilt and Gretarsson. And this has to be discussed on at least 50-70 wiki-projects themself, not only here in some back room of geeks. Grüße vom Sänger ♫(Reden) 15:50, 23 May 2017 (UTC)
 * In general editors should never have to deal with magic links directly, nor do they need to know they have to add templates themselves. For example, w:Template:Cite book and related automatically link ISBNs, so users don't see any difference. Users who have trouble with wikitext can use Citoid's citation generator which handles all templates for them. Legoktm (talk) 00:31, 12 July 2017 (UTC)


 * You propose to disable an extensively used and useful feature without any need, without any existing alternative and (therfore) without any improvement. CONTRA --Fano (talk) 16:03, 23 May 2017 (UTC)
 * The intial motivation is that we're embarking upon cleaning up wikitext, standardizing it, and modernizing it (see the related tidy migration and Linter project). This effort is being done so we can improve tools for editors, and get rid of a lot of technical debt. Magic links have always been a weird part of wikitext, and developers have often been unhappy with the concept due to how magical it is ("explicit is better than implicit"). Alternatives have been proposed and currently the one everyone agrees on is using templates. I don't disagree that for some people this will be a regression in functionality, but overall and in the long run it will be an improvement in my opinion. Legoktm (talk) 00:31, 12 July 2017 (UTC)


 * I miss any reason for abandoning these widely used features. The claims above under the header "problem" seem ridiculous to me. Un-localizable is patently untrue, as localizing them is as easy as apple-pie. Hardcoded is exactly why they are preferable to templates, they need no user activity. And while they are inflexible, that too is exactly what they are supposed to be. They fulfill one purpose, and do so very well. Unexpected is the most laughable reasoning I have read so far on this wiki. These features have been coded for a reason, they shall and do link three well defined information patterns of utmost importance to well established resources. Some years ago you removed DOI from the magic words, and I still miss it. So my proposal would be to reintroduce DOI as magic word ASAP, and certainly keep the three existing ones. --H-stt (talk) 09:26, 24 May 2017 (UTC)
 * +1 -- hgzh 11:11, 24 May 2017 (UTC)
 * +1 --Syrcro (talk) 05:45, 27 May 2017 (UTC)
 * +1 --Albinfo (talk) 19:55, 28 May 2017 (UTC)
 * +1 --ZdBdLaLaLa (talk) 20:36, 31 May 2017 (UTC)
 * +1 (a raw ISBN can be replaced by a template without disabling this feature, so it is independent) Rjh (talk) 05:55, 1 June 2017 (UTC)
 * +1 --Lucas (talk) 14:49, 14 June 2017 (UTC)
 * The stated problem: These are hardcoded, inflexible, un-localizable, and generally unexpected. Judging from H-stt's comment above (and similar comments), and his support, I'm inclined to say that this should not be considered a valid problem statement. Is there a technical, back-end, maintenance reason that magic links are difficult to work around? Why has it been okay to support it all this time and then just decide to stop supporting it because it's not okay anymore, and then claim that it wouldn't have been introduced today? D. F. Schmidt (talk) 16:49, 20 June 2017 (UTC)
 * You can find comments from developers for years (e.g. T22832 from 2009) that magic links are technical debt and only around for backwards-compatibility. As I mentioned above, there's currently a push on modernizing and cleaning up wikitext, and this has been one of the things on that list. Magic links require specialized code to handle properly, and are one thing that works differently from the rest of wikitext. I think no one has really bothered or wanted to deal with all the complaints that would come out of removing magic links. We've also advanced significantly in citation related technology (e.g. Citoid, Scribunto) that magic links aren't actually used as much as they used to. Legoktm (talk) 00:31, 12 July 2017 (UTC)
 * +1 to D. F. Schmidt's comment. The current arguments given in the Problem section seem very vague and don't describe the issue in much detail at all. K6ka (talk) 12:30, 5 July 2017 (UTC)
 * +1 -- Lostraven (talk) 18:58, 6 July 2017 (UTC)
 * If you think something is "as easy as apple-pie" it's a shame you didn't submit a patch earlier to implement that functionality - however I doubt it's as easy as you think. And while you might see hardcoded/inflexible as a feature, many people see it as exactly the opposite. As for being unexpected, see T70217, or w:Wikipedia:Village_pump_(technical)/Archive_8, etc. AFAIK DOI has never been a magic word, and I certainly never removed it. It's still an interwiki link if that's what you meant. Legoktm (talk) 00:31, 12 July 2017 (UTC)


 * Comment from technical site: It will be very helpful to change the Magics to Templates, because of a better possibility to localize or search those contents by API or as database entry as Table Templatelinks including parsing key and value. The edit on article pages is not very more difficult, but we should use our de:Template:Zitation simply a little more to catch such entries. I only can see a helpful optimization and we should do this. Doc Taxon (talk)


 * The ISBN magic link has been in use several dozen million times, I guess. This RfC is, in comparison, absolutely nothing. I can't take this seriously the way it looks. Man77 (talk) 16:08, 25 May 2017 (UTC)
 * Why do you think this RfC is "absolutely nothing"? Legoktm (talk) 00:31, 12 July 2017 (UTC)
 * Because it ain't done where those links are used, just here with the geeks. Go to the portals in the wikis, where they are heavily used and make some dozen RfC there, and then come back here. Templates are far less useful for editors as plain text, that automagically gets transformed, templates shoo new users without much interest in using strange syntax away. Grüße vom Sänger ♫(Reden) 16:10, 12 July 2017 (UTC)


 * I am agnostic on the proposal itself, but advise that the recommended list of alternatives should include, e.g.,.


 * The parts of the proposal that are shown as done are good to have (like putting in the ability of disable magic links), but I strongly oppose the idea of removing magic links in total. Even if all present ISBNs in all Wikimedia projects are migrated from templates to magic links, this change just adds yet another burden to editors and clutters page source. As per H-stt's opinion, I also support adding back DOI to magic links. If you don't like it being hardcoded, move it to an extension and make it customizable there. Removing it without considering the impact to (the future of) existing projects just does not make sense. --ネイ (talk) 15:55, 19 June 2017 (UTC)
 * A waste of editors' time, with no clear explanation. @Legoktm, the benefits mentioned were vague and the feedback was quite negative. Why did you press ahead? The decision-making process seems to have failed here. What is the correct venue to address the decision-making process? --Hobbes Goodyear (talk) 01:09, 25 June 2017 (UTC)
 * Support I assume this is used by the visual editor people? Typing ISBN/0-7475-3269-9 or ISBN  instead of ISBN 0-7475-3269-9 is not difficult. Power~enwiki (talk) 21:29, 4 July 2017 (UTC)
 * This is the first I'm learning of this. The wikis I run use magic links extensively. Using journal as an example, our citation format has been and will continue to be with a references section at the bottom. I don't claim to fully understand what you all are proposing to do. All I know is that for now I just have to plug in a DOI, PMID, etc. into the citation template and it works. It sounds like putting those IDs into the template would no longer produce links in the results shown in the references section; we'd somehow have to add an additional template and thus make it more complicated. The wiki is complicated enough, particularly for new users. Against this change, for the same reasons as H-stt. Lostraven (talk) 18:56, 6 July 2017 (UTC)
 * If you're using a standardized template, then this change affects you very little. Just have the template create the links directly (e.g., and in the articles themselves nothing will change. Legoktm (talk) 00:33, 12 July 2017 (UTC)


 * Oppose - solution which is searching for a problem. Unfourtunately the VE ist already starting to change it, making mistakes as usual. By the way: Who is we? --Eingangskontrolle (talk) 14:13, 27 March 2018 (UTC)


 * Support - the Magic links functionality should be moved to an extension that Wikimedia projects can debate about in terms of future use. Magic Links as currently set up breaks my external wikis, such as https://gowiki.tamu.edu, and others that rely on creation of page titles based on PMIDs. I have not figured out how to patch this yet, which is preventing upgrades to more recent versions of MW (currently testing with 1.29) JimHu (talk) 20:48, 8 April 2018 (UTC)


 * Support with caveats: I have used MediaWiki's magic-link type functionality in quite a few private wikis: in many organisations, referring to "Job 1234" is an unambiguous reference to a given project; so I've made  expand into  .  (Currently I do this by modifying the existing special-case code to recognise the tag and do a database lookup for the project name.  While it is certainly ugly, for these groups it is extremely useful.)  But those same wikis have ,   and   disabled, as they are not meaningful for their user groups.  So if we generalised it away into an extension, to ease   or   that would be great.  I truly understand these uses are not likely to help Wikipedia etc, but they enormously have helped the organisational wikis I've put in.  Thus I oppose simple removal, as it would become much harder to do.  Jonathan. 82.69.229.22 13:09, 27 April 2018 (UTC)

For your information, there is now Extension:MagicalLinkers. Nuno Tavares (talk) 13:09, 7 August 2018 (UTC)


 * Removing magic links seems more and more like the wrong action to me. It is advocated as an action solving a few claims; hardcoded, inflexible, un-localizable, and generally unexpected. The claim that they are hardcoded is about the current implementation. If the implementation is wrong, then fix it! The claim that they are un-localizable is also about the current implementation. Once more, if the current implementation is wrong, then fix it! Then a pretty vague claim that they are "generally unexpected". I believe that is a false statement. Something that has been on Wikipedia for so many years is not unexpected. It might be unwanted for some users, but it is not unexpected. (This is like an insurance company getting a claim about a tree unexpectedly growing up in front of a car.) This kind of functionality is infact very important as it annotate text without any user involvement. If we could generalize on the idea it could be described as a way to rewrite a pattern into a template call, which could then be redirected into a module and a gadget. A call on a three-letter currency code, followed by a number, could then be redirected to a currency-converter. Likewise a call on a value with a following unit could then be redirected to a unit-converter. A third is a converter between date and time in different timesones and different calendars. (There are several similar examples.) We can move these examples out in separate extensions, but the problem is general and should be solved in the core itself. A simple implementation is to use regular expressions to find the patterns, but this can be very heavy if users are allowed to add them. Still this is more or less whats done in the AbuseFilter extension. I'm not sure, but I wonder if the problem is that some types of natural language is closer to structured data, and in some cases the structured data is so close that we can utilize it as-is. In those cases it virtually makes no sense to wrap it in additional layers of formatting. In these cases the additional layers are just noise. ISBN codes are such structured data. Additional templates, parser functions, or whatever, are just noise. Jeblad (talk) 21:01, 18 December 2018 (UTC)