Talk:Citoid/2017

Previous archives are at /Archive 1

Bugs: pmc, redundant urls, and ref tags

Not sure where the best place to post this. But here goes:

I never use this tool myself, but I frequently am cleaning up after others that do. Three problems that I have noticed:

(1) in the PMC parameter, the value should be a integer and not prefixed with "PMC":

incorrect:|pmc=PMCxxxxxx (where xxxxx = integer)

correct: |pmc=xxxxx

The incorrect form throws an error that must be manually fixed.

(2) urls are sometimes added that are indentical to that already produced by |doi=, |pmid=, or |pmc=. In this case, the url should be suppressed since it is redundant.

(3) Ref tags that take the form of ":0", ":1", ":2". While they are unique, they are not very informative. Better to create a Harvard style ref tag, the form of first authors last name + year of publication (i.e., "Smith_2017" is much more readable than ":0"). Boghog (talk) 15:13, 2 April 2017 (UTC)

A bit of googling uncovered these:

1) https://phabricator.wikimedia.org/T157152#3125257 & https://publicaccess.nih.gov/include-pmcid-citations.htm#Examples.

3) https://phabricator.wikimedia.org/T92432.

At least for 1 it seems more like a case where the wikipedias don't want to follow the recommended (or required) referencing style of the publishers. 197.218.88.122 (talk) 16:39, 2 April 2017 (UTC)

Thanks for the links, however is important to distinguish between a template imbedded in wiki markup and how that template is rendered. The NIH recommendations only concern how the citations are rendered, not how they are entered in a {{cite journal}} template. The NIH has no recommendation on the syntax of templates.

The {{cite journal}} template renders the citation very close to the NIH recommendations (PMCxxxxx). The only difference is that {{cite journal}} adds a space between PMC and xxxxx. PMC in turn contains a wikilink to a Wikipedia article that explains what PMC stands for.

It is completely unnecessary prepend the parameter value with "PMC". This is redundant. The template itself produces the correct rendering. Hence the problem is with Citoid, not {{cite journal}}. Boghog (talk) 17:56, 2 April 2017 (UTC)

Well, that's a double edged sword. It is true that template markup differs from rendering, that means that it doesn't make any difference to a template if it receives a pmc = pmc XXXX or pmc = XXXX since it can be removed by parser functions or lua. So it comes down to aesthetics or personal preferences of some users because the template can be changed to accept both if there's consensus to do so.

The WMF developer's point in the phabricator task seems to be the same as the one in the publisher's site. The id is "PMC XXXXX", and the site also recommends "PMCID : PMC XXXX". Pages like Cancer AND Cholera don't seem to follow that recommendation anyway. While wikimedia users may prefer to use only integers to identify it, citoid and many wikimedia tools are also used by third parties, and such exemptions may not be wanted by them. There's also no guarantee that all wikipedias use the same format as english wikipedia, so it is possible that other wikis were doing the exact opposite, e.g. adding PMC where it only had an integer. The only alternatives are to change the code only for the wikis that prefer it that way, or leave it as is.

They stripped it previously (https://phabricator.wikimedia.org/T78144) without doing the research to understand the "official" preferred value, this time they decided against it after doing the research.

Anyway, I'm not WMF developer, nor associated with wikimedia, you'll have to convince them :). 197.218.88.122 (talk) 18:28, 2 April 2017 (UTC)

Can you cite a single example of another Wikipedia including "PMC" in the parameter value? Most other Wikipedias don't support pmc to begin with, and the ones that do have generally have followed the English Wikipedia's lead.

Citation Style 1 templates were created before Citoid. Hence it is reasonable that Citoid be compatible with Citation Style 1 templates, not the other way around. Furthermore, none of the other citation generation tools include "PMC" in the parameter value. I suppose that Citation Style 1 templates could be modified to optionally accept PMC in the parameter value, but this unnecessarily clutters citation templates with redundant characters. The parameter name is pmc, why does the parameter value also need to include pmc?

I have reopened the case on wikimedia. Boghog (talk) 19:04, 2 April 2017 (UTC)

Can you cite a single example of another Wikipedia including "PMC" in the parameter value?

You sparked my curiosity, and here's one example:

https://el.wikipedia.org/wiki/%CE%91%CE%BC%CE%B9%CE%BD%CE%BF%CF%84%CE%B5%CE%BB%CE%B9%CE%BA%CF%8C_%CE%AC%CE%BA%CF%81%CE%BF

Citation Style 1 templates were created before Citoid

True, but that would be over fitting. There are more than a hundred encyclopedias, and the tool should be neutral and not cater to a single one. That's what it does, it returns the id as requested. Stripping it means that it just returns a number.

The parameter name is pmc, why does the parameter value also need to include pmc?

Knowing the full PMC means that users can quickly verify if citoid actually gave them the right data, instead of guessing. This is particularly important because a number can mean just about any random thing (e.g. a PMID number instead of a PMC). Just because it adds it as a parameter doesn't mean it is necessarily correct. 197.218.88.122 (talk) 19:40, 2 April 2017 (UTC)

In the Greek Wikipedia, using a pmc parameter value where PMC is prepended is optional. However the rendering with when PMC is included in the parameter value looks strange (PMC is displayed twice and PMCID is not displayed). This should be fixed.

The whole purpose of Citoid is to automate citation generation process so the editor doesn't have to worry about the accuracy of the parameter values. Since the data is downloaded from PubMed, the parameter values with a high degree of confidence are correct.

Numbers are not any less random if PMC is prepended to the parameter value. The best test to verify that the the parameter value is correct is to follow the rendered PMC link. Boghog (talk) 20:07, 2 April 2017 (UTC)

Boghog, I tend to think that the IP is correct: it would be good for {{cite journal}} to accept the "formally correct" id number. This has been discussed for a couple of months now, and I've not yet heard any technical reason for the template to choke when it's given the "official" id number. I don't see a need to require the official id number, but it should be able to cope with it. Whatamidoing (WMF) (talk) 06:06, 3 April 2017 (UTC)

One of the bugs mentioned above is mentioned in m:Community Wishlist Survey 2019/Citations/VisualEditor: Allow references to be named. Voting is open for about another four days. Whatamidoing (WMF) (talk) 18:33, 26 November 2018 (UTC)

Two years on, and I'm still getting this error… The automated citation bot returns PMC=PMCPMCnnnnn (where n is a digit), and Wikipedia—correctly—identifies it as an error.

It's also generating dates that are wrong, according to Wikipedia own's style manual. Guarapiranga (talk) 22:41, 29 May 2019 (UTC)

The pmc=PMCPMCxxxxx is a new bug. I guess if two PMCs is good, three must be better ;-) I still do not understand the rationale for prefixing the numeric PMC value in citation templates. The pmc parameter name makes it clear what it is. In any case the redundant PMCs are being systematically removed from the English Wikipedia by gnomes and bots. Boghog (talk) 04:55, 1 June 2019 (UTC)

The (obviously wrong) double PMC problem has been fixed.

In case this comes up again, the point behind "prefixing the numeric PMC value" was that PubMed said that the complete identifier includes the "PMC" code. The English Wikipedia has historically used a truncated representation of the full id code. This is the equivalent of someone saying that their "correct" telephone number includes the local number but excludes the country and city/area codes. It's functional in some contexts, but it's not complete or unambiguous. Whatamidoing (WMF) (talk) 18:18, 2 August 2019 (UTC)

Why is the context outside of Wikipedia even relevant? Will someone ever harvest the raw cite journal template to use outside of Wikipedia? I doubt it. Even if they did, it would be a trivial matter to do a search and replace of "pmc=xxxxx" with "PMCxxxxx".

Within Wikipeida, the template, the pmc parameter name, and the rendered citation all make crystal clear the context. Why do we need to specify pmc=PMCxxxxx? The second PMC is completely redundant and unnecessary. pmc=xxxxx is clear enough. The redundant PMC prefix is being treated as a maintenance issue (see https://en.wikipedia.org/wiki/Category:CS1_errors:_PMC) and is systematically being removed by bots and gnomes.

In addition, PubMed itself is inconsistent. The PMC ID is prefixed by PMC, but the PMID ID is not prefixed with PMID? Why is that?

Finally It is true that NIH grant applications must specify PMCID: PMCxxxxxx (see https://publicaccess.nih.gov/include-pmcid-citations.htm). Based on that logic, then we would need to replace pmc=PMCxxxxx with pmc=PMCID: PMCxxxxx. Right? But how often does an NIH grant applicant resort to copying a Wikipedia citation template for one of their own publications? I bet that it has never ever happened.

Boghog (talk) 19:57, 2 August 2019 (UTC)

I think that if you asked people familiar with the English Wikipedia's core community, then you'd likely find that they're (we're) generally perceived to be fanatical about getting the facts as absolutely correct as humanly possible. Consequently, it feels strange to me that these same people seem to shrug their shoulders and say "Yup, there exists a canonical form for that identifier, but I'd rather do it my way than get it right".

The fact that a bot is "un-correcting" the canonical form (when the template could be changed to accept both forms) is particularly strange. Can you think of any other instance in which an identifier or similar objective fact is deliberately rendered in the "wrong" form? CheBI, ChEMBL, DTXIDs all get their prefixes.  Why not this one? Whatamidoing (WMF) (talk) 20:43, 3 August 2019 (UTC)

One needs to distinguish between data stored in a template and how it is rendered. This discussion is about the former.

In both {{infobox chemical}} and {{infobox drug}}, the CheBI parameter only accepts an integer value without a prefix. If one prepends the integer accession number with "CheBI", it generates a link that triplicates the CheBI prefix (e.g., CheBI: CheBiCheBixxxx) and the rendered external link is non functional. Ditto for ChEMBL. I am not sure what DTXID is.

Therefore, the cite journal pmc parameter is similar to the infobox chemical/drug parameters CheBI and ChEMBL except the pmc parameter will accept the pmc prefix, strip the (redundant) prefix from the rendered citation, and generate a functional link, but in addition, will flag it as a maintenance category.

The pmc value is stored in a parameter that is called pmc. So why do we need to repeat pmc in the parameter value? Boghog (talk) 21:48, 3 August 2019 (UTC)

One additional item. The NCBI eutils search engine (NCBI administers PubMed Central) allows searches without the PMC prefix:

https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=212403

and also returns PMC accession numbers without the prefix. The output of the above URL reads in part:

<article-id pub-id-type="pmc">212403</article-id>

This suggests that the NCBI stores PMC accession numbers in their internal databases without the prefix and adds the prefix when rendered in PubMed pages. {{cite journal}} templates are analogous to a database that does not store the prefix in the pmc parameter value, but does display the prefix when the template is rendered in a Wikipedia article.

The NIH regulations do not specify how pmc accession numbers should be stored in databases. They only specify how they should be displayed in a written grant application. Boghog (talk) 11:18, 4 August 2019 (UTC)

API shows the word "pages" only and I can't see any template fields

I have followed all the steps in the documentation over a week ago, and until now the template items don't show in the visual editor citation module.

I followed the steps in this section here: Citoid#Empty_references_appear, but all what appears in the api is this:

{
    "pages": {}
}

the template definitely has data in the :

"maps": {

    "citoid": { 

and also has the correct templatedata codes.

I executed the jobqueue several times to make sure there are no pending jobs, and null-edited the templates several times.

Any suggestions are welcome, please. Thanks in advance. Atef81 (talk) 05:00, 5 May 2017 (UTC)

Which wiki are you trying to fix? Whatamidoing (WMF) (talk) 17:37, 29 May 2017 (UTC)

Sorry I missed this message. I am trying to fix a Mediawiki installation I have. it is not part of the Wikimedia project. is this what you are asking or I missed the question? thanks. Atef81 (talk) 16:14, 13 June 2017 (UTC)

That was my question. I don't know how to fix your problem, but if it was a WMF wiki, then I could try to find someone who might be able to fix it for you.

You've already tried all the things I would recommend to you. @Mobrovac-WMF, if you see this ping, do you have any advice? Whatamidoing (WMF) (talk) 17:33, 16 June 2017 (UTC)

ISBNs citations now with autofill!

Hi all, in the Wikipedia Library program at the Wikimedia Foundation we have been working with OCLC to make autofilled ISBN citations available, through using their WorldCat database. We have deployed the feature on all language Wikipedias: you can learn more about it on the Wikimedia blog: https://blog.wikimedia.org/2017/05/11/wikimedia-oclc-partnership/

Cheers, Jake Ocaasi (WMF) (talk) 19:39, 11 May 2017 (UTC)

I've had problems with the publisher being entered in the author fields using the ISBN lookup. Is this a known problem, and do you want examples? PKM (talk) 06:30, 25 May 2017 (UTC)

Not a known problem. Examples please! Even better if you file in Phabricator here: https://phabricator.wikimedia.org/T145462 Ocaasi (WMF) (talk) 06:58, 25 May 2017 (UTC)

Will do. Both Museum publications, PKM (talk) 20:21, 25 May 2017 (UTC)

and done. It's consistent for books published by musuems (three out of three tries). I just love edge cases. - Paula PKM (talk) 23:13, 25 May 2017 (UTC)

Should be added to the main description? Thanks Jake! Merrilee (talk) 17:17, 18 May 2017 (UTC)

Citoid: several publishers recently missing?

Hi,

I think Citoid is the most fantastic service for mediawiki and I started using it about a year ago to get automatically information on many publications, based on the DOI. Really great work!

Not too long ago, Citoid switched to a new API endpoint and I was able to connect to that as well but I noticed that e.g. the Wiley publisher is suddenly not supported anymore (but used to be supported in the past). Here are some examples of articles of which data can no longer be retrieved anymore:

Wiley publishers: https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1046%2Fj.1365-2117.2002.00186.x

AGU publications (part of Wiley): https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1029%2F94WR00436

However, the service works great for e.g.:

https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1111%2Fj.1751-8369.2002.tb00087.x

Would the Wiley publisher become available again through the API in the foreseeable future?

Thank you!,

Albert.

I have posted this on phabricator as well ( https://phabricator.wikimedia.org/T165105) Albert Ke (talk) 22:09, 17 May 2017 (UTC)

Hi Albert,

We have historically had trouble with scraping Wiley - something to do with their redirect system puts us into a redirect loop and can cause timeouts. But the DOI SHOULD work (since then we have the DOI to work with to query crossref even if we can't access the website) so that sounds like a bug. I will look into it further.

Thanks for reporting this! Mvolz (WMF) (talk) 07:13, 18 May 2017 (UTC)

Appreciated Mvolz,

Hope it is something easy to solve!

Albert. Albert Ke (talk) 13:54, 18 May 2017 (UTC)

Update: It looks like this is (mostly?) a problem involving RESTBase. I don't know when it will get fixed, but they're working on it. Whatamidoing (WMF) (talk) 17:35, 29 May 2017 (UTC)

scribunto

Is scribunto really needed for working with citoid? If not, consider to remove it from the documentation here Eran (talk) 11:12, 25 August 2017 (UTC)

Publication date field for some scientific journals are not provided anymore

Tracked in Phabricator
Task T180408

Hi,

It looks like for certain scientific journals the 'publication date' field isn't provided anymore. For example for the "Earth Surface Dynamics" journal you will see the publication date (See e.g.: https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.5194%2Fesurf-5-21-2017 which has the publication date provided as: "date":"2017-01-16").

However, the "Geology" journal, published by "geoscienceworld", has no publication date through zotero (See e.g.: https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1130%2Fg38665.1 (only access date is shown)).

Would there be a quick fix such that publication date can be pulled through zotero as well?

Thanks, Albert. Albert Ke (talk) 02:12, 10 November 2017 (UTC)

I don't know how to check for this myself, but usually, when something like this happens, it means that the 'broken' sources have rearranged their websites, and that the Zotero translator needs to be updated as a result. Whatamidoing (WMF) (talk) 17:12, 10 November 2017 (UTC)

Thank you @Whatamidoing (WMF). I'm a bit new to these Zotero formats. Is this where the translators are: https://github.com/zotero/translators?

And if so, would the 'crossref' translator be the one to edit given that citoid links to crossref when searching on doi by: https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1130%2Fg38665.1

Thanks, Albert. Albert Ke (talk) 21:02, 13 November 2017 (UTC)

I don't know how to edit the Zotero translators. However, @Czar and @Mvolz do. Whatamidoing (WMF) (talk) 18:23, 16 November 2017 (UTC)

Looks like Citoid does its own DOI import (not through Zotero): https://github.com/zotero/translators/issues/1476

Mvolz will be best equipped to handle this. We can continue at phab:T180408 czar 18:47, 16 November 2017 (UTC)

Bad JavaScript Error "OO.ui.TabPanelLayout is not a constructor"

Hi all, hope this is the good place to get some help. I'm trying to install Citoid extension on a personal wiki, with VisualEditor, and I get a bad JS error.

I have first installed mediawiki (1.29), enabled only a couple of extensions (Cite, Scribunto, TemplateData...), and Visual Editor. All works fine as expected. Besides, I have a parsoid+citoid (+ zotero/translation-server) server, working well too. I followed the setup instructions (here, here, etc.).

When I enable Citoid extension, I get no mediawiki error (in debug mode). But when I try to use the “Source” function, I get 2 JS errors I don’t get without Citoid (the “Source” function in VE works well with Citoid not enabled).

I tried on different browsers, with different skins, the result remains the same.

What I can see in the browser’s console is below. The 1st error make me think the oojs-ui is not well loaded though it’s installed (with composer).

Where to look to search the cause?

Thanks !

Janiko

(1) Uncaught TypeError: OO.ui.TabPanelLayout is not a constructor

   at VeUiCiteFromIdInspector.ve.ui.CiteFromIdInspector.initialize (load.php?...)

   at VeUiCiteFromIdInspector.OO.ui.Window.setManager

Code:        this.modePanels = {

           auto: new OO.ui.TabPanelLayout('auto',{

               label: ve.msg('citoid-citefromiddialog-mode-auto')...

(2) Uncaught TypeError: Cannot read property 'setDisabled' of undefined

   at VeUiCiteFromIdInspector.<anonymous>

Code:    ve.ui.CiteFromIdInspector.prototype.getSetupProcess = function(data) {

       return ve.ui.CiteFromIdInspector.super.prototype.getSetupProcess.call(this, data).next(function() {

           var fragment;

           this.lookupPromise = null;

           this.staging = !1;

           this.results = [];

           this.lookupButton.setDisabled(true);

           this.inDialog = data.inDialog || '';

           this.replaceNode = data.replace &&... Janiko (talk) 21:34, 16 November 2017 (UTC)

To me this sounds most likely to be a version mismatch. Branches get cut for mediawiki weekly and you'll need to have the same branch for VisualEditor an Citoid, and OOUI/OOJS as they won't be compatible with other branches (although composer should be giving you the correct version of ooui :/). I seem to remember tab layout getting changed little ways back.

Where are you getting your versions? From git, or packaged somewhere?

The best place to ask for help would probably be somewhere there are VisualEditor devs, #mediawiki-visualeditor on IRC maybe? https://www.mediawiki.org/wiki/VisualEditor/Setup Or you might actually have more luck with people experienced in dealing with versioning issues i.e. #wikimedia-tech

If all else fails you can report a bug, https://phabricator.wikimedia.org/ and tag with VisualEditor/citoid. Mvolz (WMF) (talk) 17:51, 17 November 2017 (UTC)

Thank you for your answer. Unfortunately I've reinstalled VE and Citoid, checked the branch label (both VE and Citoid with -b REL1_29 option while git cloning), relaunched git submodule update --init in VE directory and still got the same behaviour: VE works well until I try to load Citoid. I will try on phabricator. Janiko (talk) 21:07, 17 November 2017 (UTC)

404 URL returns data anyway that is wierd

When I submit this URL,

https://www.thecaterer.com/business/companies/33812/barracuda-group-limited

Citoid acts like it is real, even though it know that is is 404, and sends this back

[

 {
   "url": "https://www.thecaterer.com/business/companies/33812/barracuda-group-limited",
   "itemType": "webpage",
   "websiteTitle": "{{metaTags.other['og:site_name']}}",
   "title": "{{metaTags.other['og:title']}}",
   "abstractNote": "{{metaTags.other['og:description']}}",
   "accessDate": "2017-11-20",
   "author": [
     [
       "",
       "Template:MetaTags.other.author"
     ]
   ],
   "source": [
     "citoid"
   ]
 }

] AManWithNoPlan (talk) 00:16, 20 November 2017 (UTC)

Thanks for reporting!

Unfortunately this seems to be a problem with the website. Whilst the website *says* it's a 404, it's not actually a real 404 page. The http response header it claims it's a 200 ok response :D. The weird metadata is also because they're using a templating system that returns the raw template tags in the metadata of the page.

Unfortunately I'm not sure how actionable this is on our end- perhaps the best thing to do is to report the issue to the webmaster of the website. Mvolz (WMF) (talk) 09:44, 20 November 2017 (UTC)

Citoid should reject any proposed citation data with double curly braces in it. It causes error messages. Jonesey95 (talk) 12:39, 20 November 2017 (UTC)

Yeah, VE should be escaping this: probably https://phabricator.wikimedia.org/T143453? Mvolz (WMF) (talk) 16:46, 20 November 2017 (UTC)

JSTOR is sometimes a journal and sometimes a website

https://en.wikipedia.org/api/rest_v1/data/citation/mediawiki/https%3A%2F%2Fwww.jstor.org%2Fstable%2F3073766

Sometimes the json is correct (a journal with doi, etc), and other times (often), Citoid thinks this is a website. I find this confusing, since it is not consistent. AManWithNoPlan (talk) 15:07, 14 December 2017 (UTC)

More information. Once citoid goes bad, it stays bad for a while, and then suddenly works again. I have tried coming in from other IP address ranges and with other jstor IDs just to make sure that I was not personally being blocked. Thoughts: jstor some how blocks you, but you still get title with “ - on JSTOR” appended. There are multiple and different citoid severs running, but then I would think it would be more random and not last for a time period. AManWithNoPlan (talk) 16:54, 14 December 2017 (UTC)