Talk:Citoid

Jump to navigation Jump to search

About this board

Previous archives are at /Archive 1

GreenC (talkcontribs)

How can we get User:Citation bot whitelisted it does high volume and exceeds policy -- they have their own Citoid install but it is not as good as the Wikipedia install which gives better results.

AManWithNoPlan (talkcontribs)

To reduce load we do no query urls for citations that are already "complete". Since historically we are used in the sciences, we do not consider a citation complete without a volume number. We have added a short list of popular websites (CNN.com and such, it if purely my own writing based upon a few wiki pages) that do not have volumes and such to flag more as "complete". If we switch to Citoid, then it would be nice to get some feedback on what URLs websites are the most common, so we can add more to that last.

Mvolz (WMF) (talkcontribs)
Mobrovac-WMF (talkcontribs)

Hm, there are no limits on the citation API end points AFAIK. Could you provide a sample response?

AManWithNoPlan (talkcontribs)

en.wikipedia.org Citoid blocks us, so we run our own on the tool server.

AManWithNoPlan (talkcontribs)
Mvolz (WMF) (talkcontribs)
AManWithNoPlan (talkcontribs)

We got blocked so we now use our own on the tool servers. It does seem to time out at times also.

Mvolz (talkcontribs)

So I've asked operations, and we don't block IPs. However we do have rate limits of 1000 per 10 seconds (100 requests per second) which applies to all the mediawiki and restbase APIs. If you are receiving a 429 response, you just need to make sure you are adding a timeout in between requests so as not to exceed the limit.

Is 429 the response code you are getting?

(Also, it does take really long sometimes, depending on how long the time out is in your request package it may exceed it, so you may want to set a longer time out on your end.)

AManWithNoPlan (talkcontribs)

I will investigate. If it is 429, then we can sleep a little while and try again. The bot can be being run by lots of people so, we might hit the limit. How long of a time-out are we talking about?

Mvolz (WMF) (talkcontribs)

The docs say "1000/10s (100/s long term, with 1000 burst)" - so I think if you exceed 1000 and wait 10s it should reset, but this is just from reading the docs.

For timeout I tried to find an outside value for you; in tests we allow a request to take to take up to 40 seconds, but there have even been cases of something taking 75 seconds to return in the wild: https://phabricator.wikimedia.org/T165105#4666586

And the caching layer sets a timeout of 360 seconds, so you will not get any responses that take longer than that.

Reply to "citation bot"

We couldn't make a citation for you.

11
Timmy87 (talkcontribs)

Hello. I have a problem with citoid. After I inserted url, doi, pmid,the error message is shown like this: "We couldn't make a citation for you. You can create one manually using the "Manual" tab above."


I think the "zotero transalation server" is functioning because 'c-url test' or 'npm test' result had no problem. And I had the problem in following citoid instellation manual. I cannot clone not REL1-29 but also REL1_31. The error message on my shell is "Remote branch REL1_31 not found in upstream origin.


If you want to do an anonymous checkout:

git clone https://gerrit.wikimedia.org/r/p/mediawiki/services/citoid

Like VisualEditor, the master branch requires alpha builds of MediaWiki. If you're installing on an other mediawiki version, use the right branch like git clone -b REL1_29 https://gerrit.wikimedia.org/r/p/mediawiki/services/citoid


I cannot find REL1_31 services/citoid


Thanks in advance.

ps. I'm not programmer, just i followed mediawiki manual. So my explanation of error is lack of information.


MediaWiki Certified by Bitnami 1.31.1-2 on Ubuntu 16.04

Timmy87 (talkcontribs)

I checked chrome debugging, I found error message like this :

"OPTIONS http://localhost:1970/api?action=query&format=mediawiki&search=123412345 net::ERR_CONNECTION_REFUSED"




$ curl -X GET --header 'Accepapplication/json; charset=utf-8' 'http://localhost:1970/api?action=query&search=forhappywomen.com&format=mediawiki'


But, this process is working. What's the problem...


Mvolz (talkcontribs)
Timmy87 (talkcontribs)

@Mvolz

Thanks for answer! But I have followed those ways.

  1. Citoid#Configure Citoid on a Citoid-enabled wiki
  2. Citoid#Get "could not make a citation for you" every time

But, It doesn't work.


So I followed citoid again. But, I couldn't find the right branch of sevices/citoid (Not Extensions/citoid)

I think my problem is I coudn't clone REL1_31. Could you explain how can I get REL1_31 sevices/citoid?


(Although I changed following sentence 'REL1_29' to 'REL1_31', I see the error message :Cloning into 'citoid'...

fatal: Remote branch REL1_31 not found in upstream origin)



#Citoid page

Like VisualEditor, the master branch requires alpha builds of MediaWiki. If you're installing on an other mediawiki version, use the right branch like git clone -b REL1_29https://gerrit.wikimedia.org/r/p/mediawiki/services/citoid




I cannot find REL1_31 services/citoid

Mvolz (talkcontribs)
Timmy87 (talkcontribs)

Mvolz


Thanks for answer. I agree that. I solved problem with configuration of AWS security options. Acutally It's absolutely my mistake. I didnt open port1970 in AWS setting.


I solved it and the url/citoid is well operating. But. Other function(DOI, ISBN, PMID...) have a problem  :-(

Mvolz (talkcontribs)

Unfortunately ISBN won't work anymore without a developer key from worldcat, since they shuttered their xisbn service a year or so ago :/. DOI and PMID should be working.

Timmy87 (talkcontribs)

ah! I understood.

Whenever I try to enter the pmid, I can see following message.

Lua error in Module:Citation/CS1/Identifiers at line 47: attempt to index field 'wikibase' (a nil value).

I can't find someone who struggled with same problem.

PMID auto-search citation is what I really want to use and why I started mediawiki. :-(

Timmy87 (talkcontribs)
Mvolz (talkcontribs)
Mvolz (WMF) (talkcontribs)
Reply to "We couldn't make a citation for you."

Bloomberg sites say "are you a robot?"

2
Summary by Mvolz (WMF)

Tracked in task T210871

Roy17 (talkcontribs)

I just tried adding an article from https://www.bloomberg.com by citoid, but apparently bloomberg thinks citoid is a bot and asks for a captcha test. Is there anything Citoid developers can do? Or just up to bloomberg IT staff?

PerfektesChaos (talkcontribs)

AFAIK the citoid query at external website declares itself to be Citoid, not a user agent (browser).

One should not try to cheat by pretending to be a browser, since that will be discovered easily on the following dialogue.

Yes, you are right, Bloomberg staff should make a silent exception, but that could be exploited by every other grabber then.

Reply to "Bloomberg sites say "are you a robot?""
Dvorapa (talkcontribs)

Hello, in Citoid/itemTypes the "case" type is completely missing there. Also there are multiple examples missing and some links even doesn't work. Could someone update the page a little?

Dvorapa (talkcontribs)
Mvolz (WMF) (talkcontribs)

I fixed a few things but we don't really maintain the page, another volunteer made it. Which links didn't work? Feel free to remove ones that don't.

Dvorapa (talkcontribs)

I will not remove any as it is better to find archive link than have no clue, what for example "hearing" stays for.


The link for thesis doesn't work, but I think I understand this one's meaning.

Dvorapa (talkcontribs)
Reply to "Missing case in itemTypes"
Tinss (talkcontribs)

When setting $wgCitoidServiceUrl to https://en.wikipedia.org/api/rest_v1/#!/Citation/getCitation, I get an XMLHttpRequest error in my browser when adding a citation and it seems citoid is querying this url with the old API. I'm using REL1_31 of the Citoid extension. Citoid/API, indicates that the API has changed. Is this change only available through restbase and not $wgCitoidServiceUrl?

We run a small wiki and figured we'd piggy back off Wikipedia's citoid service instead of hosting our own.

Thanks for any help!

Mvolz (WMF) (talkcontribs)

Yes, the "old" API is actually citoid's native API, and that's what wgCitoidServiceURL expects, but we deprecated the public version of that a few years back, and now we're only running the one behind restbase. If mw.config.get( 'wgVisualEditorConfig' ).fullRestbaseUrl exists, then it Citoid uses the restbase one, but I'm pretty sure if you set that in the VE config it will mess up the rest of your VE installation :).

Feel free to file a bug and/or submit a PR: https://phabricator.wikimedia.org/

In the meantime you could also just manually set fullRestbaseUrl here instead of getting it from the VE config: https://github.com/wikimedia/mediawiki-extensions-Citoid/blob/master/modules/ve.ui.CiteFromIdInspector.js#L109

Tinss (talkcontribs)

@Mvolz (WMF), I've managed to deploy restbase on my wiki and was wondering how to forward requests for /data/citation/{format}/{query} to https://fr.wikipedia.org/api/rest_v1/data/citation/{format}/{query}?

I've also noticed you've patched Citoid to have the option to make use of a public API behind restbase. Until that codes makes it past review, I'll try to use the aforementioned way.

Thanks a lot!

Mvolz (WMF) (talkcontribs)

fr and en wikipedia APIs perform identically, so there's probably no particular need to do that; internationalisation is not great, but what it does do is pass on the accept-language header. The content of the accept-language header is mw.config.get( 'wgContentLanguage' ) so as long as the content language of your wiki is fr, it'll get French when available.

Tinss (talkcontribs)

Thanks for the info! Any idea how I'm supposed to configure my local restbase to forward requests to another remote API?

Mvolz (WMF) (talkcontribs)

The change is merged, so you don't need to configure restbase, you should just be able to configure the citoid extension (if you update to 1.0.0): Citoid#Citoid extension

Reply to "Use of public API"
Pigsonthewing (talkcontribs)

The ability to fetch metadata from Wikidata by specifying only a QID is great, but please can we also have this in the wikicode editor?

Mvolz (WMF) (talkcontribs)

You can currently do this in refToolBar as well by putting the QID or the full wikidata url in the url field and pressing the "search" eyeglass button.

Unfortunately in terms of having a dedicated QID field in refToolBar, this first requires the citation templates on en wiki to add a QID parameter.

Pigsonthewing (talkcontribs)

Thank you. Yes, that works - but is undocumented, and counter-intuitive.

Jdforrester (WMF) (talkcontribs)

It's been available rom the start in the 2017 wikitext editor. If you mean the 2010 wikitext editor, that's generally not getting new features, including not getting the automatic citations ("citoid") tool. There are locally hacky gadgets on a few wikis that try to replicate that; if you're using one of those, you should ask whoever is supporting that gadget to extend their tool if it doesn't Just Work™ already.

Pigsonthewing (talkcontribs)

I mean the 2010 editor, on en.Wikipedia. I have no idea who maintains it.

Jdforrester (WMF) (talkcontribs)
Pigsonthewing (talkcontribs)
Reply to "QID in Wikitext editor"

Bugs: pmc, redundant urls, and ref tags

9
Boghog (talkcontribs)

Not sure where the best place to post this. But here goes:

I never use this tool myself, but I frequently am cleaning up after others that do. Three problems that I have noticed:

(1) in the PMC parameter, the value should be a integer and not prefixed with "PMC":

incorrect:|pmc=PMCxxxxxx (where xxxxx = integer)

correct: |pmc=xxxxx

The incorrect form throws an error that must be manually fixed.

(2) urls are sometimes added that are indentical to that already produced by |doi=, |pmid=, or |pmc=. In this case, the url should be suppressed since it is redundant.

(3) Ref tags that take the form of ":0", ":1", ":2". While they are unique, they are not very informative. Better to create a Harvard style ref tag, the form of first authors last name + year of publication (i.e., "Smith_2017" is much more readable that ":0").

197.218.88.122 (talkcontribs)
Boghog (talkcontribs)

Thanks for the links, however is important to distinguish between a template imbedded in wiki markup and how that template is rendered. The NIH recommendations only concern how the citations are rendered, not how they are entered in a {{cite journal}} template. The NIH has no recommendation on the syntax of templates.

The {{cite journal}} template renders the citation very close to the NIH recommendations (PMCxxxxx). The only difference is that {{cite journal}} adds a space between PMC and xxxxx. PMC in turn contains a wikilink to a Wikipedia article that explains what PMC stands for.

It is completely unnecessary prepend the parameter value with "PMC". This is redundant. The template itself produces the correct rendering. Hence the problem is with Citoid, not {{cite journal}}.

197.218.88.122 (talkcontribs)

Well, that's a double edged sword. It is true that template markup differs from rendering, that means that it doesn't make any difference to a template if it receives a pmc = pmc XXXX or pmc = XXXX since it can be removed by parser functions or lua. So it comes down to aesthetics or personal preferences of some users because the template can be changed to accept both if there's consensus to do so.

The WMF developer's point in the phabricator task seems to be the same as the one in the publisher's site. The id is "PMC XXXXX", and the site also recommends "PMCID : PMC XXXX". Pages like Cancer AND Cholera don't seem to follow that recommendation anyway. While wikimedia users may prefer to use only integers to identify it, citoid and many wikimedia tools are also used by third parties, and such exemptions may not be wanted by them. There's also no guarantee that all wikipedias use the same format as english wikipedia, so it is possible that other wikis were doing the exact opposite, e.g. adding PMC where it only had an integer. The only alternatives are to change the code only for the wikis that prefer it that way, or leave it as is.

They stripped it previously (https://phabricator.wikimedia.org/T78144) without doing the research to understand the "official" preferred value, this time they decided against it after doing the research.

Anyway, I'm not WMF developer, nor associated with wikimedia, you'll have to convince them :).

Boghog (talkcontribs)

Can you cite a single example of another Wikipedia including "PMC" in the parameter value? Most other Wikipedias don't support pmc to begin with, and the ones that do have generally have followed the English Wikipedia's lead.

Citation Style 1 templates were created before Citoid. Hence it is reasonable that Citoid be compatible with Citation Style 1 templates, not the other way around. Furthermore, none of the other citation generation tools include "PMC" in the parameter value. I suppose that Citation Style 1 templates could be modified to optionally accept PMC in the parameter value, but this unnecessarily clutters citation templates with redundant characters. The parameter name is pmc, why does the parameter value also need to include pmc?

I have reopened the case on wikimedia.

197.218.88.122 (talkcontribs)
Can you cite a single example of another Wikipedia including "PMC" in the parameter value?

You sparked my curiosity, and here's one example:

https://el.wikipedia.org/wiki/%CE%91%CE%BC%CE%B9%CE%BD%CE%BF%CF%84%CE%B5%CE%BB%CE%B9%CE%BA%CF%8C_%CE%AC%CE%BA%CF%81%CE%BF

Citation Style 1 templates were created before Citoid

True, but that would be over fitting. There are more than a hundred encyclopedias, and the tool should be neutral and not cater to a single one. That's what it does, it returns the id as requested. Stripping it means that it just returns a number.

The parameter name is pmc, why does the parameter value also need to include pmc?

Knowing the full PMC means that users can quickly verify if citoid actually gave them the right data, instead of guessing. This is particularly important because a number can mean just about any random thing (e.g. a PMID number instead of a PMC). Just because it adds it as a parameter doesn't mean it is necessarily correct.

Boghog (talkcontribs)

In the Greek Wikipedia, using a pmc parameter value where PMC is prepended is optional. However the rendering with when PMC is included in the parameter value looks strange (PMC is displayed twice and PMCID is not displayed). This should be fixed.

The whole purpose of Citoid is to automate citation generation process so the editor doesn't have to worry about the accuracy of the parameter values. Since the data is downloaded from PubMed, the parameter values with a high degree of confidence are correct.

Numbers are not any less random if PMC is prepended to the parameter value. The best test to verify that the the parameter value is correct is to follow the rendered PMC link.

Whatamidoing (WMF) (talkcontribs)

Boghog, I tend to think that the IP is correct: it would be good for {{cite journal}} to accept the "formally correct" id number. This has been discussed for a couple of months now, and I've not yet heard any technical reason for the template to choke when it's given the "official" id number. I don't see a need to require the official id number, but it should be able to cope with it.

Whatamidoing (WMF) (talkcontribs)
Reply to "Bugs: pmc, redundant urls, and ref tags"
Summary by Mvolz (WMF)

Fixed outdated links in page.

Smith609 (talkcontribs)

I note that under `install from scratch`, step 2 instructs users to perform

`git checkout 8d6eae22`

This command, however, returns the error

`error: pathspec '8d6eae22' did not match any file(s) known to git.`

and indeed, perusal of https://github.com/zotero/translation-server/commits/master establishes that no such commit exists. Presumably the repository history was wiped on 24 May 2018.

As such I am not able to install a version of translation-server that is compatible with nodejs v6, as the current versions require v8 or above.

Mvolz (WMF) (talkcontribs)

It looks like they moved the repository that used to live there to https://github.com/zotero/translation-server-old. You should be able to follow the directions using that URL instead.

We are planning to use the new version fairly soon but it is not yet compatible.

Ghybu (talkcontribs)

Hello, I would like to activate it on ku.wikipedia. Apart from creating this page, is there anything else to do?

Mvolz (WMF) (talkcontribs)
Whatamidoing (WMF) (talkcontribs)

@Ghybu, are you making progress? Do you need help?

I noticed that no citation templates are visible under the "Manual" tab. Do you want to add some, or does ku.wikipedia.org prefer non-templates?

Ghybu (talkcontribs)

Indeed, I don't understand why it is empty. I also tested TemplateData and it works correctly. I missed something or I have to report a bug on Phabricator?

Mvolz (WMF) (talkcontribs)

Hi, for the citations templates to be visible under the "Manual" tab, that is actually a separate configuration message called the citation tool: VisualEditor/Citation tool

Ghybu (talkcontribs)

@Mvolz (WMF): Thank you for your help, it works now :)

Whatamidoing (WMF) (talkcontribs)

Yay! Thanks for the update. I'm glad that you got it all sorted out.

2604:2000:DD4D:3A00:B9DF:24AD:C67C:9115 (talkcontribs)

Is there a way for Citoid to work with a local database so that it doesn't need to go through Zoltero.

Whatamidoing (WMF) (talkcontribs)

I think the correct answer is: "Yes – theoretically". I think you would have to re-write parts of the software (so it could find and understand your local database).

Mvolz (WMF) (talkcontribs)

So, there is no database with citoid OR Zotero's translation-server. Both Zotero and citoid generate citations on the fly, by querying APIs (like crossref) and scraping metadata from websites.

You CAN use citoid locally without setting up a local Zotero translation-server instance. However, it won't work as well. Configuration options to use or not use zotero are in config.yaml

Reply to "Local database"