Citoid



The citoid node.js service generates citation data given a URL, DOI, ISBN, PMID, PMCID or QID. It has a companion extension, Citoid, which aims to provide use of the citoid service to VisualEditor, but also may be accessible from the wikicode editing palette, if activated.

It is currently deployed in all VisualEditor-enabled WMF-Wikis, though the extension is only configured on some of them.

Public API
To request metadata about a URL, DOI, ISBN, PMCID, PMID or QID, you can use the English language API endpoint at https://en.wikipedia.org/api/rest_v1/#!/Citation/getCitation. Or, for language localised requests, use your preferred language Wikipedia.

Issue tracker and project management
Bugs, issues, and suggestions for improvement can be added to the Citoid Phabricator project.

Installation
Citoid is a nodejs app that also requires a working installation of Zotero's translation server, which uses the Zotero translators library, and xulrunner.

Install nodejs and npm
Install nodejs and npm. When you are using Ubuntu and depending on OS version you will not end up with the most recent version of nodejs. You are recommended to use nvm to manage nodejs installations.

For other systems, see:
 * http://nodejs.org/download/
 * https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
 * (Debian Squeeze, Wheezy) https://gist.github.com/x-Code-x/2562576

Install and configure Zotero's translation server
For citoid versions 0.5.3 or earlier, see: Translation-server installation instructions

For citoid versions 1.0.0 or higher:


 * 1) Clone the repository
 * 2) If you forget the recursive flag, do   then do
 * 3) Change into the directory
 * 4) Install the libraries
 * 5) Run the server:
 * 6) You should get the following output:

Try a query to verify it's working:

Get the code
If you want to do an anonymous checkout:

Or if you plan to hack citoid, then please follow the Gerrit 'getting started' docs and use an authenticated checkout url instead, such as:

JS dependencies
Install the JS dependencies. Run this command in the citoid directory:

Modify config.yaml
Config.yaml contains the configuration for the citoid service. The defaults should work out of the box for development, however, they may need to be modified in a deployment set-up.

Run the translation server
You'll first need to run translation-server; see the directions on its github page, but generally from the translation-server directory you should run:

Run Citoid server
You should be able to start the citoid web service from the citoid directory using:

This will start the citoid service on port 1970. To test it, navigate to http://localhost:1970 in your browser. You'll be able to test sample queries from this page.

Install Citoid extension
In order to have citoid functioning on your wiki in conjunction with VisualEditor, you'll need the following: VisualEditor and Parsoid, VisualEditor's Citation Tool, and the Citoid extension.

It is recommended that you have the following extensions in your extension folder: Extension:VisualEditor, Extension:Scribunto, Extension:TemplateStyles, Extension:Cite, Extension:TemplateData, and Extension:ParserFunctions, and Extension:Citoid.

VisualEditor and Citation Tool

 * 1) Set up MediaWiki: Manual:Installation_guide
 * 2) Set up Parsoid: Parsoid/Setup
 * 3) Set up VisualEditor: Extension:VisualEditor
 * 4) Set up Citation Tool: VisualEditor/Citation tool

Citoid extension
If you want to do an anonymous checkout:

Or if you plan to hack citoid, then please follow the Gerrit 'getting started' docs and use an authenticated checkout url instead, such as:

Like VisualEditor, the master branch requires alpha builds of MediaWiki. If you're installing on another mediawiki version, use the right branch like.

Then add the following line to your wiki's  after you have downloaded the extension:

Set the location to your citoid service instance in your wiki's

Configure Citoid on a Citoid-enabled wiki
The citoid extension must be configured using special TemplateData maps and a special citoid-specific message. It is currently deployed in all VisualEditor-enabled WMF-Wikis, but it must be configured before it can be used.

Ensure each template to be used in MediaWiki:Citoid-template-type-map.json has an 'citoid' maps value
Since Citoid has its own set of fields for each document type (for instance, the journal name is called 'publicationTitle' in citoid, but 'journal' in Template:Cite_journal), each Template must have TemplateData defined that creates a map between citoid's fields and the Template's field. Calling the map 'citoid' lets the citoid extension know which map to look for. If the map 'citoid' doesn't suit your purposes for use with say, a userscript, you may create a citoid service related map that is called something else; an unlimited number of maps with unique keys are allowed in the maps object. Note that you can only see TemplateData maps in edit mode; they are not visible in the TemplateData table.

Fields issn and isbn can have Arrays [] in the citoid map; using them ensures that only one ISBN is in the field. If you do not place the parameter inside an Array (i.e. isbn: "[ISBN]"), multiple ISBNs or ISSNS will be concatenated in the field (i.e. "issn: 1234-5678, 7777-7777'). All 'person' fields, e.g. author, editor, translator, contributor etc, require a 2D Arrays in the citoid map. See sample templateData below for examples.

Examples of a map objects that are compatible with the citoid extension are on English Wikipedia:
 * https://en.wikipedia.org/wiki/Template:Citation/doc
 * https://en.wikipedia.org/wiki/Template:Cite_journal/doc
 * https://en.wikipedia.org/wiki/Template:Cite_web/doc
 * https://en.wikipedia.org/wiki/Template:Cite_news/doc
 * https://en.wikipedia.org/wiki/Template:Cite_book/TemplateData

Configure special MediaWiki namespace Citoid message
You'll need to configure a special MediaWiki: namespace message. This message maps the native citoid types (website, book, journalArticle) to the appropriate template (Cite web, Cite book, Cite journal). You should match a template to every single citoid type; there is no default behaviour if no template is matched to a particular type. It's better to have a bad match (there may be some fields in common between video liner notes and a book, or video liner notes and a video, for example) than none at all.

You may consider using en wiki's Template:Citation as a catch-all for types where there is no good type match as it is designed for this situation.

An sample namespace message is found here: Citoid/MediaWiki:Citoid-template-type-map.json

Every available citoid type is listed as a key in the sample namespace message.

Troubleshooting Citoid VisualEditor Extension
Open your browser's console. The Citoid extension will log errors here.

Inspector does not appear in the toolbar
An icon for the inspector should appear in the toolbar menu. This means either citoid is not installed properly or not configured properly. Verify the extension and dependant extensions are installed on the page localhost:/wiki/Special:Version.

If the extensions are installed and the icon does not appear in the toolbar, it most likely means there's a problem with MediaWiki:Citoid-template-type-map.json. If there is no message at that location, or if the JSON is invalid, the inspector will not load. Look at the messages logged to your browser's console to diagnose the specific problem with the message. Alternatively, you may need to refresh your JavaScript cache.

Get "could not make a citation for you" every time
You may consistently get this error if the citation template being inserted contains no maps data, or if the maps data is there but not making it to the MediaWiki API. First, determine the template that the inspector is attempted to insert, for example,. View source of the template or documentation page to verify that

"maps": {

"citoid": { is present and contains fields. Then verify that these data are making it to the MediaWiki API by visiting the API page, i.e. http://localhost/api.php?action=templatedata&titles=Template:Cite%20web/doc&format=jsonfm on your local installation, or https://en.wikipedia.org/w/api.php?action=templatedata&titles=Template:Cite%20web&format=jsonfm on en wiki.

If the maps object is present in TemplateData, but not in the API response, try editing the template where the TemplateData is transcluded i.e.  (but making no changes) and saving it, a.k.a. a "null edit". There is a known bug with transcluded TemplateData where it can take a long time for the API to update ; null edits force the change.

If the response from the API looks okay, there may be an issue with installation of the citoid service. Open up your browser's debugger to confirm that the responses from the citoid server are OK.

The inspector is still "pending" after a really long time following insertion
This typically means there is a bug. If you open your JavaScript console, you will likely find error messages that will help you debug.

Access date is formatted differently on my wiki
The dates are in ISO format, which is an international standard. On the back-end, we're sticking to ISO and in the future all dates will all be in ISO, not just access date. This is because it is an unambiguous way to present the date in any language. If the community doesn't like the way this looks to the user, it is possible to edit the citation template to format the ISO dates to something that is standard in your language. For instance, you can add logic to the template such that if the date is detected to be in ISO yyyy/mm/dd format, the date is reformatted *to appear* to be dd/mm/yyyy on the page. However, if you do this, the underlying data (i.e. when you edit the wikitext, or the form in VisualEditor) will still remain the same.

My favourite site isn't recognised by citoid and only gets basic information
The Citoid service relies on the brilliant Zotero community for much of the "magic", as Zotero translators need to be written for each site. You can see a list of all Zotero translators at https://github.com/zotero/translators. Right now, Zotero best supports English-language sources. We need your help to improve coverage of other sites. You can write your own Zotero translator. Start by looking at Citoid/Creating Zotero translators.

Testing for the translators using the "server" option or 'v' flag
To test with translation-server, download and install https://github.com/zotero/translation-server Your translator will need to have the 'v' flag enabled for 'browserSupport'. More here on that: https://www.zotero.org/support/dev/translators As an example, see https://github.com/zotero/translators/blob/master/A%20Contra%20Corriente.js ; you will see there are a bunch of letters, one of which is v, which corresponds to translation-server. If server support is not enabled by testing it/then adding the 'v' flag, we won't be able to use the translator.

Installation
If you are using vagrant, you can enable the citoid role to hack on the citoid service.

If you plan to hack on the Citoid extension, you should enable the following roles:

After enabling the roles, you will further need to add wiki pages to get citoid working. The most expedient way to do this is to export the following pages from en-wiki:

And then import them: localhost:8080/wiki/Special:Import.

After importing the Templates, you may need to navigate to the template, hit the edit button, and then hit save (a "null" edit), in order for the templatedata to propagate to the DB.

citoid service
runs eslint and unit tests.

runs the full set of tests. (You can run  if you only have mocha installed locally)

runs the tests and reports code coverage.

In order for all tests to pass, you will need a WorldCat wskey in your config.dev.yaml file, which you can request at here.

Another reason why some tests may fail erroneously is if your DNS will redirect invalid domains to a valid IP (Such as BT Internet's DNS); in some cases, this causes a 520 response instead of a 400 response to be returned. This can be fixed by configuring your internet connection to use OpenDNS or another DNS that does not do this.

Citoid extension
See: Manual:JavaScript unit testing

Formatting the citation output of Citoid
Citoid depends on Extension:TemplateData for mapping variables from a wiki's citation templates to citoid's fields. However, it may not be apparent that the formatting output of citations from Citoid isn't managed explicitly by Citoid but rather by TemplateData. As such, if a particular citation format is required, you'll need to adjust the "format" in your citation templates' TemplateData. Guidance is provided in Help:TemplateData. You may, for example, see  at the bottom of many pre-existing TemplateData Citoid maps, which means the resulting citation output will appear inline. However, this will not put any spacing in the resulting citation text. If the output should have a space before each pipe character but no spaces before or after an equals sign in the citation, one may set. The code would typically be placed before closing out with the final curly brace, as such:

Consult Help:TemplateData for more formatting options.

It's worth noting that if you want to change this format on a large wiki project, you should probably discuss it somewhere with the community as users may have strong feelings about formatting in citations. Additionally, note that when "format" is changed on an existing wiki, it will cause VisualEditor to update the formatting of each citation whenever someone makes changes to it. On large wikis like Wikipedia, this can cause annoying dirty diffs and even accidental edit wars with bots that maintain citations.