Citoid

Citoid is a node.js service which currently locates citation data given an URL, DOI, PMID, or PMCID. It has a companion extension of the same name which aims to provide use of the citoid service to VisualEditor. Neither the service nor the extension are presently in production except in beta.

API
There are two endpoints available. To request metadata about a url, you can use the url endpoint and provide a POST request with the parameters "url" and "format". "url" will take any URI, including those without protocol (urls with no protocol specified are queried as http).

To request metadata about a URI, DOI, PMCID or PMID, you can use the api endpoint and provide a GET request with the parameters "search" and "format", with the requested identifier as the value for the "search" parameter.

"Format" takes "zotero", "mediawiki" and "mwDeprecated."

Current design

 * 1) Queries Zotero translator-server for translator.
 * 2) *Lists of supported services exist
 * 3) *Many applications already expose the data in a way that Zotero understands
 * 4) If no translator found, follow redirects (for the case of shortened URLs- this is not done initially because in many cases the redirect will be to a log-in url).
 * 5) Query Zotero translator-server for translators again.
 * 6) If no translator found, send to a native scraper that generally pulls out more limited metadata than what would be obtained from Zotero had a translator been available.

Issue tracker and project management
Bugs, issues, and suggestions for improvement can be added to the Citoid phabricator project.

Installation
Citoid is a nodejs app that also requires a working installation of Zotero's translation server, which uses the Zotero translators library, and xulrunner. These are included in a deployment repository, or you can install each component individually. Please note that the most recent version of xpcshell doesn't work with translation-server; the deploy directory contains a version which is known to work (29.0).

Install nodejs and npm
Install nodejs 0.10 and npm. When you are using Ubuntu and depending on OS version you will not end up with the most recent version of nodejs.

For other systems, see:
 * http://nodejs.org/download/
 * https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
 * (debian squeeze, wheezy) https://gist.github.com/x-Code-x/2562576

Install and configure Zotero's translation server
See: Translation-server installation instructions

Note: The most recent version of xpcshell does not work with translation-server. Install version 29.0.

Get the code
If you want to do an anonymous checkout:

Or if you plan to hack citoid, then please follow the Gerrit 'getting started' docs and use an authenticated checkout url instead, such as:

JS dependencies
Install the JS dependencies. Run this command in the citoid directory:

Create localsettings.js file
Copy localsettings.js.sample to localsettings.js

Run the server
You'll first need to run translation-server; see the directions on its github page, but generally you should run:

You should be able to start the citoid web service from the citoid directory using:

This will start the citoid service on port 1970. To test it, try a sample query:

Install from deploy repository
The deploy repository is used by Wikimedia in production. It uses binaries compiled for 64 bit Ubuntu- on any other system you will need to install from scratch.

Checkout deploy repo
If you want to do an anonymous checkout:

Or if you plan to hack citoid, then please follow the Gerrit 'getting started' docs and use an authenticated checkout url instead, such as:

Update submodules
This will put the source of citoid in the /src/ directory. This may not always be the most recent version of citoid- to get the most recent (potentially unstable) version,

Translation-server and the translator libraries are not under source control within the deploy repo. If you want to have the most up-to-date versions of these, you should install from scratch.

Configure translation-server
From the deploy directory,

Line one, change the translators directory path to the location your "translators" directory.

Run translation-server
From the deploy directory,

Configure and run citoid
From the deploy directory,

Alternatively, you can use the -c flag to specify the location of the localsettings.py file.

Install and configure Citoid user scripts
Users can make use of citoid user scripts if user javascript is enabled on your wiki using the $wgAllowUserJs option. You can also install VisualEditor gadgets on a site-wide basis.

There is currently one user script than can be used on English Wikipedia in conjunction with VisualEditor and a citoid service running on labs: CiteFromURL user script.

There is one user script for wikitext editing that can be used on English Wikipedia, also using the labs instance: Citoid Wikitext citations.

Install and configure Citoid extension
In order to have citoid functioning on your wiki in conjunction with VisualEditor, you'll need the following: VisualEditor and Parsoid, VisualEditor's Citation Tool, and the Citoid extension.

It is recommended that you have the following extensions in your extension folder: Extension:VisualEditor, Extension:Scribunto, Extension:Cite, Extension:TemplateData, and Extension:ParserFunctions, and Extension:Citoid.

VisualEditor and Citation Tool

 * 1) Set up MediaWiki: Manual:Installation_guide
 * 2) Set up Parsoid: Parsoid/Setup
 * 3) Set up VisualEditor: Extension:VisualEditor
 * 4) Set up Citation Tool: VisualEditor/Citation tool

Citoid extension
If you want to do an anonymous checkout:

Or if you plan to hack citoid, then please follow the Gerrit 'getting started' docs and use an authenticated checkout url instead, such as:

Then add the following line to your wiki's  after you have downloaded the extension:

Set the location to your citoid service instance in your wiki's

Configure special MediaWiki namespace Citoid message
You'll need to configure a special MediaWiki: namespace message. This message maps the native citoid types (website, book, journalArticle) to the appropriate template (Cite web, Cite book, Cite journal). You should match a template to every single citoid type; there is no default behaviour if no template is matched to a particular type. It's better to have a bad match (there may be some fields in common between video liner notes and a book, or video liner notes and a video, for example) than none at all.

You may consider using en wiki's Template:Citation as a catch-all for types where there is no good type match as it is designed for this situation.

An sample namespace message is found here: Citoid/MediaWiki:Citoid-template-type-map.json

Ensure each template specified in MediaWiki:Citoid-template-type-map.json has an 'extension/Citoid/ve.ui.CiteFromIDDialog.js' maps value
Since Citoid has its own set of fields for each document type (for instance, the journal name is called 'publicationTitle' in citoid, but 'journal' in Template:Cite_journal), each Template must have TemplateData defined that creates a map between citoid's fields and the Template's field. Calling the map 'extension/Citoid/ve.ui.CiteFromIDDialog.js' lets the citoid extension know which map to look for. If the map 'extension/Citoid/ve.ui.CiteFromIDDialog.js' doesn't suit your purposes for use with say, a userscript, you may create a citoid service related map that is called something else;  an unlimited number of maps with unique keys are allowed in the maps object.

The most up-to-date maps objects that are compatible with the citoid extension are on beta:
 * http://en.wikipedia.beta.wmflabs.org/wiki/Template:Cite_news/doc
 * http://en.wikipedia.beta.wmflabs.org/wiki/Template:Cite_journal/doc
 * http://en.wikipedia.beta.wmflabs.org/wiki/Template:Cite_web/doc
 * http://en.wikipedia.beta.wmflabs.org/wiki/Template:Cite_book/doc
 * http://en.wikipedia.beta.wmflabs.org/wiki/Template:Citation/doc

Dialog does not appear in the toolbar
The dialog should appear in the "Cite" menu; the title is "Autofill from URL." It may appear in different locations in the list (top, bottom, or middle). If the dialog does not appear, it most likely means there's a problem with MediaWiki:Citoid-template-type-map.json. If there is no message at that location, or if the JSON is invalid, the dialog will not load. Alternatively, you may need to refresh your Javascript cache.

Empty references are being inserted
Empty references most commonly inserted when the citation template being inserted contains no maps data, or if the maps data is not making it to the api. First, determine the template that the dialog is attempted to insert, for example, Template:Cite_web/doc. View source of the template or documentation page to verify that "maps": {

"extension/Citoid/ve.ui.CiteFromIDDialog": { is present and contains fields. Then verify that these data are making it to the mediawiki api by visiting the api page, i.e. http://localhost/api.php?action=templatedata&titles=Template:Cite%20web/doc&format=jsonfm.

If the maps object is present in the template data, but not in the API response, try editing the Template where the TemplateData is transcluded i.e. Template:Cite_web (but making no changes) and saving it. There is a known bug with transcluded TemplateData where it can take a long for the API to update; null edits force the change.

If the response from the API looks okay, it may be that you are running an outdated version of the citoid service as it is not always backwards compatible.

The dialog is still "pending" after a really long time following insertion
This typically means there is a bug. If you open your javascript console you will likely find error messages that will help you debug.

Related extensions and services

 * Zotero translation server -- xpcshell-based Zotero translation server, and zotero-node, an abandoned nodejs equivalent
 * Zotero field mappings, including vs. some cite templates
 * en:Category:Citation_templates
 * en:Help:Citation tools
 * VisualEditor/Citation tool