Wikibase/Indexing/RDF Dump Format/diq

This page describes the RDF dump and export format produced by Wikidata and used for export and indexing purposes. Note that while it is close to the format used by the, it is not the same code and not the same format. While we strive to keep divergence to a minimum, there may be differences and one should use documentation only for the format that is actually being consumed.

This document describes the RDF dump as can be downloaded from Wikimedia dump source, and while it can be used to create queries for Wikidata query service, the service can have small differences in how the data there look like. See the WDQS data differences chapter for the full list.

The canonical URI of the Wikibase RDF ontology is http://wikiba.se/ontology. The current version can be found at http://wikiba.se/ontology-1.0.owl.

Changes to the RDF mapping are subject to the Stable Interface Policy.

Data model
The RDF format is a binding for on the Wikibase data model and represents an export format for it. That means, in particular, that if/when the data model changes, the export format will be changed accordingly. This document will be updated for such changes. The following description assumes familiarity with the data model and the terminology used.

This RDF binding is based on the one designed for the Wikidata Toolkit by Denny Vrandecic and Markus Krötzsch, see http://korrekt.org/papers/Wikidata-RDF-export-2014.pdf.

The following description uses prefixes to describe the IRIs of the RDF resources mentioned. See the Prefixes chapter for the full description. All examples below are given in Turtle format.

Versions
The version of the data model is specified by  predicate of the   node, which is either dump node for the dump or entity data node  for single entity page. Released versions:

Header
For the RDF dump, there is the header node  containing information about the license, the software version of the generator and the date the data was produced. In single-entity export, this data is attached to the data node (see below).

Example header:


 * - specifies the IRI of the license that applies to the whole RDF document.
 * - specifies which version of the dump format is being used (currently ), will be updated when format changes, once the format is out of the beta period. The version updates will be done along the lines of semantic versioning, with major changes being BC breaking ones, minor being major BC-compatible changes and patch part changes on minor tweaks.
 * - specifies the date of the dump's data validity. Some data that is contained in a dump may be more recent than this date, but it is guaranteed that no data in the dump is older than this date. The date should be close to the time of the oldest data contained in the dump, but for technical reasons may not be exactly the same as the time of the oldest data in the dump.

Entity representation
The entity is described in two nodes - data node and entity node. For entity Q1, data node is  and entity node is.

Data node describes the metadata about the entity record in the Wikibase - i.e. data which are not part of the entity's information but instead describe the status of the entity inside Wikibase. It has type of  and contains the following metadata:


 * Information about the entity revision - this is a counter that increases with each modification of the entity data
 * Last modification time of the entity data - as an    timestamp
 * Link to the entity node with  predicate

Example:

Entity node describes the actual entity data and has type  or   depending on the kind of entity. Other entity types can be introduced in the future.

Entity description includes the following:


 * Entity labels - the main name of the entity. Labels are defined as,   and   predicates with objects being language-tagged string literals.
 * Entity aliases - the secondary names of the entity. Aliases are defined as  predicates with objects being language-tagged string literals.
 * Entity description - the longer description of the entity. Defined as  predicates with objects being language-tagged string literals.
 * Truthy statements (see below)
 * Predicates linking it to full statements

Example of the entity definition:

Page properties
Entity node can also carry additional information about the entity, such as number of links or statements.

The data is sourced from page properties and can be specified in config file. For example:

specifies how many statements this entity has,  specifiers the number of statements with External Id values, and   specifies the number of sitelinks. Additional statements can be introduced in the future.

Items
Entities that represent items have the common entity data as described above, plus can have sitelinks attached to them, as described below.

Properties
Entities that represent properties additionally feature the property type using  predicate. The object of the predicate is the property type described in Value representation below, with  prefix and each word capitalized, with no separators. I.e.,  becomes.

Each property is also linked to the predicates that are derived from it. Example:

The property predicates also have type definitions:

The type depends on the type of the original property - whether its value is literal or IRI. However,,   ,   and   predicates would always be.

Note that wdno:P22 mentioned above is not a predicate, unlike others, but a class. See the full description of it in Novalue section.

Lexemes
Please see full description at Lexeme RDF mapping.

Lexemes are represented according to Lexeme RDF mapping. Example:

MediaInfo
Please see full description at MediaInfo RDF mapping.

MediaInfo entities are represented according to MediaInfo RDF mapping. Example:

This example demonstrates the MediaInfo data on wikimedia commons when used in federation with wikidata.

Statement types
The RDF format represents statements in two forms - truthy and full statements.

Truthy statements
Truthy statements represent statements that have the best non-deprecated rank for given property. Namely, if there is a preferred statement for property P2, then only preferred statements for P2 will be considered truthy. Otherwise, all normal-rank statements for P2 are considered truthy.

Truthy statement predicates have prefix  with the property name (e.g.  ) and the object is the simple value (see below) for the statement. The qualifiers are ignored.

If the value has simple value normalization (currently true only for external ID), normalized value is listed under  prefix, e.g..

Full statements
Full statements represent all data about the statement in the system. Full statement is represented as separate node, with prefix  with the id of the statement (e.g.  ). There is no guaranteed format or meaning to the statement id.

The statements are linked to the entity with the predicate with prefix  and the name of the property (e.g.  ).

Statement representation
The statement node represents a single statement about the entity. It has type. The statement can contain the rank, the simple value (see below) of the statement, the link to the full value, the qualifiers and the references.

The statement rank is represented by the predicate  and the object being one of: ,  ,.

The statement that has the best rank for the property (i.e., preferred if there are any preferred statements in the property, otherwise normal) is also has type of.

The simple value is represented by the predicate with prefix  and the name of the property (e.g.  ) and the object being the simple value.

The full value (if required by the type) is represented by the predicate with prefix  (e.g.  ) and the object being the full value node.

The statement always has no more than one value, but can have multiple qualifiers and references.

Qualifiers
The qualifiers are represented by predicates with prefix  and the name of the property (e.g.  ) and the object being the simple value of the qualifier.

The full value (if required by the type) is represented by the predicate with prefix  (e.g.  ) and the object being the full value node.

Constraints
The service also may contain data for Wikibase Quality Constraints violations, via  predicate. The predicate links violating statement to the statement describing the constraint, e.g.:

Note that constraint violations are loaded from constraint cache and are not guaranteed to be up-to-date or present for all items (which means if you find no constraint violation statements for an item, that doesn't mean it doesn't have any - check Wikibase Quality tools for more up-to-date information).

Reference representation
References represent provenance information about statements.

Reference is represented as node, with prefix  and the local name being the hash derived from the reference contents (e.g.  ). The exact value of the hash is not guaranteed beyond the fact that the same references (i.e. ones with identical content) will generate the same hash, and the different one will generate the different one. The same reference (i.e. reference having the same properties with the same values) will be usually represented with single node, though duplicate reference nodes are possible in the data.

The type of the node is a.

The reference values are represented the same as statement values, with simple values using predicates with  prefix (e.g.  ) and full values with prefix   (e.g.  ) and the object being the full value node. Unlike statements, references can have any number of values.

Example of the reference node:

Value representation
In the RDF format, the values are represented as two forms - simple value and full value. Simple value is always a literal or IRI, and is used as direct value that is convenient to search, index and match. The full value contains additional information about the value, such as ranges, precision, calendar used, etc. Note that while for many queries simple values will be enough, for other, more complex values, only full values will be adequate.

If the statement has a value (i.e. is not set to novalue) then the simple value will always be present.

Full values are represented as nodes having prefix  and the local name being the hash of the value contents (e.g.  ). There is no guarantee of the value of the hash except for the fact that different values will be represented by different hashes, and same value mentioned in different places will have the same hash. Value node has type. The content of the node is defined by the type of the value (see below).

Example of the value node:

The following describes the handling of each kind of value, depending on the type of the value and the type of the property. Note that not all aspects of the data model are represented in RDF currently, some aspects that are currently unused (such as units or before/after values for dates) are omitted since they currently do not carry any useful information. This may change in the future if/when these aspects come into use by Wikidata.

String
Strings have value type  and property type.

String is represented as a string literal. Strings only have simple value.

Commons media
Media on have value type   and property type.

Commons media is represented as an IRI with the full Commons resource URL, derived from the Commons filename in the underlying data item. E.g.:. It has only simple value.

URL
URL values have value type  and property type.

URL is represented as a an IRI matching the URL string (e.g. . It has only simple value.

External Id
External Id values have value type  and property type. They are represented by a string literal. It has only simple value.

If the property has a formatter URI for RDF configured (on Wikidata: ), the RDF will also have normalized value, e.g.:

Wikibase Entity Id
Wikibase Entity Id values have value type  and property type.

The entity is represented by its IRI, e.g.. It has only simple value.

Monolingual text
Monolingual text values have value type  and property type.

The text is represented as a string literal with language tag. It has only simple value.

Globe coordinate
Coordinate text values have value type  and property type.

The simple value of the coordinate is the WKT string with the coordinates, with type, e.g.:. The order of the coordinates in WKT is longitude, latitude (since format version 0.0.2).

The full value has latitude, longitude and precision as double, and the globe as IRI.

Example:

Quantity
Quantity values have value type  and property type.

The simple value of the quantity is the specified amount, as a decimal literal.

The full value includes amount, unit URI (the default for unit-less values being http://www.wikidata.org/entity/Q199), and optionally upper and lower bound. If no upper an lower bound are given, the uncertainty of the quantity is undefined. Exact values are represented by quantities that have the same value for amount, upper bound and lower bound.

Example:

Time
Time values have value type  and property type.

The simple value of the time value is either datetime value of type, if the value can be converted to Gregorian date in ISO format, or a string as represented in the database, if not. The  dates follow XSD 1.1 standard, which uses the proleptic Gregorian calendar, and represents the year 1 BCE as +0000. This is in contrast the JSON representation of Julian and Gregorian dates, which follows the traditional year numbering, representing the year 1 BCE as -0001.

The full value includes the simple value above under, precision and timezone as integers and calendar model as IRI. Note that the calendar model is the original values calendar model even if  was converted to Gregorian.

Example:

Normalized values
Some values can be represented in several forms, depending on the purpose. For example, length can be expressed in different units - feet, inches, meters, miles, etc. In order to provide means to unify these forms and thus make data more friendly for automatic processing, the normalized values are introduced, which represent diverse data in a unified way.

Right now, the only value normalization that is supported is converting units for quantities into base units - e.g. length to meters. In the future, more units and more normalizations may be added, which will be documented here. The conversion table is available on the Mediawiki gerrit if needed.

The only normalized simple values are external IDs (see below).

Normalized quantity
Normalized quantity values are value nodes that are parallel to the original data nodes but represented in base units. They are connected to their parent nodes by predicates with prefix having "v" replaced with "n" - i.e.,   and  , for example:

Original quantity value is connected to the normalized value by  predicate:

The normalized value has  pointing to itself.

If the value is already normalized - i.e. is expressed in base units - then both "v" and "n" predicates point to the same value, and  for this value points to itself.

Quantities with no units or with units that are not normalizable (have no base unit they can be reduced to) do not have normalized predicates and normalized values and do not include.

The recommendation is to have no more than one base unit per property. Base units depend on Wikibase configuration and usually are chosen to represent universally accepted standard units, such as SI units.

Normalized External ID
For external IDs, normalization converts string value to URL, if the URL formatter for that purpose is defined in property data (via  setting), then the normalized value will be listed as   value for truthy values, and as normalized value for the statements in ,   and   predicates, depending on the context where the value appears.

Special values
Wikibase data model has two special kinds of snaks - PropertySomeValueSnak, specifying a value that exists but whose identity or value is unknown, and PropertyNoValueSnak, specifying that a value does not exist.

Somevalue
Unknown value is represented as RDF blank node in both simplified and full statements:

Novalue
Novalue is represented not by a regular value but as a class of the entity or statement or reference, with prefix  and the name of the property. Example:

The entity has a  class if it has a truthy novalue statement for that property. Novalue in the main snak or qualifiers of a statement corresponds to a  class on the statement node, and novalue in a reference snak corresponds to a   class on the reference node.

The classes for  are defined as follows:

Sitelinks
The links are represented as set of predicates describing the link URL. The type of the node is  and it linked with the entity via   predicate.

Badges are described with  predicates. predicate holds the plain-text name of the article, in the language of the linked wiki.

Example:

The subject URL is composed from the language site prefix and the article name, URL-encoded according to, e.g.:

More specifically, encoding used is as follows:


 * 1) Normalize the title by replacing spaces with underscores.
 * 2) Apply wfUrlencode function, which percent-encodes all non-alphanumeric characters except " ".

Redirects
Redirected entities are implemented as  predicates, for example if Q6 redirects to Q1, the dump would be:

Prefixes used
The prefixes are used in RDF formats that allow short prefixes (such as Turtle and RDF). For other formats, the full URL is used.

All prefix URLs that do not contain hostname are prefixed with the hostname of the generating wiki. All prefix URLs that contain hostname are fixed and do not depend on generating wiki.

Standard prefixes used:

Full list of prefixes
This list can be used for queries in SPARQL:

Ontology
This compiles the list of all objects and predicates that are internal to the format. For the meaning of the prefixes, see the prefixes list.

Predicates
Italicized names mean that any property name can be substituted instead of example name P123.

The following predicates are used in deep values for the values of specific types. All these predicates have the domain of  and the range depending on type below.

WDQS data differences
The Wikidata query service has the data in the format described above, but there are small differences that can be important while writing SPARQL queries: See also SPARQL query examples for how to query the data using WDQS service.
 * 1) Types (  or  ) for ,  ,   ,  ,  ,   are currently omitted for performance reasons.
 * 2) Data nodes  are not stored, all the information like version, revision and page props is stored in the entity node  instead. This is done for performance reasons.
 * 3) For labels, only   is stored but not   or  . Since they all have the same data, storing all three is redundant.
 * 4) Redirects are recorded but currently have no additional semantics implemented.
 * 5) WDQS now uses local IRIs (Skolem IRIs) instead of blank nodes to represent somevalue.  These can be tested with   rather than