Structured Data Across Wikimedia

Jump to navigation Jump to search

Structured Data Across Wikimedia (SDAW) is a project that will help us to structure content on wikitext pages in a way that will be machine-recognizable and -relatable, to make reading, editing, and searching easier and more accessible across projects and on the Internet.

It will help users associate contents between Wikimedia projects, help readers dive deeper in the Wikimedia knowledge ecosystem, and help contributors disseminate information across projects and beyond them in a Wikidata-like way. The project will also provide a venue for experimentation with computer-aided editing tools to make editing easier and more accessible to more editors around the world.


This project is a follow-up to similar development that was completed on Commons, as part of the previous Structured Data on Commons (SDC) grant, and will be partially funded by a three year grant from the Sloan Foundation. Work on SDC made us aware of the need for more advanced metadata for all content and APIs to provide better search results, which would make in turn content more accessible, discoverable, translatable and usable for other needs.

The project has three high level goals:

  1. To allow machines to recognize Wikimedia content and to suggest relationships to other Wikimedia content. We are exploring this first via the image recommendations project.
  2. To design a way to structure articles and pages to enable new content formats – such as content served in smaller, easily digestible pieces that is more accessible for readers to use and share.
  3. To give Wikimedia users a more inviting, more efficient way to search and find content, building on MediaSearch, and exploring new ways to improve search across Wikipedias using Structured Data.

What is changing[edit]

The goal of the project is to design and prototype a new system that aims to be flexible enough to serve all the kinds of metadata we might need to support in the near future.

The first area of action that has been identified is topical metadata to describe what a section of a Wikipedia article is about. This will be supported by data storage infrastructure that can structure section data in wikitext as its own entity and associate topical metadata with each section entity. This will contribute in the following ways:

  1. Tagging sections with relevant structured, language-agnostic Wikidata concepts will help users to discover, translate and localise content. It will also help us matching content between projects (i.e. between Wikipedia and Wikimedia Commons), helping with illustrating articles and growing contributions.
  2. Structuring wikitext content into discrete sections will make it easier to program machines to answer discrete questions and provide quick facts. This would support external platforms or tools that can generate concise answers, and facilitate translation and knowledge parity.
  3. Investing in a flexible and scalable metadata system is an important part of our Evolutionary Architecture. It will be useful for potential upcoming projects such as Shared Citations and Wikifunctions/Abstract Wikipedia, as well as already existing extensions such as Wikibase's ArticlePlaceholder.

The project is currently investigating link analysis systems and concept relationships as ways to determine the topical metadata of a Wikipedia article's sections, via the blue interwiki links in Wikipedia articles. Relationships between items in the Wikidata ontology are also being considered to infer, and potentially identify, relevant concepts that are not explicitly mentioned in the text.

How we plan to use this topical metadata[edit]

While we see many potential use cases that can take advantage of this metadata, we will start by using it to design new ways to improve search on the Wikipedias, like we used Structured Data on Commons to create MediaSearch.

Another possibility we are researching at the moment is to use structured data to improve our image recommendation tools, by allowing users to find images that match to a particular section, instead of just an entire article.

What do we not want to do?[edit]

  1. Leave users out of the process
  2. Overwhelm users with too much new content to moderate
  3. Add any additional bias to Wikimedia projects
  4. Add additional vectors for vandalism
  5. Introduce too much complexity into our systems


The rough example shown here illustrates what a user interface for adding and updating the topical metadata (shown here as "concepts") represented in a selected section might look like, if we learn from discussions that editors want full participation through the entire topical metadata creation process. You can see both unconfirmed machine-detected concepts and confirmed concepts, along with an option to add a custom concept by searching Wikidata. Each concept includes the Q-ID, a link to its Wikidata page, and a description to help the user decide if the concept is an appropriate fit for this section.

The following mockups are a rough representation of how editors might interact with a tool that allows them to attach concepts or topics to sections in an article. There are many aspects of this early representation that are in flux and still need to be discussed.

We're looking for feedback on these ideas so that we can continue to evolve and build on this early prototype. We may, for example, learn that this level of full "human-in-the-loop" interaction with the machine-detected concepts isn't necessary, and instead explore something more lightweight.

Status updates[edit]

May-August 2021[edit]

February 2021[edit]

  • Looking for feedback about these ideas.
  • Working on rough wireframes and mockups to help explore these ideas.
  • Exploring infrastructure to support this work via the Technical Decision Making Forum process. See task T274181

Second half of 2020[edit]


Project feedback is and will always be welcome. We are especially interested in your ideas about the extent to which you want to keep the “human-in-the-loop” throughout the topical metadata creation process. We are looking forward to hearing from you about the following open questions:

  1. Your expectations about the project
    1. What do users expect from this project? What are the necessary actions to be addressed?
    2. How do you envision this metadata being used? Can you think of ways it would aid in your workflows?
  2. Metadata moderation
    1. Is moderation necessary to avoid vandalism and/or bias?
    2. If moderation is necessary, how can it be effectively managed?
  3. Adding and confirming metadata
    1. Do users want to be able to approve or reject metadata suggested by the automated system?
    2. Do users want to be able to add additional metadata beyond what is suggested by the automated system?
    3. Do you think it may just be sufficient for users to have the opportunity to send feedback with suggestions on how to improve the machine generated metadata, when necessary?
  4. Privileges for visualising and editing
    1. Do we want metadata to be visible for all users or only for certain classes of users?
    2. Do we want metadata to be editable for all users or only for certain classes of users?

Also, more specific feedback about related projects can generally be left on the projects' talk pages:


Partial funding for this work is provided by a follow-up restricted grant from the Alfred P. Sloan Foundation, to further the work done by the first round of funding to develop Structured Data on Commons.