Upstream projects

From MediaWiki.org
Jump to navigation Jump to search

MediaWiki developers and users[1] rely on several upstream projects for their architecture or processes. At the same time, they develop software projects used by others, which are downstream to them. These links define our location in the free software map, our neighbor communities, and the flows of feedback, patches, and contributors.

Let's identify our upstream projects, our neighbor communities, and the contributors that connect us with them.

Motivation[edit]

This is a first step to improve our relations with other communities, to increase the contributions received, and our influence in the projects that matter to us.

Our mid-term goals include:

  • identify the projects where we want to see significant development, to the point of sending patches as well
  • identify the communities where Wikimedia should be regularly active and heard
  • identify the people in the Wikimedia and upstream communities that know each other and act as bridge
  • identify organizations and events we should get in touch and be part of
  • get involved in bigger development efforts regularly, become a regular FOSS player

Invented here[edit]

Projects we maintain that we want others to use and contribute to.

Own libraries used by MediaWiki core
Name Description Language Repository Issues Main contacts
MediaWiki The collaborative editing engine that runs Wikipedia. PHP Diffusion Phabricator MediaWiki Platform team, Tim Starling, Brion Vibber
OOjs Library for working with objects in JavaScript. JavaScript Gerrit Phabricator Timo Tijhof, Roan Kattouw
OOUI OOjs-based UX toolkit. JavaScript, PHP Gerrit Phabricator James Forrester, Bartosz Dziewoński
CSSJanus Convert stylesheets from left-to-right to right-to-left. PHP GitHub GitHub Timo Tijhof, Roan Kattouw
CLDRPluralRuleParser Parse plural rules specified in the CLDR project. PHP Gerrit Phabricator
RemexHtml HTML5 parsing library to replace Tidy. PHP Gerrit Phabricator Tim Starling
wikimedia/cdb Pure PHP replacement for dba functions. PHP Gerrit Phabricator Tim Starling, Chad Horohoe
wikimedia/at-ease Safe replacement to PHP's @ for suppressing warnings. PHP Gerrit Phabricator Tim Starling, Legoktm
wikimedia/utfnormal Unicode normalisation routines. PHP Gerrit Phabricator Brion VIBBER
wikimedia/ip-set Match IPs against CIDR sets. PHP Gerrit Phabricator Brandon Black
wikimedia/running-stat Efficient way to compute statistics over many observations. PHP Gerrit Phabricator Performance Team
wikimedia/base-convert Convert an arbitrarily-long string from one numeric base to another PHP Gerrit Phabricator
ScopedCallback Automatically run code once a reference is out of scope. PHP Gerrit Phabricator
WaitConditionLoop Poll state changes while executing other code concurrently. PHP Gerrit Phabricator Performance Team, Aaron Schulz
WrappedString Compact strings with common start/end pairs. PHP Gerrit Phrabricator Performance Team, Timo Tijhof
composer-merge-plugin Merge multiple composer.json files at runtime. PHP GitHub GitHub Bryan Davis
testing-access-wrapper Helper library for unit tests. PHP Gerrit Phabricator
php-session-serializer Abstract session_ functions and related INI settings. PHP Gerrit Phabricator Brad Jorsch
purtle Fast streaming RDF serializer PHP Gerrit WMDE, Stas Malyshev
Own libraries used by Wikimedia-deployed extensions and services
Name & URL Description Language(s) Repository Bug tracker Main contacts
Parsoid Two-way MediaWiki wikitext↔HTML parser. Node.js Diffusion Phabricator (under "Parsoid") Subbu Sastry,

C. Scott Ananian

VisualEditor Rich HTML editor. JavaScript Gerrit Phabricator James Forrester, Ed Sanders
UnicodeJS String manipulation based on Unicode standards (word breaks, graphemes, character classes). JavaScript Gerrit Phabricator David Chan, James Forrester, Ed Sanders
wikimedia/css-sanitizer A CSS tokenizer, parser and grammar matcher. PHP Gerrit Phabricator
wikimedia/equivset Detect visually equivalent UTF-8 characters. PHP Gerrit Phabricator Brion VIBBER
Milkshake jQuery.uls, Language selection JavaScript GitHub GitHub Santhosh Thottingal
jQuery.IME, input method editor library JavaScript GitHub GitHub Santhosh Thottingal
jQuery.i18n, Internationalization library JavaScript GitHub GitHub Santhosh Thottingal
jQuery.webfonts, Font embedding capabilities JavaScript GitHub GitHub Santhosh Thottingal
Offline content tools mw-ocg-bundler, article/resource spider tool Node.js GitHub GitHub C. Scott Ananian
mw-ocg-latexer, article PDF conversion tool Node.js GitHub GitHub C. Scott Ananian
textcat Text categorization library in PHP PHP Gerrit Phabricator Stas Malyshev, Trey Jones
Other projects and libraries maintained by the MediaWiki engineering community
Name & URL Description Language(s) Repository Bug tracker Main contacts
translatewiki.net Where we organise localisation of MediaWiki. PHP phab:diffusion/GTWN/, MLEB translatewiki:Support Nikerabbit (founder), Siebrand (co-maintainer), Raymond (MediaWiki updates responsible)
Pywikibot Python Wiki Robot Framework to automatize tasks Python Diffusion Phabricator (under "Pywikibot") xqt
Orgchart tool Generates organisational charts. JavaScript plus some node.js hackery Diffusion Phabricator (under "Utilities") MarkTraceur
Bingle Pipes bugs form Bugzilla into tools like Mingle and Trello; obsolete after migration to Phabricator Python Github Github User:Awjrichards

More stuff (hundreds of projects) by fellow MediaWiki developers and Wikimedians is also listed at:

Components[edit]

Projects that contribute to the architecture of MediaWiki, key extensions, and applications. They are included in the downloadable packages of key Wikimedia projects, or they are identified as required dependencies.

Upstream components
Name & URL Description Language(s) Repository Bug tracker Main contacts
GNU diff and diff3 Powers recent changes, history and undo button (i.e. most editing features).
Elasticsearch Distributed real-time search and analytics engine. Java GitHub GitHub
jQuery Lots of DOM traversing, and other DOM-related helpers, for JavaScript. JavaScript GitHub GitHub Timo Tijhof

(@Krinkle on GitHub)

OpenStack Nova OpenStack compute service. Python OpenStack Gerrit Launchpad Ryan Lane
OpenStack Glance OpenStack image service. Python OpenStack Gerrit Launchpad Ryan Lane
OpenStack Keystone OpenStack identity service. Python OpenStack Gerrit Launchpad Ryan Lane
OpenStack Swift OpenStack object storage service. Python OpenStack Gerrit Launchpad Wikimedia SRE
Puppet Configuration management system. Ruby GitHub Jira Wikimedia SRE
SaltStack Orchestration and remote execution system. Python GitHub GitHub Ryan Lane
Unihan Database For Chinese language conversion Used as database
Smart Common Input Method For Chinese language conversion Used as database
libtabe For Chinese language conversion Used as database
ImageMagick general purpose image scaling C No repo ImageMagick.tar.gz Forum
Lua Scripting language C
ACE Web based code editor JavaScript, HTML GitHub GitHub User:TheDJ (has signed contributor license)
VIPS Image scaling of large images (VipsScaler) GitHub GitHub User:Bryan
libRsvg PNG renderer for SVG images C GNOME git GNOME Gitlab
MathJax Math rendering in HTML/SVG/MathML; used by Math extension JavaScript GitHub GitHub
FFmpeg / libav Video conversion and scaling C ffmpeg / libav ffmpeg / libav
MWLib mediawiki parser and converter Python GitHub GitHub
Pygments.org Syntax highlighting Python Bitbucket Bitbucket
Unicode CLDR (Common Locale Data Repository) All we manage to take: language names, plural rules, language-territory information, ... [1] Nemo (CLDR Survey Tool manager)
CLDRPluralRuleParser Plural support based on CLDR JavaScript GitHub GitHub User:Santhosh.thottingal
Apertium Free software, rule-based machine translation; used by Translate, ContentTranslation and others. Soureforge [2] Francis Tyers, Firespeaker, Unhammer
Ceph Distributed storage system C++ GitHub Ceph
Cortado Media streaming Java applet Java xiph [3]
GlusterFS Distributed file system [4]
HHVM GitHub [5]
HTML Tidy corrects and cleans up HTML and XML documents by fixing markup errors and upgrading legacy code to modern standards Sourceforge Sourceforge
Kaltura / mwEmbed used by TimedMediaHandler [6]
Leaflet Maps on Mobile JavaScript [7]
Lilypond used by the Score extension [8]
MariaDB Database [9]
memcached GitHub [10]
multilightbox used by Extension:MultimediaViewer [11]
MySQL Database [12]
PHP Programming language that MediaWiki is written in [13]
Rangy JavaScript range and selection library, used by VisualEditor [14]
Swift Distributed storage system [15]
lightncandy HTML templating (mustache and handlebars) in PHP PHP GitHub GitHub EBernhardson, Kaldari, ?
Composer Dependency manager for PHP libraries - used for managing MediaWiki libraries and CI PHP getcomposer.org GitHub Legoktm, Bryan Davis, Reedy
Blazegraph RDF-graph database supporting RDF and SPARQL - used by WDQS Java Blazegraph.com Jira
(what else?)

MediaWiki development process[edit]

Projects that contribute to our development process. They are installed in Wikimedia servers and they are used in the release process of key Wikimedia projects.

Upstream tools
Name & URL Description Language(s) Repository Bug tracker Main contacts
Jenkins Continuous integration Java GitHub Jira Hashar, MarkTraceur
Jenkins Job Builder Job creation system Python Git Launchpad Hashar, Krinkle, MarkTraceur
Zuul Gerrit/Jenkins bridge trigger Python Git Launchpad Hashar, MarkTraceur
Gerrit Code review tool Java Gerrit Monorail (left empty for our gerrit admin), QChris
Etherpad Lite Real-time collaborative document editing JavaScript, mostly node.js GitHub GitHub MarkTraceur, Mutante
PHPunit Unit testing framework for PHP PHP GitHub [16]
QUnit Unit testing framework for JavaScript. JavaScript GitHub GitHub Krinkle
Grunt Task runner for build and test scripts. JavaScript GitHub GitHub
Karma Automation for running unit tests in web browsers and capturing their results. JavaScript GitHub GitHub Krinkle
Vagrant Github [17]
Logstash A tool that can be used to collect, process and forward events and log messages JRuby GitHub Jira bd808
Kibana A browser based analytics and search interface for Elasticsearch that was developed primarily to view Logstash event data JavaScript GitHub GitHub bd808
Graphite/Carbon Real-time graphing system and time series Python GitHub GitHub
Grafana Metrics visualisation Go and JavaScript GitHub GitHub
JSDuck Documentation generator Ruby GitHub GitHub issues
Mailman Mailing list server / management (see also m:Mailing lists/Administration) Python Gitlab Gitlab
git-review Command line tool for using Gerrit GitHub [18]
Phabricator Task tracker and software development tool used by Wikimedia, see https://phabricator.wikimedia.org PHP GitHub phab:tag/phabricator-upstream and secure.phabricator.com Andre et al.
Continuous integration/PHP CodeSniffer Continuous integration PHP GitHub GitHub Legoktm (?)
(what else?)

Non-MediaWiki but still WMF-used stuff[edit]

Upstream tools
Name & URL Description Language(s) Repository Bug tracker Main contacts
Varnish Reverse proxy server [19]
nginx Reverse proxy / web server [20]
RT (Request Tracker) Ticket system used by Wikimedia Operations team [21]
Selenium Automated Browser Testing [22]
Icinga Network system monitoring (icinga.wikimedia.org) [23]
OTRS Ticket system used for answering queries, see https://ticket.wikimedia.org/otrs [24]
CiviCRM Used for WMF Fundraising (now upstreaming patches) [25]
WordPress https://blog.wikimedia.org PHP Trac (mailing lists)
Rawdog https://en.planet.wikimedia.org/ etc Python http://offog.org/git/rawdog/
(what else?)

Client-side interaction[edit]

Software heavily interacting or interconnected with MediaWiki functionality and work, but user side.

Integrated on Wikimedia projects[edit]

Projects that are used in some way (embedding, linking) on Wikimedia projects, often installed in Wikimedia servers like ToolLabs.

Upstream tools
Name & URL Description Language(s) Repository Bug tracker Main contacts
CodeMirror a browser based text editor (Used by the CodeMirror extension to provide syntax highlighting in the 2010 wikitext editor.) Javascript GitHub [https://github.com/codemirror/codemirror/issues Github
IIPImage Viewing and zooming of ultra high-resolution images (linked by default on Wikimedia Commons) C++ SourceForge SourceForge
Hundreds of other packages on Tools
(what else?)

Editing tools[edit]

For a full listing, see the lists m:Open Source Toolset, m:FLOSS-Exchange, d:Q6584911.

Upstream tools
Name & URL Description Language(s) Repository Bug tracker Main contacts
Multimedia software and SVG tools Software used to contribute/work on multimedia in MediaWiki wikis, like Inkscape and GIMP. May play well with MediaWiki or not.
LibreOffice (formerly OpenOffice) Productivity suite, can e.g. produce charts and export/publish documents to wikitext (via libreoffice-wiki-publisher). [26]
Browser spellcheckers Aiding editors to prevent errors before they're saved. Over 120 in Firefox, see respective descriptions.
KIPI uploader and other third party upload tools Multimedia contribution software by third parties, usually made for integration in their software (e.g. digiKam, Gwenview).
(what else?)

Web browsers[edit]

Name & URL Description Language(s) Repository Bug tracker Main contacts
Chromium/Chrome Browser by Google chromium.org (Wikimedia issues are specifically tracked)
Firefox Browser by Mozilla mozilla.org
Opera Browser by Opera Software opera.com
Internet Explorer Browser by Microsoft microsoft.com
Safari Browser by Apple webkit.org
(what else?)

Notes[edit]

  1. For instance, whether volunteers or for work: maintainers and contributors of MediaWiki core and extensions and other developers; Wikimedia projects editors and sysadmins; sysadmins of MediaWiki wikis.

See also[edit]