Parsoid/de



Parsoid ist eine Bibliothek, die es erlaubt, zwischen Wikitext und HTML hin und her zu konvertieren. Die ursprüngliche Anwendung ist in JavaScript geschrieben (unter Verwendung von Node.js) und war seit Dezember 2012 auf dem Wikimedia-Cluster in Betrieb. Im Jahr 2019 wurde Parsoid auf PHP portiert, und die PHP-Version ersetzte die JS-Version auf dem Wikimedia-Cluster im Dezember 2019. Parsoid wird gerade in den Core von MediaWiki integriert, mit dem Ziel, irgendwann den derzeitigen nativen Parser von MediaWiki zu ersetzen.

Parsoid (die PHP-Version) wurde seit Version 1.35, die im September 2020 veröffentlicht wurde, von Natur aus gebündelt mit MediaWiki. Für Installationen außerhalb von Wikimedia wurde Parsoid/JS bis zum Lebensende von MediaWiki 1.31 (LTS) im September 2021 unterstützt.



Technische Details
Parsoid ist eine Anwendung, die zwischen der Wikitext-Syntax von MediaWiki und einem äquivalenten HTML/RDFa-Dokumentenmodell mit erweiterter Unterstützung für automatisierte Verarbeitung und umfassende Bearbeitung hin und her übersetzen kann.

Es wurde von einem Team der Wikimedia Foundation seit 2012 entwickelt. Es wird derzeit ausgiebig von, , und anderen Anwendungen verwendet.

Parsoid soll eine einwandfreie Hin- und Her-Konvertierung ermöglichen, z.B. um Informationsverlust zu vermeiden und auch "schmutzige Unterschiede" zu verhindern.

Auf Wikimedia-Wikis wird Parsoid derzeit für mehrere Anwendungen hinter proxyisiert, der das von Parsoid übersetzte HTML speichert. Es wird erwartet, dass RESTBase letztendlich durch einen Cache ersetzt wird, der enger in MediaWiki integriert ist.

Für weitere Informationen zum Gesamtprojekt, siehe diesen Blog-Beitrag vom März 2013. Um mehr über das verwendete HTML-Modell zu erfahren, sehen Sie sich die MediaWiki-DOM-Spezifikation an.

Ursprünglich wurde Parsoid als Webdienst strukturiert und in JavaScript geschrieben, wobei  verwendet wurde. Ein Technischer Vortrag aus Februar 2019 (Folien) und ein Blog-Beitrag beschreiben den Prozess der Portierung zu PHP. Die Parsoid-Erweiterungs-API befindet sich derzeit in aktiver Entwicklung; eine technische Präsentation aus August 2020 beschreibt diese Arbeit.

GitHub-Repository: https://github.com/wikimedia/parsoid

Verwendung

 * - Liste der für Parsoid veröffentlichten Versionen:
 * - für die Web-API
 * - um den HTML-Code, den Sie von der API erhalten, sinnvoll zu verarbeiten und als zukünftiges Speicherformat nützlich zu sein
 * - Notizen zur Implementierung von Parsoid von.

Installation
In MediaWiki 1.35 ist Parsoid/PHP im Paket enthalten und wird automatisch vom Visual Editor geladen. Es ist keine Konfiguration für MediaWiki 1.35 notwendig, wenn es auf einem einzelnen Server verwendet wird.

Wenn Sie eine MediaWiki-Version neuer als 1.35 verwenden, ist seit dem 24. August 2020 eine ausdrückliche Ladung von Parsoid erforderlich (der automatische Ladetrick wurde in 1.36-wmf.6 entfernt). Fügen Sie zu Ihrer LocalSettings.php hinzu:

Entwicklung
Development happens in the. Code review happens in Gerrit. See Gerrit/Getting started to set up an account for yourself.

If you use the development environment using a virtual machine, you can simply add the role   to it and it will set up a working Parsoid along with. (This may have been broken by the switch to Parsoid/PHP: T258940)

Note that the most-recently released version of Parsoid is written in PHP, and installation of Parsoid/PHP is what is described below. This is what you should use if you are running MediaWiki 1.35 or later. Check Parsoid/JS if you are running the old version of Parsoid written in JavaScript, and used for MW 1.34 and earlier.



Verlinken eines Entwickler-Checkouts von Parsoid
In a standard MediaWiki installation, Parsoid code is bundled in two different ways: first, Parsoid is included from MediaWiki as a composer library,. This contains the main codebase, but does not contain the REST API used by VisualEditor and RESTBase. In order to enable the REST API, the extension code included in the Parsoid library can be loaded with a call to  in your.

For development purposes you usually want to use a git checkout of Parsoid, and not the version bundled in MediaWiki core as a composer library. The following lines added to allow use of a git checkout of Parsoid (optionally), load the Parsoid REST API with  (rather than using the version bundled in VisualEditor) and manually do the Parsoid configuration which is usually done by VisualEditor:

These lines are not necessary for most users of VisualEditor, who can use auto-configuration and the bundled Parsoid code included in MediaWiki 1.35 and VisualEditor, but they will be required for most developers.

If you're serving MediaWiki with Nginx, you'll need to also add something like this in your server block (Assuming your MediaWiki setup has its files residing in ):

To test proper configuration, visit  where  is the hostname in your. (Note that production WMF servers do not expose the Parsoid REST API to the external network.)



Die Tests ausführen
To run all parser tests and mocha tests:

The parser tests have quite a few options now which can be listed using.

If you have the environment variable  pointing to a configured MediaWiki installation, you can run some additional tests with:



Einfachen Wikitext konvertieren
You can convert simple wikitext snippets from the command line using the  script in the   directory:

echo 'Foo' | php bin/parse.php

The parse script has a lot of options. gives you information about this.



Parsoid debuggen (für Entwickler)
See for debugging tips.

Continuous Integration
 As of October 2021 

Parsoid is always available as a library since it is a composer dependency of MediaWiki core. But two pieces are not enabled:


 * Parsoid ServiceWiring
 * Parsoid's external REST api

The test runner Quibble would enable it if it detects  has been cloned as part of the build. In which case it:


 * points the autoloader for  to the cloned code (effectively replacing the version installed by composer)
 * Load the extension

The ServiceWiring should be enabled in MediaWiki starting with 1.38.

The REST API would theorically never get merged in MediaWiki: a) it has never been exposed to the public in production, it is an internal API used by RESTBase which is going away; b) it never has been security audited and c) it is redundant with the enterprise MediaWiki API. The solution will be for VisualEditor to invoke Parsoid directly via the VisualEditor Action API which would save a round trip through the REST API.

Loading the extension is thus a hack which enables using interfaces subject to change and which we don't really want people to use yet.

For most purposes, parsoid should thus not be added as a CI dependency, the only exception as of October 2021 is the Disambiguator MediaWiki extension.

Loading parsoid as an extension let us run MediaWiki integration test jobs against  (such as Quibble, apitesting) and ensure Parsoid and MediaWiki work together.

An extension may be able to write tests with Parsoid even when the repository has not been cloned. Since it is a composer dependency of MediaWiki core the  namespace is available, but the service wiring part is not (it is   in the Parsoid repository and exposed as the   namespace). The  code would only run the parser tests if Parsoid has been loaded (which should be the default with MediaWiki 1.38).

For CI, Parsoid is tested against the tip of mediawiki, whereas mediawiki is tested with the composer dependency. In case of a breaking change, the Parsoid change get merged first (which breaks its CI but not MediaWiki one) and MediaWiki get adjusted when Parsoid is updated. It is thus a one way change.

Release build
For MediaWiki release builds, we have an integration of Parsoid ServiceWiring into VisualEditor in order to have VisualEditor work without further configuration (beside a ). The release build also enables the REST API and hook everything us so that parsoid works out of the box. This is done by copying a bit of parsoid code into VisualEditor which is not in the master branch of VisualEditor since that would be obsolete as soon as Parsoid is updated. Instead the code is maintained in two places.



Technische Dokumente

 * Parsoid/Internals: documentation about Parsoid internals with links to other details.
 * PHP Porting notes and help-wanted tasks
 * Parsoid deployment agenda on Wikimedia cluster (code normally deployed every Monday and Wednesday between 1pm - 1:30pm PST)
 * Parsoid/Round-trip testing: The round-trip testing setup we are using to test the wikitext -> HTML DOM -> wikitext round-trip on actual Wikipedia content.
 * Parsoid/Visual Diffs Testing: Info about visual diff testing for comparing Parsoid's html rendering with php parser's html rendering + a testreduce setup for doing mass visual diff tests.
 * Parsoid/limitations: Limitations in Parsoid, mainly contrived templating (ab)uses that don't matter in practice. Could be extended to be similar to the preprocessor upgrade notes (Might need updating)
 * Parsoid/Bibliography: Bibliography of related literature



Links für Parsoid-Entwickler

 * See Parsoid/Debugging for debugging tips.
 * Upgrading or adding packages to Parsoid
 * See these instructions for syncing Parsoid's copy of parser tests to/from core
 * Parsoid has a limited library interface for invoking it programatically.
 * Tech Talk about Retargeting extensions to work with Parsoid
 * So you want your extension to work with Parsoid
 * Parsoid HTML Specification Versioning
 * So you are going to change Parsoid output



Links für Parsoid-Einsatzkräfte (zum Wikimedia-Cluster)

 * Parsoid/Deployments
 * RT testing commits (useful to check regressions and fixes)
 * Deployment instructions for Parsoid
 * Kibana dashboard
 * Grafana dashboard for wt2html metrics
 * Grafana dashboard for html2wt metrics
 * Prometheus breakdown for the Parsoid cluster on eqiad
 * Prometheus breakdown for the Parsoid cluster on codfw
 * Jenkins Job Builder docs for updating jenkins jobs



Siehe auch

 * API
 * RESTBase: a caching / storing API proxy for page HTML translated by Parsoid
 * Quarterly review meetings of the Parsoid team: April 2015, January 2015 (earlier)
 * Future/Parser plan: Early (now relatively old) design ideas and issues
 * Special:PrefixIndex/Parsoid/: Parsoid-related pages on this wiki
 * Extension:ParsoidBatchAPI (archived)
 * parsoid-jsapi: a high-level interface for extraction and transformation of wikitext, similar to the mwparserfromhell API.
 * Alternative parsers
 * Parsoid/Parser Unification



Externe Links

 * Source code (GitHub mirror)
 * JS Documentation (old version of Parsoid)
 * PHP Documentation
 * Parsoid on the Wikimedia Commons

Kontakt
If you need help or have questions/feedback, you can contact us in or the wikitext-l mailing list. If all that fails, you can also contact us by email at  at the   domain.