Parsoid
Parsoid
双向运行时维基文本解析器。 使用RDFa在wikitext和HTML/XML DOM之间相互转换。
|
Parsoid是一个允许在wikitext和HTML之间来回转换的库。 原始应用程序是用JavaScript编写的(借助Node.js),并自2012年12月起在维基媒体集群上运行。 在2019年,Parsoid移植到PHP,维基媒体集群于2019年12月以PHP版本取代了JS版本。 Parsoid已经集成到MediaWiki核心,其目标是最终取代MediaWiki当前的本地解析器。 In early 2024, Parsoid began to be used on some production wikis as the default parser for read views.
Parsoid (PHP版本) 已经原生捆绑到2020年9月发布的MediaWiki版本1.35或更高版本中。 对于非维基媒体安装,Parsoid/JS在2021年9月MediaWiki 1.31(LTS)生命周期结束之前一直受支持。 The legacy parser will still be supported in MediaWiki 1.43 (LTS).
技术细节
Parsoid是一个应用程序,可以在MediaWiki的wiki文本语法和等效的HTML/RDFa文档模型之间来回转换,并增强了对自动处理和丰富编辑的支持。
自2012年以来,它一直由维基媒体基金会的一个团队开发。 它目前被VisualEditor 、Structured discussions 、内容翻译 和其他应用广泛使用。
Parsoid旨在提供完美的来回转换,即避免信息丢失并防止“脏差异”。
On Wikimedia wikis, for several applications, Parsoid is currently proxied behind RESTBase , which stores the HTML translated by Parsoid. It is expected that RESTBase will eventually be replaced with a cache more tightly integrated with MediaWiki.
For more on the overall project, see this blog post from March 2013. To read about the HTML model being used, see MediaWiki DOM spec.
Parsoid was originally structured as a web service and written in JavaScript, making use of Node.js
.
A tech talk from February 2019 (slides) and blog post describes the process of porting it to PHP.
The Parsoid extension API is currently under active development; a tech talk from August 2020 describes this work.
Github仓库: https://github.com/wikimedia/parsoid
用法
- Parsoid/Releases - Parsoid发布的版本列表
- Parsoid/API - 用于网页API
- MediaWiki DOM spec - 用于理解您从API获取的HTML代码,旨在用作未来的存储格式
- Parsoid/LanguageConverter - 关于Parsoid实现LanguageConverter 的说明
- Parsoid/Known differences with Core Parser output
安裝
在MediaWiki 1.35中,Parsoid/PHP包含在捆绑包中,并由可视化编辑器自动加载。 MediaWiki 1.35及更高版本无需配置。
Parsoid exports an internal REST API which was historically used by RESTBase and not accessible outside the WMF internal cluster. This is no longer required for Visual Editor or core read views, and the internal API is being deprecated and is planned for removal in MW 1.43.
Parsoid is nominally a composer library used by mediawiki core. If you still require the internal API for some reason, you can explicitly load Parsoid "as an extension" by adding the following to LocalSettings.php:
wfLoadExtension( 'Parsoid', "$IP/vendor/wikimedia/parsoid/extension.json" );
Any remaining third-party users of the internal Parsoid API are strongly encouraged to migrate to the core REST HTML page endpoint which provides equivalent functionality.
开发
开发地址在Parsoid Git repository。 代码审查地址在Gerrit。 请参阅Gerrit/教程为自己创建一个帐户。
If you use the MediaWiki-Vagrant development environment using a virtual machine, you can simply add the role visualeditor
to it and it will set up a working Parsoid along with Extension:VisualEditor .
(This may have been broken by the switch to Parsoid/PHP: T258940)
Note that the most-recently released version of Parsoid is written in PHP, and installation of Parsoid/PHP is what is described below. This is what you should use if you are running MediaWiki 1.35 or later. Check Parsoid/JS if you are running the old version of Parsoid written in JavaScript, and used for MW 1.34 and earlier.
链接Parsoid的开发人员签出
In a standard MediaWiki installation, Parsoid is included from MediaWiki as a composer library, wikimedia/parsoid
.
For development purposes you usually want to use a git checkout of Parsoid, and not the version bundled in MediaWiki core as a composer library. The following lines added to LocalSettings.php allow use of a git checkout of Parsoid (optionally), load the Parsoid REST API with wfLoadExtension (rather than using the version bundled in VisualEditor) and manually do the Parsoid configuration which is usually done by VisualEditor:
$parsoidInstallDir = 'vendor/wikimedia/parsoid'; # 捆绑包副本
#$parsoidInstallDir = '/my/path/to/git/checkout/of/Parsoid';
// 写给开发人员:确保Parsoid从$parsoidInstallDir执行,
// (不是默认包含在mediawiki-core中的版本)
// 必须在wfLoadExtension()*之前*执行
if ( $parsoidInstallDir !== 'vendor/wikimedia/parsoid' ) {
function wfInterceptParsoidLoading( $className ) {
// Only intercept Parsoid namespace classes
if ( preg_match( '/(MW|Wikimedia\\\\)Parsoid\\\\/', $className ) ) {
$fileName = Autoloader::find( $className );
if ( $fileName !== null ) {
require $fileName;
}
}
}
spl_autoload_register( 'wfInterceptParsoidLoading', true, true );
// 在MW 1.39中添加了AutoLoader::registerNamespaces
AutoLoader::registerNamespaces( [
// 与下文的“autoload”子句保持同步
// $parsoidInstallDir/composer.json
'Wikimedia\\Parsoid\\' => "$parsoidInstallDir/src",
] );
}
wfLoadExtension( 'Parsoid', "$parsoidInstallDir/extension.json" );
# 手动配置Parsoid
$wgVisualEditorParsoidAutoConfig = false;
$wgParsoidSettings = [
'useSelser' => true,
'rtTestMode' => false,
'linting' => false,
];
$wgVirtualRestConfig['modules']['parsoid'] = [
// Parsoid实例的URL。
// 如果Parsoid未在本地运行,则应更改$wgServer以匹配非本地主机
// 在macOS中使用Docker时,可能需要将$wgServer替换为http://host.docker.internal:8080
// 在linux中使用Docker时,可能需要将$wgServer替换为http://172.17.0.1:8080
'url' => $wgServer . $wgScriptPath . '/rest.php',
// Parsoid“域名”,见下文(可选,很少需要)
// 'domain' => 'localhost',
];
unset( $parsoidInstallDir );
These lines are not necessary for most users of VisualEditor, who can use auto-configuration and the bundled Parsoid code included in MediaWiki 1.35 and VisualEditor, but they will be required for most developers.
If you're serving MediaWiki with Nginx, you'll need to also add something like this in your server block (Assuming your MediaWiki setup has its files residing in /w/
):
location /w/rest.php/ {
try_files $uri $uri/ /w/rest.php?$query_string;
}
To test proper configuration, visit {$wgScriptPath}/rest.php/{$domain}/v3/page/html/Main%20Page
where $domain
is the hostname in your $wgCanonicalServer
.
(Note that production WMF servers do not expose the Parsoid REST API to the external network.)
运行测试
要运行所有解析器测试和mocha测试,请执行以下操作:
$ composer test
解析器测试现在有很多选项,可以使用php bin/parserTests.php --help
命令列出。
如果您的环境变量MW_INSTALL_DIR
指向已配置的MediaWiki安装,则可以使用以下命令运行一些额外的测试:
$ composer phan-integrated
转换简单的wikitext
您可以使用bin/
目录中的parse.php
脚本从命令行转换简单的wikitext片段:
echo 'Foo' | php bin/parse.php
解析脚本有很多选项。
php bin/parse.php --help
命令能为您提供相关信息。
调试Parsoid(对于开发人员)
请参阅Parsoid/Debugging 了解调试提示。
持续集成
As of October 2021
Parsoid is always available as a library since it is a composer dependency of MediaWiki core. But two pieces are not enabled:
- Parsoid ServiceWiring
- Parsoid's external REST api
The test runner Quibble would enable it if it detects mediawiki/services/parsoid.git
has been cloned as part of the build.
In which case it:
- points the autoloader for
Wikimedia\Parsoid
to the cloned code (effectively replacing the version installed by composer) - 载入扩展
wfLoadExtension( 'Parsoid', '/path/to/cloned/repo' );
The ServiceWiring should be enabled in MediaWiki starting with 1.38.
The REST API would theorically never get merged in MediaWiki: a) it has never been exposed to the public in production, it is an internal API used by RESTBase which is going away; b) it never has been security audited and c) it is redundant with the enterprise MediaWiki API. The solution will be for VisualEditor to invoke Parsoid directly via the VisualEditor Action API which would save a round trip through the REST API.
Loading the extension is thus a hack which enables using interfaces subject to change and which we don't really want people to use yet.
For most purposes, parsoid should thus not be added as a CI dependency, the only exception as of October 2021 is the Disambiguator MediaWiki extension.
Loading parsoid as an extension let us run MediaWiki integration test jobs against mediawiki/services/parsoid.git
(such as Quibble, apitesting) and ensure Parsoid and MediaWiki work together.
An extension may be able to write tests with Parsoid even when the repository has not been cloned.
Since it is a composer dependency of MediaWiki core the MediaWiki\Parsoid
namespace is available, but the service wiring part is not (it is extension/src
in the Parsoid repository and exposed as the \MWParsoid
namespace).
The ParsoidTestFileSuite.php
code would only run the parser tests if Parsoid has been loaded (which should be the default with MediaWiki 1.38).
For CI, Parsoid is tested against the tip of mediawiki, whereas mediawiki is tested with the composer dependency. In case of a breaking change, the Parsoid change get merged first (which breaks its CI but not MediaWiki one) and MediaWiki get adjusted when Parsoid is updated. It is thus a one way change.
发布版本
For MediaWiki release builds, we have an integration of Parsoid ServiceWiring into VisualEditor in order to have VisualEditor work without further configuration (beside a wfLoadExtension( 'VisualEditor' )
).
The release build also enables the REST API and hook everything us so that parsoid works out of the box.
This is done by copying a bit of parsoid code into VisualEditor which is not in the master branch of VisualEditor since that would be obsolete as soon as Parsoid is updated.
Instead the code is maintained in two places.
技术文档
- Parsoid/Internals: documentation about Parsoid internals with links to other details.
- PHP Porting notes and help-wanted tasks
- Parsoid deployment agenda on Wikimedia cluster (code normally deployed every Monday and Wednesday between 1pm - 1:30pm PST)
- Parsoid/Round-trip testing: The round-trip testing setup we are using to test the wikitext -> HTML DOM -> wikitext round-trip on actual Wikipedia content.
- Parsoid/Visual Diffs Testing: Info about visual diff testing for comparing Parsoid's html rendering with php parser's html rendering + a testreduce setup for doing mass visual diff tests.
- Parsoid/limitations: Limitations in Parsoid, mainly contrived templating (ab)uses that don't matter in practice. Could be extended to be similar to the preprocessor upgrade notes (Might need updating)
- Parsoid/Bibliography: Bibliography of related literature
Parsoid开发者的相关链接
- 请参阅Parsoid/Debugging了解调试提示。
- 升级或添加软件包到Parsoid
- 请参阅以下说明,了解如何将Parsoid的解析器测试副本同步到核心/从核心同步
- Parsoid有一个有限的库接口用于以编程方式调用它。
- 关于重定向扩展以使用Parsoid的技术讲座
- 如果您希望将扩展与Parsoid一起使用
- Parsoid HTML规范版本控制
- 如果您要更改Parsoid输出
对于Parsoid部署者的链接(到维基媒体集群)
- Parsoid/Deployments
- RT testing commits (useful to check regressions and fixes)
- Deployment instructions for Parsoid
- Kibana仪表盘
- Grafana dashboard for wt2html metrics
- Grafana dashboard for html2wt metrics
- Prometheus breakdown for the Parsoid cluster on eqiad
- Prometheus breakdown for the Parsoid cluster on codfw
- Jenkins Job Builder docs for updating jenkins jobs
参见
- API
- RESTBase: 由Parsoid翻译的页面HTML的缓存/存储API代理
- Parsoid 团队的季度审查会议:2015年4月,2015年1月(更早)
- Future/Parser plan: 早期(现在相对较旧)的设计理念和问题
- Special:PrefixIndex/Parsoid/: 此wiki中与Parsoid相关的页面
- Extension:ParsoidBatchAPI (存档)
- parsoid-jsapi:用于提取和转换wikitext的高级接口,类似于mwparserfromhell API。
- 替代解析器
- Parsoid/Parser Unification
外部链接
- 源代码(GitHub镜像源)
- JS文档(旧版Parsoid)
- PHP文档
- 维基媒体共享资源上的Parsoid
联络
如果您需要帮助或有问题/反馈,您可以在#mediawiki-parsoid connect或wikitext-l邮箱列表与我们联系。
如果所有这些方式都失败了,您也可以通过电子邮件与我们联系,地址为wikimedia.org
域名的content-transform-team
。
Parsoid由the Content Transform Team维护。
获取帮助:
|