Talk:Parsoid/2021
Add topic| This page used the Structured Discussions extension to give structured discussions. It has since been converted to wikitext, so the content and history here are only an approximation of what was actually displayed at the time these comments were made. |
Report a new bug in Phabricator
Previous discussion was archived at Talk:Parsoid/Archive on 24 June 2015.
Parsoid is not working in non-english language
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
I tried several time to setup my wiki in spanish languate with MediaWiki 1.35.1, and always shows a Parasoid/Rest error curl 7, however when I choose english as the wiki languate, everything goes fine.
Hope this can be fixed, cause my users doesn't talk english. 186.151.92.120 (talk) 06:20, 30 January 2021 (UTC)
- How did you install Parsoid? Arlolra (talk) 17:55, 17 February 2021 (UTC)
Whitespace in headings?
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
What is the intended behavior when parsingehavior when parsing:
== Foo ==
I would have expected the whitespace around Foo to be preserved, but it's not. The example at Parsoid/API#POST 2 implies that it is, but when I try it, the whitespace is gone:
wget -q -O - 'http://en.wikipedia.org/api/rest_v1/page/html/User:RoySmith%2Fsandbox%2Fparsoid-whitespace-example'
<!DOCTYPE html>
<html prefix="dc: http://purl.org/dc/terms/ mw: http://mediawiki.org/rdf/" about="https://en.wikipedia.org/wiki/Special:Redirect/revision/1005513872"><head prefix="mwr: https://en.wikipedia.org/wiki/Special:Redirect/"><meta property="mw:TimeUuid" content="b56a5f60-69b0-11eb-876b-49aa12313550"/><meta charset="utf-8"/><meta property="mw:pageId" content="66664679"/><meta property="mw:pageNamespace" content="2"/><link rel="dc:replaces" resource="mwr:revision/0"/><meta property="mw:revisionSHA1" content="9fa2ea02674418d1bab8d09bd0c639bcf220a57b"/><meta property="dc:modified" content="2021-02-08T01:54:45.000Z"/><meta property="mw:html:version" content="2.2.0"/><link rel="dc:isVersionOf" href="//en.wikipedia.org/wiki/User%3ARoySmith/sandbox/parsoid-whitespace-example"/><title>User:RoySmith/sandbox/parsoid-whitespace-example</title><base href="//en.wikipedia.org/wiki/"/><link rel="stylesheet" href="/w/load.php?lang=en&modules=mediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Csite.styles&only=styles&skin=vector"/><meta http-equiv="content-language" content="en"/><meta http-equiv="vary" content="Accept"/></head><body id="mwAA" lang="en" class="mw-content-ltr sitedir-ltr ltr mw-body-content parsoid-body mediawiki mw-parser-output" dir="ltr"><section data-mw-section-id="0" id="mwAQ"></section><section data-mw-section-id="1" id="mwAg"><h2 id="Foo">Foo</h2></section></body></html> RoySmith (talk) 02:13, 8 February 2021 (UTC)
Parsoid starts but fails to connect with curl
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
OS is Fedora 31
I downloaded parsoid from bluespice wiki
ii. create service under /etc/system/system/parsoid.service
[Unit]
Description=Mediawiki Parsoid web service on node.js
Documentation=
Wants=local-fs.target network.target
After=local-fs.target network.target
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
User=nobody
Group=nobody
WorkingDirectory=/opt/parsoid
#EnvironmentFile=-/etc/parsoid/parsoid.env
ExecStart=/usr/bin/nodejs /opt/parsoid /bin/server.js
KillMode=process
Restart=on-success
PrivateTmp=true
StandardOutput=syslog
iii. Under /opt/parsoid/config.yaml
worker_heartbeat_timeout: 300000
logging:
level: info
services:
- module: lib/index.js
entrypoint: apiServiceWorker
conf:
localsettings: ./localsettings.js
iv. Under /opt/parsoid/localsettings.js
/
* This is an example configuration for a BlueSpiceWikiFarm setup
* In this case 'httpd' is used as wiki webserver machine name as it is in our
* docker environment.
/
'use strict';
exports.setup = function(parsoidConfig) {
parsoidConfig.dynamicConfig = function(domain) {
var baseUrl = Buffer.from( domain, 'base64').toString();
parsoidConfig.setMwApi({
uri: baseUrl + '/api.php',
domain: domain,
strictSSL: false
});
}
};
The nodejs is at version 10 and parsoid is v0.10
Here's the output of curl
[root@wiki-server BlueSpice3]# curl http://127.0.0.1:8000
<!DOCTYPE html>
<#html lang="en">
<#head>
<#meta charset="utf-8">
<#title>Error<#/title>
<#/head>
<#body>
<#pre>Internal Server Error<#/pre>
<#/body>
<#/html>
(I've added a # in the bracket to show more info)
SELINUX is disabled, firewall is open and listening port 8000, although netstat doesn't show that parsoid service is using the port
[root@wiki-server BlueSpice3]# netstat -aon | grep 8000
tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN off (0.00/0/0)
httpd is configured with SSL domain certificate and https enabled.
Why does this fail? Johnjin216326 (talk) 20:15, 16 February 2021 (UTC)
- Try restarting the Parsoid service and see what information is logged to syslog? Arlolra (talk) 17:53, 17 February 2021 (UTC)
TagWhiteList in PHP Parsoid
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
Is there a TagWhiteList in PHP parsoid? I used it in JS parsoid for the Draw.io Extension (added 'A' and 'IMG' to WikitextConstants.js) to show the images in VisualEditor. Dueni.f (talk) 08:46, 18 February 2021 (UTC)
- You probably want AllowedLiteralTags
- https://github.com/wikimedia/parsoid/blob/master/src/Config/WikitextConstants.php#L86-L113 Arlolra (talk) 16:09, 18 February 2021 (UTC)
- Exactly right. Thank you very much. Dueni.f (talk) 06:51, 19 February 2021 (UTC)
Parsoid git clone installation fails
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
I used to install Parsoid for MW 1.31 (supported until June 2021) with:
# git clone --branch v0.10.0 https://gerrit.wikimedia.org/r/p/mediawiki/services/parsoid /usr/lib/parsoid
but the URL is no longer valid, it returns 404. How to install it? S0ring (talk) 08:50, 18 February 2021 (UTC)
- That command works for me. Maybe your client isn't following redirects? Try without the /p in the url, it's now just https://gerrit.wikimedia.org/r/mediawiki/services/parsoid Arlolra (talk) 16:07, 18 February 2021 (UTC)
- Thanks, but https://gerrit.wikimedia.org/r/mediawiki/services/parsoid redirects to https://gerrit.wikimedia.org/r/mediawiki/services/parsoid and returns Not Found S0ring (talk) 09:17, 19 February 2021 (UTC)
- This works
# git clone --branch v0.10.0 https://github.com/wikimedia/parsoid.gitS0ring (talk) 10:01, 19 February 2021 (UTC) - https://gerrit.wikimedia.org/r/admin/repos/mediawiki/services/parsoid Arlolra (talk) 17:39, 19 February 2021 (UTC)
- This URL returns the following error:
# git clone --branch v0.10.0 https://gerrit.wikimedia.org/r/admin/repos/mediawiki/services/parsoid /usr/lib/parsoidCloning into '/usr/lib/parsoid'...fatal: https://gerrit.wikimedia.org/r/admin/repos/mediawiki/services/parsoid/info/refs not valid: is this a git repository?S0ring (talk) 08:42, 22 February 2021 (UTC)- Sorry, I was suggesting you visit that url in a browser (https://gerrit.wikimedia.org/r/admin/repos/mediawiki/services/parsoid)
- It lists the url I had supplied earlier for cloning,
- git clone "https://gerrit.wikimedia.org/r/mediawiki/services/parsoid" Arlolra (talk) 15:44, 23 February 2021 (UTC)
Parsoid - Memory exhaustion on a big page
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
Hello, I'm having some problems with mediawiki parsoid regarding memory exhaustion, can someone help me?
on a very big page (can't tell the exact size, but the original written on MS Word has more than 70 pages) I get the following issue
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 135168 bytes) in /var/www/html/mediawiki-1.35.1/vendor/wikimedia/parsoid/src/Html2Wt/WikitextSerializer.php on line 1683. Is an explode function
As you can see, it says the memory limit is 128M, but my phpinfo says 750M, configured via php.ini in several places to make sure (php.ini, php-fpm.conf)
from my phpinfo
memory_limit 750M 750M
here's a grep -r memory_limit on my /etc
php-fpm.d/www.conf:php_admin_value[memory_limit] = 750M
php.ini:memory_limit = 750M
so, both php.ini and fpm are configured with 750M
I already tryed to fix the memory_limit on the LocalSettings.php, but also no deal
PHP 7.4.14 (fpm-fcgi)
MediaWiki 1.35.1
Lua 5.1.5
ICU 65.1
MySQL 5.6.35-80.0-log
wikimedia/parsoid 0.12.1
Can someone help me? This is preventing me and my team to create long and important documents.
Thank you! 189.9.10.125 (talk) 16:51, 19 February 2021 (UTC)
- Oh, and I did stop/restart php, phpfpm and httpd. Even restarted the OS. 189.9.10.125 (talk) 16:54, 19 February 2021 (UTC)
- I also tryed to set ini_set( 'memory_limit', '750M' ); on wikitext2html and WikitextSerializer.php, inside serializeDOM, but it raises the same error 189.9.10.125 (talk) 17:58, 19 February 2021 (UTC)
- This isn't an inherent problem with Parsoid. On the WMF cluster, Parsoid runs with an ~1.4G memory limit, which it occasionally hits, but is certainly not limited to 128M
- https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L18558-L18560
- Something in your setup is enforcing that limit. Maybe it's the OS, maybe it's the HTTP server, or PHP configurations you're mentioning.
- Try isolating it. You can run Parsoid on the command line with bin/parse.php
- Pass it your large page and see if you run up against the memory limit there Arlolra (talk) 18:21, 19 February 2021 (UTC)
- Ok, i'll try that,
- thank you 189.9.10.125 (talk) 18:25, 19 February 2021 (UTC)
Parsoid with Kerberos and Auth_Remoteuser
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
Hello all,
I'm facing around with some problems with Parsoid and the Remoteuser Authentication with Kerberos.
First my setup:
Ubuntu 20.04.
Mediamywiki 1.31.12
PHP 7.4.3
BlueSpice 3.2.0
Parsoid 0.10.0
Kerberos SSO is working fine. Now the problem is, that when editing an article with Visual Editor, the page turns white and when trying to save an "HTTP 500" error appears. In the syslog I have an "401 Unauthorized" Error. I have the following configs. Parsoid is running on the same server as my Apache Webserver and is accessible at port 8000 (via cli and curl and also via browser). Also, when creating a new section or page, the Visual Editor is working.
___________________________________________
Apache vhost:
<VirtualHost *:443>
ServerName mywiki.mydomain.com
ServerAlias mywiki
DocumentRoot /path/to/mediawiki
<Directory /path/to/mediawiki>
Options Indexes FollowSymLinks MultiViews
AllowOverride None
<RequireAny>
AuthType Kerberos
AuthName "Kerberos Login"
KrbAuthRealms mydomain.COM
KrbServiceName HTTP/serviceusr.mydomain.com
Krb5KeyTab /etc/apache2/kerberos/mykeytab.keytab
KrbLocalUserMapping On #Strips @REALM
KrbMethodNegotiate on
KrbMethodK5Passwd off
Require valid-user
Require ip 127.0.0.0/255.0.0.0
</RequireAny>
</Directory>
___________________________________________
LocalSettings.php:
$wgAuthRemoteuserUserName = function() {
global $wgDBname;
$user = '';
if( isset( $_SERVER[ 'REMOTE_USER' ] ) ) {
$user = $_SERVER[ 'REMOTE_USER' ] . '@mydomain.com';
}
if( isset( $_SERVER[ 'REMOTE_ADDR' ] ) && substr( $_SERVER[ 'REMOTE_ADDR' ], 0, 4 ) == '127.' ) {
if( empty( $user ) ) {
$user = $_COOKIE[$wgDBname.'304f3058RemoteToken'] . '@mydomain.com';
}
}
return $user;
};
___________________________________________
settings.d\020-VisualEditor.php:
// Creating base64 encoded path
$fullPath = $GLOBALS['wgServer'] . $GLOBALS['wgScriptPath'];
$encFullPath = base64_encode( $fullPath );
// Linking with Parsoid
$wgVirtualRestConfig['modules']['parsoid'] = array(
// URL to the Parsoid instance
// Use port 8142 if you use the Debian package
'url' => 'http...127.0.0.1:8000', // I wasn't allowed to post it with "://"
'domain' => $encFullPath,
'forwardCookies' => true
);
$wgVisualEditorEnablemywikitext = true;
___________________________________________
Parsoid config.yaml:
worker_heartbeat_timeout: 300000
logging:
level: info
services:
- module: lib/index.js
entrypoint: apiServiceWorker
conf:
localsettings: ./localsettings.js
___________________________________________
Parsoid localsettings.js:
'use strict';
exports.setup = function(parsoidConfig) {
parsoidConfig.dynamicConfig = function(domain) {
var baseUrl = Buffer.from( domain, 'base64').toString();
parsoidConfig.setMwApi({
uri: baseUrl + '/api.php',
domain: mydomain,
strictSSL: false
});
}
};
___________________________________________
Maybe I miss the obvious here, but I'm facing around with this issue for a few days now and i think it is time to ask the community for help ;) Wikweng (talk) 11:13, 26 March 2021 (UTC)
How to convert Wikitionary dump to html?
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
I've just found from this link that Parsoid which is a perfect tool to convert Wikitionary dump to html. I've downloaded the latest dump from here. However, I could not find any instruction to use Parsoid on this offline dump. Could you please elaborate on this issue?
Thank you so much for your help!
Dung Le. DungLe94 (talk) 21:03, 4 April 2021 (UTC)
- There hasn't been any effort to make Parsoid usable with those dumps.
- There are often questions about how to use Parsoid offline. See past discussions,
- https://lists.wikimedia.org/pipermail/wikitext-l/2020-February/000994.html
- https://lists.wikimedia.org/pipermail/wikitext-l/2020-April/000999.html
- https://www.mediawiki.org/wiki/Talk%3AParsoid/2018#h-The_most_basic_things-2018-09-13T17%3A08%3A00.000Z
- But so far Parsoid is mostly useful when it has access to a MediaWiki API to fetch configuration and resolve templates.
- If you wanted to use the source from the dump, you could do something like
cat "text from source" | php bin/parse.php --domain fr.wiktionary.org --wt2htmland that will output some html. Alternatively, you can use the titles from the dump and fetch the html from the REST API, https://fr.wiktionary.org/api/rest_v1/page/html/bonjour Arlolra (talk) 17:55, 6 April 2021 (UTC)
How to use thenets/parsoid in Docker in Windows 10?
[edit]I've installed thenets/parsoid on Docker on Windows 10. I want to convert the text file F:\zim\pomme.txt to html. I tried
docker run --name myparsoid -d -t -i -v /f/zim:/zim thenets/parsoid:latest sh
type /zim/pomme.txt | docker exec myparsoid php bin/parse.php --wt2html --offline
but it returns an error
Microsoft Windows [Version 10.0.19042.928]
(c) Microsoft Corporation. All rights reserved.
C:\Users\Akira>docker run --name myparsoid -d -t -i -v /f/zim:/zim thenets/parsoid:latest sh
7912b0cef8fba4244b2519f4f9603ec8e278b67bcc4fe08f4658721b98f941f3
C:\Users\Akira>type /zim/pomme.txt | docker exec myparsoid node bin/parse.js --wt2html --offline
The syntax of the command is incorrect.
internal/modules/cjs/loader.js:638
throw err;
^
Error: Cannot find module '/bin/parse.js'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)
Could you please shed some light on how to fix the error? DungLe94 (talk) 00:59, 14 April 2021 (UTC)
- The error you're getting is confusing since you're trying to invoke php and getting javascript errors. Arlolra (talk) 20:17, 1 November 2021 (UTC)
Can Parsoid fully work without Internet connection?
[edit]Assume that I have set up MediaWiki along with Parsoid properly. I don't import the `.xml.bz2` dump into MediaWiki. I also don't have Internet connection. Is it possible for Parsoid to convert the wikitext in the dump into a complete HTML, just as the Standard HTML REST API call from the wiki website does? DungLe94 (talk) 11:57, 2 May 2021 (UTC)
- If Parsoid has access to a local MediaWiki api, it can use that get the configuration it needs. However, if you don't import the templates, you won't get the expansion you expect. Arlolra (talk) 20:20, 1 November 2021 (UTC)
Installation section shall reflect current version
[edit]1.36.1 seems to be the last release so the comment about developers using 1.36 slightly seems to be out of date. Grin (talk) 17:25, 13 July 2021 (UTC)
- Thanks, feel free to update the documentation where it makes sense. We don't always have the resources to keep on top of it. Arlolra (talk) 20:23, 1 November 2021 (UTC)
parsoid offline mode
[edit]hi,
I see a offline option, I want to render the infoboxes offline, is there a way I can fetch the templates and then render the html without making calls to wikipedia/media wiki?
Thanks 2001:4998:EFFD:7804:0:0:0:104C (talk) 21:32, 12 August 2021 (UTC)
- The offline mode effectively turns off template fetching. At present, Parsoid relies on MediaWiki for template expansion. With some work, you can mock MediaWiki's action api or, Parsoid does have native template expansion used in parserTests, but nothing I can recommend out of the box. Arlolra (talk) 20:14, 1 November 2021 (UTC)
after upgrade to Debian 11: Error loading data from server: apierror-visualeditor-docserver-http-error
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
Hello
After upgrading from debian buster to bullseye, when the edit key is pressed this message shows:
Error loading data from server: apierror-visualeditor-docserver-http-error: ⧼apierror-visualeditor-docserver-http-error⧽. Would you like to retry?
edit source mode works.
version info:
Product Version
MediaWiki 1.34.4
PHP 7.4.21 (apache2handler)
MariaDB 10.5.11-MariaDB-1
ICU 67.1
/var/log/parsoid/parsoid.log :
{"name":"parsoid","hostname":"wiki-v134","pid":932,"level":30,"levelPath":"info/service-runner","msg":"master(932) initializing 2 workers","time":"2021-08-22T18:49:14.192Z","v":0}
{"name":"parsoid","hostname":"wiki-v134","pid":1028,"level":60,"code":"MODULE_NOT_FOUND","requireStack":["/usr/lib/parsoid/node_modules/service-runner/lib/base_service.js","/usr/lib/parsoid/node_modules/service-runner/lib/master.js","/usr/lib/parsoid/node_modules/service-runner/service-runner.js"],"moduleName":"../src/lib/index.js","levelPath":"fatal/service-runner/worker","msg":"Cannot find module '../src/lib/index.js'\nRequire stack:\n- /usr/lib/parsoid/node_modules/service-runner/lib/base_service.js\n- /usr/lib/parsoid/node_modules/service-runner/lib/master.js\n- /usr/lib/parsoid/node_modules/service-runner/service-runner.js","time":"2021-08-22T18:49:14.692Z","v":0}
{"name":"parsoid","hostname":"wiki-v134","pid":932,"level":40,"message":"first worker died during startup, continue startup","worker_pid":1028,"exit_code":1,"startup_attempt":1,"levelPath":"warn/service-runner/master","msg":"first worker died during startup, continue startup","time":"2021-08-22T18:49:15.701Z","v":0}
from the log I can not see which module is missing.
I would appreciate any advice or suggestions to fix this. RobFantini (talk) 20:57, 22 August 2021 (UTC)
- i found this: https://www.mediawiki.org/w/index.php?title=Talk%3AParsoid/2019#c-31.166.159.122-2019-12-01T20%3A33%3A00.000Z-210.195.7.193-2019-07-01T14%3A35%3A00.000Z
- after changing this in /etc/mediawiki/parsoid/config.yaml :
- old:
services:- module: ../src/lib/index.js- new:
services:- module: /usr/lib/parsoid/src/lib/index.js- and restarting parsoid the issue is fixed . RobFantini (talk) 23:00, 22 August 2021 (UTC)
How to limit size of error_log?
[edit]can i set to auto-purge old errors? or some other way to limit size?
If not, how to prevent any error logging?
My error log grew to 14 GB and maxed out my webhosting. Johnywhy (talk) 19:57, 28 August 2021 (UTC)
- Are the logs coming from Parsoid? Can you paste an example? Arlolra (talk) 20:09, 1 November 2021 (UTC)
Alternative value for $wgServer in Docker (Linux)
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
// While using Docker in macOS, you may need to replace $wgServer with http: //host.docker.internal:8080
'url' => $wgServer . $wgScriptPath . '/rest.php',
I found out that http: //container_name for $wgServer is also working.
In my specific case I am also using HTTP Auth globally for the wiki instance, so I'm using http: //user:password@container_name. 217.18.182.130 (talk) 10:50, 12 November 2021 (UTC)
Parsoid is not converting piped wiki links to html correctly
[edit]The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
Versions
[edit]- Mediawiki: 1.34.4
- Node.js: 8.11.11
- Parsoid: 0.11.0
- OS: CentOS 7.9.2009
Issue
[edit]I am currently using Parsoid to convert wikitext to html. However, when I have an inter-wiki link that includes a pipe such as [[Main Page|test 123]], Parsoid converts it to href="http://127.0.0.1:8080/wiki1/Main_Pagetest_123" title="Main Pagetest 123" (omitting some of the other html output for the link)
If anyone has any understand why this is happening or how to work around it, I'd really appreciate it! Ammcgillivray (talk) 12:56, 1 December 2021 (UTC)
- Odd. Is it actually a vertical bar or some other unicode character?
- Does it happen on the command line as well?
echo "test 123" | node bin/parse.jsArlolra (talk) 15:35, 1 December 2021 (UTC) - Oh my goodness, this is 100% user error. It does work from the command line. The reason it doesn't work from inside my program is because I was stripping pipe characters from the text being converted 🤦♀️
- False alarm. Thank you for your help and quick response time that allowed me to spot my own idiocy! Ammcgillivray (talk) 17:19, 1 December 2021 (UTC)