Extension talk:SyntaxHighlight

About this board

Previous discussion was archived at Extension talk:SyntaxHighlight/Archive 2017 on 2017-03-29.

1.38.1 - Pygments Initialization Error

5
158.48.133.2 (talkcontribs)

I have two instances of Mediawiki running on Windows Server 2016. One instance has been running for a few years and I am working on updating mediawiki from 1.35.1 (old) to 1.38.1 (new) in a separate instance to avoid breaking the production environment before it is functional. Example url on the same server.

  • server/wiki/index.php?title=Main_Page (1.35.1)
  • server/wiki1.38.1/index.php?title=Main_Page (1.38.1)

I've had this python extension installed and running for the current 1.35.1 instance for a long time. In the new 1.38.1 instance I have specified $wgPygmentizePath in the LocalSettings.php for Mediawiki as the instructions specify, pointing to the same location as the current 1.35.1 path. This code exists in both LocalSettings.php files.


#-----------------------------------------------------
# Pygments for SyntaxHighlight
$wgPygmentizePath = "E:/Program Files (x86)/Python/Scripts/pygmentize.exe";

I also tried below, and both paths seem to call pygmentize.exe and generate the error. If I comment out the $wgPygmentizePath the page loads fine but there is no syntax highlighting.


#-----------------------------------------------------
# Pygments for SyntaxHighlight
$wgPygmentizePath = "E:\\Program Files (x86)\\Python\\Scripts\\pygmentize.exe";

Each instance is pointed to its own MySQL database instance but other than that everything else is the same since these are both running on the same server. Same version of php, same version of python, same version of pygments, there is no special IIS application pool settings. So far the only difference should be the newly installed version of mediawiki (1.38.1). When I visit any page that calls <syntaxhighlight> I get this error now in 1.38.1... while for the same page in 1.35.1 pygmentize continues to execute and render fine.

[ba1074563ef8de0d563c6bb9] /wiki1.38.1/index.php/Automation_Logon_Script MediaWiki\SyntaxHighlight\PygmentsException: Fatal Python error: _Py_HashRandomization_Init: failed to get random numbers to initialize Python Python runtime state: preinitialized Backtrace: from E:\inetpub\wwwroot\wiki1.38.1\extensions\SyntaxHighlight_GeSHi\includes\Pygmentize.php(87) #0 E:\inetpub\wwwroot\wiki1.38.1\includes\libs\objectcache\BagOStuff.php(202): MediaWiki\SyntaxHighlight\Pygmentize::MediaWiki\SyntaxHighlight{closure}(integer) #1 E:\inetpub\wwwroot\wiki1.38.1\extensions\SyntaxHighlight_GeSHi\includes\Pygmentize.php(91): BagOStuff->getWithSetCallback(string, integer, Closure) #2 E:\inetpub\wwwroot\wiki1.38.1\extensions\SyntaxHighlight_GeSHi\includes\Pygmentize.php(156): MediaWiki\SyntaxHighlight\Pygmentize::getVersion() #3 E:\inetpub\wwwroot\wiki1.38.1\extensions\SyntaxHighlight_GeSHi\includes\SyntaxHighlight.php(58): MediaWiki\SyntaxHighlight\Pygmentize::getLexers() #4 E:\inetpub\wwwroot\wiki1.38.1\extensions\SyntaxHighlight_GeSHi\includes\SyntaxHighlight.php(190): SyntaxHighlight::getLexer(string) #5 E:\inetpub\wwwroot\wiki1.38.1\extensions\SyntaxHighlight_GeSHi\includes\SyntaxHighlight.php(315): SyntaxHighlight::highlightInner(string, string, array) #6 E:\inetpub\wwwroot\wiki1.38.1\extensions\SyntaxHighlight_GeSHi\includes\SyntaxHighlight.php(132): SyntaxHighlight::highlight(string, string, array, Parser) #7 E:\inetpub\wwwroot\wiki1.38.1\includes\parser\Parser.php(3979): SyntaxHighlight::parserHook(string, array, Parser, PPFrame_Hash) #8 E:\inetpub\wwwroot\wiki1.38.1\includes\parser\PPFrame_Hash.php(353): Parser->extensionSubstitution(array, PPFrame_Hash) #9 E:\inetpub\wwwroot\wiki1.38.1\includes\parser\Parser.php(2932): PPFrame_Hash->expand(PPNode_Hash_Tree, integer) #10 E:\inetpub\wwwroot\wiki1.38.1\includes\parser\Parser.php(1579): Parser->replaceVariables(string) #11 E:\inetpub\wwwroot\wiki1.38.1\includes\parser\Parser.php(697): Parser->internalParse(string) #12 E:\inetpub\wwwroot\wiki1.38.1\includes\content\WikitextContentHandler.php(294): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer) #13 E:\inetpub\wwwroot\wiki1.38.1\includes\content\ContentHandler.php(1705): WikitextContentHandler->fillParserOutput(WikitextContent, MediaWiki\Content\Renderer\ContentParseParams, ParserOutput) #14 E:\inetpub\wwwroot\wiki1.38.1\includes\content\Renderer\ContentRenderer.php(47): ContentHandler->getParserOutput(WikitextContent, MediaWiki\Content\Renderer\ContentParseParams) #15 E:\inetpub\wwwroot\wiki1.38.1\includes\Revision\RenderedRevision.php(267): MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(WikitextContent, Title, integer, ParserOptions, boolean) #16 E:\inetpub\wwwroot\wiki1.38.1\includes\Revision\RenderedRevision.php(238): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean) #17 E:\inetpub\wwwroot\wiki1.38.1\includes\Revision\RevisionRenderer.php(221): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array) #18 E:\inetpub\wwwroot\wiki1.38.1\includes\Revision\RevisionRenderer.php(158): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array) #19 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision{closure}(MediaWiki\Revision\RenderedRevision, array) #20 E:\inetpub\wwwroot\wiki1.38.1\includes\Revision\RenderedRevision.php(200): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array) #21 E:\inetpub\wwwroot\wiki1.38.1\includes\poolcounter\PoolWorkArticleView.php(137): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput() #22 E:\inetpub\wwwroot\wiki1.38.1\includes\poolcounter\PoolCounterWork.php(162): PoolWorkArticleView->doWork() #23 E:\inetpub\wwwroot\wiki1.38.1\includes\page\ParserOutputAccess.php(302): PoolCounterWork->execute() #24 E:\inetpub\wwwroot\wiki1.38.1\includes\page\Article.php(702): MediaWiki\Page\ParserOutputAccess->getParserOutput(WikiPage, ParserOptions, MediaWiki\Revision\RevisionStoreCacheRecord, integer) #25 E:\inetpub\wwwroot\wiki1.38.1\includes\page\Article.php(517): Article->generateContentOutput(User, ParserOptions, integer, OutputPage, array) #26 E:\inetpub\wwwroot\wiki1.38.1\includes\actions\ViewAction.php(80): Article->view() #27 E:\inetpub\wwwroot\wiki1.38.1\includes\MediaWiki.php(544): ViewAction->show() #28 E:\inetpub\wwwroot\wiki1.38.1\includes\MediaWiki.php(321): MediaWiki->performAction(Article, Title) #29 E:\inetpub\wwwroot\wiki1.38.1\includes\MediaWiki.php(910): MediaWiki->performRequest() #30 E:\inetpub\wwwroot\wiki1.38.1\includes\MediaWiki.php(564): MediaWiki->main() #31 E:\inetpub\wwwroot\wiki1.38.1\index.php(53): MediaWiki->run() #32 E:\inetpub\wwwroot\wiki1.38.1\index.php(46): wfIndexMain() #33 {main}

Appreciate any ideas, I am not sure where to look.

Tystnaden (talkcontribs)

What version of Python do you have installed on the system?

158.48.133.2 (talkcontribs)
  • python 3.9.1
  • pygments 2.12
  • php 7.4.14
  • mysql 5.7

Thanks,

Tystnaden (talkcontribs)

Is there some reason why you cannot use the Pygments which is bundled with the extension? I don't run windows, but I would expect the extension to work out of the box.

Tystnaden (talkcontribs)

Note that in MW 1.38.1 the extension has Pygments 2.11.2 bundled.

Reply to "1.38.1 - Pygments Initialization Error"
Novem Linguae (talkcontribs)

Is lang="regex" supported? Is there an alias for it that I'm missing? If not supported, what's the proper repo to request its addition?

Dinoguy1000 (talkcontribs)

Regex might be supported as part of Perl (that's the first thing I'd try after lang="regex", at least). If it's not supported at all (or you can't find any other language that supports it), the place to request support would be the Pygments project, after which you'd need to file a task on Wikimedia Phabricator to have the Pygments version used in this extension updated.

Stang (talkcontribs)
Dinoguy1000 (talkcontribs)

Yes, that's what I said.

PerfektesChaos (talkcontribs)

Well, the problem is that there is no unique RegExp syntax but a large variety of dialects. They all need different lexers since some elements are permitted in the one language but unknown and an error within the other context.

Reply to "RegEx"

"Failed to invoke Pygments"

8
Amire80 (talkcontribs)

Hi,

I've installed Scribunto and then SyntaxHighlight_GeSHi on my local MediaWiki installation. All the extensions are the latest versions from Gerrit, and I'm using Ubuntu and MediaWiki-Docker.

I created a Scribunto module page. When I view, I see this error message at the top:

Notice: Failed to invoke Pygments: [Called from MediaWiki\SyntaxHighlight\SyntaxHighlight::highlightInner in /var/www/html/w/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php at line 293] in /var/www/html/w/includes/debug/MWDebug.php on line 507

"pygmentize" appears to have the necessary permissions: -rwxrwxr-x  1 amir amir 1063938 May 25 15:15 pygmentize

When I simply use the "syntaxhighlight" tag on a page in the main space, the text is set in a monospace font, but without actual highlighting, and the page is auto-added to the "Pages with syntax highlighting errors" category.

I tried the "Troubleshooting" section here, but couldn't find something that will help. What could be wrong?

Tystnaden (talkcontribs)

Do you have the Python3 executable 'python3' installed on the server and located on the PATH of the PHP interpreter?

Tystnaden (talkcontribs)
Amire80 (talkcontribs)

Oh, that's probably the problem. The docker image doesn't have Python. The solution is supposed to be something like MediaWiki-Docker/Extension/SyntaxHighlight, but it doesn't seem to work. When I run docker-compose build, towards the end I get this:

Status: Downloaded newer image for docker-registry.wikimedia.org/dev/stretch-php72-fpm:2.0.0
 ---> 7f47f1dab953
Step 2/2 : RUN apt update && apt install -y python3
 ---> Running in 5b103ed36b7b

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Get:1 http://security.debian.org stretch/updates InRelease [53.0 kB]
Ign:2 http://mirrors.wikimedia.org/debian stretch InRelease
Get:3 http://security.debian.org stretch/updates/main amd64 Packages [977 kB]
Ign:4 https://apt.wikimedia.org/wikimedia stretch-wikimedia InRelease
Get:5 http://mirrors.wikimedia.org/debian stretch-updates InRelease [93.6 kB]
Get:6 http://mirrors.wikimedia.org/debian stretch-backports InRelease [91.8 kB]
Err:7 https://apt.wikimedia.org/wikimedia stretch-wikimedia Release
  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Get:8 http://mirrors.wikimedia.org/debian stretch Release [118 kB]
Get:9 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages [790 kB]
Get:10 http://mirrors.wikimedia.org/debian stretch-backports/contrib amd64 Packages [12.0 kB]
Get:11 http://mirrors.wikimedia.org/debian stretch Release.gpg [3177 B]
Get:12 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages [9610 kB]
Reading package lists...
E: The repository 'https://apt.wikimedia.org/wikimedia stretch-wikimedia Release' does not have a Release file.
The command '/bin/sh -c apt update && apt install -y python3' returned a non-zero code: 100
ERROR: Service 'mediawiki' failed to build : Build failed

The instructions at MediaWiki-Docker/Extension/SyntaxHighlight probably need an update, but I know practically nothing about Docker and advanced Debian, so I can't do it myself. Any help?

Tystnaden (talkcontribs)

Sorry, you said that you know "practically nothing" about Docker, and I know zero about it, or about installing MediaWiki with it. I have my test wiki running natively on a machine.

Tystnaden (talkcontribs)

Seems to me that they really need to add python3 to the Docker image so that it works out of the box.

Tacsipacsi (talkcontribs)

First, Debian Stretch is no longer supported on Wikimedia (when this page was created a year ago, it was still supported), so I think (but haven’t tried it out) that the dockerfile should rather be

FROM docker-registry.wikimedia.org/dev/buster-php72-fpm:2.0.0
RUN apt-get update && apt-get install -y python3

(Or buster-php72-fpm:2.0.0-s1, I don’t know which one is the newer. The second line’s change is just to silence APT’s warnings about the stable CLI interface.) APT’s HTTPS support has been steadily improving in the last years, so if you’re lucky, switching to Buster will automatically fix the actual issue (server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none).

Amire80 (talkcontribs)

Thanks @Tacsipacsi, this worked! It also required a reboot of docker—python3 appeared not installed at first, but after the reboot it worked.

Reply to ""Failed to invoke Pygments""
Nicolas senechal (talkcontribs)

Hello, I have multiple bugs with this extension, but I don’t know what to search on the internet or what did wrong…

The first time I use this extension everything goes well for the second time it goes wrong (per part of code).

So the first bug it's with visualEditor: when I install it for the first time it works but now the edit following the cursor (so it's very difficult to modify some code).

The second it’s syntax highlight with sql the first time I save it and everything works well. But I edit the code again and the highlight disappears… so I try with sources and the same first time it works second time it disappears…

It is really weird that the first time it works and seconde the bug appears and it still here. I try to change the skin and the bug is still here.

Here is my configuration of this extension.

wfLoadExtension('SyntaxHighlight_GeSHi');

$wgPygmentizePath = "c:\\Python27\\Scripts\\pygmentize.exe";

I install pygmentize with easy_install. Wikimedia 1.37.1 running on xampp 8.1.2 in localhost with a window 10.

Any help is apprised thank you!

Want (talkcontribs)

Sorry, but using WYSIWYG for editation of the wikicode is the pure evil.

Reply to "Some UI bug I have"

Is it possible to create links inside <syntaxhighlight ></syntaxhighlight > tags?

3
194.44.177.25 (talkcontribs)

Sometimes I need to create internal links from inside the <syntaxhighlight ></syntaxhighlight > tags. Is it possible and, if yes, than how can it be done?

Shirayuki (talkcontribs)
Want (talkcontribs)

Yes. Solution exists. I use it on my help page for example code, but in wiki must be installed Extension:Variables.

  1. First I set {{#vardefine:temp|10 000,-}} value of the variable
  2. Next use for view exemple code {{#tag:syntaxhighlight|{{#var:temp}}|lang=haskell}}
  3. And finally I use {{#var:temp}} for the interpretation
Reply to "Is it possible to create links inside <syntaxhighlight ></syntaxhighlight > tags?"

Is it possible to use HTML tags inside the <syntaxhighlight></syntaxhighlight> tags?

7
MrJConstantine (talkcontribs)

I'm dealing with a programming language that's based on Delphi but has some differences.

There are some keywords that do not exist in Delphi, so they won't be colored and highlighted the proper way.

Is it somehow possible to use the HTML tags like <b> or <i> inside the <syntaxhighlight lang=delphi line></syntaxhighlight> blocks?

I would appreciate the help.

ExE Boss (talkcontribs)

No, because <syntaxhighlight> is a <nowiki>‑like tag.

PerfektesChaos (talkcontribs)

You would need to extend the upstream variety of offered languages. However, perhaps there is already another Delphi derivative which will fit your needs?

MrJConstantine (talkcontribs)

Actually, no. It's our internal system and it is not officially available anywhere except our company.

Dinoguy1000 (talkcontribs)

It's possible to work around this by using #tag, e.g. {{ #tag: syntaxhighlight | sample code here with a bit of <b>bolding</b> |lang=delphi |line }}, but it's likely to be pretty finicky (and you're not likely to get much help working around that finickiness). PerfektesChaos' suggestions would be better.

MrJConstantine (talkcontribs)

I've tried it like this but it doesn't work for me.

I get the following result:

<b>componentlist</b>
Tacsipacsi (talkcontribs)

Pygments (the library which does the actual highlighting) is open source, so you can write your own lexer based on the Delphi/Pascal one. I haven’t tried doing so, so I don’t know how complicated it is (especially installing the modified Pygments), but it should be possible.

Reply to "Is it possible to use HTML tags inside the <syntaxhighlight></syntaxhighlight> tags?"

Failed to invoke Pygments: Error clone: main.c:2608 main: Operation not permitted [Called from SyntaxHighlight::highlightInner

10
Lwangaman (talkcontribs)

With MediaWiki 1.35.3, I was able to get the SyntaxHighlight extension to work by using an external pygmentize:

$wgPygmentizePath = '/usr/local/bin/pygmentize';

However after upgrading to MediaWiki 1.36.2, pygmentize is no longer working correctly. If I turn on error reporting:

error_reporting(E_ALL);
ini_set("display_errors", 1);

$wgShowExceptionDetails = true;
$wgShowDBErrorBacktrace = false;
$wgShowSQLErrors = true;
$wgResourceLoaderDebug = false;
$wgDebugToolbar = true;
$wgDevelopmentWarnings = true;
$wgShowDebug = true;

I then get these errors:

Notice: Failed to invoke Pygments: Error clone: main.c:2608 main: Operation not permitted [Called from SyntaxHighlight::highlightInner in /path/to/mediawiki_1_36/w/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php at line 316] in /path/to/mediawiki_1_36/w/includes/debug/MWDebug.php on line 499

Same or similiar problem for lilypond:

Unable to obtain LilyPond version:
Error clone: main.c:2608 main: Operation not permitted

In the debug data I see the following information:

[exec] Executing: '/usr/bin/firejail' '--quiet' '--profile=/path/to/mediawiki_1_36/w/includes/shell/firejail.profile' '--blacklist=/path/to/mediawiki_1_36/w/LocalSettings.php' '--noroot' '--seccomp' '--private-dev' '--net=none' -- /bin/bash '/path/to/mediawiki_1_36/w/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/local/bin/pygmentize'\'' '\''-l'\'' '\''js'\'' '\''-f'\'' '\''html'\'' '\''-O'\'' '\''cssclass=mw-highlight,encoding=utf-8,linenos=inline'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=180; SB_CGROUP='\'''\''; SB_MEM_LIMIT=314572800; SB_FILE_SIZE_LIMIT=104857600; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes'
[exec] Error running '/usr/bin/firejail' '--quiet' '--profile=/path/to/mediawiki_1_36/w/includes/shell/firejail.profile' '--blacklist=/path/to/mediawiki_1_36/w/LocalSettings.php' '--noroot' '--seccomp' '--private-dev' '--net=none' -- /bin/bash '/path/to/mediawiki_1_36/w/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/local/bin/pygmentize'\'' '\''-l'\'' '\''js'\'' '\''-f'\'' '\''html'\'' '\''-O'\'' '\''cssclass=mw-highlight,encoding=utf-8,linenos=inline'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=180; SB_CGROUP='\'''\''; SB_MEM_LIMIT=314572800; SB_FILE_SIZE_LIMIT=104857600; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes': Error clone: main.c:2608 main: Operation not permitted
[objectcache] fetchOrRegenerate(global:highlight:f3eb16b7c32d4a36f595352942cb4e1b): miss, new value computed
[error] [YWLA1o0mAVG7arGqGzx1GAAAAA4] /wiki/User_talk:Johnrdorazio ErrorException: PHP Notice: Failed to invoke Pygments: Error clone: main.c:2608 main: Operation not permitted
[Called from SyntaxHighlight::highlightInner in /path/to/mediawiki_1_36/w/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php at line 316]
[warning] Failed to invoke Pygments: Error clone: main.c:2608 main: Operation not permitted
[Called from SyntaxHighlight::highlightInner in /path/to/mediawiki_1_36/w/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php at line 316]
Lwangaman (talkcontribs)

Does MediaWiki 1.36 require the usage of a local shellbox proxy? Or is that optional? I thought I would try adapting the instructions for setting up MediaWiki Shellbox to work in my Plesk setup, but I haven't been successful yet, so I've kept these two lines commented out:

//$wgShellboxUrl = 'http://shellbox.internal/shellbox';
//$wgShellboxSecretKey = 'random_generated_key_here';

But I still can't quite figure out if this is required by MediaWiki 1.36 or simply supported by MediaWiki 1.36.

Lwangaman (talkcontribs)

Running

/usr/local/bin/pygmentize -l sql -f html -O cssclass=mw-highlight,encoding=utf-8 < /dev/null

on the command line outputs:

<div class="mw-highlight"><pre><span></span>
</pre></div>

Same goes for

/path/to/mediawiki_1_36/w/extensions/SyntaxHighlight_GeSHi/pygments/pygmentize -l sql -f html -O cssclass=mw-highlight,encoding=utf-8 < /dev/null

. So they seem to work from the command line. Seems to me the Shell::Command is not working correctly to invoke pygmentize, or lilypond, or perhaps any shell command?

Lwangaman (talkcontribs)

I'm running MediaWiki/Shellbox 1.0.4, so I don't believe that Phabricator T274474 is the issue.

To further test this, I am attempting to run the same pygmentize under firejail from the command line. I'm explicitly whitelisting limit.sh when running from the command line, and this kind of seems to work:

firejail --profile=/path/to/mediawiki_1_36/w/includes/shell/firejail.profile --blacklist=/path/to/mediawiki_1_36/w/LocalSettings.php --whitelist=/path/to/mediawiki_1_36/w/vendor/wikimedia/shellbox/src/Command/limit.sh --noroot --seccomp --private-dev --net=none -- /bin/bash /path/to/mediawiki_1_36/w/vendor/wikimedia/shellbox/src/Command/limit.sh '/usr/local/bin/pygmentize -l sql -f html -O cssclass=mw-highlight,encoding=utf-8,linenos=inline < /dev/null' SB_INCLUDE_STDERR=; SB_CPU_LIMIT=180; SB_CGROUP=''; SB_MEM_LIMIT=314572800; SB_FILE_SIZE_LIMIT=104857600; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes

This produces:

Reading profile /path/to/mediawiki_1_36/w/includes/shell/firejail.profile
Parent pid 3252769, child pid 3252770
Child process initialized in 24.94 ms
<div class="mw-highlight"><pre><span></span><span class="linenos">1</span>
</pre></div>

Parent is shutting down, bye...

However it only works if I add that " < /dev/null". Without the "< /dev/null" the process just hangs. Is it possible that this is missing from MediaWiki Shell Command?

Lwangaman (talkcontribs)

I tried editing wikimedia/shellbox/src/Command/BashWrapper.php like this:

        if ( $time > 0 || $mem > 0 || $filesize > 0 || $wallTime > 0 ) {
            $cmd = '/bin/bash ' . escapeshellarg( __DIR__ . '/limit.sh' ) . ' ' .
                escapeshellarg( $command->getCommandString() . ' < /dev/null' ) . ' ' .
                escapeshellarg(
                    "SB_INCLUDE_STDERR=" . ( $command->getIncludeStderr() ? '1' : '' ) . ';' .
                    "SB_CPU_LIMIT=$time; " .
                    'SB_CGROUP=' . escapeshellarg( $this->cgroup ) . '; ' .
                    "SB_MEM_LIMIT=$mem; " .
                    "SB_FILE_SIZE_LIMIT=$filesize; " .
                    "SB_WALL_CLOCK_LIMIT=$wallTime; " .
                    "SB_USE_LOG_PIPE=yes"
                );
            $command->unsafeCommand( $cmd )
                ->useLogPipe();
            if ( $command->getAllowedPaths() ) {
                // If specific paths have been allowed, make sure we explicitly
                // allow limit.sh. We don't do this unconditionally because it
                // doesn't work as expected in firejail, see T274474, T182486
                $command->allowPath( __DIR__ . '/limit.sh' );
            }
        }

Notice the extra . ' < /dev/null' after $command->getCommandString(). However this still doesn't help much, I'm getting the same error as before:

[exec] Executing: '/usr/bin/firejail' '--quiet' '--profile=/path/to/mediawiki_1_36/w/includes/shell/firejail.profile' '--blacklist=/path/to/mediawiki_1_36/w/LocalSettings.php' '--noroot' '--seccomp' '--private-dev' '--net=none' -- /bin/bash '/path/to/mediawiki_1_36/w/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/local/bin/pygmentize'\'' '\''-l'\'' '\''js'\'' '\''-f'\'' '\''html'\'' '\''-O'\'' '\''cssclass=mw-highlight,encoding=utf-8,linenos=inline'\'' < /dev/null' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=180; SB_CGROUP='\'''\''; SB_MEM_LIMIT=314572800; SB_FILE_SIZE_LIMIT=104857600; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes'
[exec] Error running '/usr/bin/firejail' '--quiet' '--profile=/path/to/mediawiki_1_36/w/includes/shell/firejail.profile' '--blacklist=/path/to/mediawiki_1_36/w/LocalSettings.php' '--noroot' '--seccomp' '--private-dev' '--net=none' -- /bin/bash '/path/to/mediawiki_1_36/w/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/local/bin/pygmentize'\'' '\''-l'\'' '\''js'\'' '\''-f'\'' '\''html'\'' '\''-O'\'' '\''cssclass=mw-highlight,encoding=utf-8,linenos=inline'\'' < /dev/null' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=180; SB_CGROUP='\'''\''; SB_MEM_LIMIT=314572800; SB_FILE_SIZE_LIMIT=104857600; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes': Error clone: main.c:2608 main: Operation not permitted
Lwangaman (talkcontribs)

I see others have had a similar issue when running in a docker container (see https://github.com/netblue30/firejail/issues/1856), however I'm not running in a docker container. To make sure permissions are not an issue, I did a 'sudo su - mediawikiApacheUser' and then issued the same pygmentize under firejail from the command line, this time as the mediawiki apache user, and got the same result as above without any permission errors.

Lwangaman (talkcontribs)

I see someone else mentioned something about an old kernel being the issue, with missing NAMESPACE support.

My current kernel, on Ubuntu 20.04, is 5.4.0-73-generic. If I cd /usr/src/linux-headers-5.4.0-73-generic and egrep '_NS' .config, I seem to have all of the necessary NAMESPACE support:

CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NET_NSH=m
CONFIG_PATA_NS87415=m
CONFIG_PATA_NS87410=m
CONFIG_NS83820=m
CONFIG_GAMEPORT_NS558=m
CONFIG_TCG_NSC=m

So that doesn't seem to me to be the issue...

Tacsipacsi (talkcontribs)

(edit conflict)×3

If you don’t add < /dev/null, the process tries to read from the standard input, i.e. it waits for you to type something. Press Ctrl+D to tell it you’re done with typing.

I’m not really familiar with Firejail, but doesn’t its behavior depend on the user executing it? By using

sudo -u www-data firejail ...

instead of

firejail ...

you should become closer to what actually happens when MediaWiki tries to call Pygmentize (assuming you use Apache, different web servers may use different users).

Lwangaman (talkcontribs)

In fact I did give that a try, as mentioned here. Seems to be working ok, without any particular errors... So I'm not sure why the MediaWiki Shellbox::Command is producing Error clone: main.c:2608 main: Operation not permitted.

Lwangaman (talkcontribs)

For the time being I have just set $wgShellRestrictionMethod to false, rather than to 'firejail'. At least Lilypond and Pygments are working again. It would be nice for it to work with firejail...

Reply to "Failed to invoke Pygments: Error clone: main.c:2608 main: Operation not permitted [Called from SyntaxHighlight::highlightInner"
Pneuma01 (talkcontribs)

Excuse me. On my local MediaWIki, when I install SyntaxHighlighter, I get an error message when displaying Gadget:

Notice: fwrite(): write of 7804 bytes failed with errno=32 Broken pipe in D:\mediawiki-1.36.0\vendor\wikimedia\shellbox\src\Command\UnboxedExecutor.php on line 315

When I save this script, the gadget does not work, but it does work if I remove SyntaxHighlighter from LocalSettings.php beforehand and save it.

Can anyone tell me what is causing this?

63.245.160.132 (talkcontribs)

I'm getting the same error on my new install of 1.36.0

Seems to be looking for the OS type, but not sure how to fix it.

143.196.163.1 (talkcontribs)

I "fix" the problem with this code on line 315 : $res = @fwrite( $pipe, $buffers[$fd], 65536 );

Lwangaman (talkcontribs)

I don't think that really fixes anything, that just hides/silences the error message? But the error is still there. I am getting the same error now on 1.36.2 after upgrading from 1.35.3.

Reply to "Linter Error on 1.36.0 ?"

how to batch replace <source> with <syntaxhighlight>

4
Summary by Ernstkm

There are several options: 1) use a (semi)automated editing tool such as AutoWikiBrowser or Pywikibot; 2) Extension:ReplaceText; or 3) use a tool such as sed to edit the raw article text of a SQL database dump, then re-import from that SQL dump. The first two can be done on a live site; the third would require special care (such as setting $wgReadOnly) to ensure no user edits are lost in the interim.

Hawk914 (talkcontribs)

Hi, I am new to mediawiki. But I have to upgrade an old mediawiki that contains lots of page with source tags. Since it's deprecated by syntaxhighlight tag, how do I replace all source tags in all pages with syntaxhighlight tag in a batch way?

Dinoguy1000 (talkcontribs)

The way I'd do it is via AutoWikiBrowser (AWB) or a similar (semi)automated editing tool (if you're handy with a programming language and don't mind spending time on it, you could also write such a tool yourself). If you can install extensions yourself, or can request to have them installed, another option is Extension:ReplaceText , though personally I don't recommend this versus running AWB or similar.

2001:A61:BDD:901:C911:EA05:A98:C376 (talkcontribs)

Another possibility for linux is the pywikibot replace script (see Manual:Pywikibot/replace.py) with the following user-fix.py entry:

fixes['source_tag'] = {
    'regex': False,
    'msg': {
        '_default': 'source-tag ersetzt durch syntaxhighlight',
    },
    'replacements': [
        (r'<source', '<syntaxhighlight'),
        (r'</source', '</syntaxhighlight'),
    ]
}

called like this:

python3 pwb.py replace -fix:source_tag -start:!
Mustakioren (talkcontribs)

that worked for me

cat wiki.sql | sed 's/<source/<syntaxhighlight/g' | sed 's/<\/source/<\/syntaxhighlight/g' > wiki.new.sql

make sure to backup and check thoroughly after importing back

Reply to "how to batch replace <source> with <syntaxhighlight>"

The code field is a keyboard trap

5
MLRodrigue (talkcontribs)

I noticed that it is impossible to get out of the code entry field when you have to rely on keyboard navigation. (Mediawiki 1.31). Would it be hard to fix this?

Tacsipacsi (talkcontribs)

The issue is that the tab key is used to indent/outdent the code. I don’t think it would be hard to change this behavior to navigating between form fields, but then there was no way to change the indentation (except for copy-pasting the tabulator character or entering it by ASCII/Unicode codepoint, neither of them is convenient). So there’s no good solution for this; the developers seem to have chosen a solution that’s better for those who prefer indenting their code with tabs but worse for those who rely on keyboard navigation.

Dinoguy1000 (talkcontribs)

Could Shift+Tab be adapted for this purpose? AFAIK it has no special function within the field, and it also doesn't in the browser or OS (Ctrl+Tab switches between open tabs in Firefox and (IIRC) Chrome, and Alt+Tab switches between open program windows in Windows).

Tacsipacsi (talkcontribs)

It has: that’s the outdent. (Shift, as usual, reverses the direction: here it’s outdent instead of indent, if it wasn’t overridden, it would go to the previous form field instead of the next one, Ctrl+Shift+Tab goes to the previous tab instead of the next one, Alt+Shift+Tab goes to the least recently used window instead of the most recently used one, and so on.)

Dinoguy1000 (talkcontribs)

Aah, right, I always forget about outdenting for some reason. So much for that idea, then.

Reply to "The code field is a keyboard trap"