Extension talk:SyntaxHighlight

• Extension talk:SyntaxHighlight/REL1.25 (LTS)

Previous discussion was archived at Extension talk:SyntaxHighlight/Archive 2017 on 2017-03-29.

I want to have a template that contains precisely the source file to be displayed, and to drop it into another page syntax-highlighted.

Conceptually, what I want is something like

<syntaxhighlight> {{msgnw:MySourceFile}} </syntaxhighlight>

but of course that doesn't expand the template.

{{Pre|{{msgnw:MySourceFile}}}} sort of works, but it's not quite the same formatting as <syntaxhighlight>.

Is there a way?

If you haven't already, try {{ #tag: syntaxhighlight | {{MySourceFile}} |lang="lang" }}.

Thanks but, alas, turns a semicolon into &#59;.

Precisely what I tried - itself inside a template - is

{{ #tag: syntaxhighlight | {{msgnw:User:Jordan Brown/sandbox/{{{name}}}}}|lang="C" }}

where the page pointed at contains cube(10);.

But I didn't know anything about #tag, so I'll look into it further.

You might have luck without the msgnw (I haven't tested this myself):

{{ #tag: syntaxhighlight | {{:User:Jordan Brown/sandbox/{{{name}}}}}|lang="C" }}

No, didn't work. With

{{ #tag: syntaxhighlight | {{{{BOOKNAME}}/examples/{{{name}}}}}|lang="C"|line="line" }}

and the input being (a deliberate torture test)

cube(10);|| foo && bar {{baz}} <pre></pre> &lt;
x < 5 && y > 6

the result was

cube(10);|| foo && bar [[:Template:Baz]] '"`UNIQ--pre-00000000-QINU`"' <
x < 5 && y > 6

Thanks for your help. Other editors on the book have looked at it and decided that my idea of keeping the examples in separate pages would be too awkward, so I'm abandoning the effort for now.

If you see `UNIQ--pre-*-QINU` stripped tags, this is because the transcluded content was parsed by MediaWiki. The only safe way to transclude a content not written in the Mediawiki syntax and that must not be parsed by it, is to use the "msgnw:" prefix before the transcluded full pagename, then you can put it inside a "#tag:" which allows the raw transclusion (via "msgnw:") or the parsed transclusion (without "msgnw:") to work.

However, it should be possible to use a simpler approach with <syntaxhighlight page="fullpagename" />, without even having to set its lang="*" attribute, because every page has metadata specifying its content-type (aka "page content model"): the syntaxhighlight extension should know every supported "content model" of MediaWiki, and notably MediaWiki, JSON, CSS, Javascript, XML, Lua, config/INI files, CSV files (the variant defined by the most common RFC using ASCII commas as field delimiters, ASCII double-quotes for escapes, and numbers in English format using dots and not commas, or otherwise escaping numbers within quotes, and non-mandatory quotation marks surrounding fields, and whitespaces between fields not significant, but where newlines may be present within fields escaped by quotation marks within which whitespaces are significant), and plain text, that should all be mappable to a language code/name supported by a Pygment parser.

Is lang="regex" supported? Is there an alias for it that I'm missing? If not supported, what's the proper repo to request its addition?

Regex might be supported as part of Perl (that's the first thing I'd try after lang="regex", at least). If it's not supported at all (or you can't find any other language that supports it), the place to request support would be the Pygments project, after which you'd need to file a task on Wikimedia Phabricator to have the Pygments version used in this extension updated.

No; Request should be left at upstream.

Yes, that's what I said.

Well, the problem is that there is no unique RegExp syntax but a large variety of dialects. They all need different lexers since some elements are permitted in the one language but unknown and an error within the other context.

Since the transition to pygments, there's no longer any support for any regular expression syntax, notably lang="pcre" (which was commonly used in various talk pages, notably on Wikidata which uses PCRE extensively), or the legacy lang="regexp". All these talk pages are now listed in a tracking category for all kinds of errors produced by "syntaxhighlight" (without distinguishing any tracking category for languages indicated in the "lang" attribute" but that are not (or no longer) recognized. PCRE regular expressions are not complex to parse, but this should recognize all features of PCRE 5+, including multiline expression with whitespace compression, comments (important in talk pages), and distinctive coloring for inners of character classes, open/close brackets, operators, escapes, defined label names, and a few keywords in special constructs. The "regexp" language is just a subset (without multiline/comments and a much more limited syntax). As well we should have a recognizer for Lua patterns (lang="luapattern"?) which are much more restricted (notably no support for repetitions in a limited range, or for '|' alternates, or defined subpatterns, or extended character classes based on POSIX or Unicode character properties).

As well, before we could use lang="HTML5", which was working for colorizing most the Mediawiki wiki syntax (but not lang="html"). Now we have to use lang="html" (but not lang="html5", which is not recognized). There's an evident need to create some common aliases.

Running MediaWiki 1.38.4 on

• Windows Server 2019
• IIS 10
• PHP 8.1.13
• Python 3.11
• Pygmentize 2.13.0

For unknown reasons some lexers work while other don't and instead generate the "Pages with syntax highlighting errors"-category. The page below, as an example, generates the powershell and bash code correctly, but not the yaml.

If I swap the yaml for bash without changing the block of code below, it does not generate an "Pages with syntax highlighting" category of it, and of course it doesn't highlight it correctly either as it is trying to bash highlight a yaml snippet.

<syntaxhighlight lang="powershell">
# This is a PowerShell script
Write-Host "Hello world"
</syntaxhighlight>

<syntaxhighlight lang="bash">
# This is a shell script in bash
ls -l /var/lib
</syntaxhighlight>

<syntaxhighlight lang="yaml">
# This is a yaml block
- martin:
    job: Developer
</syntaxhighlight>

I did not try all lexers supported by pygmentize. But it seems some work, some not. powershell and bash works every time, yaml and text fails everytime. It doesn't matter what order the blocks are on the page, they can be alone and it still fails. I checked the file SyntaxHighlight.lexers.php and all the lexers that pygmentize support are there, marked as true.

This is what debug output looks for the above page, relevant section. Note it only runs pygmentize four times. First time with version check, second with lexers check, third and fourth is the syntax highlight blocks that work.

[DBQuery] MediaWiki\Storage\SqlBlobStore::fetchBlobs [0s] 127.0.0.1: SELECT old_id,old_text,old_flags FROM `text` WHERE old_id = 1822
[objectcache] fetchOrRegenerate(global:SqlBlobStore-blob:hkr_mediawiki:tt%3A1822): miss, new value computed
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b8cf480f6bf3322
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-V"" 2>&1
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b8cf480f6bf3322/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b8cf480f6bf3322/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b8cf480f6bf3322"
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-86fd2e5bba940d09
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-L" "lexer"" 2>&1
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-86fd2e5bba940d09/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-86fd2e5bba940d09/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-86fd2e5bba940d09"
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-237f871e31353545
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-l" "powershell" "-f" "html" "-O" "cssclass=mw-highlight,encoding=utf-8" "file""
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-237f871e31353545/file"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-237f871e31353545/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-237f871e31353545/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-237f871e31353545"
[objectcache] fetchOrRegenerate(global:highlight:1e04b7e8e2e671872099e468f594fa58): miss, new value computed
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-5a0641ea93a9910c
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-l" "bash" "-f" "html" "-O" "cssclass=mw-highlight,encoding=utf-8" "file""
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-5a0641ea93a9910c/file"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-5a0641ea93a9910c/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-5a0641ea93a9910c/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-5a0641ea93a9910c"
[objectcache] fetchOrRegenerate(global:highlight:79a0bec1f4c29b0a4e1f837a4b70f4a9): miss, new value computed
[DBConnection] Wikimedia\Rdbms\LoadBalancer::getLocalConnection: reused a connection for local/0
[DBQuery] LCStoreDB::get [0s] 127.0.0.1: SELECT lc_value FROM `l10n_cache` WHERE lc_lang = 'sv' AND lc_key = 'messages:syntaxhighlight-error-category' LIMIT 1																									
[DBConnection] Wikimedia\Rdbms\LoadBalancer::getLocalConnection: reused a connection for local/0
[DBQuery] LCStoreDB::get [0s] 127.0.0.1: SELECT lc_value FROM `l10n_cache` WHERE lc_lang = 'sv' AND lc_key = 'linkPrefixExtension' LIMIT 1
[DBConnection] Wikimedia\Rdbms\LoadBalancer::getLocalConnection: reused a connection for localAutoCommit/0

When swapping the yaml for bash it processes correctly and doesn't throw errors and doesn't get the "Pages with syntax highlighting errors". It now runs pygmentize five times, with the last working bash-block.

[DBQuery] MediaWiki\Storage\SqlBlobStore::fetchBlobs [0s] 127.0.0.1: SELECT old_id,old_text,old_flags FROM `text` WHERE old_id = 1823
[objectcache] fetchOrRegenerate(global:SqlBlobStore-blob:hkr_mediawiki:tt%3A1823): miss, new value computed
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-c3bb7d17062047d8
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-V"" 2>&1
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-c3bb7d17062047d8/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-c3bb7d17062047d8/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-c3bb7d17062047d8"
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0484336c3f645690
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-L" "lexer"" 2>&1
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0484336c3f645690/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0484336c3f645690/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0484336c3f645690"
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b05e1edd661e368
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-l" "powershell" "-f" "html" "-O" "cssclass=mw-highlight,encoding=utf-8" "file""
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b05e1edd661e368/file"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b05e1edd661e368/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b05e1edd661e368/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-0b05e1edd661e368"
[objectcache] fetchOrRegenerate(global:highlight:1e04b7e8e2e671872099e468f594fa58): miss, new value computed
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-d08c021bfe1b0311
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-l" "bash" "-f" "html" "-O" "cssclass=mw-highlight,encoding=utf-8" "file""
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-d08c021bfe1b0311/file"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-d08c021bfe1b0311/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-d08c021bfe1b0311/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-d08c021bfe1b0311"
[objectcache] fetchOrRegenerate(global:highlight:79a0bec1f4c29b0a4e1f837a4b70f4a9): miss, new value computed
[exec] Creating base path C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-90a3aca3d2df8f1f
[exec] Executing: cmd /s /c ""c:\Progra~1\Python311\Scripts\pygmentize.exe" "-l" "bash" "-f" "html" "-O" "cssclass=mw-highlight,encoding=utf-8,linenos=inline,hl_lines=4" "file""
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-90a3aca3d2df8f1f/file"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-90a3aca3d2df8f1f/sb-stderr"
[exec] Removed file "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-90a3aca3d2df8f1f/sb-stdout"
[exec] Removed directory "C:\WINDOWS\TEMP\mwtmp-IUSR/shellbox-90a3aca3d2df8f1f"
[objectcache] fetchOrRegenerate(global:highlight:3ad02a4ec124ea3009c02acf078da02e): miss, new value computed
[DBConnection] Wikimedia\Rdbms\LoadBalancer::getLocalConnection: reused a connection for local/0
[DBQuery] LCStoreDB::get [0s] 127.0.0.1: SELECT lc_value FROM `l10n_cache` WHERE lc_lang = 'sv' AND lc_key = 'linkPrefixExtension' LIMIT 1
[DBConnection] Wikimedia\Rdbms\LoadBalancer::getLocalConnection: reused a connection for localAutoCommit/0

I'm so out of ideas.

What do you think?

Upgrading to MediaWiki 1.39.4 appears to have fixed this issue. All lexers work in this version.

The problem still persisted in MediaWiki 1.39.2 but an upgrade to 1.39.4 fixed it.

I was experiencing an issue on Windows where the lexer list was being incorrectly parsed, identifying language names with a colon at the end, i.e. as 'sql:', 'c++:', etc. My setup was as follows:

Product	Version
MediaWiki	1.39.3
PHP	8.0.7 (apache2handler)
MySQL	8.0.25
ICU	68.2
Pygments	2.15.1

Using the latest Python 3.11 for Windows.

I managed to fix this on my machine by making one modification to includes\Pygmentize.php (add '\r' to trim character list for fetchLexers function, on line 214):

Can we please incorporate this into the production code?

There is no need because this has been implemented for SyntaxHighlight for a future MediaWiki version.

Optionally, one can use the python -m pygments -L lexer --json command, which outputs the lexer list formatted as JSON. This feature has been added in Pygments 2.11.0.

Pygments lexers like "lua", "html", and "css" put pages using <syntaxhighlight> into "Category:Pages with syntax highlighting errors", despite these lexers being correct. Maybe there are other lexers causing this problem - I don't know. But the three I mentioned above are the ones I commonly use for template/module documentation.

I don't know why this happens. The wiki I contribute to runs on MediaWiki 1.39 with Extension:SyntaxHighlight version 2.0. The Pygments documentation states that these Short names: for the lexers I mentioned above are correct, so why are the pages that use them with <syntaxhighlight> placed in the tracking category?

Does anyone know?

Does the code get highlighted? (It’s always good to share a link so that I can answer such simple questions on my own.) The error category gets added whenever the highlighting is unsuccessful – this includes not only incorrect lexer names, but also when the code is too big or calling Pygments fails for any reason.

The code gets highlighted, so I don't know why pages using <syntaxhighlight> end up in the tracking category. There are pages in the tracking category for this wiki that use syntax highlighting correctly but still end up categorized.

Could you please provide a link? It’s very hard to debug without being able to see what exactly is happening.

I apologize for the 2-month late response User:Tacsipacsi. I tried replying sooner, but an error with Extension:StructuredDiscussions on MediaWiki was preventing me from entering a response about a month ago.

I just wanted to let you know that the lexers I defined 2 months ago as not working do work again. The pages on the wiki I contribute to no longer end up being incorrectly categorized under this extension's tracking category when using <syntaxhighlight> correctly.

This issue I had may have been fixed with a major version update to Pygments. During the time I had the categorization issue, this extension was using Pygments 2.11.2. It now uses Pygments 2.15.0 as of April 10 of this year (commit 9f9c13b).

The <code> tag is an HTML5 construct and not related to the SyntaxHighlight extension.

For example:

h t t p s : / / www.youtube.com/watch?v=pRXoS_P0lk

This mixes the code tag with the bold - note that there is a whitespace added after the = character. Why? And how can I fix it?

the extra spaces in the URL are to get around the linkspam filter

Dangit. It didn't do it here. I guess I need to fix something on my won wiki. NVM

The <code> tag is an HTML5 construct and not related to the SyntaxHighlight extension.

You will find live real examples in wp en:Template:Maplink . Here is just a demo:

The table's caption

Column header 1	Column header 2	Column header 3
Row header 1	Cell 2	Cell 3
Row header A	Cell B	Cell C

{{maplink|frame=yes|text=Ten countries|raw={
  "type": "ExternalData",
  "service": "geoshape",
  "query": "
SELECT * WHERE {
  ?id wdt:P463 wd:Q1065 .
} LIMIT 10
"}|frame-width=600|frame-height=400|frame-lat=25|frame-long=10|zoom=1}}

The visible code box is a "pre" box inside the "SyntaxHighlight" but we can't apply any styling to this "pre" box.

Styling the pre tag itself is possible using TemplateStyles. See Template:Codesample and it's related style sheets as an example.

Nice to know, thank you. But I'm not sure that I will be able to set the width of a code box within a Wikipedia page by this way, or... Jean-Louis² (talk) 17:39, 20 June 2023 (UTC)

Sorry, some more information are necessary because we can't easily see wikicode here as in Wikipedia. <syntaxhighlight style="min-width:60em; border: 3px dashed blue;" The syntaxhighlight box in dashed blue do not "see" the table next to it. Styling his size is impossible. Styling some child box inside is impossible too.

According to the extension´s documentation, it should run on Windows, can anyone confirm that?

Apache Error.log message: fwrite(): Write of 17 bytes failed with errno=22 Invalid argument in C:\\xampp\\htdocs\\wiki\\includes\\shell\\Command.php

(Unfortunately, I get no other messages even with more debug options on)

• MediaWiki: 1.35.3

• Windows 10 64bit
• Python 3.11 and Pygments package installed

There are several hack suggestions in the talk pages: Topic:Vibz8ak3wnpu9mv8, Topic:V4n3pjg78pvmh12n (no temp out files created), Topic:Ugtwpe4lyuly6q98. None of them helped me so far.

I also found this entry: https://phabricator.wikimedia.org/T199989

After editing Command.php with the proposed patch (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/762736/4/includes/shell/Command.php), the extension works. However, I would prefer a proper solution without hacking the MediaWiki core.

I suggest that you upgrade to MediaWiki 1.39.X, which is the current LTS release. Note that MediaWiki 1.35 is going EOL in September 2023. See https://phabricator.wikimedia.org/T199989 for the details about this issue.

An update to MW 1.39 ist definitely planned, my understanding: the issue is solved there with Shellbox.

However, the installation does not look trivial to me... Does it run on Windows?

In my case, the whole idea of the Windows version is to offer a ready-to-run package for colleagues without direct access to our original Wiki (which runs on Linux).

Since passing parameters and results to the pygmentizer through stdin/stdout does not work on Windows,

i tried to use files for input and output. Pleas adjust the temporary directory to your environment.

Works for me on MediaWiki 1.33.

Greetings,

Ralf Naujokat

function ( $oldValue, &$ttl ) use ( $code, $lexer, $options, &$error ) {
	$optionPairs = [];
	foreach ( $options as $k => $v ) {
		$optionPairs[] = "{$k}={$v}";
	}
	
	$tmp = tempnam( 'D:\\MediaWiki\\tmp', 'SH' );
	$tmp_out = $tmp . ".out";
	$tmp_in  = $tmp . ".in";
	file_put_contents( $tmp_in, $code );
	
	$result = Shell::command(
		self::getPygmentizePath() .
		' -l ' . $lexer .
		' -f ' . 'html' .
		' -O ' . implode( ',', $optionPairs ) . 
		' -o ' . $tmp_out . 
		' '    . $tmp_in 
	)
		// ->input( $code )
		->restrict( Shell::RESTRICT_DEFAULT | Shell::NO_NETWORK )
		->execute();

	if ( $result->getExitCode() != 0 ) {
		$ttl = WANObjectCache::TTL_UNCACHEABLE;
		$error = $result->getStderr();
		return null;
	}

	$out = file_get_contents( $tmp_out );
	
	unlink( $tmp );
	unlink( $tmp_in );
	unlink( $tmp_out );
	
	return $out;
}

Oops, forgot to mention:

Modify the file 'includes\SyntaxHighlight.php', starting at line 305

Thank you so much for sharing this - I was ripping my hair out!

Also thank you, but have you get syntax highlighter working when attribute highlight is used (highlight="1,5,6" for example)?

I´ve tried with MediaWiki 1.35.3, but the tmp.out files are not created for the results, thus not working so far.

Enabling this extension lead to error: 1319 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught ExtensionDependencyError: SyntaxHighlight requires "shell" ability: Unable to run external programs, proc_open() is disabled." All pages is blank. MW 1.39.1.

Any ideas what to do?

If proc_open() is disabled, you must enable it in php.ini (you can't enable it from anywhere else). Contact your hosting provider if you can't get root access.

OK, thanks, done. But it's strange that message don't just tell to enable proc_open and confuse me with additional info. Finally, it works (while CodeEditor still don't :( ).