Extension talk:SyntaxHighlight

Previous discussion was archived at Extension talk:SyntaxHighlight/Archive 2017 on 2017-03-29.

Hello,

I've just migrated and upgraded our company's MediaWiki to a new server. Windows Server 2012 R2, Apache 2.4.33 64bit + PHP 7.1.19 64bit + Python 3.6 64bit, MediaWiki 1.31.0.

Everything works except the SyntaxHighlight extension. The error message:

Notice:  Failed to invoke Pygments: 'C:\Python36\Scripts\pygmentize.exe" "-l" "css" "-f" "html" "-O" "cssclass' is not recognized as an internal or external command, operable program or batch file.

[Called from SyntaxHighlight::highlight in C:\Apache24\htdocs\wiki\extensions\SyntaxHighlight_GeSHi\includes\SyntaxHighlight.php at line 336] in C:\Apache24\htdocs\wiki\includes\debug\MWDebug.php on line 309

The PATH is set correctly, and from a sample test php file I can call Pygments, and it works:

<?php

$output = shell_exec('C:\Python36\Scripts\pygmentize.exe -l php -f html -O cssclass C:\Apache24\htdocs\test2.php');

echo "$output2";

?>

I have tried several options, like using the exe: $wgPygmentizePath = "C:\\Python36\\Scripts\\pygmentize.exe"; or the bytecode from the cgi directory:

$wgPygmentizePath = "C:\\Apache24\\cgi-bin\\pygmentize.pyc"; but nothing helped. Please advise how to debug further this problem.

I'm having the same problem, there are some system config differences (for instance, I run MediaWiki 1.32 master on XAMPP with PHP 7.2.2), but having the same identical error. Having two versions of Python in two different directories, I tried both of them (2.7 and 3.3), changed wgPygmentizePath to all possible combinations, set full user rights on pygmentize, added everything to PATH, but I'm still getting the same error. And, as Glanthor Reviol, I can indeed execute it from wiki folder using windows' cmd and also directly from SyntaxHighlight using exec. I really can't think of something more to do, sounds like there's some specific problem with Windows. BTW, SyntaxHighlight used to work correctly with an old version (can't recall which one) of the extension itself and MediaWiki, using Python 2.7.

I seem to have the same problem on a MediaWiki 1.31.0, with both Python27 and Python37.

I read somwhere that there were som changes in Shell::command from MediaWiki 1.31.0 - and I traced the problem down to somwhere in that area, but could not find a fix for it.

Hope someone can find a fix for this problem.

I wrote a short and shallow explanation on phabricator, see https://phabricator.wikimedia.org/T199989. Basically, I fear that this might be a very upstream problem, i.e. a PHP bug which won't be fixed (a couple of links on the task) since Windows doesn't actually provide the needed logic. More specifically, the problem as far as I understood it should be that on Windows you cannot set streams as not blocking, which then produces a conflict within shell pipes, entering in a vicious circle. If this is true, there's probably no easy or clean solution to this problem: we could only solve it with a workaround or by refactoring the shell code. But I guess we should really find a solution, since this problem affects a core feature (Shell) on a whole OS.

Hello everyone.

I have the same problem on my side. SyntaxHighlight isn't working since 1.32 update.

I understand the problem with PHP, sream problem etc but I'm looking for a solution, a workaround and since 2 months, I'm unable to find such solution.

Does somebody have a miracle for me ?

Regards

Personally, I can't find one. Actually, I'm not completely sure about the cause, too. Any update will come on Phabricator, anyway.

Thanks for reply.

I am afraid that there is no updates on phabricator too ^^

Wait & see...

Any news? I have the same problem

I'm using MediaWiki 1.28.2 and SyntaxHighlight 2.0. My system is a 64 bit Windows 10 machine.

The code ends up in a plain PRE tag and the page is added to category "Pages with syntax highlighting errors". Setting $wgShowExceptionDetails did not trigger any output. On a hunch I renamed pygmentize to pygmentize.exe, tried to run it via the command line and got this error message:

Unsupported 16-Bit Application

The program or feature "\\...\pygmentize.exe" cannot start or run due to incompatibility with 64-bit versions of Windows. Please contact the software vendor to ask if a 64-bit Windows compatible version is available.

Is that the problem? Do I need a different version of pygmentize? Where do I get that?

So you installed a x64 version of Python ?

Maybe you should try a x32 version Python and install Pygments again.

In case anyone's interested, here's the solution:

1. Find the file called pygmentize in the extensions\SyntaxHighlight_GeSHi folder.
2. Move that file to your Apache's cgi-bin. If your server is set up reasonably well only executables in that magic directory will get executed.
3. Install Python 2.7 because pygmentize is not really an executable. It's Python byte code and requires the python.exe to run.
4. Make sure that the Python install directory is added to your system's PATH.
5. Rename pygmentize to pygmentize.pyc. This is Windows. Extensions are magic here. No extension, no magic.
6. Find the cgi-bin's AddHandler directive in your httpd.conf and add .pyc to the list if it isn't there already.
7. Edit your LocalSettings.php and add this line somewhere:
   

   $wgPygmentizePath = '...\\cgi-bin\\pygmentize.pyc';
   

8. Restart your server.

That worked for me.

This didn't worked for me (XAMPP-solution), Windows 10, Python 3.7 and Python 2.7. Thank you for sharing your solution, but it didn't worked...

Thank you very much! Your info helps me a lot. With the difference: I use IIS as Webserver.

MediaWiki Version 1.29

1. You need to install Python 2.73 in default directory
2. Map cgi extensions like here: http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html(The Extension is "pyc" and not like in the link "py"!)
3. Find the file called pygmentize in the extensions\SyntaxHighlight_GeSHi folder.Rename pygmentize to pygmentize.pyc. This is Windows. Extensions are magic here. No extension, no magic.
4. Add this to LocalSettings.php

$wgPygmentizePath = 'C:\inetpub\wwwroot\Wiki\extensions\SyntaxHighlight_GeSHi\pygments\pygmentize.pyc';

wfLoadExtension( 'SyntaxHighlight_GeSHi' );

wfLoadExtension( 'HighlightJS' );

$wgHighlightJsMagic = 'syntaxhighlight';

$wgHighlightJsStyle = 'tomorrow-night-bright';

I've tried this and it's still not working.

Server 2012 R2

IIS 8.5

python 2.73 installed in c:\Python27

Followed instructions above...scratching head. Is there a way to output the error....why it's not highlighting?

None of this worked for me either, I have gone through the PHP itself and there is little to no information I could find that helped solve the problem.

Instead I have opted to replace the faulty section of code with some basic PHP script that seems to work (at least for my machine).

Try this fix:

Open SyntaxHighlight.class.php in the mediawiki\extensions\SyntaxHighlight_GeSHi folder

Scroll to around line 211 to find the highlight function

Towards the end of this function, you should see an if statement like:

if ( $output === false) {

// does stuff here

}

Simply replace this entire if block with the following script, and it should fix the problem for you (edit the Python executable path as needed for your installation of course):

if ( $output === false ) {

           $optionPairs = [];

           foreach ( $options as $k => $v ) {

               $optionPairs[] = "{$k}={$v}";

           }           

           $execCmd = 'C:\\Python27\\python.exe ' . self::getPygmentizePath() . ' -f html -l ' . $lexer . ' -O ' . implode( ',', $optionPairs );           

           $descriptorSpec = array(

               0 => array('pipe', 'r'),

               1 => array('pipe', 'w')

           );           

           $proc = proc_open($execCmd, $descriptorSpec, $pipes);           

           fwrite($pipes[0], $code);

           fclose($pipes[0]);           

           $output = stream_get_contents($pipes[1]);

           fclose($pipes[1]);

           proc_close($proc);

           $cache->set( $cacheKey, $output );

       }

---

This should hopefully be a fix, if the results don't appear straight away it might be due to caching, in which case you can add $output = false; just before the if statement to reset it.

Hope it works for anyone else that's having problems, tested this on IIS 10

I battled with this myself. No matter what i tried it didnt work. Until i discovered this : Extension:Highlightjs Integration. This worked so well for me and i didnt need // wfloadextension( 'syntaxhighlight_geshi' ); in my LocalSettings.php. All i needed to add after following standard instructions (download, paste in folder etc...) was this line : wfloadextension( 'Highlightjs_Integration' ); in my LocalSettings.php

Is there a way to make it work in VisualEditor? For me the lines with code are displayed as blank lines.

Hi RockSheep. I dont know the answer to this since I haven't yet played with VisualEditor before. I am sure that there must be a way to get it working since Highlightjs should work in any browser really...

Also as a sidenote if you want to change the style change it in extension.json in the Highlightjs_Integration folder in the "styles" section. Should look like this  :

"highlight/styles/vs.css"

to view what styles are available check here :

https://highlightjs.org/static/demo/

Rory's solution needs a slight tweak in 1.31, but it works mostly, line 307:

I set this in LocalSettings.php and also renamed the pygmentize to pygmentize.pyc and setup a Script Mapping Handler in IIS.

$wgPygmentizePath = 'C:\inetpub\wwwroot\extensions\SyntaxHighlight_GeSHi\pygments\pygmentize.pyc';

$output = $cache->getWithSetCallback(
            $cache->makeGlobalKey( 'highlight', self::makeCacheKeyHash( $code, $lexer, $options ) ),

            $cache::TTL_MONTH,
            function ( $oldValue, &$ttl ) use ( $code, $lexer, $options, &$error ) {

                $optionPairs = [];
                foreach ( $options as $k => $v ) {
                    $optionPairs[] = "{$k}={$v}";
                }             

                $execCmd = self::getPygmentizePath() . ' -f html -l ' . $lexer . ' -O ' . implode( ',', $optionPairs );           

                $descriptorSpec = array(
                    0 => array('pipe', 'r'),
                    1 => array('pipe', 'w')
                );           

                $proc = proc_open($execCmd, $descriptorSpec, $pipes);           

                fwrite($pipes[0], $code);
                fclose($pipes[0]);

                $res = stream_get_contents($pipes[1]);

                fclose($pipes[1]);
                proc_close($proc);

                return $res;

            }
        );

SyntaxHighlight uses Python Pygmentize

If you are running MediaWiki on a Shared Host without cpanel installed, forget about using SyntaxHighlight.

It won't work! And that is very very sad!

Why are there python packages needed to make an extension? Don't do that! It is helping the developer to make it easier, but makes the portability of the extension not better, even worse or not working at all. Again very very sad issue here

I'm on a shared host with SiteGround and use cpanel. SyntaxHighlight isn't working for me any more. I guess I'll try another code highlighting extension.

Is there any news on this failure? We have a rather busy but recently built server. Along with the lack of colorization we also have each of the 98 pages where the tag is called showing up on Category:Pages with syntax highlighting errors. MediaWiki 1.31.0 PHP 7.2.9 (cgi-fcgi) MySQL 5.7.23-log

Will we be better off looking into a different solution like https://www.mediawiki.org/wiki/Extension:SyntaxHighlighterAndCodeColorizer which also shows as stable but a bit older?

If you upload whole mediawiki installation or just this extension via FTP manager that can use ASCII mode (for Filezilla use Edit -> Settings -> Transfers -> FTP:File Types and check OFF "Treat files without extension as ASCII files") for file transfer this extension wont work with error "zipimport.ZipImportError: bad local file header:".

If you're using an FTP manager to upload files to your server, and you mess with the ASCII mode settings, you're generally going to have a lot of problems of this nature; it's nothing specific to this extension (the default settings are default for a reason).

The thing is Filezilla defaults to "Treat files without extension as ASCII files" and pygmentize is falling into this category.

Filezilla uploads extension damaged.

I was unable to find zipimport error message anywhere on this page and it gives no clue about the actual problem.

LS,

I have installed mediawiki version 1.30 and trying to get this working on my page:

===eind schakelaars===

// Mechanical endstop with COM to ground and NC to Signal uses "false" here (most common setup).
#define X_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
#define Y_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
#define Z_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
#define X_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
#define Y_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
#define Z_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
#define Z_MIN_PROBE_ENDSTOP_INVERTING false // set to true to invert the logic of the probe.

It places a correct border around the code but no line numbers infront ore highlighted ones.

So whats wrong here??

try

<syntaxhighlight lang="c++" line start="530" highlight="2-3" style="border:2px dashed red">
// Mechanical endstop with COM to ground and NC to Signal uses "false" here (most common setup).
#define X_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
#define Y_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
#define Z_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
#define X_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
#define Y_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
#define Z_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
#define Z_MIN_PROBE_ENDSTOP_INVERTING false // set to true to invert the logic of the probe.
</syntaxhighlight>

The language you specified, does not exist (there is no language "c / c++"). Either use "c" or "c++". The example above looks like:

530 // Mechanical endstop with COM to ground and NC to Signal uses "false" here (most common setup).
531 #define X_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
532 #define Y_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
533 #define Z_MIN_ENDSTOP_INVERTING true // set to true to invert the logic of the endstop.
534 #define X_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
535 #define Y_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
536 #define Z_MAX_ENDSTOP_INVERTING false // set to true to invert the logic of the endstop.
537 #define Z_MIN_PROBE_ENDSTOP_INVERTING false // set to true to invert the logic of the probe.

• edit* strangely, when I tested it, lines 2-3 were highlighted correctly. Now, the highlight is gone and I do not seem to get it to work again. sorry.

But under supported languages it says:....

• BrainFuck
• C / C++
• C#

This is not realy clear that one should only use C++

edit:

It looks also to be a version problem As I have wiki version1.30 installed where it doesnt work, and this website uses 1.33 where it works.

I help to maintain a small wiki on Wikia (aka Fandom) for the PICO-8 fantasy video game console, which is programmed in a modified version of Lua. We've got syntaxhighlight working for Lua code blocks, which is great, but it would be nice if we could add built-in keywords (only functions, I think) that aren't present in vanilla Lua. It's not crucial, but it would be nice.

Obviously we only have just so much access, as we don't maintain the Fandom servers, so it's quite possible that what we'd need to configure is out of our reach, but I figured it can't hurt (much) to ask. Even knowing it's not possible is useful information.

Thanks!

Edit: Not sure exactly what version is installed on Wikia, but I think it's at least the Pygments version.

What you're asking for would require server-side access, which you don't have on most server farms (including Wikia/Fandom). Even before that, though, your first step would be writing (or finding someone else to write for you) a Pygments lexer for your language; it sounds like you could just modify the Lua lexer, which would greatly simplify that process. Once that's done, you should contact Wikia/Fandom staff to find out if they'll add the lexer for your wiki (or, if they prefer not to fragment the SyntaxHighlight configuration between their wikis, if they'll add it for all wikis). Alternatively, you could try to get the lexer added to Pygments itself and then try to get the WMF to update the Pygments version/lexer library packaged with the extension, then try to get Wikia/Fandom to upgrade to the new extension version, but obviously that route would be far more work (with far more places it could stall out).

If none of this appeals to you, of course, you could look into Javascript-based syntax highlighting; this would only require Wikia/Fandom staff to approve the JS code (though you would also have to support the code itself for as long as you used it).

I think the lexer idea might be the best choice, and if that's not an option, then maybe just let it go and use the vanilla highlighting. As I said, even knowing it's not likely to happen is useful info. :) Thanks!

I would like to use the <source> tag to present MediaWiki wikicode, but I couldn't find any related lexer. Is there any one I can use?

Nope. See phab:T29828.

Thanks for the link. Too bad MediaWiki doesn't support its own syntax!

Two points:

1. Try html. It's better than nothing.
2. The tag source is deprecated. Use syntaxhighlight instead.

Try "moin" as well. Mediwiki format was orignated from Moinmoin.

See https://www.wikimatrix.org/compare/mediawiki+moinmoin

MediaWiki's syntax was mostly derived from UseModWiki's, not Moinmoin's, which shouldn't be particularly surprising since UseMod was its predecessor on WMF wikis. (Where the syntax ultimately comes from is a completely different question, of course, and one I'm not sure would be particularly easy to answer.)

Of course, this doesn't really matter much for syntax highlighting; as long as the markup systems are sufficiently similar to each other, using Moinmoin's lexer for MediaWiki markup should still be a decent stopgap.

Yet, I tried 'lang=moin', and it doesn't really do anything concerning syntax highlight. Precision: what I'm trying to show in the code is template inclusion. Anyway, I just would like that the page I created were not automatically categorized as a "Page with syntax highlighting errors". I think the only way to do so is not to use this tag...

I suggest adding this to Extension:SyntaxHighlight#Syntax highlighting error category:

"The category may also be added if there are more than 1000 lines or more than 100 kB text.^[1]"

It was reported at w:Wikipedia:Village pump (technical)#Syntax highlighting. I have not actually been able to produce the error with 5000 lines at enwiki but it does happen above 100 kB.

Added. Thanks for the good notes. :-)

The extension SyntaxHighLight uses the access to shell of the Webserver it is running on.

Most hosting provider do not give the user access to the shell.

The code of SyntaxHighLight uses the MediaWiki Class Shell which checks if the user has access to the proc_open. See:

/**
* Checks if this class is effectively disabled via php.ini config.
*
* @return bool disabled
*/
public static function isDisabled() {
  static $disabled = null;

  if ( is_null( $disabled ) ) {
    if ( ! function_exists( 'proc_open' ) ) {
      wfDebug( "proc_open() is disabled\n" );
      $disabled = true;
    } else {
      $disabled = false;
    }
  }
  return $disabled;
}

There seems not to be another solution available.

My advice to not use the shell for extensions. It won't work.

The other remark is to make a requirements chapter for every extension which gives the user this information BEFORE he is trying to use such an extension!

This requirement chapter can explain which functions and which languages are needed.

There are a number of PHP functions that are disabled in the php.ini by the hosting providers. Examples are disk_total_space, diskfreespace, exec, system, popen, proc_open, proc_nice, shell_exec, passthru, dl

Check your php.ini file or create a website with the function phpinfo();

I would also suggest to use PHP as de-facto without the need to use other languages.

My advice to not use the shell for extensions. It won't work.

In the other direction, our advice is not to use hosting providers that don't give you shell access, as it won't work for lots of functionality.

That said, I've added a note reminding sysadmins of this.

Product	Version
MediaWiki	1.31.0-alpha
HHVM	3.21.3 (srv)
MariaDB	10.0.32-MariaDB-0+deb8u1
ICU	52.1

SyntaxHighlight

2.0 (330ac62)12:31, 25 February 2018

GPL-2.0-or-later

Provides syntax highlighting <syntaxhighlight> using Pygments - Python syntax highlighter

Brion Vibber, Tim Starling, Rob Church, Niklas Laxström, Ori Livneh and Ed Sanders

# /usr/share/nginx/html/wiki/extensions/SyntaxHighlight_GeSHi/pygments/pygmentize -V

Pygments version 2.2.0, (c) 2006-2017 by Georg Brandl. -- PS! I also tried v2.0.2 and specifying the path in LocalSettings.

[error] 683#683: *648 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Call to undefined method MediaWiki\Shell\Command::input() in /usr/share/nginx/html/wiki/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php on line 320" while reading response header from upstream, client: 1.2.3.4, server: mycoolhost.net, request: "POST /wiki/api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mycoolhost.net"   

                $output = $cache->getWithSetCallback(

                        $cache->makeGlobalKey( 'highlight', self::makeCacheKeyHash( $code, $lexer, $options ) ),

                        $cache::TTL_MONTH,

                        function ( $oldValue, &$ttl ) use ( $code, $lexer, $options, &$error ) {

                                $optionPairs = [];

                                foreach ( $options as $k => $v ) {

                                        $optionPairs[] = "{$k}={$v}";

                                }

                                $result = Shell::command(

                                        self::getPygmentizePath(),

                                        '-l', $lexer,

                                        '-f', 'html',

                                        '-O', implode( ',', $optionPairs )

                                )

                                        ->input( $code )

                                        ->restrict( Shell::RESTRICT_DEFAULT | Shell::NO_NETWORK )

                                        ->execute();

                                if ( $result->getExitCode() != 0 ) {

                                        $ttl = WANObjectCache::TTL_UNCACHEABLE;

                                        $error = $result->getStderr();

                                        return null;

                                }

                                return $result->getStdout();

                        }

                );

Any tips?

Hi,

i want to highlight the ABAP-Coding in my Mediawiki. In the List of Supported languages ABAP is' listet, but the available lexers pygments.lexers.business.ABAPLexer

are still listet on the page http://pygments.org/docs/lexers/#lexers-for-php-and-related-languages.

What can i do, to have my highlight ABAP-Coding back???

Greetings

Meinolf

The ABAP lexer is the first one listed under http://pygments.org/docs/lexers/#lexers-for-business-oriented-languages. Have you actually tried using it yet?

Thank you,

i know about this ABAP lexer. But I don't know how to use it with SyntaxHighlight. I don't know how to download it (on the listet page there it no download-link), I do't know how to install this lexer and so on.

Greetings

Meinolf

It should be installed as long as you have this extension installed. To use it you'd use <syntaxhighlight lang="abap">code here</syntaxhighlight>, as can be seen on Wikipedia's ABAP article.

Yes, when i use it in this Kind the page get the Category:Pages_with_syntax_highlighting_errors.

And i think ist's a Problem of the Extension, because the ar a Lot of Page with syntax-Highlightling-erros. Look at the link Category:Pages with syntax highlighting errors. eg the lang php or sql doset work also as ABAP.

Greeting

Meinolf

Try twiddling the capitalization, e.g. Abap or ABAP instead of abap. Apparently the extension is case-sensitive, but there doesn't seem to be a 100% comprehensive list of correct values for the lang attribute.

Sorry, but this proposal does not solve the problem.

even using <syntaxhighlight> without the parameter lang generates the error category.

greeting

Meinolf

The category is always added if you don't have a lang attribute, since the extension doesn't have a default language, and doesn't have any language auto-detection feature. However, having a lang attribute doesn't guarantee that the categoy won't be added; it is also added if the value of the lang attribute is invalid (though as I said above, there isn't any completely comprehensive list of valid values as far as I'm aware). But, one of the ways the value can be "invalid" is if it isn't the expected capitalization, even though the actual language matching doesn't seem to be case-sensitive in most cases. As long as the actual syntax highlighting works correctly, changing the lang attribute's value to prevent the category from being added isn't terribly important. So, is your code being correctly highlighted when you use <syntaxhighlight lang="abap">?

No, the code isn't highlighted! The page, the code looks like using the TAGs <pre>code</pre>

Wen i use <syntaxhighlight lang="abap"> the Page will displayed with the Category:Pages with syntax highlighting errors.

greetings

Meinolf Gerling

Is it being highlighted if you use any other capitalization of abap? If not, I'm afraid I'm out of ideas, beyond maybe making sure your MediaWiki installation and the extension are both the latest version, and double-checking the extension's configuration (both if you're able to, of course).