Extension talk:SyntaxHighlight

Jump to navigation Jump to search

About this board


Archives

Previous discussion was archived at Extension talk:SyntaxHighlight/Archive 2017 on 2017-03-29.

Slowking4 (talkcontribs)
Reply to "pencil icon on editing toolbar"
Glanthor Reviol (talkcontribs)

Hello,

I've just migrated and upgraded our company's MediaWiki to a new server. Windows Server 2012 R2, Apache 2.4.33 64bit + PHP 7.1.19 64bit + Python 3.6 64bit, MediaWiki 1.31.0.

Everything works except the SyntaxHighlight extension. The error message:

Notice:  Failed to invoke Pygments: 'C:\Python36\Scripts\pygmentize.exe" "-l" "css" "-f" "html" "-O" "cssclass' is not recognized as an internal or external command, operable program or batch file.

[Called from SyntaxHighlight::highlight in C:\Apache24\htdocs\wiki\extensions\SyntaxHighlight_GeSHi\includes\SyntaxHighlight.php at line 336] in C:\Apache24\htdocs\wiki\includes\debug\MWDebug.php on line 309

The PATH is set correctly, and from a sample test php file I can call Pygments, and it works:

<?php

$output = shell_exec('C:\Python36\Scripts\pygmentize.exe -l php -f html -O cssclass C:\Apache24\htdocs\test2.php');

echo "$output2";

?>

I have tried several options, like using the exe: $wgPygmentizePath = "C:\\Python36\\Scripts\\pygmentize.exe"; or the bytecode from the cgi directory:

$wgPygmentizePath = "C:\\Apache24\\cgi-bin\\pygmentize.pyc"; but nothing helped. Please advise how to debug further this problem.

Daimona Eaytoy (talkcontribs)

I'm having the same problem, there are some system config differences (for instance, I run MediaWiki 1.32 master on XAMPP with PHP 7.2.2), but having the same identical error. Having two versions of Python in two different directories, I tried both of them (2.7 and 3.3), changed wgPygmentizePath to all possible combinations, set full user rights on pygmentize, added everything to PATH, but I'm still getting the same error. And, as Glanthor Reviol, I can indeed execute it from wiki folder using windows' cmd and also directly from SyntaxHighlight using exec. I really can't think of something more to do, sounds like there's some specific problem with Windows. BTW, SyntaxHighlight used to work correctly with an old version (can't recall which one) of the extension itself and MediaWiki, using Python 2.7.

Bar-ucholstebro (talkcontribs)

I seem to have the same problem on a MediaWiki 1.31.0, with both Python27 and Python37.

I read somwhere that there were som changes in Shell::command from MediaWiki 1.31.0 - and I traced the problem down to somwhere in that area, but could not find a fix for it.

Hope someone can find a fix for this problem.

Daimona Eaytoy (talkcontribs)

I wrote a short and shallow explanation on phabricator, see https://phabricator.wikimedia.org/T199989. Basically, I fear that this might be a very upstream problem, i.e. a PHP bug which won't be fixed (a couple of links on the task) since Windows doesn't actually provide the needed logic. More specifically, the problem as far as I understood it should be that on Windows you cannot set streams as not blocking, which then produces a conflict within shell pipes, entering in a vicious circle. If this is true, there's probably no easy or clean solution to this problem: we could only solve it with a workaround or by refactoring the shell code. But I guess we should really find a solution, since this problem affects a core feature (Shell) on a whole OS.

Reply to "Failed to invoke Pygments on Windows"
AhmadF.Cheema (talkcontribs)

Aside from the MediaWiki:Common.css page, any page (MediaWiki:Common.js, Module pages, pages including syntaxhighlight tags) that needs to call Extension:SyntaxHighlight throws out the following process timed out error:

Fatal exception of type "Symfony\Component\Process\Exception\ProcessTimedOutException"

The problematic process according to the logs:

The process "'.../extensions/SyntaxHighlight_GeSHi/pygments/pygmentize' '-l' 'lua' '-f' 'html' '-O' 'cssclass=mw-highlight,encoding=utf-8'" exceeded the timeout of 60 seconds.
  • Tested versions: MediaWiki core (REL1_30) - SyntaxHighlight_GeSHi (REL1_30) - PHP (5.6; 7.0; 7.1; 7.2)

In the case of REL1_31 branch, the process continues to run (and the browser continues to load) indefinitely without throwing out the timed out error. Even when the browser loading has been stopped, the php-cgi command keeps on running in the background and has to be manually terminated through SSH. This could possibly be because composer update removes the "symfony" package.

  • Tested versions: MediaWiki core (REL1_31) - SyntaxHighlight_GeSHi (REL1_31) - PHP (7.0; 7.1; 7.2)
  • Tested versions: MediaWiki core (REL1_31) - SyntaxHighlight_GeSHi (master) - PHP (7.2)

The pygmentize file has execute permissions (755). Also updated composer.

Platform is shared hosting CentOS environment. Process used to work previously, hosting support confirmed not to have made any changes to the server and even tried to run the script after disabling all firewall protections to no avail.

Anyone has any ideas?

Ciencia Al Poder (talkcontribs)

Does it happen on every CSS/JS page, or only some of them? Try to test with a simple CSS/JS page.

I had a similar problem with one specific page, which timed out, probably due to complex code or a bug in the syntax highlight code.

AhmadF.Cheema (talkcontribs)

Aside from MediaWiki:Common.css, on every page.

A confusing update.

  1. $wgMainCacheType (CACHE_MEMCACHED) turned Off - CSS/JS pages work fine.
  2. $wgMainCacheType turned On - CSS/JS pages continue to work fine.
  3. $wgMainCacheType again turned Off - CSS/JS pages start getting timed out.
    1. Comment out any extension - CSS/JS pages start working again.
    2. Re-enable the extension - CSS/JS pages continue to work fine.

Hard-refresh was done after every step. This situation was reproduce-able, however for the moment the issue with CSS/JS pages and any page containing syntaxhighlight tags code blocks appears to have stopped occurring.

However, during none of the steps above, do Module pages work. They continue to get timed out.

Update:

Module pages also appear to be working now, waiting for response from hosting provider regarding any changes on their end which might have fixed the issue.

Update II:

Hosting provider confirmed no changes on the server.

Apparently, will never get to the bottom of this.

Will close this issue in the next couple of days, unless any new information comes along.

Ciencia Al Poder (talkcontribs)

Maybe SyntaxHighlight is slow, and it eventually gets the syntax highlight correctly and caches it, so it works correctly next time you visit the page. Of course, this doesn't make sense when you turn off $wgMainCacheType and it still works fine. Maybe the page was cached on your browser and you didn't get to the server when you tried it...

AhmadF.Cheema (talkcontribs)

Possible, though seems doubtful, since I attempted this probably more than 10 times, with hard-refreshes and browser incognito windows.

Tystnaden (talkcontribs)

Do you have Python version 3 installed on the server? The version of Pygments used by the extension bundled with MW 1.31 requires python3 to be installed. I have added a note at the top of this article about this issue.

AhmadF.Cheema (talkcontribs)

I didn't before, but this particular weird issue was probably not linked to python3 as error came and went while using both MediaWiki v1.30 and 1.31 (and their corresponding extension versions). From what I understand python3 is needed when highlighting languages such as Lua, in the absence of python3, MediaWiki just outputs the non-highlighted version with a - Notice: Failed to invoke Pygments: /usr/bin/env: python3: No such file or directory - error in debugging mode.

Tystnaden (talkcontribs)

Okay, that does sound like a different problem.

Daimona Eaytoy (talkcontribs)
Reply to "Pygmentize process getting TimedOut"

How to switch this off?

2
Summary by Jdforrester (WMF)

Off-topic question about CodeMirror.

Atsirlin (talkcontribs)

SyntaxHighlight was recently rolled out on Russian Wikivoyage, and now I see all my source text in purple, light-green, and other disturbing colors. What is a simple way to switch this off, at least for me personally through my common.css?

Atsirlin (talkcontribs)

My question has been answered. Sorry for trouble.

Zewas (talkcontribs)

I'm using MediaWiki 1.28.2 and SyntaxHighlight 2.0. My system is a 64 bit Windows 10 machine.

The code ends up in a plain PRE tag and the page is added to category "Pages with syntax highlighting errors". Setting $wgShowExceptionDetails did not trigger any output. On a hunch I renamed pygmentize to pygmentize.exe, tried to run it via the command line and got this error message:

Unsupported 16-Bit Application

The program or feature "\\...\pygmentize.exe" cannot start or run due to incompatibility with 64-bit versions of Windows. Please contact the software vendor to ask if a 64-bit Windows compatible version is available.

Is that the problem? Do I need a different version of pygmentize? Where do I get that?

Lanthanis (talkcontribs)

So you installed a x64 version of Python ?

Maybe you should try a x32 version Python and install Pygments again.

Zewas (talkcontribs)

In case anyone's interested, here's the solution:

  1. Find the file called pygmentize in the extensions\SyntaxHighlight_GeSHi folder.
  2. Move that file to your Apache's cgi-bin. If your server is set up reasonably well only executables in that magic directory will get executed.
  3. Install Python 2.7 because pygmentize is not really an executable. It's Python byte code and requires the python.exe to run.
  4. Make sure that the Python install directory is added to your system's PATH.
  5. Rename pygmentize to pygmentize.pyc. This is Windows. Extensions are magic here. No extension, no magic.
  6. Find the cgi-bin's AddHandler directive in your httpd.conf and add .pyc to the list if it isn't there already.
  7. Edit your LocalSettings.php and add this line somewhere:
    $wgPygmentizePath = '...\\cgi-bin\\pygmentize.pyc';
    
  8. Restart your server.

That worked for me.

94.217.149.6 (talkcontribs)

Thank you very much! Your info helps me a lot. With the difference: I use IIS as Webserver.

MediaWiki Version 1.29

  1. You need to install Python 2.73 in default directory
  2. Map cgi extensions like here: http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html(The Extension is "pyc" and not like in the link "py"!)
  3. Find the file called pygmentize in the extensions\SyntaxHighlight_GeSHi folder.Rename pygmentize to pygmentize.pyc. This is Windows. Extensions are magic here. No extension, no magic.
  4. Add this to LocalSettings.php

$wgPygmentizePath = 'C:\inetpub\wwwroot\Wiki\extensions\SyntaxHighlight_GeSHi\pygments\pygmentize.pyc';

wfLoadExtension( 'SyntaxHighlight_GeSHi' );

wfLoadExtension( 'HighlightJS' );

$wgHighlightJsMagic = 'syntaxhighlight';

$wgHighlightJsStyle = 'tomorrow-night-bright';

209.37.106.160 (talkcontribs)

I've tried this and it's still not working.

Server 2012 R2

IIS 8.5

python 2.73 installed in c:\Python27

Followed instructions above...scratching head. Is there a way to output the error....why it's not highlighting?

Rory.fewell (talkcontribs)

None of this worked for me either, I have gone through the PHP itself and there is little to no information I could find that helped solve the problem.

Instead I have opted to replace the faulty section of code with some basic PHP script that seems to work (at least for my machine).

Try this fix:

Open SyntaxHighlight.class.php in the mediawiki\extensions\SyntaxHighlight_GeSHi folder

Scroll to around line 211 to find the highlight function

Towards the end of this function, you should see an if statement like:

if ( $output === false) {

// does stuff here

}

Simply replace this entire if block with the following script, and it should fix the problem for you (edit the Python executable path as needed for your installation of course):

if ( $output === false ) {

           $optionPairs = [];

           foreach ( $options as $k => $v ) {

               $optionPairs[] = "{$k}={$v}";

           }           

           $execCmd = 'C:\\Python27\\python.exe ' . self::getPygmentizePath() . ' -f html -l ' . $lexer . ' -O ' . implode( ',', $optionPairs );           

           $descriptorSpec = array(

               0 => array('pipe', 'r'),

               1 => array('pipe', 'w')

           );           

           $proc = proc_open($execCmd, $descriptorSpec, $pipes);           

           fwrite($pipes[0], $code);

           fclose($pipes[0]);           

           $output = stream_get_contents($pipes[1]);

           fclose($pipes[1]);

           proc_close($proc);

           $cache->set( $cacheKey, $output );

       }

---

This should hopefully be a fix, if the results don't appear straight away it might be due to caching, in which case you can add $output = false; just before the if statement to reset it.

Hope it works for anyone else that's having problems, tested this on IIS 10

Egon.allison (talkcontribs)

I battled with this myself. No matter what i tried it didnt work. Until i discovered this : Extension:Highlightjs Integration. This worked so well for me and i didnt need // wfloadextension( 'syntaxhighlight_geshi' ); in my LocalSettings.php. All i needed to add after following standard instructions (download, paste in folder etc...) was this line : wfloadextension( 'Highlightjs_Integration' ); in my LocalSettings.php

Egon.allison (talkcontribs)

Also as a sidenote if you want to change the style change it in extension.json in the Highlightjs_Integration folder in the "styles" section. Should look like this  :

"highlight/styles/vs.css"

Egon.allison (talkcontribs)
Reply to "Syntax highlighting does not work"
Mayazcherquoi (talkcontribs)

How can I force Pygmentize to render output using the "colorful" style, as per: http://pygments.org/docs/styles/ ?

I have tried setting the following:

$wgPygmentizePath = '/usr/bin/pygmentize -S colorful';

However, this will just cause Pygmentize to error out on any pages containing a SyntaxHighlight block.

Cheers.

2001:8B0:CA07:C57A:127B:44FF:FE93:FAC4 (talkcontribs)

I too would be interested in how to do this. The <tt>-S</tt> option is not the correct one. I think you would need to edit the source of the extension to add

options['style'] = 'colorful';

However, I think you would also need to edit the <tt>maintenance/updateCSS.php</tt> script to also use that style, and run it, in order to update the CSS files in the <tt>modules/</tt> directory, as these are what define what the output actually looks like.

I tried all of that and although my colour scheme did change, it didn't come out looking correct. I wanted a dark style like a white-on-black terminal, but it persists in giving me a light grey background. Some further clues are missing.

For now I have switched to using the Highlightjs_Integration extension.

Reply to "Control Pygmentize style?"
Aschroet (talkcontribs)

On our 1.29.1-Installation we activated this plugin. The highliting works properly. Also the integration into the VE is generally working, i. e. when adding a code block it asks about language and line numbers. However, when "inserting" i only see the ugly wikitext but not a preview how it will look like after saving. Don't know if this is a bug or a missing feature. Any help appriciated.

Jdforrester (WMF) (talkcontribs)

I think this is T164120, an issue that was fixed about a year ago, and was released in the 1.30 branch of the code. If I'm right, when you upgrade this should be resolved for you. If I'm not and it's a different issue, please give some more details so we can track that.

Aschroet (talkcontribs)

I think it is another issue since i select a valid language and as a result i am getting a string shown in VE like <syntaxhighlight lang="java"> test </syntaxhighlight>. This string is marked in blue and there is a bubble "code block" with the edit button. Unfortunately, i cannot upgrade to 1.30 in my environment. Could you help to debug this issue. I already checked the parsoid log without any result.

Reply to "Preview within VE does not work"

Call to undefined method MediaWiki\Shell\Command::input()

1
77.88.121.37 (talkcontribs)
Product Version
MediaWiki 1.31.0-alpha
HHVM 3.21.3 (srv)
MariaDB 10.0.32-MariaDB-0+deb8u1
ICU 52.1
SyntaxHighlight 2.0 (330ac62)12:31, 25 February 2018 GPL-2.0-or-later Provides syntax highlighting <syntaxhighlight> using Pygments - Python syntax highlighter Brion Vibber, Tim Starling, Rob Church, Niklas Laxström, Ori Livneh and Ed Sanders

# /usr/share/nginx/html/wiki/extensions/SyntaxHighlight_GeSHi/pygments/pygmentize -V

Pygments version 2.2.0, (c) 2006-2017 by Georg Brandl. -- PS! I also tried v2.0.2 and specifying the path in LocalSettings.

[error] 683#683: *648 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Call to undefined method MediaWiki\Shell\Command::input() in /usr/share/nginx/html/wiki/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php on line 320" while reading response header from upstream, client: 1.2.3.4, server: mycoolhost.net, request: "POST /wiki/api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "mycoolhost.net"   

                $output = $cache->getWithSetCallback(

                        $cache->makeGlobalKey( 'highlight', self::makeCacheKeyHash( $code, $lexer, $options ) ),

                        $cache::TTL_MONTH,

                        function ( $oldValue, &$ttl ) use ( $code, $lexer, $options, &$error ) {

                                $optionPairs = [];

                                foreach ( $options as $k => $v ) {

                                        $optionPairs[] = "{$k}={$v}";

                                }

                                $result = Shell::command(

                                        self::getPygmentizePath(),

                                        '-l', $lexer,

                                        '-f', 'html',

                                        '-O', implode( ',', $optionPairs )

                                )

                                        ->input( $code )

                                        ->restrict( Shell::RESTRICT_DEFAULT | Shell::NO_NETWORK )

                                        ->execute();

                                if ( $result->getExitCode() != 0 ) {

                                        $ttl = WANObjectCache::TTL_UNCACHEABLE;

                                        $error = $result->getStderr();

                                        return null;

                                }

                                return $result->getStdout();

                        }

                );

Any tips?

Reply to "Call to undefined method MediaWiki\Shell\Command::input()"
Tgds003 (talkcontribs)

i have a suggestion.

i want to reduce height of syntaxhighlight element when user doubleclick, so i added some code to Mediawiki:common.js.

----

$('.mw-highlight pre').dblclick(function(event) {

    if ($(event.target).parent().attr('class') == "scrolldiv") {

        $(event.target).unwrap();

    } else {

        $(event.target).wrap('<div class="scrolldiv"/>');

        $('.scrolldiv').css({ height: '300px', overflow: 'auto' });

    }

});

---

i thought it would be useful for someone.:)

thank you!

Dinoguy1000 (talkcontribs)

Instead of height, I'd suggest using max-height on .scrolldiv, so that the box won't be unnecessarily tall if there's only a handful of lines of code.

Tgds003 (talkcontribs)

yes, you are right.

but, max-height doesn't seem to work.

i think we should use the 'if' statement rather than max-height.

i edited a code, as follows

----

$('.mw-highlight pre').dblclick(function(event) {

if (parseInt($(this).css('height'), 10)>300){

if ($(this).parent().attr('class') == "scrolldiv") {

    $(this).unwrap();

} else {

    $(this).wrap('<div class="scrolldiv"/>');

    $('.scrolldiv').css({ height: '300px', overflow: 'auto' });

}

}

});

Reply to "collapsible syntaxhighlight"
Aka sektor (talkcontribs)

Not work <syntaxhighlight lang="lua">

Aka sektor (talkcontribs)

Any news?

99.90.196.227 (talkcontribs)

keep format as <nowiki><pre>..</pre></nowiki> but not colourize

on 1.29.1 either

Reply to "MW 1.29.0 bugs"