Extension talk:SyntaxHighlight

• Extension talk:SyntaxHighlight/REL1.25 (LTS)

Previous discussion was archived at Extension talk:SyntaxHighlight/Archive 2017 on 2017-03-29.

We have a page with lots of (~500) <syntaxhighlight> blocks. Since the code block rendering command pygmentize is call for each block separetely the rendering of the wiki page ends up with very poor performance (2 minutes). This specially breaks the VE and general user experience when a page is previewed or saved.

Is there possibility to improve the performance in this case or do we have just a not described usecase with so many code blocks?

Here an already existing ticket: https://phabricator.wikimedia.org/T122510

I tried for, I guess, more than a year, make it work, but it simply doesn't work, following all the possible available guides i found on the Internet. So I don't know if someone really made it work normally.

I'm using Windows, Download Mediawiki from Github, as same as Syntaxhighlight, followed all the possible instructions, but it doesn't work.

Hi All,

As in the post just under, I have recently upgraded mediawiki to version 1.34.4 (from 1.24) in a fresh directory and found that SyntaxHighlight extension is not working well.

There is no color applied for all the langs I tried (especialy ksh one).

As in the under post, I guess there is a link with the Python version used.

Both Python V2.7.9 and V3.4.2 are available on the server but I can not figure out wich one is used by the PHP interpreter.

Her are the specs :

OS: Linux Distribution

Hosting: OVH Shared hosting

PHP version: 7.3.20 (cgi-fcgi)

Mediawiki version: 1.34.4

SyntaxHighlight: 2.0 (embended)

Does anybody have a clue ?

Regards

Hi All,

I have recently upgraded mediawiki to version 1.35.0 and found that SyntaxHighlight extension is not working.

I have read the prerequisite for SyntaxHighlight extension that it requires Python version 3 to be installed to get it work. But my server has installed Python 2.6 and due to shared hosting I could not upgrade to Python 3.

Please guide me how to get it worked this extension in Mediawiki 1.35.0

OS: Linux Distribution

Hosting: Shared hosting

PHP version: 7.4.9

Mediawiki version: 1.35.0

It seems that the pygmentize library uses the mediawiki Shell command to execute bash on Linux.

This in return needs /bin/bash to be accessible.

I understand that you will need to give the web user access to /bin/bash so the pygmentize library will work. Is that true?

I am very hesitant to give a web user access to bash for safety reasons

Hello,

I've just migrated and upgraded our company's MediaWiki to a new server. Windows Server 2012 R2, Apache 2.4.33 64bit + PHP 7.1.19 64bit + Python 3.6 64bit, MediaWiki 1.31.0.

Everything works except the SyntaxHighlight extension. The error message:

Notice:  Failed to invoke Pygments: 'C:\Python36\Scripts\pygmentize.exe" "-l" "css" "-f" "html" "-O" "cssclass' is not recognized as an internal or external command, operable program or batch file.

[Called from SyntaxHighlight::highlight in C:\Apache24\htdocs\wiki\extensions\SyntaxHighlight_GeSHi\includes\SyntaxHighlight.php at line 336] in C:\Apache24\htdocs\wiki\includes\debug\MWDebug.php on line 309

The PATH is set correctly, and from a sample test php file I can call Pygments, and it works:

<?php

$output = shell_exec('C:\Python36\Scripts\pygmentize.exe -l php -f html -O cssclass C:\Apache24\htdocs\test2.php');

echo "$output2";

?>

I have tried several options, like using the exe: $wgPygmentizePath = "C:\\Python36\\Scripts\\pygmentize.exe"; or the bytecode from the cgi directory:

$wgPygmentizePath = "C:\\Apache24\\cgi-bin\\pygmentize.pyc"; but nothing helped. Please advise how to debug further this problem.

I'm having the same problem, there are some system config differences (for instance, I run MediaWiki 1.32 master on XAMPP with PHP 7.2.2), but having the same identical error. Having two versions of Python in two different directories, I tried both of them (2.7 and 3.3), changed wgPygmentizePath to all possible combinations, set full user rights on pygmentize, added everything to PATH, but I'm still getting the same error. And, as Glanthor Reviol, I can indeed execute it from wiki folder using windows' cmd and also directly from SyntaxHighlight using exec. I really can't think of something more to do, sounds like there's some specific problem with Windows. BTW, SyntaxHighlight used to work correctly with an old version (can't recall which one) of the extension itself and MediaWiki, using Python 2.7.

I seem to have the same problem on a MediaWiki 1.31.0, with both Python27 and Python37.

I read somwhere that there were som changes in Shell::command from MediaWiki 1.31.0 - and I traced the problem down to somwhere in that area, but could not find a fix for it.

Hope someone can find a fix for this problem.

I wrote a short and shallow explanation on phabricator, see https://phabricator.wikimedia.org/T199989. Basically, I fear that this might be a very upstream problem, i.e. a PHP bug which won't be fixed (a couple of links on the task) since Windows doesn't actually provide the needed logic. More specifically, the problem as far as I understood it should be that on Windows you cannot set streams as not blocking, which then produces a conflict within shell pipes, entering in a vicious circle. If this is true, there's probably no easy or clean solution to this problem: we could only solve it with a workaround or by refactoring the shell code. But I guess we should really find a solution, since this problem affects a core feature (Shell) on a whole OS.

Hello everyone.

I have the same problem on my side. SyntaxHighlight isn't working since 1.32 update.

I understand the problem with PHP, sream problem etc but I'm looking for a solution, a workaround and since 2 months, I'm unable to find such solution.

Does somebody have a miracle for me ?

Regards

Personally, I can't find one. Actually, I'm not completely sure about the cause, too. Any update will come on Phabricator, anyway.

Thanks for reply.

I am afraid that there is no updates on phabricator too ^^

Wait & see...

Any news? I have the same problem

I want to add that I have this problem as well, on the latest 1.33, latest Syntax_Highlighter from the repo, etc. I've fought with it for hours. Has anyone successfully used pygment on Windows? It seems like it'd almost be easier, at this point, just to go back to a version that worked, and figure out how to get 'json' support, which is what spawned this whole upgrade for me.

To add to my last comment, after upgrading to mediawiki-1.33.0, PHP 7.1.1, on Windows 7, and trying every bloody combination suggested in this thread (including editing the Syn*.php file, adding handler to httpd.conf and just on and on), there is absolutely no combination that 'works'. The closest is specifying the exe in a path AND editing the 'fixed' Rory solution below, but that only half works. Some items are colorized, but others are flat out hidden, with no error (if there's a highlight or lines tag). This is pathetic. I've wasted almost a whole day messing with this, and it just frustrates me I can't even downgrade now. (Older versions, pre-pygmentize, show no color at all?!!!).

The HightlightJs and Highlight-Integrate extensions don't seem to support highlighting code lines, or displaying line numbers.

Is this really that hard? Can anyone come up with an actual, fully working install for this bloody thing using MW1.33.0+ with xammp?

The problem has not been solved yet......I'm try to use syntaxhighlight, but no matter how the test is unsuccessful. Finally I found this article and I won't try anymore until the problem is solved.

OS: Windows 10 Professional Edition

XAMPP: Version 7.3.11 with Apache 2.4.41 and PHP 7.3.11 (VC15 X86 64bit thread safe) + PEAR

MySQL: Version 5.7.21

MediaWiki: Version 1.33.1

Same issue here (Windows Server, IIS8.5, PHP7.4, Python 3.8.2, mw 1.34 - see Topic:Vibz8ak3wnpu9mv8 for more details

just solved (MW 3.4 IIS on Windows Server) it by

• editing SyntaxHighlight.php as shown here
• 'pip install pygment' manually

• setting $wgPygmentizePath='C:/Python/Scripts/pygmentize.exe';

               

History says that access to the skin is required and in the current version, this need is not met !!!

This extension requires access to the shell.

@Jdforrester (WMF)

This is in case I can not use this plugin without proc_open changes???

Fatal error: Uncaught ExtensionDependencyError: SyntaxHighlight requires "shell" ability: Unable to run external programs, proc_open() is disabled in

Sorry, I don't know. @ToBeFree declared that they had "got the extension working without having used the shell for it once" , but I don't understand how. Maybe they can advise.

As far as I understand, ToBeFree managed to install the extension without using the shell. This doesn’t mean that it doesn’t want to run external programs at runtime. (It apparently wants.)

Ah, right. Yeah, I think we should add the note back.

thanks for the ping – perhaps the wording could be changed to prevent such confusion. To "require access to the shell" can mean "the extension must be able to run shell programs using proc_open()" or "the extension can only be run/maintained/installed by someone with shell access". I interpreted it as the latter due to the chmod instruction "In Linux, set execute permissions for the pygmentize binary".

So do you confirm that this extension should have proc_open access?

Hello

I use this template:

<div style="overflow-x: auto;">
<div style="padding: 8px 20px;margin-bottom: 25px; text-align: {{{align}}};direction: {{{ltr}}}; clear: both; box-shadow: none!important; overflow-x: auto; background-color: none;"><h3>{{{title|}}}</h3>
{{{title2|}}}
 <!--[[Image:Crystal Clear action apply.png|25px]]-->
{{#ifeq:{{{showlinenumbers|n}}}|y
|{{#tag:syntaxhighlight|{{{source|{{{1|}}}}}}|lang="{{{lang|html}}}"|highlight="{{{highlight|}}}"|enclose="div"|line="y"|style="border-left: 4px solid #4CAF50; background: none;"}}
|{{#tag:syntaxhighlight|{{{source|{{{1|}}}}}}|lang="{{{lang|html}}}"|highlight="{{{highlight|}}}"|enclose="div"|style="border-left: 4px solid #4CAF50; background: none;"}}
}}{{سخ}}
<div style="text-align: right; clear: both;">{{Clickable button 2|{{{textbutton|مشاهدهٔ نتیجه »}}}|class=mw-ui-{{{colorbutton|progressive}}}|url=https://wikicod.ir/tryit/?file=/{{{dars|}}}/{{{link|}}}&type={{{output|client}}}}}
</div>
</div>
</div>
<noinclude>{{Documentation}}</noinclude>

But after the article in the source section, for example:

<!--This is a comment. Comments are not displayed in the browser-->

<p>This is a paragraph.</p>

I put, the output shows only this:

<p>This is a paragraph.</p>

And does not show this part:

<!--This is a comment. Comments are not displayed in the browser-->

What is the problem?

Thank you for your help

Thanks

Two issues at first glance:

1. Please omit quotes when providing wiki parameters, they will be put around again: |lang={{{lang|text}}}
2. If the comment is really part of the source input, it will be displayed. However, I guess it will we removed already when you provide your input. You will need some smart method for encoding to keep it alive.

|{{{lang|html}}}

Thank you for your reply

I " picked the quote

Did not matter and re-show comments in the output

If possible explain more

Sincerely

• You are feeding your template from somewhere.
• When you are providing parameter values, any comment will be removed and the template or module code will never ever know about that comment.
• When you are feeding your template code, it will not know about any comment.
• You need to escape e.g. the leading < but resolve that before further processing. Not easy and not possible to explain here since quite tricky.

_{(edit conflict)} When you use the {{#tag: form, the wikitext is preprocessed before passing it to SyntaxHighlight. This includes substituting template parameters, which you want to take advantage of, but also includes removing HTML-style comments (just like they are removed from the HTML output). I found that <nowiki/> can be used to prevent them from being processed as comments:

{{Template|source=<!-<nowiki/>-This is a comment. Comments are not displayed in the browser-->

<p>This is a paragraph.</p>}}

(I’d probably “break” the closing --> as well: while it’s not needed for this example to work, not “breaking” it can lead to unexpected results when one wants to comment out this part of the article.)

Hello

Thank you for your guidance

My problem was solved

Thanks & Regards

@Dinoguy1000: <source> is deprecated, but not removed yet. I think it should not be removed from the documentation as long as it works—feel free to add a huge warning advising users not to use it, but not mentioning it while it’s still used on all over Wikimedia creates even larger confusion than what was before. For example a user may wonder why a page ends up in Category:Pages with syntax highlighting errors when there’s no <syntaxhighlight> on the page at all (only <source> tags); if <source> is also mentioned, it’s more clear the two are synonyms (at this time).

It is still mentioned, at the end of the "Usage" section. I don't believe, given its current status and pending removal, that more than that mention is warranted (though the tracking category that gets added when it's used should be included).

Here's more general details on the removal: https://phabricator.wikimedia.org/T237267

Please don't remove the "source" alias, before letting the extension generate a tracking category, and allow tracked pages to be updated.

The problem is that "source" is much easier for most wiki users, that have difficulties to properly get the orthograph "syntaxhighlight" (or is it "syntaxhilite"?) without typos (and what is the correctc capitalisation between the two words?) May prefer "source", which is evident (don't be limtied to the fact that internally you use some highlighting library like GeShi or other, that uses the term "highlight" or "hilite"). So there are in fact MORE needed aliases with the "syntaxhighlight" term (which is very errorpone), but none for "source", which is definitely the best choice and should be the canonical name of the tag in MediaWiki ("syntaxhighlight" is a legacy).

If "source" does not make evidence that the source code will be highlighted (or because it has another use in HTML5) or this is indicating source code, then use "beautify" or "colorcode", or "programming" which also makes more evident that the parameter "lang" is not specifying a humane language (BCP47) but a programming language or just "syntax" (drop the term "highlight" kept only as a legacy alias); or support it only as an extended attribute of the "code", "tt", or "pre" elements (but then change "lang" parameter name into "syntax" or "as" for the XML-like syntax of this extension):

<code #tag:syntax:as="C">some code in C</code>

{{#tag:syntax|some code in C|as=C}}

(I wonder if MediaWiki supports hooks for custom attributes in the HTML/XML-like syntax, instead of just custom elements).

And note that there's still the problem of the unsafe embedding for the source code (see the other message about HTML-unescaping, which is needed to avoid havoc with the wiki parser, falsely interpreting the source code and modifying it)

@Verdy p: The extension is already generating a tracking category on pages using the old source tags.

@Dinoguy1000: I think it’s much more confusing for readers for the time being; especially for those who get there using a section link like Extension:SyntaxHighlight#Syntax highlighting error category. I don’t think it makes sense to remove references for it everywhere (I’m speaking about such documentation pages, not actual usage, of course). The deprecation notice could be made more prominent (like adding a colored box on the top of the page), but that’s all. And if you think keeping them will make replacement bots more complicated, you’re wrong—they should be made more sensitive anyway to stop doing replacements like the one I reverted here—<source> is in nowiki, and has nothing to do with the SyntaxHighlight extension at all…

@Verdy_p: See phab:T251116; further discussion about whether removing <source> is a good idea or whether there should be another short name is off-topic here.

Note that the addition of the tracking category is very recent (I did not see it until very recently, this was not announced and I do not see it on every wiki, so the "deprecation" was decided by developers even if this was annouvned as their "intent" a long time ago. But the change to use "syntaxhighlight" only was never voted, and it is a nightmare for most editors, so the prefered name chosen is really on topic: ir could have just been "syntax" or we could have just used a "namespaced attribute" to standard HTML "pre", "tt", "code" tags, like " ... ).

As well you kept the non-conforming use of the required lang="*" attribute (whose value is not the BCP 47 language tag, but a symbolic name for the programming language, but could also have been the type="*" attribute like for standard "script" elements)....

The reason for the deprecation of the "source" element is also very fuzzy/doubtful:

• we are actually NOT parsing HTML5 input, we just parse a MediaWiki syntax (that will then be used to *generate* HTML5) which has its own rules for acceptable elements/attributes and acceptable rules for self-closing tags, plus it adds its own private tags that are not HTML5 and the extra wiki-specific syntax for lists, tables, wikilinks and a few styles (bold, italic, monospaced blocks, and headings). There's absolutely no need for the Mediawiki text to be HTML5 compliant: it has never been compliant with HTML, and it will never be (unless we abandon the Mediawiki syntax completely for editing pages in pure HTML5, but then how will we even support templates or even this syntax coloring extension?).
• we still don't accept random HTML5 input, so we still don't support on input any newer semantic elements like "heading"/"section"/"footnote"/"summary"/..., or and not even some older but standard and non-dangerous useful elements from HTML4 like "colgroup" (wanted since long!), or "thead"/"tfoot"/"tbody"/"rowgroup"...
• The only convincing reason for deprecating the "source" tag is that because you want to add it to Mediawiki so that it will match the HTML5 semantic tag. But why only this one? Why we still don't have "colgroup"?

Clearly the deprecation anticipates a possible future change that has not even been prepared and fully designed, never been discussed (for its scope of application). For now there's clearly no need at all. And the replacement tag proposed is horrible to type for most people and its current full name was not approved by anyone.

Such deprecation just annoys people needlessly (just like the previous deprecation of self-closing tags when tags may have no content (like <s id="x"/> like in the XHTML standard syntax, which is shorter and simpler than <s id="x"></s> to place a content-less anchor on a page: you've created unnecessary work on every reviewers to maintain lot of pages without any added benefit)

@Tacsipacsi: mentioning source without including the fact that it is deprecated and soon to be removed, at every mention, is unacceptable. It doesn't matter if one mention of it does specify that, if any others do not, especially if they're on a completely different part of the page. This is plainly and clearly demonstrated by the fact that the page has listed source as "discouraged" since May 2009, and yet, over a decade later, it's still used so widely as to require a full and proper deprecation period, including tracking category and bot runs to clean up.

Adding an ambox at the top of the page isn't going to do much either; Wikipedia's extensive use of them for article issues has successfully trained people to ignore them, exactly the same as banner ads from past decades did for getting people to ignore any banner-style image elements on web pages.

Any further mentions of source that are added to this page must be clearly in the context of its deprecation and eventual removal, so there is absolutely no ambiguity about whether it should be used, especially for people who arrive at the page via a deep-link directly to a section.

So what about writing

<syntaxhighlight> or (deprecated) <source>

(listing <source> second everywhere; probably without the parentheses)? That would keep the references, but it would make clear deprecation is now a real thing. (And I think a banner is worth adding in addition to other notices, including this parenthesized one—probably not everyone will notice it, but it still helps the people who do.)

By the way, I don’t interpret that 2009 wording (which was used till this April) that the short name was discouraged. This wording states that the long name “may help avoid conflicts if your source code itself contains <source>”—most source codes don’t contain <source>, in which case (in my interpretation) this sentence doesn’t encourage either tag name over the other. (This is why I happily used the short version for years even though I knew about this part of the documentation.)

I just don't see the benefit on including a mention of source with every mention of syntaxhighlight. Other deprecated markup doesn't get this treatment. If the deprecated tag is mentioned every time the current tag is as well, it makes it sound like the documentation doesn't really think the tag should be deprecated, and encourages confusion and doubt in readers. To flip this on its head, the documentation used source largely without mention of syntaxhighlight for over a decade, and to my knowledge, no one complained about it. Why is it suddenly so important that we do the opposite?

The full wording is "In older versions [...] the extension used [...] This is still supported, but [...]"; this has always read to me as a clear discouragement of using source.

I don’t know what happened for over a decade and I don’t know when this over a decade was (my diff link above shows a state where the two were already equally represented). I’m not aware of any other deprecated markup either. I just see the current changes and would like to keep as much information as possible as long as it’s relevant (which is until it’s finally and fully decommissioned). But I see I won’t succeed…

IP is trying to do something that SH doesn't have the functionality to do due to the nature of the implementation.

Hi,

I'm using several nested templates within SyntaxHighlight v2.0 on MediaWiki v.1.33.3 to try to create a shallow documentation generator. So far, I have an template feeding inputs into another template that calls syntaxhighlight. My problem is this: I'm trying to create links within a syntaxhighlighted code block that allows for external links, by calling another template within it. Is this possible within this extension? I've been trying to play around with it for a few hours now, to no avail.

Thanks.

Basically, you can do that and it is done already on wikis by template transclusion stack.

You will need syntax {{#tag:syntaxhighlight|1=code|lang=xyz}} and {{!}} to escape pipes. Good luck if double }} occur in your program code at the same level. Content and sections of other pages may be transcluded.

Basically the feature is dedicated to illustrate encyclopedic articles by static code snippets. A very sophisticated dynamic transclusion system might require advanced knowledge of wiki programming.

Thanks for the response. My knowledge of MediaWiki's inner mechanics is very superficial, I mostly use templates and the odd Lua script. I would link my existing template if I didn't trip the abuse filter. I should have been a bit more specific on the implementation - rereading it now, it's kind of vague (or I'm not understanding how to implement what I want)

Just to make sure I understand, given the following (omitting more complicated parts to just get to the crux of what I'm trying to do):

• a "link" template that takes an input and returns something like [whateverurlhere?search={{{1}}} {{{1}}}]

• a template that has feeds one or more of the above to syntaxhighlight

would create the following, given an input of test

{{#tag:syntaxhighlight|[whateverurlhere?search=test test]|lang=c#}}

How would I escape the provided first parameter to be processed as wikitext, instead of appearing verbatim? Apologies if I'm being dense, or going about this the wrong way.

If you are using Lua anyway, I would recommend to use frame:callParserFunction( "syntaxhighlight", { [1] = "printf(%"Hello world%")", lang = "c#" } )

For template syntax, the part in between will be evaluated wrt double or triple brackets.

• You might try to escape by <nowiki> if brackets indicate a URL.
• However, that URL would not be linked, since it has single brackets and those are passed as parameter 1= to syntaxhighlight and that is hiding it from being evaluated.
• You may put other wikitext as 1=code parameter, e.g. template transclusions or transcluding sections from other pages.
• The major problem is if there is a pipe symbol | occurring in your c# code, e.g. bitwise or / || or. Those need to be escaped by {{!}} otherwise it will terminate that source code parameter if present at the same level like the #tag: function. If transcluded from somewhere else, as parameter value or result of any other thing, it does no harm.
• Value of {{{1}}} will be shown as current value (being replaced by current content).
• The inner parameter value of 1=code is evaluated first, until next | or }} visible at the same level, then #tag: will be evaluated with that string.
• Pipes within complete template transclusion or complete internal link are assigned to those wrapping elements, they do not influence #tag: evaluation.

I don't think you understand what the IP is asking; they want to post-process the output of <syntaxhighlight> to, for example, add links to it, which I don't think there is any supported way of doing that, although I can think of a very hacky way.

Just to provide context before I'm able to start hacking away at it - here is a snippet of the template.

{{Code/Pre|lang={{{lang|}}}|{{#if:{{{modifier|}}}|{{{modifier|}}} |}}{{#if:{{{keywords|}}}|{{{keywords|}}} |}}{{#if:{{{type|class}}}|{{API/ref|{{{type|class}}}}}{{#if:{{{generic|}}}|| |}}|}}{{#if:{{{generic|}}}|<{{{generic|}}}> |}}{{#if:{{{name|}}}|{{{name|}}} |}}{{#if:{{{parent|}}}|: {{API/ref|{{{parent|}}}}}|}}{{#if:{{{parentGeneric|}}}|| |}}

Aside from potential refactoring and code shortening (trying to get this to work before anything else), Code/pre is a template containing the SwitchHighlight call and API/ref is a template containing a switch statement with a link to a search query for external docs, or an internal link to a page (which is where I currently am in trying to add hyperlinks using wikitext). I'm trying to get the type identifiers to be searchable for faster navigation between documentation pages. All the nowiki tags are for syntax purposes and will be replaced later. Based on what I understood from PerfektesChaos, the URL can't be linked in the way I am attempting to do it as it all becomes evaluated by a template.

The URL wouldn’t be clickable since it is displayed as text, and whatever you are feeding into syntaxhighlight is shown as source code rather than active element. That is the rationale behind syntaxhighlight. If you want to get a clickable URL you need to place an external link out of the syntaxhighlight region, e.g. using a table.

An entirely different approach would be a JavaScript gadget which is to be started by some miracle when that page is displayed, and which could read any element content, find some hidden URL and equip any element with a listener for a click.

Alright. Thanks for the help, I'll look for another avenue to try to get this functionality implemented.