I'm developing an extension and I'm using the resource loader to load my stylesheet and script files. Now that I've developed on multiple branches for a while, I can see that the resource loader responses contain my stylesheet multiple times: Once in its current version and once in an old version or a version that may come from a different branch of my project. How can I purge the cache of the resource loader to remove that second sheet from the response?
ResourceLoader loads old/wrong stylesheet
Is it possible to load resources from a different URL
I'm trying to load all the static resources (js, css and image files) from a separate URL (a CDN actually), and using $wgStylePath has the images working just fine, but all the css and js loaded through the ResourceLoader still comes from the original domain.
Is it possible (now in 1.17.0 or in the future) to make these files load from a different URL?
Thanks for the answer.
From my testing, setting $wgLoadScript also means the CDN has to be able to execute PHP (load.php), so in other words it has to be a full-on webserver, not just a static file server (as 99% of all CDNs are). The fact that it also needs to access the db and LocalSettings.php means it probably has to be on the same subnet, etc.
I'm starting to wonder about this resourceloader - it goes to a lot of trouble to minify a ton of css and js on every single request, using php to do so, and hitting the main web server (in this case, memory-hungry Apache).
Wouldn't it be better to minify as many of those files as possible beforehand and just serve them as they are (and then they can also come from a static-file-server CDN somewhere else on the internet (ie. entirely different subnet, etc.) ?
Is there any way to achieve this?
Well depending on your set up, it obviously is not intended to go through all minification, combination, embedding, localization etc.
Because of various unique combinations in the URL such as the version timestamps that ResourceLoader includes in requests to load.php, these requests are highly cachaeble!
Because of the wide range of scenarios that MediaWiki has to support to be able to scale to a platform like Wikipedia, it needs access to the filesystem and the database. (Consider localization into over 300 languages including right-to-left and non-latin languages, support for user-preferences that allow users to change the skin on a per-user base (and thus changing the modules to be loaded), gadgets (enabling custom scripts), scripts from extensions loaded only under certain conditions)
The way Wikipedia has this set up is by using a reverse-proxy like Squid or Varnish that serve a static cache of all resources and, because of the timestamps, it can cache these requests "for ever". Whenever a module has changes (which won't happen for an average MediaWiki install unless you upgrade MediaWiki, change configuration files or install/uninstall extensions), it will use the new timestamp in the request for that module, thus changing the url to load.php. Then the bits-server will only initiate MediaWiki if there is no static cache for the url.
According to the stats as of August 2011, bits.wikimedia.org/../load.php has a cache hit ratio of 98.2%. For all those requests MediaWiki was not initialized, no database connection etc.
However a simple static-file-server CDN does not suffice. It's not impossible to use a static-file-server CDN, but no implementation for that was made as Wikipedia uses Varnish as a reverse-proxy for cache. You could contact User:Catrope if you're interested in building in support for a static-file-server CDN (e.g. somehow upload static files through FTP or something to that CDN when new ones need to be generated and embed or 301-redirect to those urls directly.
Thanks for the great explanation Krinkle, I understand the situation much better now. The timestamp-on-the-resourceLoader-URLs makes a lot of sense for a reverse-proxy cache like Varnish
I had always intended to put Varnish in front of MediaWiki/Apache at some point, so I'll focus my efforts there instead of trying to offload more stuff into a static file-serving CDN.
You meant 99.8% ;) . Also, all Wikimedia wikis run off four web servers with a total of nine Varnish servers (in two different data centers) in front of them. And those four Apaches barely get any load; their CPU usage is like 10% so in theory ResourceLoader could run off a single Apache. See also our OSCON presentation, and the slide with these numbers. --Catrope 19:14, 18 October 2011 (UTC)
From your description, we could setup anything for load.php to goto the CDN. If the CDN uses our server as the origin (and includes the query params as part of the caching) then the very first request hits our server and what it produces is imminently cachable by the CDN (forever).
So why couldnt we use a CDN (with no Varnish, no access to the DB etc.) with load.php?
Nobody is saying you can't use a CDN. If you have a CDN that works the way you just described (automatically pulling uncached urls from your origin web server, including query parameters), then by all means, go for it :)
If you'd like to know more about Wikipedia's configuration feel free to join IRC.on
Suggested feature: What links here?
I'm happy with your exercise and I'm really looking forward to see the delivery. If it is not too late, I have one suggestion:
- The address may be enclosed in double brackets.
- Reason: the code
.load('[[User:Foo/helpful.js]]')generates entries on What links here?
- Until now it has been common practice to ask users to add a comment
// [[User:Foo/helpful.js]]in order to get knowledge about other users.
- That might be helpful to be aware of the number of other people influenced by changes, if the script is developed for common usage. It might be necessary to Talk: to others if behaviour of script changes significantly.
- To avoid caching problems after bugfixes, friends were to be informed and asked to clear the cache -- I understood that the RevID will be appended to the URL in the future.
- Stripping off enclosing brackets seems to me quite easy.
Thanks for taking the time to write up this feedback. It's most welcome!
To get right to the subject, right now ResourceLoader is not and can not be used to load user scripts. User scripts are a (powerful) invention by the community that has shown great results and makes sharing really easy, however they are currently not natively supported by the MediaWiki software. Scripts are stored on (technically article-like) normal wiki pages, and the method to load the raw wikitext (
action=raw) is used to load it without rendering and thus, when the url is inserted into an HTML
To make this easier to do for the community, the developers have added the popular
importScript() function to the core software so that it doesn't have be created from scratch on each wiki.
Also Extension:Gadgets was created which makes writing scripts even easier and is ultimately how scripts should be managed. Since Gadgets are built as an extension and connect into the right MediaWiki hooks, all the native features are available for Gadgets. That includes
- efficient loading of scripts and styles (combined and compacted into a small package, re-uses this cached package whenever possible, based on:
- automatisch cache clearing (cache identified with a unique module name and timestamp of when it was last modified)
- (and, soon also:) internationalization / translation (Gadgets 2.0 is on it's way, as part of the RL2 project).
- "usage" tracking (through the preferences). See w:Wikipedia:Database_reports/User_preferences#Gadgets for example.
- loading other modules as part of your module ('dependencies')
- and more..
But what it can't do is create gadgets on a per-user level (only on a per-site level). This is a much requested feature, but currently not possible yet in an efficient and scalable way (requires a few other changes as well). But I expect that this will be implemented as part of Gadgets 3.0.
So to get back at your suggestion. Although you can use
mw.loader.load() right now to load a url of choice (including, but not limited to, urls to the current wiki where your user script is stored), that is just loading it as a url (just like
importScriptURI does), that is not really a feature of ResourceLoader. And as such, since user scripts technically don't exist as a feature, ResourceLoader can't create an alias for it in the existing system.
Since the environment in which user scripts are written has changed a lot in the past few months/years, I don't think it is a good idea to make any more changes that require (or motivate) a lot of changes (such as changing to format to
.load('[[Page name]]'). Instead focus on keeping the environment stable, while developers work on a solution to migrate away from user scripts (the current scripts will continue to work, don't worry)... to a solution that allows the same (and more) functionality in a better way that
- does integrate with ResourceLoader
- allows creation and modification of gadgets on a user-level
- easy sharing with other users (and other wikis)
..in other words: Gadgets 3.0, in which users can create their own fully-featured Gadgets!
So for now I'd recommend to stick with what already users know, use and will work fine:
in the cache (objectcache table in MySQL) with expiration date 2038. I guess such entries are never hit (as they contain timestamps, which are often unique for unregistered users) and since of the long expiration date they fill the cache -- in a couple of weeks it grew to 690MB.
What's the best way to circumvent this problem?
Ouch, that's a problem. The cache keys in question probably contain the word 'minify' and an md5 hash, right? If so, it's the JS minifier cache, which caches the result of JS minification, and because the same original code always results in the same minified code, it's cached forever as it can never change.
I'm afraid that for now you'll have to periodically purge the objectcache table of keys with this pattern; I've filed a bug about it, and once it's fixed, there will be a patch that you can apply that addresses this behavior. But it may take up to a few months for this patch to get written, unfortunately. An alternative workaround is to use memcached instead of the DB cache; if you have enough users that you're producing >30MB/day in cache cruft, that might be a good idea anyway. memcached uses a fixed maximum size for its cache, and when the cache fills up, it will start throwing out (evicting) old entries, with the least recently used (LRU) entries getting evicted first. This is what we use on our production wikis (Wikipedia and sister sites) as well.
Thanks for the report, this is a quite nasty issue that we hadn't heard about before. We never experienced it ourselves because we use memcached, which is not affected by this problem as it automatically removes unused entries if the cache grows too large.
Thanks for the fast reply! Yes, the keys contain "minify". We will consider switching to memcached, but for now I've just hacked ResourceLoader.php -- instead of
$cache->set( $key, $result);
we now use
$cache->set( $key, $result, time()+86400);
I guess this hack would be a bad thing to do on a large wiki, but it should be OK for our medium sized one (~5k pageviews per day).
That should be fine, it'll expire the minifier cache after a day that way. Even on a large wiki, that would just be a minor annoyance, not a real problem. In fact, we might actually end up with something similar to that as a fix.
I was wondering what the bugId number was for this issue. We too have been having this issue (mw 1.18) on our corporate site where the objectcache table grows past a a few GB (at one point it was 28gb). After a while I just go in there and run
TRUNCATE TABLE objectcache; to fix it up, however this is not the correct solution.
I recently checked the latest build, 1.20.4 to see if something like the suggested fix was placed in ResourceLoader.php, but did not see it. I was hoping to find the modify for
$cache->set( $key, $result ); to something like,
$cache->set( $key, $result, time()+86400);.
We love using MW here for our development and support personnel. This bug is our biggest show-stopper and want to upgrade to the latest version, but i want to make sure before I take our site down for maintenance.
Thank you! and keep up the good work
BOM characters need stripping
I've noticed some issues loading resources where source files might have had byte order markers at the start \uFEFF
Not sure how I got them there but I had to use a Notepad++ to strip them for resourceloader to work correctly, might be worth checking for such characters before emebding in resource script object.
Is a script or stylesheet assembled by ResourceLoader unambiguously defined by its URI, regardless of the user for whom it is served?
In other words, can on safely cache such resources with a caching proxy?
Hi Alex, this is not an authoritative answer, but AFAIK the answer is yes, and I certainly do so on my wiki (I use Varnish). The RL module are language dependant, but not exactly user-dependant - user information, such as the config variable wgUserName, are injected into the HTML document itself, which should not be cached for logged-in users.
Wikimedia's own configuration files for Varnish can be found here, if you want to take a look: https://github.com/wikimedia/operations-puppet/tree/production/modules/varnish/templates/vcl
I'm using MW 1.18.1, and all requests going to MW have "Accept-Encoding gzip, deflate".
The following examples are the only two Resource Loader files being compressed:
Which are both text/css.
The following examples are not being compressed:
I have verified zlib is correctly installed in Apache. I am NOT using $wgDisableOutputCompression
Thanks, -Dangrec 00:40, 14 February 2012 (UTC)
I am seeing the same thing in MW 1.21 (i found these because Google analytics page speed analysis complained about them). Any suggestions? Thanks a lot.
Found the solution: edit /etc/php.ini and set zlib.output_compression to "On". Then test with http://www.whatsmyip.org/http-compression-test/ or Google Analytics.
Thanks a lot. This helped me too. Google complained about 5 load.php statements not being compressed (as far as I can see 4 scripts and 1 style). After setting zlib.output_compression to "On" all is compressed :)
is there a way to prefix all css?
the css of wikipedia is messing with other css of the skin and website. i want to limit it to only the rectangle the wiki is inside.
i want to do like:
and all the css that it will return will be prefixed
Using LESS with @media and IE8
IE8 doesn't support @media queries. Is there any way to make rendered LESS work with IE8 conditionals (as in include the file again without the @media for IE8, but only IE8)?
I'm simply accepting the penalty for IE8 users (upgrade, gosh darn it!), and using Respond to load @media rules - by conditionally including Respond itself.
After an edit without my magic word, my script JS disappears
I'm developing an extension with ResourceLoader. I load the script JS in the render of my magic word.
It works but when I change a page in my Wiki without my magic word, ResourceLoader forget my script JS. I have to purge the page with my magic word to see my script in ResourceLoader.
Have I forgotten something ? The "$wgOut->addModules(" is not in the good function ?