Talk:Requests for comment/MediaWiki libraries

Composer managed contents
In my implementation for Requests for comment/Structured logging in I created a   directory that is managed using Composer. Would getting that implementation merged be a reasonable resolution to this RFC? The concept of this form of library manage was vetted and approved by Tim in his review of that RFC. --BDavis (WMF) (talk) 22:12, 23 April 2014 (UTC)


 * Yep, that solves it.--Ryan lane (talk) 22:24, 23 April 2014 (UTC)

Discussion on April 29
We'll be discussing this RfC Tuesday 29 April, 2200-2300 UTC. Sharihareswara (WMF) (talk) 17:40, 25 April 2014 (UTC)
 * Summary of discussion: We just need to merge "Add Composer managed libraries" https://gerrit.wikimedia.org/r/#/c/119939/ in order to complete part I of this. Then Part II is actually moving everything else into that directory, and also making it reasonable to move some extensions into that directory. Then we started talking about the Composer RfC. :-) Log:


 * 22:02:57 OK, now on to the first topic today, Ryan_Lane's MediaWiki libraries RfC. Then we'll move on to parent5446's RfC on 3rd-party components
 * 22:03:01 #topic MediaWiki libraries
 * 22:03:12 #link https://www.mediawiki.org/wiki/Requests_for_comment/MediaWiki_libraries Ryan Lane's proposal (better handling and versioning libraries that are MW extensions)
 * 22:03:12 * marktraceur frows at Book Management
 * 22:03:36 #chair sumanah brion andrewbogott TimStarling
 * 22:03:36  Current chairs: TimStarling andrewbogott brion sumanah
 * 22:03:41 So I have a question about this RfC. It mentions LdapAuthentication as an example. What library in LdapAuthentication would we be packaging with core?
 * 22:03:46 #info See https://www.mediawiki.org/wiki/Talk:Requests_for_comment/MediaWiki_libraries - we just need to merge "Add Composer managed libraries" https://gerrit.wikimedia.org/r/#/c/119939/ in order to complete part I of this.
 * 22:03:47 #info Then Part II is actually moving everything else into that directory, and also making it reasonable to move some extensions into that directory.
 * 22:04:04 B/c I understand the RFC in the context of the Gerrit patch
 * 22:04:19 Libraries that are being used in core will need to be managed somehow
 * 22:04:23  parent5446: a number of extension use LDAP as a library
 * 22:04:31 i’m not sure i understand this rfc; is the idea to package specific libraries in mediawiki core?
 * 22:04:39  I'd split the necessary functionality out of the LDAP extension into a common library
 * 22:04:44 or to provide a general place to put libraries?
 * 22:04:47 ...how? Are there extensions that use LDAP other than LdapAuthentication?
 * 22:04:53  I think the idea is to create a separate class of extensions that you put in libs/ instead of extensions/
 * 22:05:05  TimStarling: yep
 * 22:05:13 first question: why?
 * 22:05:25 second question: what needs to be different other than the filename ‘extensions’ -> ‘libs’?
 * 22:05:36  because there's now a lot of extensions being used as libraries and they aren't being versioned with mediawiki
 * 22:05:40 OK, but the gerrit patch makes it look like we'd be packaging them with core. So it makes it sound like we'd be packaging libraries within core that are not actually used in core.
 * 22:05:45 third question: does that patch adding a libs directory help or hinder this rfc’s recommendations?
 * 22:05:53 (the patchset is by bd808 in case he wants to comment)
 * 22:05:57  most of these could be bundled with core
 * 22:06:11  we have includes/libs already for things that are bundled
 * 22:06:13  and should definitely be versioned with releases
 * 22:06:28  TimStarling: that's for things that core depends on, but not for things extensions may need to depend on
 * 22:06:45 so is this directory’s contents something is fixed, that extensions can depend on always being present?
 * 22:06:55 or is it something extensible, that extensions can rely on having a way to install things there?
 * 22:07:09 I don't really think it's a good idea to package extension libraries as part of core if core is not using it.
 * 22:07:23 The best solution for Ryan's RFC would probably be enabling the full use of Composer (or a similar system) with wm-core
 * 22:07:27 Maybe an extension needs a different library version, and can't use it because core decided to package it
 * 22:07:48 * sumanah invited Jeroen to discuss composer-y stuff with us in this chat but - ah, there's JeroenDeDauw!
 * 22:08:03  Just saw calendar notification now
 * 22:08:07  Anout to head off
 * 22:08:14  so, maybe I should describe my issue, rather than this specific solution?
 * 22:08:18 JeroenDeDauw: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140429.txt starting at 2200 in case you wanna see our talk till now
 * 22:08:26 s/talk/chatter/
 * 22:08:29  with composer, I'd like to know whether you think extensions should be in mediawiki/* or mediawiki/extensions/*
 * 22:08:32 Ryan_Lane: yes please :D
 * 22:08:42  in the package names
 * 22:08:44 #info the Composer RfC: https://www.mediawiki.org/wiki/Requests_for_comment/Extension_management_with_Composer
 * 22:08:57  so, we have extensions. those extensions for a long time only had a dependency on a specific mediawiki version
 * 22:09:02 <TimStarling> because JeroenDeDauw put a few in mediawiki/*, but it seems a bit weird to me
 * 22:09:14 <Ryan_Lane> now we have extensions that only exist as libraries for use by extensions
 * 22:09:14 <JeroenDeDauw> TimStarling: vendor/package
 * 22:09:18 <JeroenDeDauw> TimStarling: two levels
 * 22:09:24 (many extensions require external dependencies as well…)
 * 22:09:52 extensions can also be hooks onto other extensions -- but that doesn't make the 'parent' extension a library
 * 22:11:03 * aude waves
 * 22:11:09 <TimStarling> hmm
 * 22:11:12 this seems to be mostly an argument about our release process... in theory, everything in a release is stably versioned and has release branches for backports
 * 22:11:43 <Ryan_Lane> well, it's more an issue that we have no reasonable way to specify extension compatibility
 * 22:12:00 <Ryan_Lane> and now we need to specify extension compatibility with mediawiki and with other library extensions
 * 22:12:02 I mean, this is basically what composer was made to do
 * 22:12:21 Ryan_Lane: don’t you just grab them from the same version branch?
 * 22:12:21 Except people don't want to use composer in core
 * 22:12:34 * sumanah sees parent5446's point. If this is a configuration/dependency management problem, maybe we should go with the thing that does that
 * 22:12:40 <Ryan_Lane> brion: very few people use version branches
 * 22:12:52 Ryan_Lane: how would this solve that?
 * 22:12:59 we're cutting the version branches automatically now
 * 22:13:14 <greg-g> mwalker: but everyone else still uses tarballs
 * 22:13:15 <Ryan_Lane> my proposal was to have the common libraries shipped with mediawiki and locked with the version
 * 22:13:22 …
 * 22:13:31 so something that’s common to a group of extensions, and has no relation to mediawiki core...
 * 22:13:34 … would be shipped with core?
 * 22:13:38 Shipping common libraries is kind of like a band-aid to the actual problem
 * 22:13:39 <TimStarling> I think it's fine to have extensions that depend on other extensions
 * 22:13:50 parent5446: I think we are open to using composer in core, but new need to figure out how to do that in a way that works for JeroenDeDauw's use case and mine (which is basically the same as Ryan_Lane's)
 * 22:14:07 <Ryan_Lane> the issue is that most of these library extensions probably should just be core
 * 22:14:09 <TimStarling> drupal has a DIY system for this
 * 22:14:18 <TimStarling> or of course we could use composer
 * 22:14:25 I thought the main problem was that people didn't like how it didn't interact properly with Git?
 * 22:14:29 <Ryan_Lane> if 20 extensions are using an extension, why isn't it a core feature?
 * 22:14:30 Emufarmers: Dantman: would love to have your input here as people who work on non-WMF MediaWiki installations
 * 22:14:32 <TimStarling> or support any other package management system, like PEAR or .deb or whatever
 * 22:14:50 Ryan_Lane: are those 20 extensions all specific and only used by a tiny minority of installations?
 * 22:14:54 greg-g, we bundle extensions into our release tarballs...
 * 22:14:55 or are they used all the time by most?
 * 22:15:05 <Ryan_Lane> to be fair, I'd be fine with composer, if it can be used reasonably
 * 22:15:23 Well many extensions have already been made to work as composer libraries
 * 22:15:24 <TimStarling> if a facility is widely used, I think it can be in core, you know that I am not one of the people who has a narrow view of what core should be
 * 22:15:25 <greg-g> mwalker: yeah. (/me doesn't want to derail, it was a minor point)
 * 22:15:27 it feels like the solution here is “a dependency-managing installation system for extensions"
 * 22:15:38 <Ryan_Lane> TimStarling: I'm not saying you are :)
 * 22:15:42 <TimStarling> but yes, we need extension dependencies for various reasons
 * 22:15:42 Which means we should check out https://www.mediawiki.org/wiki/Requests_for_comment/Extension_management_with_Composer
 * 22:15:51 and maybe expand it
 * 22:16:04 <Dantman> sumanah: I'm not 100% certain of the topic, or even whether "extensions as composer modules" is included in this or excluded.
 * 22:16:18 <TimStarling> not just for libraries -- some extensions hook others, for example
 * 22:16:28 <Ryan_Lane> yep
 * 22:16:41 <Ryan_Lane> it's something we've been punting on forever
 * 22:16:43 Dantman: sounds like we're talking about including it, maybe
 * 22:16:44 <TimStarling> some use RL modules from another extension
 * 22:17:09 Dantman: the topic is https://www.mediawiki.org/wiki/Requests_for_comment/MediaWiki_libraries Ryan Lane's proposal (better handling and versioning libraries that are, or are used by, MW extensions)
 * 22:17:22 <Ryan_Lane> so, let's drop this proposal and maybe talk about the composer suggestions?
 * 22:17:43 <Ryan_Lane> or is that not how this process works? heh
 * 22:17:52 Well if we need to itemize this to dependencies on individual RL modules and hooks, there's not really any system that handles that
 * 22:18:05 With composer it's either you depend on the entire extension or you don't
 * 22:18:12 <Dantman> Well this idea of extensions as composer modules has annoyed me over and over again.
 * 22:18:14 Unless you separate into further libraries
 * 22:18:15 Ryan_Lane: I'm fine with the process working like that, if you want to abandon your RfC and become a coauthor on the Composer one :)
 * 22:18:53 Extension management with composer would allow managing dependent libraries as well, for the extensions. My use-case of wanting an external library for core is slightly different, but could pervert the system by using a basically empty extension just to get the library management.
 * 22:19:00 andrewbogott: I guess one question for you as Ops representative is how Ops feels about Composer (although I assume if Ryan's fine with it then you would be too)
 * 22:19:02 <Ryan_Lane> wasn't that another RfC that was in the queue for today?
 * 22:19:21 Ryan_Lane: no, it wasn't; it is one that I wanted people to have fresh in their memory because I predicted it would interrelate
 * 22:19:24 <Ryan_Lane> the current implementation of composer doesn't fit with our deployment process
 * (And then we moved onto the Composer RfC proper.) Sharihareswara (WMF) (talk) 23:42, 29 April 2014 (UTC)

Dependency management
It seems like the root problem is the lack of dependency management in extension installation when manually fetching extensions instead of grabbing everything from a consistent git checkout.

I don't think stashing random libraries unused by core into MediaWiki would really help here, while some tool for managed installation and updates would be more useful. --brion (talk) 22:17, 29 April 2014 (UTC)