Talk:Requests for comment/MediaWiki libraries

From mediawiki.org
Latest comment: 9 years ago by Sharihareswara (WMF) in topic Discussion on April 29

Composer managed contents[edit]

In my implementation for Requests for comment/Structured logging in Gerrit change 119939 I created a libs directory that is managed using Composer. Would getting that implementation merged be a reasonable resolution to this RFC? The concept of this form of library manage was vetted and approved by Tim in his review of that RFC. --BDavis (WMF) (talk) 22:12, 23 April 2014 (UTC)Reply

Yep, that solves it.--Ryan lane (talk) 22:24, 23 April 2014 (UTC)Reply

Discussion on April 29[edit]

We'll be discussing this RfC Tuesday 29 April, 2200-2300 UTC. Sharihareswara (WMF) (talk) 17:40, 25 April 2014 (UTC)Reply

Summary of discussion: We just need to merge "Add Composer managed libraries" https://gerrit.wikimedia.org/r/#/c/119939/ in order to complete part I of this. Then Part II is actually moving everything else into that directory, and also making it reasonable to move some extensions into that directory. Then we started talking about the Composer RfC. :-) Log:
22:02:57 <sumanah> OK, now on to the first topic today, Ryan_Lane's MediaWiki libraries RfC. Then we'll move on to parent5446's RfC on 3rd-party components
22:03:01 <sumanah> #topic MediaWiki libraries
22:03:12 <sumanah> #link https://www.mediawiki.org/wiki/Requests_for_comment/MediaWiki_libraries Ryan Lane's proposal (better handling and versioning libraries that are MW extensions)
22:03:12 * marktraceur frows at Book Management
22:03:36 <sumanah> #chair sumanah brion andrewbogott TimStarling
22:03:36 <wm-labs-meetbot> Current chairs: TimStarling andrewbogott brion sumanah
22:03:41 <parent5446> So I have a question about this RfC. It mentions LdapAuthentication as an example. What library in LdapAuthentication would we be packaging with core?
22:03:46 <sumanah> #info See https://www.mediawiki.org/wiki/Talk:Requests_for_comment/MediaWiki_libraries - we just need to merge "Add Composer managed libraries" https://gerrit.wikimedia.org/r/#/c/119939/ in order to complete part I of this.
22:03:47 <sumanah> #info Then Part II is actually moving everything else into that directory, and also making it reasonable to move some extensions into that directory.
22:04:04 <parent5446> B/c I understand the RFC in the context of the Gerrit patch
22:04:19 <parent5446> Libraries that are being used in core will need to be managed somehow
22:04:23 <Ryan_Lane> parent5446: a number of extension use LDAP as a library
22:04:31 <brion> i’m not sure i understand this rfc; is the idea to package specific libraries in mediawiki core?
22:04:39 <Ryan_Lane> I'd split the necessary functionality out of the LDAP extension into a common library
22:04:44 <brion> or to provide a general place to put libraries?
22:04:47 <parent5446> ...how? Are there extensions that use LDAP other than LdapAuthentication?
22:04:53 <TimStarling> I think the idea is to create a separate class of extensions that you put in libs/ instead of extensions/
22:05:05 <Ryan_Lane> TimStarling: yep
22:05:13 <brion> first question: why?
22:05:25 <brion> second question: what needs to be different other than the filename ‘extensions’ -> ‘libs’?
22:05:36 <Ryan_Lane> because there's now a lot of extensions being used as libraries and they aren't being versioned with mediawiki
22:05:40 <parent5446> OK, but the gerrit patch makes it look like we'd be packaging them with core. So it makes it sound like we'd be packaging libraries within core that are not actually used in core.
22:05:45 <brion> third question: does that patch adding a libs directory help or hinder this rfc’s recommendations?
22:05:53 <sumanah> (the patchset is by bd808 in case he wants to comment)
22:05:57 <Ryan_Lane> most of these could be bundled with core
22:06:11 <TimStarling> we have includes/libs already for things that are bundled
22:06:13 <Ryan_Lane> and should definitely be versioned with releases
22:06:28 <Ryan_Lane> TimStarling: that's for things that core depends on, but not for things extensions may need to depend on
22:06:45 <brion> so is this directory’s contents something is fixed, that extensions can depend on always being present?
22:06:55 <brion> or is it something extensible, that extensions can rely on having a way to install things there?
22:07:09 <parent5446> I don't really think it's a good idea to package extension libraries as part of core if core is not using it.
22:07:23 <bd808> The best solution for Ryan's RFC would probably be enabling the full use of Composer (or a similar system) with wm-core
22:07:27 <parent5446> Maybe an extension needs a different library version, and can't use it because core decided to package it
22:07:48 * sumanah invited Jeroen to discuss composer-y stuff with us in this chat but - ah, there's JeroenDeDauw!
22:08:03 <JeroenDeDauw> Just saw calendar notification now
22:08:07 <JeroenDeDauw> Anout to head off
22:08:14 <Ryan_Lane> so, maybe I should describe my issue, rather than this specific solution?
22:08:18 <sumanah> JeroenDeDauw: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140429.txt starting at 2200 in case you wanna see our talk till now
22:08:26 <sumanah> s/talk/chatter/
22:08:29 <TimStarling> with composer, I'd like to know whether you think extensions should be in mediawiki/* or mediawiki/extensions/*
22:08:32 <brion> Ryan_Lane: yes please :D
22:08:42 <TimStarling> in the package names
22:08:44 <sumanah> #info the Composer RfC: https://www.mediawiki.org/wiki/Requests_for_comment/Extension_management_with_Composer
22:08:57 <Ryan_Lane> so, we have extensions. those extensions for a long time only had a dependency on a specific mediawiki version
22:09:02 <TimStarling> because JeroenDeDauw put a few in mediawiki/*, but it seems a bit weird to me
22:09:14 <Ryan_Lane> now we have extensions that only exist as libraries for use by extensions
22:09:14 <JeroenDeDauw> TimStarling: vendor/package
22:09:18 <JeroenDeDauw> TimStarling: two levels
22:09:24 <brion> (many extensions require external dependencies as well…)
22:09:52 <mwalker> extensions can also be hooks onto other extensions -- but that doesn't make the 'parent' extension a library
22:11:03 * aude waves
22:11:09 <TimStarling> hmm
22:11:12 <mwalker> this seems to be mostly an argument about our release process... in theory, everything in a release is stably versioned and has release branches for backports
22:11:43 <Ryan_Lane> well, it's more an issue that we have no reasonable way to specify extension compatibility
22:12:00 <Ryan_Lane> and now we need to specify extension compatibility with mediawiki and with other library extensions
22:12:02 <parent5446> I mean, this is basically what composer was made to do
22:12:21 <brion> Ryan_Lane: don’t you just grab them from the same version branch?
22:12:21 <parent5446> Except people don't want to use composer in core
22:12:34 * sumanah sees parent5446's point. If this is a configuration/dependency management problem, maybe we should go with the thing that does that
22:12:40 <Ryan_Lane> brion: very few people use version branches
22:12:52 <brion> Ryan_Lane: how would this solve that?
22:12:59 <mwalker> we're cutting the version branches automatically now
22:13:14 <greg-g> mwalker: but everyone else still uses tarballs
22:13:15 <Ryan_Lane> my proposal was to have the common libraries shipped with mediawiki and locked with the version
22:13:22 <brion> …
22:13:31 <brion> so something that’s common to a group of extensions, and has no relation to mediawiki core...
22:13:34 <brion> … would be shipped with core?
22:13:38 <parent5446> Shipping common libraries is kind of like a band-aid to the actual problem
22:13:39 <TimStarling> I think it's fine to have extensions that depend on other extensions
22:13:50 <bd808> parent5446: I think we are open to using composer in core, but new need to figure out how to do that in a way that works for JeroenDeDauw's use case and mine (which is basically the same as Ryan_Lane's)
22:14:07 <Ryan_Lane> the issue is that most of these library extensions probably should just be core
22:14:09 <TimStarling> drupal has a DIY system for this
22:14:18 <TimStarling> or of course we could use composer
22:14:25 <parent5446> I thought the main problem was that people didn't like how it didn't interact properly with Git?
22:14:29 <Ryan_Lane> if 20 extensions are using an extension, why isn't it a core feature?
22:14:30 <sumanah> Emufarmers: Dantman: would love to have your input here as people who work on non-WMF MediaWiki installations
22:14:32 <TimStarling> or support any other package management system, like PEAR or .deb or whatever
22:14:50 <brion> Ryan_Lane: are those 20 extensions all specific and only used by a tiny minority of installations?
22:14:54 <mwalker> greg-g, we bundle extensions into our release tarballs...
22:14:55 <brion> or are they used all the time by most?
22:15:05 <Ryan_Lane> to be fair, I'd be fine with composer, if it can be used reasonably
22:15:23 <parent5446> Well many extensions have already been made to work as composer libraries
22:15:24 <TimStarling> if a facility is widely used, I think it can be in core, you know that I am not one of the people who has a narrow view of what core should be
22:15:25 <greg-g> mwalker: yeah. (/me doesn't want to derail, it was a minor point)
22:15:27 <brion> it feels like the solution here is “a dependency-managing installation system for extensions"
22:15:38 <Ryan_Lane> TimStarling: I'm not saying you are :)
22:15:42 <TimStarling> but yes, we need extension dependencies for various reasons
22:15:42 <sumanah> Which means we should check out https://www.mediawiki.org/wiki/Requests_for_comment/Extension_management_with_Composer
22:15:51 <sumanah> and maybe expand it
22:16:04 <Dantman> sumanah: I'm not 100% certain of the topic, or even whether "extensions as composer modules" is included in this or excluded.
22:16:18 <TimStarling> not just for libraries -- some extensions hook others, for example
22:16:28 <Ryan_Lane> yep
22:16:41 <Ryan_Lane> it's something we've been punting on forever
22:16:43 <sumanah> Dantman: sounds like we're talking about including it, maybe
22:16:44 <TimStarling> some use RL modules from another extension
22:17:09 <sumanah> Dantman: the topic is https://www.mediawiki.org/wiki/Requests_for_comment/MediaWiki_libraries Ryan Lane's proposal (better handling and versioning libraries that are, or are used by, MW extensions)
22:17:22 <Ryan_Lane> so, let's drop this proposal and maybe talk about the composer suggestions?
22:17:43 <Ryan_Lane> or is that not how this process works? heh
22:17:52 <parent5446> Well if we need to itemize this to dependencies on individual RL modules and hooks, there's not really any system that handles that
22:18:05 <parent5446> With composer it's either you depend on the entire extension or you don't
22:18:12 <Dantman> Well this idea of extensions as composer modules has annoyed me over and over again.
22:18:14 <parent5446> Unless you separate into further libraries
22:18:15 <sumanah> Ryan_Lane: I'm fine with the process working like that, if you want to abandon your RfC and become a coauthor on the Composer one :)
22:18:53 <bd808> Extension management with composer would allow managing dependent libraries as well, for the extensions. My use-case of wanting an external library for core is slightly different, but could pervert the system by using a basically empty extension just to get the library management.
22:19:00 <sumanah> andrewbogott: I guess one question for you as Ops representative is how Ops feels about Composer (although I assume if Ryan's fine with it then you would be too)
22:19:02 <Ryan_Lane> wasn't that another RfC that was in the queue for today?
22:19:21 <sumanah> Ryan_Lane: no, it wasn't; it is one that I wanted people to have fresh in their memory because I predicted it would interrelate
22:19:24 <Ryan_Lane> the current implementation of composer doesn't fit with our deployment process
(And then we moved onto the Composer RfC proper.) Sharihareswara (WMF) (talk) 23:42, 29 April 2014 (UTC)Reply

Dependency management[edit]

It seems like the root problem is the lack of dependency management in extension installation when manually fetching extensions instead of grabbing everything from a consistent git checkout.

I don't think stashing random libraries unused by core into MediaWiki would really help here, while some tool for managed installation and updates would be more useful. --brion (talk) 22:17, 29 April 2014 (UTC)Reply