Topic on Talk:Requests for comment/Extension management with Composer

We'll be discussing related RfCs on Tuesday 29 April, 2200-2300 UTC and I thought people interested in this RfC might want to participate, by commenting on the other proposals, and/or by coming to the IRC meeting.

Summary of discussion: We have some open questions for Markus regarding Composer, and for administrators of MediaWiki installations. Would Composer work in your environment?

• LINK: https://www.mediawiki.org/wiki/Requests_for_comment/Extension_management_with_Composer (TimStarling, 22:19:37)
• LINK: http://www.bn2vs.com/blog/2013/11/24/introduction-to-composer-for-mediawiki-developers/ (sumanah, 22:24:03)
• <bd808> The ops concerns can be dealt with for the WMF cluster using a technique like I used in the Scholarships app (which uses Composer and is live on the cluster). The technique in patch https://gerrit.wikimedia.org/r/#/c/119939/ is very similar to what Scholarships does to freeze the external libs. (sumanah, 22:31:33)
• <parent5446> Technically it should be trivial to convert most extensions to a composer library (sumanah, 22:31:50)
• Wikidata team has some Composer experience per aude's explanation (sumanah, 22:32:48)
• discussion of whether to deal with multiple packaging systems or just one, use something like https://github.com/derrabus/composer-webui for a web UI, CLI vs web (doing web UI securely is ?) (sumanah, 22:33:38)
• IDEA: question for composer-for-extensions: best way to fit in with our source-control-based deployment? (brion, 22:37:11)
• IDEA: question for composer-for-extensions: web ui possible for small installations? (compare with wordpress’s installer & extension manager) avoid chmod 777 (brion, 22:37:11)
• IDEA: question for composer-for-extensions: avoid proliferation of package managers, but make sure we can still install extensions manually if need be or for migrations (brion, 22:37:12)
• <bd808> A composer extension can register several customization scripts to hooks -- https://getcomposer.org/doc/articles/scripts.md (sumanah, 22:40:59)
• ACTION: sumanah to get input from Wikia, ShoutWiki, wikiHow, & other 3rd-party folks on the Composer RfC talk page (sumanah, 22:42:16)
• ACTION: sumanah to bring this spirited discussion to the attention of Markus and ask him if he can move forward, incorporate suggestions, maybe talk to aude from Wikidata & bd808 for specific HOWTO recommendations to incorporate (sumanah, 22:43:01)

This is the IRC log of today's discussion - this happened in between two other RfC discussions so look at the full log for context.

22:19:45 <andrewbogott> Yeah, I was going to say: I don't think Ryan_Lane is fine with it :)

22:19:47 <sumanah> but since Markus isn't here and JeroenDeDauw needs to head off, it's suboptimal. Ah well

22:20:22 <Ryan_Lane> composer, as currently implemented, means you need to run composer on every node you deploy to. it also makes it impossible to disable any individual extension

22:20:30 <TimStarling> I don't think there's any reason to use composer in production

22:20:38 <aude> Ryan_Lane: no

22:20:44 <aude> you make a build

22:20:57 <aude> with all the stuff installed

22:20:59 <aude> run tests on it

22:21:00 <bd808> The ops concerns can be dealt with for the WMF cluster using a technique like I used in the Scholarships app (which uses Composer and is live on the cluster).

22:21:05 <aude> then deploy that

22:21:14 <aude> all the stuff checked in, including composer.lock

22:21:17 <Ryan_Lane> what about other people using similar deployment methods?

22:22:06 <mwalker> what aude has described is basically how we're having to deploy Node.JS apps

22:22:08 <aude> it's also possible to have an internal satis (packagist) system

22:22:24 <bd808> The technique in patch https://gerrit.wikimedia.org/r/#/c/119939/ is very similar to what Scholarships does to freeze the external libs.

22:22:24 <aude> that only knows about our own stuff

22:22:26 <TimStarling> I'd rather not have any kind of external dependency for source in production -- just our own git servers

22:22:33 <Ryan_Lane> so, does this assume that all WMF extensions will also have composer data?

22:22:48 <brion> can composer use private repos?

22:22:50 <TimStarling> the wikidata thing is to run composer locally and then push the result into git, right?

22:22:51 <aude> probably our own satis + a build would work

22:22:54 <aude> brion: yes

22:22:57 <parent5446> Technically it should be trivial to convert most extensions to a composer library

22:22:59 <brion> yes looks like good

22:23:00 <brion> :D

22:23:14 <aude> TimStarling: yes

22:23:28 <brion> so what does it look like to composer-ize an extension? what sort of migration path would we be looking at?

22:23:38 <aude> including it's important to carefully check composer lock and the diff

22:23:40 <brion> and would this just handle fetching files, or would it also handle installation, enabling, etc?

22:23:45 <parent5446> You make a composer.json file that tells composer to include the main extension file

22:23:47 <mwalker> brion, http://www.bn2vs.com/blog/2013/11/24/introduction-to-composer-for-mediawiki-developers/

22:23:54 <aude> plus all our tests run and then only after they pass, it gets merged

22:23:59 <TimStarling> composer fetches files and handles class autoloading

22:23:59 <brion> thanks

22:24:03 <sumanah> #link http://www.bn2vs.com/blog/2013/11/24/introduction-to-composer-for-mediawiki-developers/

22:24:36 <TimStarling> it has its own autoload system, and does clever things like scanning .php files on install for class definitions

22:24:41 <aude> some of a problem will be when some other extension wants to use same library that wikidata uses

22:24:58 <TimStarling> in addition to PSR-0 of course

22:25:01 <aude> would be nice to have it registered in a shared place

22:25:01 <brion> related question:

22:25:12 <brion> obviously small installations would love an extension installation manager.

22:25:23 <brion> would tying to composer force us to require CLI for extension setup?

22:25:37 <brion> or would it allow for a web control panel as well, given appropriate permissions on server?

22:25:42 <sumanah> ok, it sounds like there is a lot of interest in Composer usage. Ryan_Lane does this interest you enough to make you wanna collaborate on investigating things after today, working on the RfC with Markus Glaser or whoever else wants to work on this? JeroenDeDauw maybe?

22:25:43 <aude> TimStarling: we also use composer option to dump a classmap (helps optimize)

22:25:54 <sumanah> because we're not gonna figure all this out and approve it today

22:25:57 <aude> so in the end it is like AutoLoader.php

22:25:58 <TimStarling> we can support any number of pacakge management systems in parallel

22:26:01 <parent5446> brion: Yes, unless we make a PHP interface for composer, which we can

22:26:13 <brion> parent5446: that works for me

22:26:13 <TimStarling> we don't need to support composer exclusively

22:26:20 <parent5446> ^that as well

22:26:26 <brion> hmmmmmmmmmmm

22:26:30 <brion> i hate mixing package managers though

22:26:38 <mwalker> +1 to brion

22:26:40 <aude> brion: a php interface would be cool, but not aware of one

22:26:45 <TimStarling> as long as we document dependencies as well as putting them in composer.json, people can keep on installing things the old way

22:26:50 <Ryan_Lane> sumanah: I can't really spend time on this. I need this as a user of mediawiki, though

22:26:50 <Dantman> I also have doubts about being able to do a web interface around composer.

22:26:50 <aude> probably would be fairly simple

22:27:00 <Dantman> Especially a secure one.

22:27:01 <sumanah> Ryan_Lane: totally fair - just wanted to check

22:27:27 <brion> Dantman: secure is the hard part. wordpress updates for instance seem to jump through weird hoops to ssh into itself or some weird shit :D

22:27:32 <brion> it scares me

22:27:38 <brion> but is an oh-so-useful feature

22:28:04 <TimStarling> it's a bit more secure than just using the web user for installation

22:28:25 <brion> chmod 777

22:28:26 <TimStarling> at least after installation, the source tree is not writable by the webserver

22:28:31 <brion> yeah :D

22:28:35 <parent5446> Wordpress also allows using FTP to update

22:28:39 <Dantman> Most of the important metadata is still in PHP. So you can't read it without first fetching and installing the extension.

22:28:41 <brion> *shudder*

22:28:45 <bd808> Google found https://github.com/derrabus/composer-webui for me

22:29:36 <parent5446> Looking at the code for what bd808 just posted, you can basically just include composer itself, and use it's internal interface

22:29:36 <sumanah> hexmode: you talk to Markus a lot, I assume :-) Do you know whether Markus has time to respond to any of these points soon?

22:29:43 <sumanah> (since he proposed Composer-y stuff)

22:30:11 <Dantman> It's also a dead end if you can't do shell execution from php. (Don't we still sort of support that?)

22:30:12 <brion> sounds like we’ve got a lot of questions and ideas to queue up for that :D

22:30:30 <sumanah> brion: help me summarize for the notes?

22:30:36 <Dantman> Side note on composer itself.

22:30:44 <parent5446> You wouldn't need shell access to use composer from PHP

22:31:00 <Dantman> Hmmm... right.

22:31:33 <sumanah> #info <bd808> The ops concerns can be dealt with for the WMF cluster using a technique like I used in the Scholarships app (which uses Composer and is live on the cluster). The technique in patch https://gerrit.wikimedia.org/r/#/c/119939/ is very similar to what Scholarships does to freeze the external libs.

22:31:44 <Dantman> Composer is similar to other package managers that install things locally.

22:31:50 <sumanah> #info <parent5446> Technically it should be trivial to convert most extensions to a composer library

22:31:52 <Dantman> Like npm and bower.

22:32:37 <Dantman> However it's unique (in a flawed way) at being the only one that requires that every installed package be present inside the .json file.

22:32:48 <sumanah> #info Wikidata team has some Composer experience per aude's explanation

22:33:32 <Dantman> Both bower and npm can install packages locally without adding them to the .json file you commit. (If you want to add it you use --save)

22:33:36 <TimStarling> how would registration be handled?

22:33:38 <sumanah> #info discussion of whether to deal with multiple packaging systems or just one, use something like https://github.com/derrabus/composer-webui for a web UI, CLI vs web (doing web UI securely is ?)

22:33:45 <bd808> You can add arbitrary libraries using Composer at the command line, but it writes them into the lock file

22:34:06 <legoktm> [15:28:39] <Dantman> Most of the important metadata is still in PHP. So you can't read it without first fetching and installing the extension. <-- Yeah, I was thinking about that as well. I had an idea that we could just put all the extension registration stuff (hooks, classes, basically mw:User:Legoktm/extension_registration) into a JSON file, so anything can read it without actually installing the extension

22:34:21 <bd808> Which is very similar to using a require.txt file with pip in python or ruby's bundler

22:35:14 <TimStarling> would composer installation somehow register the extension? or would registration require a line to be added manually to LocalSettings.php?

22:35:32 <aude> TimStarling: can be done wither way

22:35:59 <aude> i prefer manually registration for prodution

22:35:59 <TimStarling> that may be a question for discussion on the RFC page then

22:36:20 <parent5446> TimStarling: If you set the autoloader to include the main file, it would not need lines added to LocalSettings

22:36:25 <aude> so we can enable something for one wiki but not an other (e.g.)

22:36:35 <Dantman> My point is, if it weren't for that flaw in composer. It would have never been necessary to delete the composer.json that we used to have in core.

22:36:36 <aude> like we do w/o composer

22:36:45 <bd808> A composer extension can register several customization scripts to hooks -- https://getcomposer.org/doc/articles/scripts.md

22:37:11 <brion> #idea question for composer-for-extensions: best way to fit in with our source-control-based deployment?

22:37:11 <brion> #idea question for composer-for-extensions: web ui possible for small installations? (compare with wordpress’s installer & extension manager) avoid chmod 777

22:37:11 <brion> i hope i’m using the right magic tags

22:37:12 <brion> #idea question for composer-for-extensions: avoid proliferation of package managers, but make sure we can still install extensions manually if need be or for migrations

22:37:12 <brion> brb

22:37:38 <sumanah> brion: yeah, you are :-) https://wiki.debian.org/MeetBot #idea works

22:37:46 <brion> ok ...

22:37:53 <brion> did we get the key points?

22:38:05 <sumanah> also #info #action #help #link

22:38:11 <TimStarling> maybe we should have our own admin console for registration

22:38:21 <brion> whoa colloquy just dropped the last 5 minutes on me in a batch :D

22:38:37 <TimStarling> and it would detect extensions installed by composer for the purposes of displaying the list

22:38:53 <bd808> Dantman: Part of the problem is that core is both a library and an application. Composer doesn't grok that.

22:38:55 <TimStarling> that way you can disable them, but it is still easy to use

22:38:57 <brion> TimStarling: i like that

22:39:08 <brion> composer for physical installation, but we have a separate activation as we do now

22:39:19 <brion> this works well for multi-site farms as well as for shipping defaults etc

22:40:36 <TimStarling> you could pull the list of enabled extensions out of the DB or some other web-writable place

22:40:46 * sumanah invited folks from Wikia, ShoutWiki, & wikiHow to come talk about this; will try to get them to talk on the talkpage

22:40:59 <sumanah> #info <bd808> A composer extension can register several customization scripts to hooks -- https://getcomposer.org/doc/articles/scripts.md

22:41:01 <TimStarling> iterate, invoke some autoloaded static function for registration

22:42:16 <sumanah> #action sumanah to get input from Wikia, ShoutWiki, wikiHow, & other 3rd-party folks on the Composer RfC talk page

22:42:21 <aude> so, i guess... with wikidata, we disabled autoloading of wikibase client and repo by default

22:42:39 <aude> those are then loaded via CommonSettings based on config

22:42:54 <TimStarling> should we talk about parent5446'

22:42:55 <aude> once one of those is loaded, all the dependencies are autoloaded

22:42:59 <TimStarling> should we talk about parent5446's RFC now?

22:43:01 <sumanah> #action sumanah to bring this spirited discussion to the attention of Markus and ask him if he can move forward, incorporate suggestions, maybe talk to aude from Wikidata & bd808 for specific HOWTO recommendations to incorporate

22:43:06 <sumanah> I think so TimStarling