Thread:Talk:Requests for comment/Extension management with Composer/April 29th RfC review/reply (2)

This is the IRC log of today's discussion - this happened in between two other RfC discussions so look at the full log for context.


 * 22:19:45 Yeah, I was going to say: I don't think Ryan_Lane is fine with it :)
 * 22:19:47 but since Markus isn't here and JeroenDeDauw needs to head off, it's suboptimal. Ah well
 * 22:20:22  composer, as currently implemented, means you need to run composer on every node you deploy to. it also makes it impossible to disable any individual extension
 * 22:20:30  I don't think there's any reason to use composer in production
 * 22:20:38 Ryan_Lane: no
 * 22:20:44 you make a build
 * 22:20:57 with all the stuff installed
 * 22:20:59 run tests on it
 * 22:21:00 The ops concerns can be dealt with for the WMF cluster using a technique like I used in the Scholarships app (which uses Composer and is live on the cluster).
 * 22:21:05 then deploy that
 * 22:21:14 all the stuff checked in, including composer.lock
 * 22:21:17  what about other people using similar deployment methods?
 * 22:22:06 what aude has described is basically how we're having to deploy Node.JS apps
 * 22:22:08 it's also possible to have an internal satis (packagist) system
 * 22:22:24 The technique in patch https://gerrit.wikimedia.org/r/#/c/119939/ is very similar to what Scholarships does to freeze the external libs.
 * 22:22:24 that only knows about our own stuff
 * 22:22:26  I'd rather not have any kind of external dependency for source in production -- just our own git servers
 * 22:22:33  so, does this assume that all WMF extensions will also have composer data?
 * 22:22:48 can composer use private repos?
 * 22:22:50  the wikidata thing is to run composer locally and then push the result into git, right?
 * 22:22:51 probably our own satis + a build would work
 * 22:22:54 brion: yes
 * 22:22:57 Technically it should be trivial to convert most extensions to a composer library
 * 22:22:59 yes looks like good
 * 22:23:00 :D
 * 22:23:14 TimStarling: yes
 * 22:23:28 so what does it look like to composer-ize an extension? what sort of migration path would we be looking at?
 * 22:23:38 including it's important to carefully check composer lock and the diff
 * 22:23:40 and would this just handle fetching files, or would it also handle installation, enabling, etc?
 * 22:23:45 You make a composer.json file that tells composer to include the main extension file
 * 22:23:47 brion, http://www.bn2vs.com/blog/2013/11/24/introduction-to-composer-for-mediawiki-developers/
 * 22:23:54 plus all our tests run and then only after they pass, it gets merged
 * 22:23:59  composer fetches files and handles class autoloading
 * 22:23:59 thanks
 * 22:24:03 #link http://www.bn2vs.com/blog/2013/11/24/introduction-to-composer-for-mediawiki-developers/
 * 22:24:36  it has its own autoload system, and does clever things like scanning .php files on install for class definitions
 * 22:24:41 some of a problem will be when some other extension wants to use same library that wikidata uses
 * 22:24:58  in addition to PSR-0 of course
 * 22:25:01 would be nice to have it registered in a shared place
 * 22:25:01 related question:
 * 22:25:12 obviously small installations would love an extension installation manager.
 * 22:25:23 would tying to composer force us to require CLI for extension setup?
 * 22:25:37 or would it allow for a web control panel as well, given appropriate permissions on server?
 * 22:25:42 ok, it sounds like there is a lot of interest in Composer usage. Ryan_Lane does this interest you enough to make you wanna collaborate on investigating things after today, working on the RfC with Markus Glaser or whoever else wants to work on this? JeroenDeDauw maybe?
 * 22:25:43 TimStarling: we also use composer option to dump a classmap (helps optimize)
 * 22:25:54 because we're not gonna figure all this out and approve it today
 * 22:25:57 so in the end it is like AutoLoader.php
 * 22:25:58  we can support any number of pacakge management systems in parallel
 * 22:26:01 brion: Yes, unless we make a PHP interface for composer, which we can
 * 22:26:13 parent5446: that works for me
 * 22:26:13  we don't need to support composer exclusively
 * 22:26:20 ^that as well
 * 22:26:26 hmmmmmmmmmmm
 * 22:26:30 i hate mixing package managers though
 * 22:26:38 +1 to brion
 * 22:26:40 brion: a php interface would be cool, but not aware of one
 * 22:26:45  as long as we document dependencies as well as putting them in composer.json, people can keep on installing things the old way
 * 22:26:50  sumanah: I can't really spend time on this. I need this as a user of mediawiki, though
 * 22:26:50  I also have doubts about being able to do a web interface around composer.
 * 22:26:50 probably would be fairly simple
 * 22:27:00  Especially a secure one.
 * 22:27:01 Ryan_Lane: totally fair - just wanted to check
 * 22:27:27 Dantman: secure is the hard part. wordpress updates for instance seem to jump through weird hoops to ssh into itself or some weird shit :D
 * 22:27:32 it scares me
 * 22:27:38 but is an oh-so-useful feature
 * 22:28:04  it's a bit more secure than just using the web user for installation
 * 22:28:25 chmod 777
 * 22:28:26 <TimStarling> at least after installation, the source tree is not writable by the webserver
 * 22:28:31 yeah :D
 * 22:28:35 Wordpress also allows using FTP to update
 * 22:28:39 <Dantman> Most of the important metadata is still in PHP. So you can't read it without first fetching and installing the extension.
 * 22:28:41 *shudder*
 * 22:28:45 Google found https://github.com/derrabus/composer-webui for me
 * 22:29:36 Looking at the code for what bd808 just posted, you can basically just include composer itself, and use it's internal interface
 * 22:29:36 hexmode: you talk to Markus a lot, I assume :-) Do you know whether Markus has time to respond to any of these points soon?
 * 22:29:43 (since he proposed Composer-y stuff)
 * 22:30:11 <Dantman> It's also a dead end if you can't do shell execution from php. (Don't we still sort of support that?)
 * 22:30:12 sounds like we’ve got a lot of questions and ideas to queue up for that :D
 * 22:30:30 brion: help me summarize for the notes?
 * 22:30:36 <Dantman> Side note on composer itself.
 * 22:30:44 You wouldn't need shell access to use composer from PHP
 * 22:31:00 <Dantman> Hmmm... right.
 * 22:31:33 #info The ops concerns can be dealt with for the WMF cluster using a technique like I used in the Scholarships app (which uses Composer and is live on the cluster). The technique in patch https://gerrit.wikimedia.org/r/#/c/119939/ is very similar to what Scholarships does to freeze the external libs.
 * 22:31:44 <Dantman> Composer is similar to other package managers that install things locally.
 * 22:31:50 #info Technically it should be trivial to convert most extensions to a composer library
 * 22:31:52 <Dantman> Like npm and bower.
 * 22:32:37 <Dantman> However it's unique (in a flawed way) at being the only one that requires that every installed package be present inside the .json file.
 * 22:32:48 #info Wikidata team has some Composer experience per aude's explanation
 * 22:33:32 <Dantman> Both bower and npm can install packages locally without adding them to the .json file you commit. (If you want to add it you use --save)
 * 22:33:36 <TimStarling> how would registration be handled?
 * 22:33:38 #info discussion of whether to deal with multiple packaging systems or just one, use something like https://github.com/derrabus/composer-webui for a web UI, CLI vs web (doing web UI securely is ?)
 * 22:33:45 You can add arbitrary libraries using Composer at the command line, but it writes them into the lock file
 * 22:34:06 [15:28:39] <Dantman> Most of the important metadata is still in PHP. So you can't read it without first fetching and installing the extension. <-- Yeah, I was thinking about that as well. I had an idea that we could just put all the extension registration stuff (hooks, classes, basically mw:User:Legoktm/extension_registration) into a JSON file, so anything can read it without actually installing the extension
 * 22:34:21 Which is very similar to using a require.txt file with pip in python or ruby's bundler
 * 22:35:14 <TimStarling> would composer installation somehow register the extension? or would registration require a line to be added manually to LocalSettings.php?
 * 22:35:32 TimStarling: can be done wither way
 * 22:35:59 i prefer manually registration for prodution
 * 22:35:59 <TimStarling> that may be a question for discussion on the RFC page then
 * 22:36:20 TimStarling: If you set the autoloader to include the main file, it would not need lines added to LocalSettings
 * 22:36:25 so we can enable something for one wiki but not an other (e.g.)
 * 22:36:35 <Dantman> My point is, if it weren't for that flaw in composer. It would have never been necessary to delete the composer.json that we used to have in core.
 * 22:36:36 like we do w/o composer
 * 22:36:45 A composer extension can register several customization scripts to hooks -- https://getcomposer.org/doc/articles/scripts.md
 * 22:37:11 #idea question for composer-for-extensions: best way to fit in with our source-control-based deployment?
 * 22:37:11 #idea question for composer-for-extensions: web ui possible for small installations? (compare with wordpress’s installer & extension manager) avoid chmod 777
 * 22:37:11 i hope i’m using the right magic tags
 * 22:37:12 #idea question for composer-for-extensions: avoid proliferation of package managers, but make sure we can still install extensions manually if need be or for migrations
 * 22:37:12 brb
 * 22:37:38 brion: yeah, you are :-) https://wiki.debian.org/MeetBot #idea works
 * 22:37:46 ok ...
 * 22:37:53 did we get the key points?
 * 22:38:05 also #info #action #help #link
 * 22:38:11 <TimStarling> maybe we should have our own admin console for registration
 * 22:38:21 whoa colloquy just dropped the last 5 minutes on me in a batch :D
 * 22:38:37 <TimStarling> and it would detect extensions installed by composer for the purposes of displaying the list
 * 22:38:53 Dantman: Part of the problem is that core is both a library and an application. Composer doesn't grok that.
 * 22:38:55 <TimStarling> that way you can disable them, but it is still easy to use
 * 22:38:57 TimStarling: i like that
 * 22:39:08 composer for physical installation, but we have a separate activation as we do now
 * 22:39:19 this works well for multi-site farms as well as for shipping defaults etc
 * 22:40:36 <TimStarling> you could pull the list of enabled extensions out of the DB or some other web-writable place
 * 22:40:46 * sumanah invited folks from Wikia, ShoutWiki, & wikiHow to come talk about this; will try to get them to talk on the talkpage
 * 22:40:59 #info A composer extension can register several customization scripts to hooks -- https://getcomposer.org/doc/articles/scripts.md
 * 22:41:01 <TimStarling> iterate, invoke some autoloaded static function for registration
 * 22:42:16 #action sumanah to get input from Wikia, ShoutWiki, wikiHow, & other 3rd-party folks on the Composer RfC talk page
 * 22:42:21 so, i guess... with wikidata, we disabled autoloading of wikibase client and repo by default
 * 22:42:39 those are then loaded via CommonSettings based on config
 * 22:42:54 <TimStarling> should we talk about parent5446'
 * 22:42:55 once one of those is loaded, all the dependencies are autoloaded
 * 22:42:59 <TimStarling> should we talk about parent5446's RFC now?
 * 22:43:01 #action sumanah to bring this spirited discussion to the attention of Markus and ask him if he can move forward, incorporate suggestions, maybe talk to aude from Wikidata & bd808 for specific HOWTO recommendations to incorporate
 * 22:43:06 I think so TimStarling