NOLA Hackathon 2011/Saturday

Notes from Saturday, 15 October, 2011. Part of NOLA Hackathon. Archived from Etherpad. = Saturday =

Markus Glaser wants to talk with some people about making it easier for 3rd party devs.
 * get more "core hacks" upstream
 * reduce need for "core hacks"
 * avoid work being done twice

FileRepo / File refactor
attending: Tim, Ben(MS), Russ, Brion, Aaron, Ben(WMF), robla For cleaning up Swift & related repositories http://www.mediawiki.org/wiki/Requests_for_comment/Refactor_on_File-FileRepo-MediaHandler <- some earlier notes File    <- have this do the metadata part |-- <- FileBackend goes here? LocalFile Foreign...File    SwiftFile      ForeignApiFile

most things in FileRepo will move to FileBackend Things like ForeignDBRepo, ForeignAPIRepo, SwiftRepo will mostly move to FileBackend alternate classes Need a multi-write FileBackend (to distribute write requests to both swift & classic filesystem) FileRepo will have a file store, it won't be a file store ForeignAPIRepo -> should never need to call a FileBackend
 * (FileRepo will be mostly empty for BC)
 * Pass config arrays on to the other backends etc...

ForeignDBRepo -> still does.

^ this'll help us see which to separate Example LocalRepo::cleanupDeletedBatch -- mixes raw file work like unlink with DB manipulation. Russ's poking at separating that to the FileBackend (for the file bit) and the FileRepo (for the DB bit) cooperating. Assigned to Aaron: cf last night's notes: https://www.mediawiki.org/wiki/NOLA_Hackathon/Friday#Swift_media_.26_stuff (Tim & Ben(MS) chatting on various details of splitting up FileRepo, RepoGroup etc)

Testing
<- add some info on chad's work? new http://integration.mediawiki.org/ci server mostly working as of 12:30pm local -- yay! Try clicking around! We are seeing intermittent phpunit test failures which we don't quite understand: http://integration.mediawiki.org/ci/job/MediaWiki-phpunit/41/#showFailuresLink Haven't been able to repro yet on Ubuntu 10.04 separate install or our own local installs Checking if can repro on same machine outside of jenkins... We fixed this, we were running a beta version of phpunit rather than the stable copy. Downgrading resolved the issue.
 * they look like not working (some templates, some parser functions)

From Developer Intro
The Manual:Title.php doc should be changed to not use factory method ~ Title.fromURL per the doc string on the method, this is not the function your looking for.

LinkArchive
Brion looked over some of this w/ Kevin Recommended splitting the record & generate feed portion from the spider/archiver portion Recording stuff for the feed generation can fetch currently-stored external links before the ArticleSaveComplete and compare against the ParserOutput to get a differential of just 'added' links (more or less) Spider/archiver should probably run its own logic for 'this page has been seen before', which simplifies the feed generator logic.
 * makes the record/feed code cleaner, easier to review & deploy
 * feed seems fairly clean, has an id-based continutation which should index fine
 * runs out through API; format looks sane enough; archive.org already has code to spider that format
 * archive.org is prepared to spider based on the feed, so we're not in a hurry to run our own archiver

Third-party committers help
What can we do to make things easier for 3rd-party devs to upstream stuff? Do we want to? ;) TL;DR summary key items: Some issues/pressures:
 * check perf implications of empty wfRunHooks and sprinkle more hooks around
 * continue refactoring things that are more flexible with factory functions & subclassing
 * clean up documentation, all pitch in on it
 * skin discussions: -> get people talking towards a final plan
 * MW core devs: communicate in support of markus & friends when these things come up! Ping robla if necessary, he'll poke us.
 * sometimes projects have requirements that don't seem to fit well with core
 * ^-- we could probably actually have a nice central point ('getDisplayName' etc) so it's easier to hook, even if we don't need the same feature
 * in this case there's also input / name collision issues...
 * More hooks!
 * being able to replace entire functions/methods is often also usefulish
 * hooks in inner loops can be slowish -- may be a concern? (is this a real problem? can we measure some?) -> check the actual performance overhead of bunches of extra callbacks
 * OO parts with good factories can be really easy to subclass
 * do an audit over non-factory usage? see if we have to kill more raw usages to make subclassing easier
 * Better outreach? Ask some more 3-rd party devs what they need merged back or refactored
 * ex: we're working with Wikia to slowly merge in more of their hooks & hacks
 * It's a two way street. Core devs can be more accomodating, 3rd parties can come forward with what they need.
 * bump up communication on mediawiki-l? or a new low-traffic list?
 * start asking what peoples' needs are so we can start working on them
 * invite more folks to next hackathons etc?
 * do a launch meeting with some of the biggies? (special hackathon? :)
 * There's lots of duplication of effort
 * eg: Wikia, WMF and others have all implemented "image insertion dialogs"
 * Mediawiki.org documentation update week!
 * some stuff is out of date or confusing -- running into old docs can confuse the heck out of people
 * may have to mellow out some of our older documentation! ("never ever try to restrict read access or your dog will die!" -> "let's make  sure things are factored nicely for this to work")
 * try to encourage active merging as people come across stuff
 * [markus] would like to encourage an atmosphere of extension developers reviewing each others' code
 * looking at other examples has helped him -- it's a win-win
 * the best documentation is written while working on the code
 * Need to improve how we communicate about upcoming changes (eg resourceloader coming in, and what you would need to do and *how important it is* to deal with it)
 * Versioning for core vs extensions -- how to handle extensions that want to work on multiple core versions? Core maintainers tend to update extensions on trunk that are only compatible with the latest.
 * the current 'make a branch of all extensions at the time of core change' feels awkward; they're very hard to maintain in SVN
 * probably easier to maintain each extensions' back-compat branches in git?
 * maintaining compat with multiple core versions in the current trunk can be really awkward too
 * seeing core devs' changes break your code on the old version at least lets you know a change is coming ;)
 * warnings with deprecation... try to keep a couple versions before dumping things to let ext devs have time to migrate to new usages.
 * markus indicates that the deprecation warnings in logs can be useful for him
 * git will also make it easier to maintain your own back-compat branches without them having to be in central master repos (or doing ad-hoc branch names/descriptions)
 * SKINS aaaaiiiiieeeeeee
 * there has been talk :) -> see Daniel Friesen's threads on wikitech-l
 * still needs to be finalized; some other folks working on related issues per robla
 * Wikia - Owen Taylor (oasis etc)
 * modular pieces that get assembled into a skin with headers, sidebars, banners, widgets, footers etc
 * WMF - Patrick Reilly (MobileFrontend system has some similarities in some parts)
 * think XSLT-like transformation rules on top of a simple "base layer" skin?
 * per chad - we should probably go ahead and boil the oceans; when new sensible skin system comes in let all the old ones die :)
 * we all need to push to go ahead and find and implement a solution here -- if done by a 3rd-party need to be sure we all make it work for us all!