Team Practices Group/Improving burndown charts

91

Introduction
This page is a work in progress, reflecting current thinking. It is not authoritative in any way.

Raw kickoff meeting notes: https://etherpad.wikimedia.org/p/PhabGraphKickoff

Terminology

 * Burndown Chart
 * https://en.wikipedia.org/wiki/Burn_down_chart
 * Burnup Chart
 * http://brodzinski.com/2012/10/burn-up-better-burn-down.html
 * Phabricator
 * The issue-tracking system used by the WMF
 * Phabricator Sprint Extension
 * Adds "Sprint" type projects, which add a "Story Points" field to any task in that project
 * Written by the WMF (?)
 * Phragile
 * A tool that generates graphs of data pulled from phabricator's API
 * Release
 * A "marketing" feature-driven release, OR a timeboxed release
 * For feature-driven, the question we want to answer is "When?"
 * For timeboxed, the question is "How much can we get done?"
 * Release Burnup Chart (or Product Burnup Chart)
 * A burnup chart whose scope spans multiple sprints
 * Release Cumulative Flow Chart
 * http://www.agilesherpa.org/agile_coach/metrics/cumulative_flow/
 * Sprint Burnup Chart
 * A burnup chart whose scope is limited to a single sprint

Assumptions

 * Burnup charts are fine (in lieu of Burndown charts)
 * Product/Release charts are the focus of this initiative (as opposed to Sprint charts)
 * Forward-looking prediction is the focus of this initiative (as opposed to retrospective)
 * An external chart generation tool is acceptable (it doesn't have to be built into phab)
 * At least for now, it would be acceptable for phab to export raw data, to allow fancy charts to be generated by a spreadsheet
 * We have some (unknown $) budget to spend on upstream Phabricator coding, as needed
 * We have the ability to configure our phab instance however we wish
 * We have limited WMF human hours available for phab coding
 * Any phab coding we do should be in the form of patches pushed upstream
 * Is backward-looking important, given that projects wouldn't have used the necessary conventions???
 * Long-term reliance on the Sprint Extension is problematic
 * We need Task.Estimate (aka "Story Points"), and Project.StartDate/EndDate
 * (Not sure what else Sprint Extension provides that we need

Current State

 * Sprint extension
 * Developed by Christopher at WMDE
 * Requires ongoing maintenance to remain compatible with upstream changes
 * Therefore requires testing with each new upstream release
 * Christopher doesn't have time to maintain it
 * Mukunda can maintain as needed, but not ideal
 * We really need to get sprint-y features in upstream (Story points, start/end dates)
 * Phabricator's built-in burnup charts (part of Sprint Extension):
 * Reports/Burnup, filter by a single project
 * Does not track when tasks were added/removed to/from the project
 * Odd pink color choice
 * In FF on Linux, scrolling down leaves the graph grid behind
 * Phabricator's built-in burndown charts (phact?):
 * Work OK within a single Sprint, but require a ton of work to span multiple sprints
 * Trigger on a task being in a "Done" column, which not all teams use
 * But note that "resolved" is also problematic
 * Generates Burndown charts, not burnup charts
 * Does not show the scope line moving up or down as scope changes--it always just shows as a flat line, at the *current* scope level
 * Handles weekends in ways that some people dislike, but that's really a Sprint chart issue more than a Release chart issue
 * Phact:
 * Upstream reporting tool, with a "burnup chart" feature
 * Disabled in our instance.
 * Chase: Phact is in "prototype" mode, effectively disowned by upstream
 * Generally phab doesn't have good reporting, so phact was a response to that
 * But they didn't have clear requirements, so it has gone stale
 * We could push for phact to take on a reporting role, but it doesn't seem to have much of use to build on
 * Phragile:
 * Is being developed by wmde
 * Provides ??? charts (I haven't seen examples yet)
 * Is an external tool, which would be acceptable
 * Would need to interface with phab authentication/authorization system?
 * To be able to handle security issues, yes, but for normal projects, is this true?
 * Currently provides BOTH graph generation AND sprint creation
 * Should these be separated?
 * Robla's "wbstatus" script:
 * Is written in python
 * Scrapes phab html to generate a state model of issues on a workboard
 * Could be used as a template to create a similar tool for burnup purposes
 * Example output: https://mw-core-wbstatus.wmflabs.org/?r=2015-03-16_to_2015-03-20
 * Source code: https://github.com/robla/phab-wbstatus

Project Roles

 * Kevin: Product Owner.
 * Gather and prioritize requirements
 * https://www.mediawiki.org/wiki/Team_Practices_Group/Improving_burndown_charts
 * Project management of externals (e.g. phab, phragile)
 * Reporting to stakeholders
 * Mukunda:
 * could code in phab
 * maintain phab in general (features, integration, deployment)
 * could maintain/fix the sprint extension as needed
 * Complexity of integrating Sprint extension with upstream changes (we have no Continuous Integration of Phab+Sprint extension)
 * Greg:
 * OK'ing Mukunda's time
 * Delegating interface w/Evan
 * Neither an implementor nor a (knowledgeable) consumer
 * Arthur:
 * Shepherding, supporting
 * Conduit/buffer as necessary for exec requirements
 * Chase:
 * Ops eng
 * mostly 'historically significant'
 * dealt with deployments previously (handing off to Mukunda)
 * much experience dealing with upstream
 * is pregnant ;)

Issues to Remember

 * Backlog stories may not have estimates
 * Assign arbitrary value (e.g. average story size)?
 * "Release" might mean fixed-date/train, or feature-driven
 * Not all teams will have "sprints"--some are Kanban
 * Need to avoid double-counting when task and subtasks were both estimated
 * Could be an issue both for "work completed" and for "target scope" calculations

Data Model

 * It is "relatively easy" to expose other db elements via new custom Conduit API calls
 * Helpful phab queries, if they are/were possible:
 * List all tasks that were in Column C of Project P on Date D (NO?)
 * List all projects matching wildcard name search (YES?) [Maybe not necessary]
 * List all tasks which were EVER in Project P (NO?)
 * List all transactions for Task T, with timestamps (YES except not sure about timestamps)
 * List all transactions for Project P, with timestamps (NO?)
 * If we did a nightly snapshot of all tasks in Project P,Q,R..., then where would we store that data
 * Can a Release project also be a Sprint project? (YES)
 * (would be nice to have estimated future epics only be in the Release project, and not to force them to also exist in a Sprint project that isn't a real Sprint)

Desired Datasets/Charts

 * Points "done" per sprint over time (velocity)
 * Note: Should scale by sprint length, to accommodate Kanban teams
 * Note: Is this based on a "done" column, or marked "resolved"?
 * Current points of all tasks in all columns of all sprints PLUS non-sprint backlog
 * Historical points of all tasks in all columns of all sprints PLUS non-sprint backlog
 * Average task estimate across multiple sprints (or maybe just within each sprint?)
 * To provide a placeholder size for unestimated tasks
 * Average Lead Time per story (from entering a sprint backlog until being "done")
 * Especially helpful for Kanban projects

Possible Solution(s)

 * Completed points are calculated from a set of wildcard-filtered Sprint projects
 * CONVENTION: Tasks are considered completed if they are in a workboard column named "Done"
 * Graph-requesting user could specify the column name when they give the wildcard spec
 * Brute-force algorithm:
 * For each matching Sprint project,
 * Sum story points of all tasks in the "Done" column
 * Note the timestamp at which that Sprint ended (assuming it is in the past)
 * Scope of release is calculated from a Release project
 * FEATURE REQUEST: New Phab API call: List all transactions that refer to a specific project
 * CONVENTION: The Release project must contain *every* task that is part of that release
 * Any task with subtasks must exclude any subtask estimates from its own current estimate
 * Brute-force algorithm:
 * Given a Release project, find all transactions affecting it
 * Create event history of tasks being added/removed on that project
 * Replay the history to know which tasks were in that project at any timestamp
 * Record the story point sum at desired timestamps
 * Phragile, with the following changes:
 * Ability to view graphs for sprints not created by phragile (e.g. existing sprints)
 * Make sure it will work with a months-long sprint
 * Burnup charts instead of burndown
 * NOTE: Should use historical estimates, as well as historical statuses
 * Must include tasks which used to be in the sprint, but no longer are
 * Calculate scope line based on points in the sprint at that time
 * NOTE: Must use the historical estimates, as well as historical statuses
 * Must include tasks which used to be in the sprint, but no longer are
 * Need a rule to handle unestimated tasks (TBD)
 * Ability to export underlying data that built the chart
 * Allows power users to generate alternative graphs
 * Make the "ideal points" line optional
 * Would prefer subset of the app which only does anonymous queries
 * For increased security
 * Would prefer subset of the app that doesn't require its own database
 * For simpler deployment and less ongoing maintenance
 * Only feature to drop would be snapshots?