Extension:Graph/Plans

This page is a place for WMF staff and volunteers to gather the information needed for the WMF to decide the role it will play in ensuring the needs the extension emerged to serve continue being met.

Decisions to be made
The Open questions listed below are meant to surface the information the WMF will need to make the following decisions:


 * 1)  What needs will any proposed path forward need to meet? 
 * 2)  What role will the WMF play in ensuring these needs are met? 

FAQ

 * 1) Once re-enabled, will the Graph Extension support pseudo-protocols? Yes. Once the Graph Extension is re-enabled, it will support most, if not all, of the existing protocols, subject to concerns about security and maintainability.   We will collectively discuss these concerns as we encounter them.   Of course, if there are particular ones you think ought to be supported, we'd value you naming them and sharing why on the talk page, as Snævar, Bawolff, Nikki, and Pietrasagh have started doing.     Please note: protocol support is a notably fragile part of the Graph Extension. If/when we come to learn this facet of the extension is causing problems, we might need to disable protocol support.
 * 2) How much time – if any – will elapse between the Graph Extension being re-enabled with support for Vega 5 and support for Vega 2 being discontinued?  In short, we do not yet know when support for Vega 2 will be discontinued. This is a timeline we (staff + volunteers) will need to define together.  With the above said, here is what we are certain about at it relates to Vega 2 and Vega 5 support:
 * 3) Being able to provide guidance about when Vega 2 support will be discontinued depends on us first learning how the migration to Vega 5 is going.
 * 4) We do not plan to support Vega 2 graphs indefinitely
 * 5) What can we do to preserve graphs that have not/can not be converted from Vega 2 to Vega 5?  Converting graphs of this sort on a server that allows sandboxed rendering exposes security vulnerabilities the current approach is meant to mitigate. More details in T334940#9161842.   With the above said, we estimate the graphs that have not/cannot be ported from Vega 2 to Vega 5 to be relatively small and comprised of:
 * 6) Graphs that use a custom protocol (e.g. Wikimedia API or WDQS) the sandboxed iFrame approach does not support.
 * 7) Graphs that can be ported, but no one has undertaken the work to do so.

Proposal
To safely restore access to the information and capabilities disabling the Graph Extension has left people without and promote the volunteer+staff collaboration needed to do so, we (the WMF) are committing to:

Re-enable the Graph Extension in a sandboxed iFrame with a restrictive content security policy.
 * 1) Once the Graph Extension is reenabled, it will continue to work with Vega 2 for a yet-to-be defined period of time. Note: we'll need to define this window together.
 * 2) After that "yet-to-be defined period of time," Vega 2 support will be discontinued and use of the Graph Extension will require volunteers to make graphs with Vega 5.
 * 3) As soon as possible, make the sandboxed Graph extension available on the beta cluster for testing. See: T346292.
 * 4) Investigate the viability of adding logging to increase our awareness of instances where people are exploiting the security vulnerabilities inherent with restoring support for Vega on our platform. See T346414.
 * 5) Publish the technical documentation needed for developers across the Movement to understand how we implemented the sandboxed CSP approach
 * 6) Publish a clear timeline for when you all can expect all of the above to happen
 * 7) Note: exploratory work to redeploy the Graph Extension in a sandboxed iframe has started. See T222807.
 * 8) Share regular updates about the progress we're making on the commitments named above on Phabricator and MediaWiki.
 * 9) Support volunteers with code and processes that will ease the transition from Vega 2 to Vega 5 when the time for this transition comes.

In support of the above, we'd need to depend on ya'll (volunteers) to:


 * 1) Spread awareness of this proposal and the updates that will come as we start implementation, assuming this proposal moves forward.
 * 2) Manually migrate some proportion of Vega 2-based graphs to be compatible with Vega 5. See the "Vega 2 → Vega 5 transition" section below.
 * 3) Potentially, fix/port graphs that attempted to fetch live data using methods that the sandboxing approach inhibits.
 * 4) Note: the need for the above will become clear once we decide on whether we will restore the pseudo-protocols that were used to fetch data live from the action API, the REST API, WDQS etc, and the precise sandbox parameters we select (domains/ports/http methods allowed). This decision will be made in T346291.

Vega 2 → Vega 5 Transition

 * Why do we think it's worthwhile to migrate from Vega 2 to Vega 5?
 * Vega 2 has been superseded by Vega 3, 4, then 5 upstream.  Upstream and third-party documentation exclusively refers to syntax in “Vega version 3.0 and later”, and it is difficult for new contributors to find documentation relevant to Vega 2.  The last upstream release (bugfix or security) of Vega 2.x was in January 2017.  Vega 5 was released in March 2019 and is still under active maintenance and development, with the latest 5.25.0 release in April 2023.
 * Volunteers have reported issues with Vega 2's accessibility, syntax, and overall functionality, per this 2023 wish.
 * Vega 5 has made improvements to the library's expression layer that harden it from a security perspective compared to Vega 2.  It is not perfect, but by introducing a parsed expression grammar it offers a more robust foundation for additional security hardening in the future if it proves necessary.
 * Maintaining multiple versions of Vega concurrently is unsustainable in the long run. The wiki community is taxed in the attempt to independently support software which is not being maintained upstream. Our efforts are best spent working in cooperation with upstream and third-party developers, and to do this we need to be working from the upstream Vega 5 code base.
 * What might be required to migrate graphs from Vega 2 to Vega 5?
 * Create a converter that would migrate Vega 2-based graphs to be compatible with Vega 5. @Jdlrobson started work on an initial approach in T335048#8794138.  The initial work needs to be restructured slightly to refocus it on being an aid to manual porting, instead of the automatic translator which was its original goal.  Note: We estimate this converter currently works for ~80% of graphs, with diminishing returns on additional engineering effort to cover more.  We do not plan on continuing to invest significant additional engineering resources here, but instead to simply repurpose the existing codebase as an aid to manual porting.
 * Volunteers would need to update  syntax on a case-by-case basis, aided by (1) the ability to run the existing Vega 2 and new Vega 5 specification side-by-side, (2) the partial Vega2-to-5 porting tool which handles 80% of the “obvious” keyword changes and other mechanical conversions, (3) [ https://vega.github.io/vega/docs/porting-guide/ the upstream Vega2 porting guide], and (4) additional documentation or tools which might be created by the wiki community.
 * Update the limited number of Scribunto templates on-wiki which generate  output in Vega 2 format to instead output Vega 5.  This requires both lua and Vega expertise, but fixes a larger number of Vega 2 uses on wiki at once.

Research
The research that informed the for safely and securely restoring access to the information and capabilities  has left people without.