Extension talk:Graph/Plans

Table of usage
I'm not sure how accurate using "Pages with disabled graphs" is to get a count. Graphs have been disabled for a while now, presumably people have been removing them from pages, and thus also removing the category. Maybe looking at an older dump of the category table might be better but probably quite a bit of work. Bawolff (talk) 23:18, 14 July 2023 (UTC)


 * I'm not sure how accurate using "Pages with disabled graphs" is to get a count. Graphs have been disabled for a while now, presumably people have been removing them from pages, and thus also removing the category.
 * Mmm, great spot, @Bawolff. It looks like @TheDJ spotted a similar issue.
 * As a next step, I'm going to ask some folks internally about how we might go about generating a more accurate measure of usage. Of course, if new ideas emerge in your mind for how we might go about doing this and/or if volunteers who you think would be well equipped to help out with this, please let me know. PPelberg (WMF) (talk) 16:58, 28 July 2023 (UTC)

Including personal opinions
@Bawolff: I appreciate you both being bold in adding this statement and asking whether it's appropriate to do so.

What if you prefaced the bit beginning with "The elephant in the room is that graph is a high maintenance extension... with your username so as to show this is a personal opinion that may or may not be something the information we end up gathering substantiates? How does this sound to you? PPelberg (WMF) (talk) 23:54, 14 July 2023 (UTC)


 * SGTM. Bawolff (talk) 23:59, 14 July 2023 (UTC)
 * Awesome. PPelberg (WMF) (talk) 00:03, 15 July 2023 (UTC)

Generating graphs from Wikidata
hi – two things!


 * 1) Thank you for contributing helpful information to the Extension:Graph/Plans page  🙏🏼
 * 2) Related to "1.", are you able to add an example or perhaps describe in a bit more detail what an up-to-date graph from Wikidata data generated using the Graph Extension looked like? I ask as this is the first time I'm hearing of the extension being used in this way.

Ok, thank you ^ _ ^ PPelberg (WMF) (talk) 21:37, 28 July 2023 (UTC)


 * Hi you can find one example here (see the wikicode). In short, when the graph extension was working, this template get the neutron lifetime from Wikidata and then plot these values as a function of the year.
 * So that, when I add a new value on Wikidata, the graph is automatically updated. Pamputt (talk) 21:46, 28 July 2023 (UTC)
 * Ah, I see – thank you for following up with this added context and for doing so so promptly, @Pamputt!
 * Sure enough, just after I posted the above, I also saw that @Nux helpfully added the example of en:Template:Airport-Statistics.
 * In any event, @Pamputt can you please have a quick look at the below and let me know if there is anything missing, unnecessary, and/or inaccurate about how I'm currently understanding how Wikidata, the Graph Extension, and on-wiki templates can be used to generate auto-updating graphs?
 * How Peter (me) currently understands the relationships between Wikidata, the Graph Extension, and on-wiki templates
 * 1. Use SPARQL to retrieve the information you're seeking from Wikidata. ''[https://query.wikidata.org/#%23%20Scroll%20down%20and%20hit%20blue%20arrow%20down%20to%20run%20and%20see%20the%20results%20%2B%20the%20sources%0ASELECT%20%3Fyear%20%3Fitem%20%3Fshortname%20%28MAX%28%3Fnumber%29%20AS%20%3Fpassengers%29%20%20%20%28SAMPLE%28COALESCE%28%3Freference_URL%2C%20%3Fmonthly_reference_URL2%29%29%20AS%20%3Fsample_reference_URL%29%0AWITH%0A%7B%20%20SELECT%20%3Fitem%20%3Fstatement%20%3Fdate%20%3Fyear%20%3Ftimevalue%20%3Fnumberperperiod%20%3Freference_URL%0A%20%20WHERE%20%20%7B%20%20%20%20%3Fitem%20wdt%3A%20%3Fairport_code%0A%20%20%20%20VALUES%20%3Fairport_code%20%20%20%20%20%7B%20%22%22%20%20%20%20%7D%0A%20%20%20%20%3Fitem%20p%3AP3872%20%3Fstatement.%0A%20%20%20%20%3Fstatement%20pqv%3AP585%20%3Ftimevalue%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20ps%3AP3872%20%3Fnumberperperiod.%0A%20%20%20%20%3Ftimevalue%20wikibase%3AtimeValue%20%3Fdate.%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20pq%3AP518%20%3Fapplies.%20%7D%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20prov%3AwasDerivedFrom%20%2F%20%28pr%3AP854%7Cpr%3AP4656%29%20%3Freference_URL.%20%7D%0A%20%20%20%20FILTER%20%28BOUND%28%3Fapplies%29%3Dfalse%20%7C%7C%20%3Fapplies%20%3D%20wd%3AQ2165236%20%29%0A%20%20%20%20MINUS%20%7B%20%3Fstatement%20wikibase%3Arank%20wikibase%3ADeprecatedRank%20%7D%0A%20%20%20%20BIND%20%28YEAR%28%3Fdate%29%20AS%20%3Fyear%29%0A%20%20%20%20FILTER%20%28%3Fyear%20%3E1949%29.%20%20%20%20FILTER%20%28%3Fyear%20%3C%20YEAR%28NOW%28%29%29%29%0A%20%20%7D%20%7D%20AS%20%25airport%0AWHERE%0A%7B%20%20%20%7B%20%20%20%20%23%20Get%20the%20sum%20of%20monthly%20values%20within%20a%20year%0A%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28SUM%28%3Fmax_numberperperiod%29%20AS%20%3Fnumber%29%20%28SAMPLE%28%3Fmonthly_reference_URL%29%20AS%20%3Fmonthly_reference_URL2%29%0A%20%20%20%20WHERE%0A%20%20%20%20%7B%20%20%20%20%20%20%23%20Get%20the%20maximal%20value%20and%20a%20sample%20reference%20URL%20for%20each%20unique%20month%0A%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28MAX%28%3Fnumberperperiod%29%20AS%20%3Fmax_numberperperiod%29%20%28SAMPLE%28%3Freference_URL%29%20AS%20%3Fmonthly_reference_URL%29%0A%20%20%20%20%20%20%20%20WHERE%20%20%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20%20%20INCLUDE%20%25airport%0A%20%20%20%20%20%20%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%20%3Fprec.%0A%20%20%20%20%20%20%20%20%20%20FILTER%20%28%3Fprec%20%3E%209%29%23%20precision%20more%20precise%20or%20equal%20to%20month%0A%20%20%20%20%20%20%20%20%7D%20%20%20%20%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%20%3Fdate%0A%20%20%20%20%20%20%7D%20%20%20%20%7D%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%0A%20%20%7D%20%20UNION%20%20%7B%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%209%20.%20%20%20%20BIND%20%28%3Fnumberperperiod%20AS%20%3Fnumber%29%20%20%20%20BIND%20%28%3Freference_URL%20AS%20%3Fsample_reference_URL%29%0A%20%20%20%20INCLUDE%20%25airport%0A%20%20%7D%0A%20%20OPTIONAL%20%7B%3Fitem%20wdt%3AP1813%20%3Fthis.%20%20%20%20%20%20%20%20%20%20%20%20%23%20has%20shortname%0A%20%20%20%20FILTER%28LANG%28%3Fthis%29%3D%22en%22%29%20%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%2Cen%22.%20%3Fitem%20rdfs%3Alabel%20%3FitemLabel.%7D%0ABIND%28COALESCE%28%3Fthis%2C%3FitemLabel%29%20as%20%3Fshortname%29%0A%7D%20GROUP%20BY%20%3Fitem%20%3Fshortname%20%3Fyear%20ORDER%20BY%20%3Fitem%20DESC%20%28%3Fyear%29 Example SPARQL query]''
 * 2. Use a template to ingest the data the query you wrote in "1." will have returned and convert that data into a format (e.g. JSON) that the Graph extension can understand
 * 3. Generate a graph using the Graph Extension by "calling" the data you generated in "2." by adding the  notation to the page that you'd like the graph to appear on PPelberg (WMF) (talk) 21:59, 28 July 2023 (UTC)
 * Yes, this is how it worked. The SPARLQ query directly returns the data in the JSON format. From this JSON, the key are the ones given in the SPARQL query (i.e. "main_val", "lower", etc.), and so we can call them in the graph code to format the data as we wish. Pamputt (talk) 04:00, 29 July 2023 (UTC)
 * Thank you for following up with this additional clarity, @Pamputt.
 * To be doubly certain, it sounds like it's two steps (listed below) rather than three steps (as I'd thought above), does that sound accurate?
 * Use SPARQL to retrieve the information you're seeking from Wikidata. ''[https://query.wikidata.org/#%23%20Scroll%20down%20and%20hit%20blue%20arrow%20down%20to%20run%20and%20see%20the%20results%20%2B%20the%20sources%0ASELECT%20%3Fyear%20%3Fitem%20%3Fshortname%20%28MAX%28%3Fnumber%29%20AS%20%3Fpassengers%29%20%20%20%28SAMPLE%28COALESCE%28%3Freference_URL%2C%20%3Fmonthly_reference_URL2%29%29%20AS%20%3Fsample_reference_URL%29%0AWITH%0A%7B%20%20SELECT%20%3Fitem%20%3Fstatement%20%3Fdate%20%3Fyear%20%3Ftimevalue%20%3Fnumberperperiod%20%3Freference_URL%0A%20%20WHERE%20%20%7B%20%20%20%20%3Fitem%20wdt%3A%20%3Fairport_code%0A%20%20%20%20VALUES%20%3Fairport_code%20%20%20%20%20%7B%20%22%22%20%20%20%20%7D%0A%20%20%20%20%3Fitem%20p%3AP3872%20%3Fstatement.%0A%20%20%20%20%3Fstatement%20pqv%3AP585%20%3Ftimevalue%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20ps%3AP3872%20%3Fnumberperperiod.%0A%20%20%20%20%3Ftimevalue%20wikibase%3AtimeValue%20%3Fdate.%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20pq%3AP518%20%3Fapplies.%20%7D%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20prov%3AwasDerivedFrom%20%2F%20%28pr%3AP854%7Cpr%3AP4656%29%20%3Freference_URL.%20%7D%0A%20%20%20%20FILTER%20%28BOUND%28%3Fapplies%29%3Dfalse%20%7C%7C%20%3Fapplies%20%3D%20wd%3AQ2165236%20%29%0A%20%20%20%20MINUS%20%7B%20%3Fstatement%20wikibase%3Arank%20wikibase%3ADeprecatedRank%20%7D%0A%20%20%20%20BIND%20%28YEAR%28%3Fdate%29%20AS%20%3Fyear%29%0A%20%20%20%20FILTER%20%28%3Fyear%20%3E1949%29.%20%20%20%20FILTER%20%28%3Fyear%20%3C%20YEAR%28NOW%28%29%29%29%0A%20%20%7D%20%7D%20AS%20%25airport%0AWHERE%0A%7B%20%20%20%7B%20%20%20%20%23%20Get%20the%20sum%20of%20monthly%20values%20within%20a%20year%0A%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28SUM%28%3Fmax_numberperperiod%29%20AS%20%3Fnumber%29%20%28SAMPLE%28%3Fmonthly_reference_URL%29%20AS%20%3Fmonthly_reference_URL2%29%0A%20%20%20%20WHERE%0A%20%20%20%20%7B%20%20%20%20%20%20%23%20Get%20the%20maximal%20value%20and%20a%20sample%20reference%20URL%20for%20each%20unique%20month%0A%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28MAX%28%3Fnumberperperiod%29%20AS%20%3Fmax_numberperperiod%29%20%28SAMPLE%28%3Freference_URL%29%20AS%20%3Fmonthly_reference_URL%29%0A%20%20%20%20%20%20%20%20WHERE%20%20%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20%20%20INCLUDE%20%25airport%0A%20%20%20%20%20%20%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%20%3Fprec.%0A%20%20%20%20%20%20%20%20%20%20FILTER%20%28%3Fprec%20%3E%209%29%23%20precision%20more%20precise%20or%20equal%20to%20month%0A%20%20%20%20%20%20%20%20%7D%20%20%20%20%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%20%3Fdate%0A%20%20%20%20%20%20%7D%20%20%20%20%7D%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%0A%20%20%7D%20%20UNION%20%20%7B%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%209%20.%20%20%20%20BIND%20%28%3Fnumberperperiod%20AS%20%3Fnumber%29%20%20%20%20BIND%20%28%3Freference_URL%20AS%20%3Fsample_reference_URL%29%0A%20%20%20%20INCLUDE%20%25airport%0A%20%20%7D%0A%20%20OPTIONAL%20%7B%3Fitem%20wdt%3AP1813%20%3Fthis.%20%20%20%20%20%20%20%20%20%20%20%20%23%20has%20shortname%0A%20%20%20%20FILTER%28LANG%28%3Fthis%29%3D%22en%22%29%20%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%2Cen%22.%20%3Fitem%20rdfs%3Alabel%20%3FitemLabel.%7D%0ABIND%28COALESCE%28%3Fthis%2C%3FitemLabel%29%20as%20%3Fshortname%29%0A%7D%20GROUP%20BY%20%3Fitem%20%3Fshortname%20%3Fyear%20ORDER%20BY%20%3Fitem%20DESC%20%28%3Fyear%29 Example SPARQL query]''
 * Generate a graph using the Graph Extension by "calling" the data you generated in "1." by adding the  notation to the page that you'd like the graph to appear on
 * PPelberg (WMF) (talk) 00:14, 1 August 2023 (UTC)
 * Yes, that's it :) Pamputt (talk) 04:48, 1 August 2023 (UTC)
 * Yes, that's it :) Pamputt (talk) 04:48, 1 August 2023 (UTC)

Update: 11 August
hey y'all – this update is an effort to help us:
 * 1) Align on what we've come to know and think about the Graph Extension
 * 2) Work together to find the information needed to address the remaining questions/uncertainties

I'm thinking the above will help us arrive at the requirements needed to distinguish viable from non-viable solutions and for the WMF (by way of me) to propose some paths forward for us to consider in the next update, planned for 15 September.

With this in mind, if you see something below that you disagree with, think could benefit from additional clarity/specificity, etc., please say as much by commenting below!

...this update is meant to identify any gaps and inconsistencies in what we're collectively thinking so that we can move forward together.

Now, before getting into what's become more clear since the 14 July update and what new uncertainties/questions surfaced: thank you!

Thank you to @Ahecht, @Bawolff, @Bouzinac, @Edu!, @Matma Rex, @Msz2001, @Novem Linguae, @Nux, @Pamputt, @Pppery, @Quiddity, @RobinLeicester, @User:SBassett (WMF), @Snævar, @Tgr, @TheDJ, and @User:VPoundstone-WMF for patiently and generously sharing the information and resources I've needed to understand and orient myself within the decisions we find ourselves needing to make.

Now onto the update… PPelberg (WMF) (talk) 18:55, 11 August 2023 (UTC)


 * Help needed: what question do we need help answering?
 * How might we go about inviting non-English speakers into this conversation and making them feel welcomed to contribute?
 * How might we go about making more people aware that this conversation is happening? It's important that we increase the likelihood that people who have experiences with and information about the Graph Extension be aware of, and ideally, a part of this conversation.
 * How might we go about forming a more accurate understanding of how widely used the Graph Extension was in relation to features that offer similar functionality?
 * PPelberg (WMF) (talk) 18:56, 11 August 2023 (UTC)
 * For #3, @Bawolff's old write-up is still probably the most comprehensive existing analysis and/or starting point for that question: https://www.mediawiki.org/wiki/User:Bawolff/Reflections_on_graphs SBassett (WMF) (talk) 18:42, 22 August 2023 (UTC)
 * Understood – thank you for pointing me to this, @SBassett (WMF); I've added this to the main page. PPelberg (WMF) (talk) 22:10, 23 August 2023 (UTC)
 * Clarity: what do we agree to be true about the Graph extension and the needs it was meant to meet?
 * There seem to be three distinct features the Graph Extension offered for which there are no current workarounds/alternatives:
 * Generating interactive maps/charts/graphs
 * Where "interactive" in this context refers to functionality/features like: enabling individual people viewing a graph to adjust the underlying parameters that determine the data they see and how it is visualized.
 * Generating maps/charts/graphs that update automatically
 * This is most often done by pulling in data from Wikidata.
 * Updating maps/charts/graphs on-wiki
 * The Graph Extension enables people to generate graphs using data stored on-wiki. This means, people can update graphs by editing the data on wiki, similar to how they might edit any other page/piece of information on the wikis.
 * In addition to the "distinct features" the Graph Extension offered, it also enabled people to:
 * Specify how this data is represented (e.g. chart type, colors, legend, mouseover events/interactions, etc.)
 * Store data within templates on pages separate from articles
 * Embed/transclude graphs within articles using templates
 * Write Lua modules that:
 * Convert JSON data into a well formatted wiki table
 * Extract data needed for graph and outputs as JSON
 * Create graph templates that can consume data and plot it
 * Pull in data from external sources via URLs
 * From generating article page view graphs to showing air traffic volume, volunteers depended on the Graph Extension in a variety of reader- and editor-focused contexts.
 * PPelberg (WMF) (talk) 18:57, 11 August 2023 (UTC)
 * Impressions: what seems like it could be true? 
 * Absent accessible and easy-to-use features that enable people to store, collaborate on, and present data on-wiki:
 * The data we host/present on our projects is more vulnerable to being outdated/inaccurate
 * The content we offer is more likely to lack corresponding data visualizations
 * In order for the Movement as a whole, and the various communities that comprise it, to evaluate the impact of the work it's doing, they need ways of generating, updating, and visualizing data.
 * To use the Graph Extension, you need/needed a relatively high degree of technical expertise on top of a foundation of data literacy.
 * Related to "3.", the Graph Extension has remained a relatively niche feature that one could, theoretically, never interact with (directly or indirectly) throughout your tenure as an editor.
 * PPelberg (WMF) (talk) 18:57, 11 August 2023 (UTC)
 * @PPelberg I'm interesting in Template:Graph:PageViews: we really need this mechanism working in ruwiki. MBH (talk) 05:07, 19 August 2023 (UTC)
 * "Template:Graph:PageViews: we really need this mechanism working in ruwiki."
 * @MBH: this is helpful to know – I'm glad you said something.
 * To help me more fully understand the extent to which you, and other volunteers at ru.wiki, depend on Template:Graph:PageViews, can you please share what has been made difficult or not possible as a result of it no longer functioning? [i]
 * i. E.g. Are there particular actions decisions that are more difficult, or perhaps not possible, to make because you do not have easy access to information about how views of a particular page vary over time? PPelberg (WMF) (talk) 22:06, 23 August 2023 (UTC)
 * @PPelberg 1) ruwiki extensively uses this template on articles' talk pages, 2) I own a bot for ru- and ukwiki, that generates articles' peak popularity (number of readers) monthly and yearly stats, see ru:File:Просмотры статей об айти-разработке.png, ru:ВП:Пики интереса к статьям/За год, uk:Вікіпедія:Спалахи інтересу до статей/За рік, this stats pages uses this template and now they are broken. MBH (talk) 18:05, 24 August 2023 (UTC)
 * @PPelberg 1) ruwiki extensively uses this template on articles' talk pages, 2) I own a bot for ru- and ukwiki, that generates articles' peak popularity (number of readers) monthly and yearly stats, see ru:File:Просмотры статей об айти-разработке.png, ru:ВП:Пики интереса к статьям/За год, uk:Вікіпедія:Спалахи інтересу до статей/За рік, this stats pages uses this template and now they are broken. MBH (talk) 18:05, 24 August 2023 (UTC)


 * I could really use a mechanism for generating graphs like Our World in Data; something requested at the start of the pandemic and many times since. "we should aim to have a world map graph template just like [OWID], regardless of what we have to do with the graph extension or services behind it." Sj (talk) 07:20, 30 August 2023 (UTC)

Underlying purpose inspiring the extension
Improving graphs and interactive content and Yurik's Dream of Content seem as relevant as ever! This isn't quite responsive to Peter's questions above which were specifically about use of / need for the Graph extension as currently implemented, so I'm putting these points in a separate section; feel free to refactor or merge.


 * Data visualization is an important part of a more beautiful and richly sourced references: helpful to readers, and to the identity of the projects.
 * The existence of good data viz adds incentive to get the underlying data on the projects (and to get it right)
 * The ability to do good data viz attracts people who care about that facet of knowledge production and appreciate an impactful place to contribute that to the commons. This is where having tools that let us make interactive results leads to a new ecosystem of curators and readers
 * Many of our most common use cases might instead use the same visualization libraries as OWID: their context and audience and need for customizable compact summaries from shared data sources is quite similar to ours.
 * Continuity and archival quality: if needed we can downgrade interactives to static visuals, but should not remove data visualizations without replacement. If the extension is going away, there should be a way to convert removed graphs to a static image [that can be run at scale]
 * Uncertainty dissipates community. It helps to have a maintainer and roadmap/plan of record for "how we visualize data on the projects". This is always going to be a range of uses, with a power law, some of which have many alternatives. Currently people who might help develop alternatives are still thinking about Vega migrations (cf the talk page) and may be wasting their time.
 * Losing previous content without replacement for 6 months is weird. We should at least render all previous graphs and embed the resulting images.  Not having a maintainer to do this or explain why it's not being done is weird.  An extended conversation about what we may or may not do in the future (without first doing the above as a stopgap, to avoid losing content people spent thousands of hours making) is very weird. And teaches people not to trust the wikis as a place to make and share visualizations. (Part of the challenge is that it's not obvious who should do it, or if it will be a waste of time b/c a more official solution is about to drop.)
 * We all benefit from building a shared vision for how and where data visualization happens on the projects, and how that fits into the overall development of reading and editing. This is broader and simpler than the questions about this extension.
 * Any sort of roadmap is fine. "eventually reenable a comparable extension" is clear. "we can't secure this, extension is deprecated + tasks closed, please migrate off w/ these tools; we'll find another solution" would be clear. "tools to visualize data should be maintained by community and not depend on staff or need hefty security review" would be clear. We could still debate changing the roadmap but at any point in time the current roadmap should be legible. Sj (talk) 07:20, 30 August 2023 (UTC) 13:18, 6 September 2023 (UTC)


 * @Sj responses/follow up questions to the points you raised below.
 * Before that, thank you for starting this conversation.
 * I think you are spot-on in naming the need for exploring the broader themes/capabilities the Graph Extension is an expression of and is crucial context to consider as we propose a set of next steps.[i]
 * Note: I'm going to post responses in separate comments so that we can [hopefully] explore multiple threads in parallel a bit more easily.
 * i. Propose next steps will come via the update I'll share next Friday, 15 Sep. PPelberg (WMF) (talk) 23:16, 8 September 2023 (UTC)
 * Data visualization is an important part of a more beautiful and richly sourced references: helpful to readers, and to the identity of the projects.
 * I share this assumption that readers value data visualizations.
 * Have you seen/read any research that substantiates this? Asked another way: are you able to share what you've noticed that contribute to you thinking the above is true?
 * For context, the most compelling research I've come across to-date related to the above being true is a study from the Research Team. This research found illustrated Wikipedia pages receive 4x more pageviews than those without images. That same research also found the ratio of image clicks to pageviews to be 1:30 compared to 1:340 for citations and 1:110 for links. PPelberg (WMF) (talk) 23:20, 8 September 2023 (UTC)
 * I would argue that video is a better model for data visualization than an image. It is relatively easy to improve an article with an image. Even a bad image is often better than no image if that's all that is available. Video is much harder - it can improve an article, but it is much more important that the media is of high quality and tells a "story". It is much harder to create a good video, and a bad video really does nothing for an article. I think this is what data visualization is like - it can significantly improve an article, but only if good, plot is more important, and it is much more difficult to create a good visualization regardless of the tools. Bawolff (talk) 02:37, 9 September 2023 (UTC)
 * There are all sorts of readers. Areas where this is not in question: Scholarly journals and research have prized visualizations for over a century. So have statistical or mathematical topics such as budgets, economic histories, epidemiologies, &c. For general-purpose descriptions to a lay audience, textbooks for many decades, encyclopedias since Encarta, and more recently with data journalism (in news) and Wolfram Alpha (in search), subfields or major enterprises have been built on the premise + promise of reliable, legible visualization. Good viz brings readers who better understand topics visualized that way. For current readers on an average article that includes some graphable data: I don't have specific research, but infoboxes and summary tables seem quite popular and widely referenced / screenshotted.  Sj (talk) 23:27, 11 September 2023 (UTC)
 * I would argue that video is a better model for data visualization than an image. It is relatively easy to improve an article with an image. Even a bad image is often better than no image if that's all that is available. Video is much harder - it can improve an article, but it is much more important that the media is of high quality and tells a "story". It is much harder to create a good video, and a bad video really does nothing for an article. I think this is what data visualization is like - it can significantly improve an article, but only if good, plot is more important, and it is much more difficult to create a good visualization regardless of the tools. Bawolff (talk) 02:37, 9 September 2023 (UTC)
 * There are all sorts of readers. Areas where this is not in question: Scholarly journals and research have prized visualizations for over a century. So have statistical or mathematical topics such as budgets, economic histories, epidemiologies, &c. For general-purpose descriptions to a lay audience, textbooks for many decades, encyclopedias since Encarta, and more recently with data journalism (in news) and Wolfram Alpha (in search), subfields or major enterprises have been built on the premise + promise of reliable, legible visualization. Good viz brings readers who better understand topics visualized that way. For current readers on an average article that includes some graphable data: I don't have specific research, but infoboxes and summary tables seem quite popular and widely referenced / screenshotted.  Sj (talk) 23:27, 11 September 2023 (UTC)


 * The existence of good data viz adds incentive to get the underlying data on the projects (and to get it right)
 * Mmm. The relationship you're naming between the usefulness of a knowledge format/representation and peoples' likelihood to contribute the knowledge needed to produce them seems insightful and accurate to me.
 * To the "accuracy" bit, are you able to share a bit about how you seem to have come to think that this relationship holds specifically true for data visualizations in our ecosystem?
 * No worries if this is more of an intuition at this point. Although, if there are things you can point to that cause you to feel confident in this statement, I'd value seeing them. PPelberg (WMF) (talk) 23:21, 8 September 2023 (UTC)
 * Historically across a lot of formats, first one is tried as a style (dot maps; nav templates; infoboxes; short WD descriptions), then it gets incorporated into various editing scripts and bots, then at some point one or two people dedicate themselves to populating and cleaning up the format across whole categories of relevant pages. We have scattered and inhomogenous ways to store tabular data and time series data; having a central use case like this will lead to scripts and other tools standardizing elements that make the data more findable, better updated, editable and more often corrected. Having a quick way to visualize data will over time lead to identifying and fixing mistake and outliers (visible across dozens of covid-tracking projects). Sj (talk) 23:27, 11 September 2023 (UTC)


 * The ability to do good data viz attracts people who care about that facet of knowledge production and appreciate an impactful place to contribute that to the commons. This is where having tools that let us make interactive results leads to a new ecosystem of curators and readers
 * The potential you're naming here is inspiring to me and it leads me to wonder the following…
 * Assuming it's accurate to think the interactive graphs the Graph Extension enabled volunteers to produce did NOT produce the "new ecosystem of curators and readers" you're referring to above, do you have a sense for what might have contributed to this? PPelberg (WMF) (talk) 23:22, 8 September 2023 (UTC)
 * That would not be accurate. It needs an easier onramp, and a place to store versioned tabular data that can be used as the input to such graphs, but the tens of thousands of pages using it, and the handful of specific projects + maintainers on those projects that used it extensively (and commented immediately when it went away) were members of that ecosystem. These things often have a slow takeoff before there is a critical mass of people learning from and borrowing from one another, filling in the various gaps. OWID illustrates the point nicely: we have an obvious source of freely-licensed data on hundreds of topic areas, hundreds of editors spending thousands of hours making equivalent data into static images on Wikipedia (like the 1500 revisions of this graph of COVID cases in Canada), no easy way to do that with Graph (despite a dozen people specifically trying and failing to make that happen). There was also a WikiGraphers User Group created recently, but not pointed to this extension as the natural place to start graphing. Sj (talk) 23:27, 11 September 2023 (UTC)


 * Many of our most common use cases might instead use the same visualization libraries as OWID: their context and audience and need for customizable compact summaries from shared data sources is quite similar to ours.
 * Great spot. I'm thinking this information will come in handy if/when we arrive in a place where we're exploring the possibility of building a successor to the Graph Extension. In the meantime, I've documented this idea on T345962. PPelberg (WMF) (talk) 23:25, 8 September 2023 (UTC)
 * Continuity and archival quality: if needed we can downgrade interactives to static visuals, but should not remove data visualizations without replacement.
 * +1; excellently put. PPelberg (WMF) (talk) 23:26, 8 September 2023 (UTC)
 * If the extension is going away, there should be a way to convert removed graphs to a static image [that can be run at scale].
 * This sounds right to me too. I'm checking with engineering to understand the viability of what you're proposing. I'll report back what I hear here and on T334940 (thank you for raising this there). PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)
 * Uncertainty dissipates community.
 * Agreed and I appreciate you naming this potential.
 * Related to the above: can you think of ways we could improve how we're communicating about the Graph Extension and the plans surrounding it?
 * Perhaps the question above is better suited for a "retro" of sorts once a plan is in place. Tho, I thought I'd seed the question now in case immediate ideas/suggestions came to mind. PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)
 * Currently people who might help develop alternatives are still thinking about Vega migrations (cf the talk page) and may be wasting their time.
 * I hadn't seen this comment; thank you for drawing my attention to it. I'll follow up with this person to try and offer some clarity about what we're thinking. PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)
 * We all benefit from building a shared vision for how and where data visualization happens on the projects, and how that fits into the overall development of reading and editing. This is broader and simpler than the questions about this extension.
 * +1 and I think you investing the time to articulate the above is starting the conversation necessary to help us do what you're describing…thank you!
 * In terms of what's next, here's what I'm thinking:
 * Between now and next Friday, I'll be preparing an update that will include:
 * A proposal for what we do with Graph Extension and the visualizations that have been rendered inaccessible by it being disabled
 * A broader snapshot of how we're currently thinking about the strategic relevance of supporting the creation of, and collaboration on, on-wiki data visualizations.
 * How does that sound? What – if anything – about "1." and "2." would you suggest we adjust and/or add to provide the clarity and direction I think you aptly named us all as collectively needing.
 * Note: Please know the questions I'm asking above are in service of me being newer to this context and motivated to understand how you've come to arrive at these points of view so that we can reason about this space together ^ _ ^ PPelberg (WMF) (talk) 23:30, 8 September 2023 (UTC)
 * I'd just like to echo the sub-bullet that was not quoted here, about having a legible roadmap. A proposal and strategic snapshot will be nice, but I hope that if these do not represent a firm decision that's been made, an ETA can be communicated on when such a decision will be made. For people who are vaguely aware of how software development works, "we don't know when this will be fixed" is an understandable position. "We don't know when this will be fixed, or if it will be fixed, or what will replace it or when it will be replaced" is not very encouraging. Even a decision not to support a graphing module at the core level and kicking it back to the community is a decision that will give volunteers the green light to start working on a replacement. Folly Mox (talk) 01:56, 11 September 2023 (UTC)
 * Seconded! I keep running to graphs all over en.Wikipedia that are broken, and for some articles, it has a really significant impact. We need a clear plan and communication one way or another. Ganesha811 (talk) 14:04, 11 September 2023 (UTC)
 * @Folly Mox: I hope that if these do not represent a firm decision that's been made, an ETA can be communicated on when such a decision will be made.
 * @Ganesha811: We need a clear plan and communication one way or another.
 * +1 to both of the points you both are raising which I understand to be something like: "''While a broader strategic conversation is important, we first need clear direction from you all, the WMF, about what – if anything – you're going to do to improve the current state of things."
 * Assuming I've understood y'all accurately, you can expect me to offer this clarity via an update to this page and T334940 before this Friday, (15 Sep) is over. More context here. PPelberg (WMF) (talk) 19:20, 11 September 2023 (UTC)
 * That sounds good, and yes, I think you've summarized the central point correctly. Every day that graphs are broken is a day that tens of thousands of Wikipedia articles are incomplete (often in significant ways), and therefore are not serving readers well. Ganesha811 (talk) 19:23, 11 September 2023 (UTC)
 * Excellent. Thank you for reviewing, @Ganesha811. And thank you for acting on the instinct to come here to speak up. PPelberg (WMF) (talk) 19:26, 11 September 2023 (UTC)
 * Thank you for the expectation of clarity. I think that's what people are seeking in general. The before / after the strategic conversation chronology is less meaningful to me than the "after five months" temporal milestone, but it sounds like you've got some bullet points prepared for Friday that will indicate what the overall vibe is regarding the issue, so I consider my concerns allayed. Folly Mox (talk) 20:06, 11 September 2023 (UTC)
 * Thank you for the expectation of clarity. I think that's what people are seeking in general. The before / after the strategic conversation chronology is less meaningful to me than the "after five months" temporal milestone, but it sounds like you've got some bullet points prepared for Friday that will indicate what the overall vibe is regarding the issue, so I consider my concerns allayed. Folly Mox (talk) 20:06, 11 September 2023 (UTC)

Update: 15 September
Hi y'all – this update is an effort to arrive at the clarity we need to ensure volunteers around the Movement regain access to the information and capabilities disabling the Graph Extension has left them without.

Within the next two weeks, you can expect another update about how we are thinking about the broader strategic importance of graphs.

With the above in mind, here's what we need your help with to move forward: Can you please review the "Proposal" and "Vega 2 → Vega 5 transition" sections below and share what seems unclear, missing, and/or misguided?

---

Notes: the "Proposal" that follows is an outgrowth of the discussions and information gathering that's been taking place, in large part, on this page and Extension:Graph/Plans over the past three months.

''Note: exploratory work to redeploy the Graph Extension in a sandboxed iframe has started. See T222807.'' ''Note: We estimate this converter currently works for ~80% of graphs, with diminishing returns on additional engineering effort to cover more. We do not plan on continuing to invest significant additional engineering resources here, but instead to simply repurpose the existing codebase as an aid to manual porting.''
 * Proposal
 * To safely restore access to the information and capabilities disabling the Graph Extension has left people without and promote the volunteer+staff collaboration needed to do so, we (the WMF) are committing to:
 * Re-enable the Graph Extension in a sandboxed iFrame with a restrictive content security policy.
 * Once the Graph Extension is reenabled, it will continue to work with Vega 2 for a yet-to-be defined period of time. Note: we'll need to define this window together.
 * After that "yet-to-be defined period of time," Vega 2 support will be discontinued and use of the Graph Extension will require volunteers to make graphs with Vega 5.
 * As soon as possible, make the sandboxed Graph extension available on the beta cluster for testing. See: T346292.
 * Investigate the viability of adding logging to increase our awareness of instances where people are exploiting the security vulnerabilities inherent with restoring support for Vega on our platform. See T346414.
 * Publish the technical documentation needed for developers across the Movement to understand how we implemented the sandboxed CSP approach
 * Publish a clear timeline for when you all can expect all of the above to happen
 * Share regular updates about the progress we're making on the commitments named above on Phabricator and MediaWiki.
 * Support volunteers with code and processes that will ease the transition from Vega 2 to Vega 5 when the time for this transition comes.
 * In support of the above, we'd need to depend on ya'll (volunteers) to:
 * Spread awareness of this proposal and the updates that will come as we start implementation, assuming this proposal moves forward.
 * Manually migrate some proportion of Vega 2-based graphs to be compatible with Vega 5. See the "Vega 2 → Vega 5 transition" section below.
 * Potentially, fix/port graphs that attempted to fetch live data using methods that the sandboxing approach inhibits.
 * Note: the need for the above will become clear once we decide on whether we will restore the pseudo-protocols that were used to fetch data live from the action API, the REST API, WDQS etc, and the precise sandbox parameters we select (domains/ports/http methods allowed). This decision will be made in T346291.
 *  Vega 2 → Vega 5 Transition 
 * Why do we think it's worthwhile to migrate from Vega 2 to Vega 5?
 * Vega 2 has been superseded by Vega 3, 4, then 5 upstream.  Upstream and third-party documentation exclusively refers to syntax in “Vega version 3.0 and later”, and it is difficult for new contributors to find documentation relevant to Vega 2.  The last upstream release (bugfix or security) of Vega 2.x was in January 2017.  Vega 5 was released in March 2019 and is still under active maintenance and development, with the latest 5.25.0 release in April 2023.
 * Volunteers have reported issues with Vega 2's accessibility, syntax, and overall functionality, per this 2023 wish.
 * Vega 5 has made improvements to the library's expression layer that harden it from a security perspective compared to Vega 2.  It is not perfect, but by introducing a parsed expression grammar it offers a more robust foundation for additional security hardening in the future if it proves necessary.
 * Maintaining multiple versions of Vega concurrently is unsustainable in the long run. The wiki community is taxed in the attempt to independently support software which is not being maintained upstream. Our efforts are best spent working in cooperation with upstream and third-party developers, and to do this we need to be working from the upstream Vega 5 code base.
 * What might be required to migrate graphs from Vega 2 to Vega 5?
 * Create a converter that would migrate Vega 2-based graphs to be compatible with Vega 5. @Jdlrobson started work on an initial approach in T335048#8794138.  The initial work needs to be restructured slightly to refocus it on being an aid to manual porting, instead of the automatic translator which was its original goal.
 * Volunteers would need to update  syntax on a case-by-case basis, aided by (1) the ability to run the existing Vega 2 and new Vega 5 specification side-by-side, (2) the partial Vega2-to-5 porting tool which handles 80% of the “obvious” keyword changes and other mechanical conversions, (3) the upstream Vega2 porting guide, and (4) additional documentation or tools which might be created by the wiki community.
 * Update the limited number of Scribunto templates on-wiki which generate  output in Vega 2 format to instead output Vega 5.  This requires both lua and Vega expertise, but fixes a larger number of Vega 2 uses on wiki at once.
 * PPelberg (WMF) (talk) 23:52, 15 September 2023 (UTC)


 * The open question over pseudo-protocols sounds like a pretty big sticking point. I totally understand the reluctance, even just from an operational perspective this couples a bunch of not-really-production things to production in a pretty uncontrolled way, but that would also seriously alter the applicable usecases for the extension. Bawolff (talk) 00:51, 16 September 2023 (UTC)
 * ...that would also seriously alter the applicable usecases for the extension.
 * @Bawolff: can you please say a bit more about the above? Are you most concerned about specific graphs no longer functioning were we to not restore the pseudo-protocols that were used to fetch data live from the action API, the REST API, WDQS etc.? Are you most concerned that the extension would be generally less useful? Something else? PPelberg (WMF) (talk) 23:44, 27 September 2023 (UTC)
 * i'm concerned its a pretty big decision to leave as a question mark Bawolff (talk) 08:38, 28 September 2023 (UTC)
 * I share this concern. I think that without access to data (which is really what the pseudo protocols are about), that the graphs themselves will loose a lot of their potential. Who’s writing json data arrays in wikitext or Lua modules ? —Th e DJ (Not WMF) (talk • contribs) 21:33, 28 September 2023 (UTC)
 * ''"Affording people the ability to fetch live data is core to what makes the Graph Extension uniquely valuable and difficult to find a replacement for."
 * @Bawolff + @TheDJ: assuming it's accurate for me to understand what y'all are sharing as what I've written above, then: understood. Based on what we talked about internally today, offering pseudo-protocol support sounds doable in this new sandboxed approach.
 * The question that remains is what exactly that support looks like. I expect for us (staff + volunteers) to be in a place to shape this together in the next few weeks.
 * Thank you for raising this. You doing so held us accountable to prioritizing talking about this during today's meeting.
 * On our end, we're processing the feedback everyone here has been sharing and working towards sharing an updated proposal in the coming weeks. PPelberg (WMF) (talk) 21:48, 28 September 2023 (UTC)
 * Answering the question of how much time it takes for graphs to become vega 5 is a math equation. There is an missing variable in that equation of how the pseudo-protocols get implemented. In the current gerrit master, pageviews works, commons is mostly there - it connects and there was an fix for an fetching issue, but with Graph disabled I was never able to verify it.
 * Wikidata is not supported at all, there is next to no code for it in gerrit. Wikidata support is not needed, volunteers (users) can use lua wikibase access (https://doc.wikimedia.org/Wikibase/master/php/docs_topics_lua.html), but rewriting from the current Wikidata SQL to lua wikibase is going to take three times as long as converting from vega2 to vega5, without touching the protocol bit. Snævar (talk) 09:54, 16 September 2023 (UTC)
 * Wikidata sparql and wikidata lua aren't really equivalent. There may be some things that can be converted but i would be very surprised if that was everything. Bawolff (talk) 10:02, 16 September 2023 (UTC)
 * As a Wikidata editor, I completely disagree. Wikibase's Lua functions are extremely limited in functionality and do not come close to what SPARQL can do. We mainly use graphs to display statistics (e.g. how many items are instances of something). We can't even fetch a list of inverse links using Lua, we definitely can't use it to create graphs like that. - Nikki (talk) 11:57, 19 September 2023 (UTC)
 * True, the pages using the template would need to be changed to say what item should be used. For example, lets look at a page that uses Template:Airport-Statistics, instead of specifying the airport code of the airport the graph is supposed to be of, it would specify the item id. Snævar (talk) 00:28, 9 October 2023 (UTC)
 * I do not see any signs of you reading the comments of the commits of when the protocols where being considered to be added in the former round in April. It only helps you doing so, giving you the chance to make a counter-argument. Snævar (talk) 00:50, 9 October 2023 (UTC)
 * I agree with Nux. Support for Vega 2 should be dropped with automatic translation to Vega 5 as temporary solution. There need to be clear message that graph is automatically converted and needs attention. It will motivate templates maintainers and graph users that directly use Vega syntax to migrate to Vega 5 instead of using static image as replacement for graph in article.
 * I interact with extensions mostly with Template:Graph:Chart template and in my experience this was most common use on Wikipedia articles for charts. Update on underlying Module:Graph is almost done (see |Module:Graph/sandbox, here  and here ). Test cases of template  show how well automatic translation works.
 * I'm trying to update map part of module but without wikiraw:// protocol (T335325) enabled for beta it's not possible to debug it efficiently. Same thing for other templates and modules relying on data fetch with "url".
 * Pietrasagh (talk) 16:28, 17 September 2023 (UTC)
 * Nux raises a compelling point. I think people will be OK with immediately transitioning to Vega 5 so long as it'll work "forever", as opposed to a temporary solution that'll stop working at some arbitrary time. Enterprisey (talk) 20:16, 17 September 2023 (UTC)
 * I agree with Nux, making the process larger may be interesting from a security point of view, but graphs have been broken for months, so just migrating to Vega5 would make the process better for readers, as they don't have two separate time windows with potential failure. Theklan (talk) 13:50, 18 September 2023 (UTC)
 * It sounds to me as though automatic conversion may not quite work, and people may need to see V2 and V5 side by side to test conversion / understand what's not yet working. But as long as the messaging is "V2 graphs no longer work" and v2 is only available in some side-by-side testing mode, it can be clear that we are immediately in the V5 migration period. Sj (talk) 19:38, 20 September 2023 (UTC)
 * I don't know much about the details of this (I'm only here because I was trying to find out when graphs will work again). A converter sounds like a "nice to have" feature, but I hope the priority is in getting graphs working again at all - having graphs which work if we fix them manually would be an improvement over having no graphs. - Nikki (talk) 12:12, 19 September 2023 (UTC)
 * Depending on the distribution of the Vega 2 usage, would it make sense to offer a Wikimedia/Wikipedia "simplified" graph template which would act as a front-end to Vega 5 or whatever graph module is used in the future? Is what the "pseudo-protocols" was referring to?
 * My point being: if there are a large set of simple, static (not dynamically-generated) graphs (e.g. basic pie charts, one-variable bar charts, and one-variable linear/log line graphs), can these be automatically converted to a simplified chart template ? As someone who has worked on just one stacked bar chart, I found the Vega 2 quite difficult; I imagine a less-technical user would like something they could, say, dump from a spreadsheet to make a simple graph. Jason Olshefsky (talk) 12:32, 19 September 2023 (UTC)
 * Module:Graph has served as an front-end for a lot of Vega graphs. In it's simplest form, it can be told some positions on the x-axis, some positions on the y-axis and the type of graph (f.x. line graph) and it gives the result.
 * pseudo-protocols are not a front-end. They are essentially (in a oversimplified way) shortened versions of an domain. After an protocol we specify the location of what we want. For example, "tabular://" is a protocol, pointing to commons.wikimedia.org/wiki/Data: and after that we specify what data namespace page we want. The data from that page is then used as data-points in the graph, or less commonly, as text in the graph.
 * The most simple graphs are likely to be auto-converted by the V2 to V5 converter. I do not think there is an list over the most simple graphs. Snævar (talk) 00:07, 9 October 2023 (UTC)
 * Pseudo-protocol stuff is useful info. Are these now enabled on the beta.wmflabs site? and are there ones that can access commons images and the OSM maps? RobinLeicester (talk) 11:36, 9 October 2023 (UTC)
 * No, you need to use full urls like the conversion from wikiraw in https://en.wikipedia.beta.wmflabs.org/wiki/Template:Graph:PageViews.
 * Connection to Wikidata Queries are nonexistant.
 * Tabular I have not got to work, a delevoper gave me this link https://commons.wikimedia.org/w/api.php?action=jsondata&formatversion=2&title=&format=json&origin=* but there is still some work to be done. Maybe beta does not have access to commons.wikimedia, only the beta version, IDK. Snævar (talk) 18:34, 10 October 2023 (UTC)
 * See also https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Graph/+/910613 Snævar (talk) 18:50, 10 October 2023 (UTC)
 * Thanks - with this info I have got commons images working in the beta. (hadn't thought to try the upload file name!) I have still no idea where to look for the maps that were available via mapsnapshot://, but then, I have not got any sort of handle on vega projections/transforms either, so still plenty to do. RobinLeicester (talk) 15:52, 11 October 2023 (UTC)
 * Thanks - with this info I have got commons images working in the beta. (hadn't thought to try the upload file name!) I have still no idea where to look for the maps that were available via mapsnapshot://, but then, I have not got any sort of handle on vega projections/transforms either, so still plenty to do. RobinLeicester (talk) 15:52, 11 October 2023 (UTC)

Why not just start with v5?
You ask for what time should the v2 be available. How about no time at all? It has been quite a long time when graphs were not working. Testing v2 and then doing migration seems like a waste of resources to me. As I understand using v2 and then v5 would mean that after a period of graphs working there would be again a period of graphs not working. This work-don't work cycle would not look good (would make Wikipedia look less reliable). So let's just make the new version work... Unless maybe both versions could work at the same time? Nux (talk) 12:58, 16 September 2023 (UTC)
 * Unless maybe both versions could work at the same time? Yes, at least that's what it implies to me. Edit: missed auto-migration part. I support your proposal to just auto migrate everything and just use vega 5. Edit 2: PPelberg (WMF) just thanked this reply so I think yes, both versions would work at the same time Aaron Liu (talk) 19:48, 18 September 2023 (UTC)
 * +1. If there's a compelling reason to do this in two steps rather than in one, I don't see it in what PPelberg posted above. Nardog (talk) 22:03, 18 September 2023 (UTC)

I could see both at the same time, with all V2 renders tagged as "needs conversion". Once most have been converted a scripted update could note on all [talk]pages using an unconverted graph that it's going to stop rendering again soon, then shift to no longer rendering V2, leaving the "needs conversion" tag. Sj (talk) 19:41, 20 September 2023 (UTC)
 * could note on all [talk]pages just do some hatnote in the template like a TfD hatnote, like the tag will output info that it's legacy both at the same time is indeed the graphs team (or whatever it's called)'s plan, Nux's just proposing that we just auto-migrate all graphs without doing anything else when graphs are reenabled and template editors would port the other 20%, with the fact that vega2 doesn't work being an incentive to port to vega5. Aaron Liu (talk) 20:33, 20 September 2023 (UTC)
 * Makes sense. Sj (talk) 21:59, 20 September 2023‎ (UTC)

Add step for archiving unconvertible graphs as images?
Assuming some graphs won't be converted to V5, at some point it would be good to compile a list of graphs that need to be converted to images, and do the conversion [on a server that allows sandboxed v2 rendering]. I'd love to see that included in the timeline, so they don't slip through the cracks. Sj (talk) 21:58, 20 September 2023 (UTC)


 * Somewhere to re-create the V2 graph or map would be great. In-place V2 would have the benefit of a rapid return for broken items. eg the '#tag:graph' -> 'Graph:Street map with marks' -> 'OSM Location map' -> 'final map' chain of conversion could easily be a long process of re-thinking and re-working. If so, working maps during that period would be a benefit of itself. But any method of seeing the V2 output would be a huge help in checking if the new version is matching the original features. Similarly, editors working at hand-converting a graph they added years ago, would doubtless welcome some method of comparing the new with what it is supposed to look like. RobinLeicester (talk) 17:16, 26 September 2023 (UTC)
 * There is an Vega V2 to V5 converter in the graph extension, so simple V2 graphs are shown. There has been work from Pietrasagh on Module:Graph, which covers a lot of graphs. The best way to convert the graphs in my opinion is to use Vegas own Vega2 to Vega5 transition guide, use their own editor on their website - which tells you if and what errors there are. Any protocol functionality needs to be tested on WMF wikis - beta is the only way currently. As for enabling Vega2 two out of the five security issues have available code to hack the computer viewing the graph - so yeah, if that gets enabled, I am installing NoScript and blocking all graphs displayed on my own computer. Snævar (talk) 07:33, 30 September 2023 (UTC)

Achievement horizon
Hello, just a simple question : When should we expect it to work again, within a confidence interval? Will we be waiting for weeks/months ? Is it useful to indicate that would-be timeframe in the warning message ? If it is about months to go, then shouldn't we remove graph templates in wikis ? Bouzinac (talk) 09:23, 29 September 2023 (UTC)


 * Good question. We shouldn't, but we should instead modify the uncomplete and misleading current public error message, by adding the safest (longest) imaginable delay . Their is huge need for a (even too cautiously oversized) repair time. Valmontin (talk) 01:14, 2 October 2023 (UTC)
 * When should we expect it to work again, within a confidence interval? Will we be waiting for weeks/months ? Is it useful to indicate that would-be timeframe in the warning message ?
 * @Bouzinac: in short, I expect us to be able to share an estimate for when we all can plan for graphs to be working again before October is over.
 * The above is built on thinking:
 * 1) Before next week is over, we'll invite y'all to review the roadmap/plan we're proposing we use to guide the work of reenabling the extension
 * 2) Through "1)", we (staff + volunteers) will arrive at a roadmap/plan we are both satisfied and we (staff) can use that updated plan to create an estimate of the sort you're understandably seeking
 * I recognize that in answering as I have the question you asked remains largely unaddressed. Although, I hope this bit of certainty is of some use. PPelberg (WMF) (talk) 23:31, 3 October 2023 (UTC)
 * It would be immensely useful to update the warning message. In the short term, PPelberg, can that message point people here for more detail?
 * Personally I find months-long processes of getting clarity to be somewhat draining on collective effort, and much prefer pessimistic-but-always-accurate messaging of the form "We don't currently have a plan to fix and restore this extension. Discussions of how this might happen in the future are happening here. Discussions of how to export graphs to other formats are happening here." (An internal process that can commit to outcomes w/ large confidence intervals and update iteratively, is as good, but extended uncertainty that can't even make such commitments is not.) Sj (talk) 21:22, 9 October 2023 (UTC)
 * Before next week is over, we'll invite y'all to review the roadmap/plan we're proposing we use to guide the work of reenabling the extension.
 * Update: the roadmap I committed us to delivering before today was over is delayed. You can expect this next week.
 * In the meantime, I've updated Graph/Plans and created Extension:Graph/Plans/Research to create space for it.
 * Thank you for being patient with us, y'all. I recognize every delay adds to the frustration of being without functionality you are depending on to make decisions and deliver people who come to visit the wikis with the information they're seeking. PPelberg (WMF) (talk) 00:23, 14 October 2023 (UTC)
 * We are also going to need better documentation on the protocols (the old doc is at Extension:Graph/Guide). Please keep the old doc for now, do not overwrite it, it is still useful for converting purposes.
 * I also thought, on the user side, that using the user tool Synchronizer would make the rollout time pretty quick. Basically, once an graph is ready, it could be rolled out pretty much immediately. Snævar (talk) 15:45, 14 October 2023 (UTC)

Concrete help please?
I came here as an expert programmer, hoping to help, but I very quickly got lost. Can anyone please explain how editors who don't (yet?) understand the difference between Vega 2 and 5 can help devs speed the repairs? Is there a reading list to get up to speed and a step-by-step concrete set of instructions for updating broken graphs? Sandizer (talk) 09:11, 3 October 2023 (UTC)


 * There is a guide at https://vega.github.io/vega/docs/porting-guide/ on what needs to be changed. On top of that, all of the protocols are have changed (wikidiff, wikiapi, wikifile, etc.), that part needs better documentation. Protocols should be changed to full urls. Also, use the vega editor at https://vega.github.io/editor/#/custom/vega, it will tell you what is wrong with your graph, except for protocol issues. Also, if you are working on Module:Graph or Template:Graph:Lines, then please do not start from scratch on those, some progress on those has allready been done at https://en.wikipedia.beta.wmflabs.org. --Snævar (talk) 15:05, 5 October 2023 (UTC)

I think the Vega guide is probably just showing the port from Vega2 to Vega3. There are several times I have added Vega5 things that where not in the migration guide.--Snævar (talk) 01:53, 21 October 2023 (UTC)


 * Made a new guide for Vega2 to 5 at Extension:Graph/Vega 2to5 list. Added a few things that are not in the official vega porting guide. Snævar (talk) 18:14, 22 October 2023 (UTC)

Other notes
Some links to other helpful writeups on how people are working on this:
 * RobinLeicester's notes
 * Jdlrobson on phab: 120319, T335048

added by Sj (talk) added by Snævar (talk)
 * https://en.wikipedia.beta.wmflabs.org/wiki/Template:Graph:PageViews for an example how to convert "wikiraw" links.
 * T335325 on how to avoid CORS issues with full urls
 * Pseudo-protocols and CORS workaround doesn't work on beta.wmflabs ->  Pietrasagh (talk) 08:47, 28 October 2023 (UTC)


 * Yeah, a lot of those full url hints are invalid now, since WMF is now going to make the protocol support work again. This is mainly a developer problem. Personally, I tend to add the missing data from the urls into the graphs and test them that way. If you are expecting something else than a notice, then a dev should answer. Snævar (talk) 10:14, 28 October 2023 (UTC)

Update: 20 October
Hi y'all – this update is meant to:


 * 1) Address the questions that emerged in response to the 15 September update
 * 2) Make clear the roadmap we're proposing to follow to safely and securely re-enable the Graph Extension

With the above in mind, there are two things we need your help reviewing:


 * 1) Roadmap for re-enabling the Graph Extension
 * 2) Responses to questions raised after 15 September update

We are specifically curious to know:
 * What – if anything – about Roadmap and/or FAQ is unclear/could benefit from clarification?
 * ''What – if anything – did you expect to see mentioned within the Roadmap or FAQ that you are not seeing?
 * What – if anything – about the Roadmap do you think ought to be reconsidered?

Thank you all for being patient as we prioritized the conversations and research needed to arrive at this proposal. PPelberg (WMF) (talk) 22:16, 20 October 2023 (UTC)


 * Having done a considerable number of Vega 2->5, i would say it will be wasteful to create a complex conversion tool. Most of the conversion is very straightforward and can be done by hand. It is also possible to write a simple tool using Jupyter notebook or observablehq page with a simple JSON->JSON conversion. My recommendation -- roll out Vega5 asap, preferably implementing one addition simial to Elasticsearch's Kibana -- where data.url can be an object instead of just a string - and that object could point to other data sources without any URL escaping.  This way authors can do   - this would use data from commons:Data:Ncei.noaa.gov/weather/New York City.tab Yurik (talk) 00:52, 22 October 2023 (UTC)
 * Having done a considerable number of Vega 2->5, i would say it will be wasteful to create a complex conversion tool. Most of the conversion is very straightforward and can be done by hand. It is also possible to write a simple tool using Jupyter notebook or observablehq page with a simple JSON->JSON conversion.
 * Good spot. It sounds like we're aligned in wanting to avoid work that we expect to be marginally impactful.
 * With the above in mind, can you think of any questions we ought to consider asking at the end of Phase 1 – before work on the conversion tool is scheduled to start – that could help us verify the extent to which conversion is as a straightforward as you think it might be?
 * My recommendation -- roll out Vega5 asap…
 * Understood. Is there something you saw [or didn't see!] in the roadmap that is leading you to think Vega 5 implementation is happening later than you think it ought to?
 * I ask the above thinking we're aligned in rolling out Vega 5 ASAP. Although, you raising this point is causing me to want to double check :)
 * …preferably implementing one addition similar to Elasticsearch's Kibana -- where data.url can be an object instead of just a string - and that object could point to other data sources without any URL escaping. This way authors can do url: {"type": "data", "title": "Ncei.noaa.gov/weather/New York City.tab"} - this would use data from commons:Data:Ncei.noaa.gov/weather/New York City.tab
 * Regarding the technical implementation you described above, I wonder: @Cscott: how does what @Yurik shared relate to what you've had in mind?
 * All of this aside, thank you reviewing the proposal so promptly and all the work and thinking you've done/contributed to this space.
 * ^ _ ^ PPelberg (WMF) (talk) 23:01, 26 October 2023 (UTC)
 * There are some moderately complex renames/relocations which our existing vega2->vega5 tool handles which seem to be useful, but 100% agreed that we don't plan to spend any more time on it other than make what we have already done a usable tool for assisting manual porting. That's in the roadmap, I think.  We would *love* community help building those porting tools, and if folks want to get started by taking the code we wrote go right ahead!
 * I'm not 100% certain about your suggestion about, it's something I think we can look at more closely as we implement protocol support for vega 5.  If you wanted to open a phab task (or upstream ticket at vega) and write up your proposal fully that would probably be helpful.  Protocol support definitely includes grabbing data from sister projects, and we will definitely be working w/ the community to develop a guide on best practices on how to do that, which would be a logical place to include advice like "if you use this object-style syntax for url you won't need to manually url escape the source". cscott (talk) 16:29, 27 October 2023 (UTC)
 * I am hugely encouraged to see the Roadmap. I am working towards resolving the OSM Location Map conversion, and have some working prototypes on the beta site. Unless I am missing something, Vega5 seems to not like Transforms of inline data declared as "values". I can find no useable output. My solution is to do the mercator projection stuff within a wikipedia template, and send x,y plots to a new replacement for 'Graph:Street map with marks'. That might still be usefully updated for use by wdqs and tabular file sources, but am I right that it is not useable for inline lat and lon transforms? RobinLeicester (talk) 20:27, 23 October 2023 (UTC)
 * Understood, @RobinLeicester! I've asked members of the engineering team whether there is guidance they can offer about the OSM Location Map conversion you're exploring.
 * Two things in the meantime…
 * Is this the page where we can see the work you've been doing on the OSM Location Map conversion?
 * We're glad to hear you're encouraged by the roadmap – thank you for sharing as much with us and for continuing to be a helpful voice as we find our way back to graphs being re-enabled ^ _ ^
 * PPelberg (WMF) (talk) 23:12, 26 October 2023 (UTC)
 * Yes, of the various subpages of https://en.wikipedia.beta.wmflabs.org/wiki/Special:PrefixIndex/Template:OSM_Location_map/ 'core' processes the parameter blocks, 'Getxy' turns the mercator coords into pixel xy, 'Labelitem' generates the Vega5 inline data items and 'Graphmap' is the replacement for 'Graph:Street map with marks' plus various new options from v5. It is now all working (as seen in /doc) except that the BetaCluster 'Template:if empty' seems to not work, and I have only stripped it out from the first few parameter blocks, so it onlys shows mark1 and mark2 items. RobinLeicester (talk) 23:05, 30 October 2023 (UTC)
 * I'm not enough of a vega expert to answer that, but it does seem like transforms in vega 5 are pretty powerful. That said, map projections are pretty esoteric.  Maybe ask this question to vega upstream? cscott (talk) 16:30, 27 October 2023 (UTC)
 * Vega5 can defiantly use longitudes and latitudes. In line 63 and 64 in https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Template:Graph:Street_map_with_marks we use the name of the data, here "data", so "datum.coord[0]" is "data.coord[0]". There is an example on Vega, more complex than you need, at https://vega.github.io/vega/examples/airport-connections/ . It uses https://vega.github.io/vega/data/airports.csv which has latitudes and longitudes of USA airports. How did I get that file? Well, since Vega is run client-side your computer has to get the data-points anyway. I watch the networking tab in my browser and pull the hidden data files from there. Not sure why they do not just have a link to it.
 * Also take a look at https://vega.github.io/vega/docs/projections/ for your mercator projection. Snævar (talk) 17:20, 28 October 2023 (UTC)
 * Hmm, this should’ve been in tech news. Aaron Liu (talk) 14:41, 24 October 2023 (UTC)
 * I think adding mention of this in Tech/News is a great idea. Thank you for raising this, @Aaron Liu. Done with help from @Quiddity (WMF). PPelberg (WMF) (talk) 02:46, 26 October 2023 (UTC)
 * I don’t think we should backport Vega 2 support. Do we have any timeframes for the roadmap? It’s been almost half a year Aaron Liu (talk) 15:33, 24 October 2023 (UTC)
 * I don’t think we should backport Vega 2 support.
 * @Aaron Liu, can you please say a bit more? What have you noticed/experienced/etc. that's causing you to question the value of backporting Vega 2 support?
 * Do we have any timeframes for the roadmap? It’s been almost half a year.
 * Good question. I'm thinking there will be a timeline we can share during the first or second week of November. By that time, it will have been 1-2 weeks since mention of the roadmap will have appeared in Tech/News and as a result, I anticipate we'll know whether people see any fundamental concerns with the roadmap. PPelberg (WMF) (talk) 23:06, 26 October 2023 (UTC)
 * Thanks. On Vega 2, I think that maintaining things with two versions at once would impact the timeline quite a bit and having Vega 2 templates break (or publicly display a warning incl. outside previews) would encourage editors to fix them Aaron Liu (talk) 00:23, 27 October 2023 (UTC)
 * This. Volunteers are very good at fixing things insofar as they are capable. So let them. Nardog (talk) 01:00, 27 October 2023 (UTC)
 * Does "first or second week of November" resolve to Nov. 3, 10, 17, or some other date? Tfmorris1 (talk) 16:38, 10 November 2023 (UTC)
 * @Tfmorris1, hi: you can expect to see an update sometime next week.
 * I appreciate you following up to ask...you doing so was the prompt I needed to revise the guidance I shared above. PPelberg (WMF) (talk) 17:42, 10 November 2023 (UTC)
 * I feel there is no real willingness to address the problem. The proposed roadmap comes 6 months after graphs were disabled, and we still don't have a clear timeline for reactivating them. A roadmap without a clear timeline is pointless in this case. Even a partial reactivation of the graph template on the English wikipedia, which would be easy to implement by porting it to Vega 5, would be a huge step forward. This apparent unwillingness to invest and solve this is deeply frustrating for all the volunteers that invested huge amounts of time into developing and maintaining charts that are now being deleted and forgotten after 6 months of outage. Also, the communication regarding this incident has been a disgrace. There should be a centralized page with information about the background of what happened, the status of the work to reactivate the extension, and a clear plan of action. This information should be clearly linked wherever there is a message about disabled charts. Ita140188 (talk) 01:36, 25 October 2023 (UTC)
 * The Roadmap is missing an associated schedule. :) Without it, communication about the update will be mostly of interest to technical users. Strainu (talk) 16:44, 26 October 2023 (UTC)
 * I hear you, @Strainu.
 * I'm thinking there will be a timeline we can share during the first or second week of November. I shared some context about this timing above. Although, please let me know if what you see there brings up any new questions for you.
 * Without it, communication about the update will be mostly of interest to technical users.
 * This is actually by design. Before committing to a timeline, we've been wanting to ensure the volunteers who are experienced with developing Graphs think the roadmap is viable.'''
 * With the above said, I appreciate what I shared does little to address the question you arrived here seeking help answering. Even so, I'm glad you asked…you doing so reminds me and us of how this outage continues to impact people. PPelberg (WMF) (talk) 23:08, 26 October 2023 (UTC)
 * Well, the only concern I have with the roadmap is it seems to block the deployment to a decision (and implementation) on whether to support Vega2. Can't we do that in parallel to deploying Vega5 so we unblock people who are willing to change the existing graphs? Strainu (talk) 07:22, 27 October 2023 (UTC)
 * I'm in total agreement, and  on this. In my opinion, bringing Vega 2 support back, even temporarily, is a bad idea principally because it reduces the incentive to port old graphs to the new syntax. If 80% of graphs can be auto-converted, then providing a tool to do that, combined with documentation (with examples) for editors who want to manually convert old graphs to the new format, that's the way forward; I think you'll be surprised how fast the change can be. It also reduces the engineering effort required to put back support for a dead technology that still constitutes a potential security risk, even with mitigations in place; the sandbox is a very good idea, but it should be used in addition to transitioning to new, more secure software. The rationale for re-supporting Vega 2 is, alas, equivalent to a rationale for supporting it forever, and the situation will only get worse over time. Please bite the bullet, and go directly for Vega 5 and only Vega 5, which will not only get graphs working again faster, but will also avoid analysis paralysis and the many costs of supporting two versions at once. The Anome (talk) 07:31, 31 October 2023 (UTC)
 * It will take like 3 months to move vega2 graphs to vega5 across the whole WMF sphere, once the protocols have been re-implemented. Enwiki projects a month, small wikis (as in few articles on the wiki) are usually slower. Snævar (talk) 21:52, 6 November 2023 (UTC)

Stats
I looked over some graphs and found out that once Module:Graph is complete, 40-50% of pages that use Vega graphs on the English and Spanish Wikipedias will have some graphs converted. I say some, because Module:Graph is not the only way Vega graphs are displayed on those pages. (Spanish Wikipedia and English Wikipedia have functionally the same Module:Graph code) I predict that the conversion to vega5, especially on English and Spanish Wikipedia, will happen quickly at first, and then progressively get slower. As graphs are converted in templates with fewer and fewer transclusions and then graphs used on individual pages, it is inevitable that the progress will get slower.

Also, it is falsely intimidating looking at the numbers in Category:Pages_with_disabled_graphs, with thousands of transclusions on the largest wikipedias, but the actual number of graphs in each project is in the hundreds.

A lot of analytics will need to be done before any number can be given of how long the Vega 2 to 5 conversion takes. Snævar (talk) 01:52, 21 October 2023 (UTC)

Templates are sorted into a group, all wikis in group1 (grp1) have the same version or an outdated one. Group other are all other wikis. enwiki = english wikipedia, enwikisource = english wikisource. Graph Chart - too many to count (over 10000)

Suggested PageViews group (all the same code or outdated): frwiki, commonswiki, ruwikinews, cawiktionary, ttwiki, mywiki, orwiki, srwiki, suwiki, tlwiki, zh_yuewiki, bnwikisource, enwikisource, huwiki, jawiki, kkwiki, lvwiki, rowiki, wikidatawiki.--Snævar (talk) 19:07, 23 October 2023 (UTC)


 * On ruwiki, the vast majority of graphs were generated by ru:Module:Statistical, which is already updated to Vega 5 (although the graphs are currently disabled in the module so the transclusions are not in the tracking category for graphs). The significant portion of the rest is Graph PageViews, Module:Graph etc. Colt browning (talk) 15:50, 27 October 2023 (UTC)
 * Interesting. A lot of the modules sitelinked to the Russian Module:Statistical are in fact different modules. The only wiki to update from that Russian module is krcwiki and it is way behind (63 revisions), maybe some template parameters where removed and added since then. I appreciate the addition, I am mainly figuring out where wikis can collaborate and prevent multiple wikis from working on the same thing on their own. Snævar (talk) 11:19, 28 October 2023 (UTC)
 * Actually, the differences don't matter for our purposes. The update from ruwiki for the part which generates the graphs will Just Work in all wikis (assuming that the module worked in those wikis before the graphs became broken, which is not always the case). --Colt browning (talk) 16:47, 9 November 2023 (UTC)

Module:Graph counts, Only Wikipedias counted. Total transclusions 30712.
 * Group1: total usage 6838, top users: mk (1746), es (1594), members: alswiki, arywiki, astwiki, aswiki, bawiki, bewiki, cswiki, eowiki, eswiki, fiwiki, frwiki, hewiki, hrwiki, hywiki, ltwiki, mkwiki, nowiki, skwiki, urwiki, uzwiki, vecwiki.
 * Group2: total usage: 16304, top users: en (10219), ms (1813), members: arwiki, arzwiki, aswiki, azbwiki, azwiki, banwiki, bhwiki, bswiki, bxrwiki, cawiki, cebwiki, ckbwiki, dawiki, enwiki, elwiki, etwiki, euwiki, fawiki, fowiki, guwiki, hiwiki, idwiki, kkwiki, knwiki, kuwiki, lldwiki, lvwiki, mdfwiki, mlwiki, mrwiki, mswiki, mywiki, newiki, nnwiki, orwiki, papwiki, shnwiki, slwiki, sqwiki, srwiki, svwiki, tawiki, tcywiki, tlwiki, yowiki, zhwiki.
 * Others total usage: 7570.--Snævar (talk) 12:06, 28 October 2023 (UTC)

Scope
I'm puzzled as to why at some sort of graph (bar? column? line?) of "historical population" data is affected, but a pie chart of religious affiliations (using the   template) is unaffected. Care to explain? —DIV (1.129.104.79 13:24, 28 October 2023 (UTC))
 * w:Template:Pie chart makes a graph using ad-hoc HTML. w:Template:Historical populations eventually wraps w:Module:Graph, which uses the disabled Graph extension. There's no good reason other that history for this to be the case. * Pppery * it has begun 14:34, 28 October 2023 (UTC)
 * OK. So are you implying that in an ideal world they would both be using the same mechanism: i.e. either both using ad hoc HTML, or both using a dedicated graphing module?  —DIV (1.129.111.250 00:14, 30 October 2023 (UTC))
 * Probably. * Pppery * it has begun 01:40, 30 October 2023 (UTC)

Request tracking with category, not linter flag
In Phase 5, I see Deprecate, and then remove support for Vega 2 from the codebase, creating categories or linter tags to mark any remaining Vega 2 graphs. I came here to request that a category, rather than Linter, be used for this tracking. Linter is for syntax errors, not the sorts of conditions that are tracked at Special:TrackingCategories. We already have Linter being misused for tables that are "too wide" (see ), which has resulted in editors who work on actual Linter errors having to exclude that tracking flag from reports and advise people new to Linter-error-removal to disregard the "error". This "outdated graph format" or whatever the right name is matches very nicely with Special:TrackingCategories and not with Linter. Thanks. Jonesey95 (talk) 20:06, 6 November 2023 (UTC)