Extension talk:Graph/Plans

Table of usage
I'm not sure how accurate using "Pages with disabled graphs" is to get a count. Graphs have been disabled for a while now, presumably people have been removing them from pages, and thus also removing the category. Maybe looking at an older dump of the category table might be better but probably quite a bit of work. Bawolff (talk) 23:18, 14 July 2023 (UTC)


 * I'm not sure how accurate using "Pages with disabled graphs" is to get a count. Graphs have been disabled for a while now, presumably people have been removing them from pages, and thus also removing the category.
 * Mmm, great spot, @Bawolff. It looks like @TheDJ spotted a similar issue.
 * As a next step, I'm going to ask some folks internally about how we might go about generating a more accurate measure of usage. Of course, if new ideas emerge in your mind for how we might go about doing this and/or if volunteers who you think would be well equipped to help out with this, please let me know. PPelberg (WMF) (talk) 16:58, 28 July 2023 (UTC)

Including personal opinions
@Bawolff: I appreciate you both being bold in adding this statement and asking whether it's appropriate to do so.

What if you prefaced the bit beginning with "The elephant in the room is that graph is a high maintenance extension... with your username so as to show this is a personal opinion that may or may not be something the information we end up gathering substantiates? How does this sound to you? PPelberg (WMF) (talk) 23:54, 14 July 2023 (UTC)


 * SGTM. Bawolff (talk) 23:59, 14 July 2023 (UTC)
 * Awesome. PPelberg (WMF) (talk) 00:03, 15 July 2023 (UTC)

Generating graphs from Wikidata
hi – two things!


 * 1) Thank you for contributing helpful information to the Extension:Graph/Plans page  🙏🏼
 * 2) Related to "1.", are you able to add an example or perhaps describe in a bit more detail what an up-to-date graph from Wikidata data generated using the Graph Extension looked like? I ask as this is the first time I'm hearing of the extension being used in this way.

Ok, thank you ^ _ ^ PPelberg (WMF) (talk) 21:37, 28 July 2023 (UTC)


 * Hi you can find one example here (see the wikicode). In short, when the graph extension was working, this template get the neutron lifetime from Wikidata and then plot these values as a function of the year.
 * So that, when I add a new value on Wikidata, the graph is automatically updated. Pamputt (talk) 21:46, 28 July 2023 (UTC)
 * Ah, I see – thank you for following up with this added context and for doing so so promptly, @Pamputt!
 * Sure enough, just after I posted the above, I also saw that @Nux helpfully added the example of en:Template:Airport-Statistics.
 * In any event, @Pamputt can you please have a quick look at the below and let me know if there is anything missing, unnecessary, and/or inaccurate about how I'm currently understanding how Wikidata, the Graph Extension, and on-wiki templates can be used to generate auto-updating graphs?
 * How Peter (me) currently understands the relationships between Wikidata, the Graph Extension, and on-wiki templates
 * 1. Use SPARQL to retrieve the information you're seeking from Wikidata. ''[https://query.wikidata.org/#%23%20Scroll%20down%20and%20hit%20blue%20arrow%20down%20to%20run%20and%20see%20the%20results%20%2B%20the%20sources%0ASELECT%20%3Fyear%20%3Fitem%20%3Fshortname%20%28MAX%28%3Fnumber%29%20AS%20%3Fpassengers%29%20%20%20%28SAMPLE%28COALESCE%28%3Freference_URL%2C%20%3Fmonthly_reference_URL2%29%29%20AS%20%3Fsample_reference_URL%29%0AWITH%0A%7B%20%20SELECT%20%3Fitem%20%3Fstatement%20%3Fdate%20%3Fyear%20%3Ftimevalue%20%3Fnumberperperiod%20%3Freference_URL%0A%20%20WHERE%20%20%7B%20%20%20%20%3Fitem%20wdt%3A%20%3Fairport_code%0A%20%20%20%20VALUES%20%3Fairport_code%20%20%20%20%20%7B%20%22%22%20%20%20%20%7D%0A%20%20%20%20%3Fitem%20p%3AP3872%20%3Fstatement.%0A%20%20%20%20%3Fstatement%20pqv%3AP585%20%3Ftimevalue%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20ps%3AP3872%20%3Fnumberperperiod.%0A%20%20%20%20%3Ftimevalue%20wikibase%3AtimeValue%20%3Fdate.%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20pq%3AP518%20%3Fapplies.%20%7D%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20prov%3AwasDerivedFrom%20%2F%20%28pr%3AP854%7Cpr%3AP4656%29%20%3Freference_URL.%20%7D%0A%20%20%20%20FILTER%20%28BOUND%28%3Fapplies%29%3Dfalse%20%7C%7C%20%3Fapplies%20%3D%20wd%3AQ2165236%20%29%0A%20%20%20%20MINUS%20%7B%20%3Fstatement%20wikibase%3Arank%20wikibase%3ADeprecatedRank%20%7D%0A%20%20%20%20BIND%20%28YEAR%28%3Fdate%29%20AS%20%3Fyear%29%0A%20%20%20%20FILTER%20%28%3Fyear%20%3E1949%29.%20%20%20%20FILTER%20%28%3Fyear%20%3C%20YEAR%28NOW%28%29%29%29%0A%20%20%7D%20%7D%20AS%20%25airport%0AWHERE%0A%7B%20%20%20%7B%20%20%20%20%23%20Get%20the%20sum%20of%20monthly%20values%20within%20a%20year%0A%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28SUM%28%3Fmax_numberperperiod%29%20AS%20%3Fnumber%29%20%28SAMPLE%28%3Fmonthly_reference_URL%29%20AS%20%3Fmonthly_reference_URL2%29%0A%20%20%20%20WHERE%0A%20%20%20%20%7B%20%20%20%20%20%20%23%20Get%20the%20maximal%20value%20and%20a%20sample%20reference%20URL%20for%20each%20unique%20month%0A%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28MAX%28%3Fnumberperperiod%29%20AS%20%3Fmax_numberperperiod%29%20%28SAMPLE%28%3Freference_URL%29%20AS%20%3Fmonthly_reference_URL%29%0A%20%20%20%20%20%20%20%20WHERE%20%20%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20%20%20INCLUDE%20%25airport%0A%20%20%20%20%20%20%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%20%3Fprec.%0A%20%20%20%20%20%20%20%20%20%20FILTER%20%28%3Fprec%20%3E%209%29%23%20precision%20more%20precise%20or%20equal%20to%20month%0A%20%20%20%20%20%20%20%20%7D%20%20%20%20%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%20%3Fdate%0A%20%20%20%20%20%20%7D%20%20%20%20%7D%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%0A%20%20%7D%20%20UNION%20%20%7B%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%209%20.%20%20%20%20BIND%20%28%3Fnumberperperiod%20AS%20%3Fnumber%29%20%20%20%20BIND%20%28%3Freference_URL%20AS%20%3Fsample_reference_URL%29%0A%20%20%20%20INCLUDE%20%25airport%0A%20%20%7D%0A%20%20OPTIONAL%20%7B%3Fitem%20wdt%3AP1813%20%3Fthis.%20%20%20%20%20%20%20%20%20%20%20%20%23%20has%20shortname%0A%20%20%20%20FILTER%28LANG%28%3Fthis%29%3D%22en%22%29%20%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%2Cen%22.%20%3Fitem%20rdfs%3Alabel%20%3FitemLabel.%7D%0ABIND%28COALESCE%28%3Fthis%2C%3FitemLabel%29%20as%20%3Fshortname%29%0A%7D%20GROUP%20BY%20%3Fitem%20%3Fshortname%20%3Fyear%20ORDER%20BY%20%3Fitem%20DESC%20%28%3Fyear%29 Example SPARQL query]''
 * 2. Use a template to ingest the data the query you wrote in "1." will have returned and convert that data into a format (e.g. JSON) that the Graph extension can understand
 * 3. Generate a graph using the Graph Extension by "calling" the data you generated in "2." by adding the  notation to the page that you'd like the graph to appear on PPelberg (WMF) (talk) 21:59, 28 July 2023 (UTC)
 * Yes, this is how it worked. The SPARLQ query directly returns the data in the JSON format. From this JSON, the key are the ones given in the SPARQL query (i.e. "main_val", "lower", etc.), and so we can call them in the graph code to format the data as we wish. Pamputt (talk) 04:00, 29 July 2023 (UTC)
 * Thank you for following up with this additional clarity, @Pamputt.
 * To be doubly certain, it sounds like it's two steps (listed below) rather than three steps (as I'd thought above), does that sound accurate?
 * Use SPARQL to retrieve the information you're seeking from Wikidata. ''[https://query.wikidata.org/#%23%20Scroll%20down%20and%20hit%20blue%20arrow%20down%20to%20run%20and%20see%20the%20results%20%2B%20the%20sources%0ASELECT%20%3Fyear%20%3Fitem%20%3Fshortname%20%28MAX%28%3Fnumber%29%20AS%20%3Fpassengers%29%20%20%20%28SAMPLE%28COALESCE%28%3Freference_URL%2C%20%3Fmonthly_reference_URL2%29%29%20AS%20%3Fsample_reference_URL%29%0AWITH%0A%7B%20%20SELECT%20%3Fitem%20%3Fstatement%20%3Fdate%20%3Fyear%20%3Ftimevalue%20%3Fnumberperperiod%20%3Freference_URL%0A%20%20WHERE%20%20%7B%20%20%20%20%3Fitem%20wdt%3A%20%3Fairport_code%0A%20%20%20%20VALUES%20%3Fairport_code%20%20%20%20%20%7B%20%22%22%20%20%20%20%7D%0A%20%20%20%20%3Fitem%20p%3AP3872%20%3Fstatement.%0A%20%20%20%20%3Fstatement%20pqv%3AP585%20%3Ftimevalue%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20ps%3AP3872%20%3Fnumberperperiod.%0A%20%20%20%20%3Ftimevalue%20wikibase%3AtimeValue%20%3Fdate.%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20pq%3AP518%20%3Fapplies.%20%7D%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20prov%3AwasDerivedFrom%20%2F%20%28pr%3AP854%7Cpr%3AP4656%29%20%3Freference_URL.%20%7D%0A%20%20%20%20FILTER%20%28BOUND%28%3Fapplies%29%3Dfalse%20%7C%7C%20%3Fapplies%20%3D%20wd%3AQ2165236%20%29%0A%20%20%20%20MINUS%20%7B%20%3Fstatement%20wikibase%3Arank%20wikibase%3ADeprecatedRank%20%7D%0A%20%20%20%20BIND%20%28YEAR%28%3Fdate%29%20AS%20%3Fyear%29%0A%20%20%20%20FILTER%20%28%3Fyear%20%3E1949%29.%20%20%20%20FILTER%20%28%3Fyear%20%3C%20YEAR%28NOW%28%29%29%29%0A%20%20%7D%20%7D%20AS%20%25airport%0AWHERE%0A%7B%20%20%20%7B%20%20%20%20%23%20Get%20the%20sum%20of%20monthly%20values%20within%20a%20year%0A%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28SUM%28%3Fmax_numberperperiod%29%20AS%20%3Fnumber%29%20%28SAMPLE%28%3Fmonthly_reference_URL%29%20AS%20%3Fmonthly_reference_URL2%29%0A%20%20%20%20WHERE%0A%20%20%20%20%7B%20%20%20%20%20%20%23%20Get%20the%20maximal%20value%20and%20a%20sample%20reference%20URL%20for%20each%20unique%20month%0A%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28MAX%28%3Fnumberperperiod%29%20AS%20%3Fmax_numberperperiod%29%20%28SAMPLE%28%3Freference_URL%29%20AS%20%3Fmonthly_reference_URL%29%0A%20%20%20%20%20%20%20%20WHERE%20%20%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20%20%20INCLUDE%20%25airport%0A%20%20%20%20%20%20%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%20%3Fprec.%0A%20%20%20%20%20%20%20%20%20%20FILTER%20%28%3Fprec%20%3E%209%29%23%20precision%20more%20precise%20or%20equal%20to%20month%0A%20%20%20%20%20%20%20%20%7D%20%20%20%20%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%20%3Fdate%0A%20%20%20%20%20%20%7D%20%20%20%20%7D%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%0A%20%20%7D%20%20UNION%20%20%7B%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%209%20.%20%20%20%20BIND%20%28%3Fnumberperperiod%20AS%20%3Fnumber%29%20%20%20%20BIND%20%28%3Freference_URL%20AS%20%3Fsample_reference_URL%29%0A%20%20%20%20INCLUDE%20%25airport%0A%20%20%7D%0A%20%20OPTIONAL%20%7B%3Fitem%20wdt%3AP1813%20%3Fthis.%20%20%20%20%20%20%20%20%20%20%20%20%23%20has%20shortname%0A%20%20%20%20FILTER%28LANG%28%3Fthis%29%3D%22en%22%29%20%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%2Cen%22.%20%3Fitem%20rdfs%3Alabel%20%3FitemLabel.%7D%0ABIND%28COALESCE%28%3Fthis%2C%3FitemLabel%29%20as%20%3Fshortname%29%0A%7D%20GROUP%20BY%20%3Fitem%20%3Fshortname%20%3Fyear%20ORDER%20BY%20%3Fitem%20DESC%20%28%3Fyear%29 Example SPARQL query]''
 * Generate a graph using the Graph Extension by "calling" the data you generated in "1." by adding the  notation to the page that you'd like the graph to appear on
 * PPelberg (WMF) (talk) 00:14, 1 August 2023 (UTC)
 * Yes, that's it :) Pamputt (talk) 04:48, 1 August 2023 (UTC)
 * Yes, that's it :) Pamputt (talk) 04:48, 1 August 2023 (UTC)

Update: 11 August
hey y'all – this update is an effort to help us:
 * 1) Align on what we've come to know and think about the Graph Extension
 * 2) Work together to find the information needed to address the remaining questions/uncertainties

I'm thinking the above will help us arrive at the requirements needed to distinguish viable from non-viable solutions and for the WMF (by way of me) to propose some paths forward for us to consider in the next update, planned for 15 September.

With this in mind, if you see something below that you disagree with, think could benefit from additional clarity/specificity, etc., please say as much by commenting below!

...this update is meant to identify any gaps and inconsistencies in what we're collectively thinking so that we can move forward together.

Now, before getting into what's become more clear since the 14 July update and what new uncertainties/questions surfaced: thank you!

Thank you to @Ahecht, @Bawolff, @Bouzinac, @Edu!, @Matma Rex, @Msz2001, @Novem Linguae, @Nux, @Pamputt, @Pppery, @Quiddity, @RobinLeicester, @User:SBassett (WMF), @Snævar, @Tgr, @TheDJ, and @User:VPoundstone-WMF for patiently and generously sharing the information and resources I've needed to understand and orient myself within the decisions we find ourselves needing to make.

Now onto the update… PPelberg (WMF) (talk) 18:55, 11 August 2023 (UTC)


 * Help needed: what question do we need help answering?
 * How might we go about inviting non-English speakers into this conversation and making them feel welcomed to contribute?
 * How might we go about making more people aware that this conversation is happening? It's important that we increase the likelihood that people who have experiences with and information about the Graph Extension be aware of, and ideally, a part of this conversation.
 * How might we go about forming a more accurate understanding of how widely used the Graph Extension was in relation to features that offer similar functionality?
 * PPelberg (WMF) (talk) 18:56, 11 August 2023 (UTC)
 * For #3, @Bawolff's old write-up is still probably the most comprehensive existing analysis and/or starting point for that question: https://www.mediawiki.org/wiki/User:Bawolff/Reflections_on_graphs SBassett (WMF) (talk) 18:42, 22 August 2023 (UTC)
 * Understood – thank you for pointing me to this, @SBassett (WMF); I've added this to the main page. PPelberg (WMF) (talk) 22:10, 23 August 2023 (UTC)
 * Clarity: what do we agree to be true about the Graph extension and the needs it was meant to meet?
 * There seem to be three distinct features the Graph Extension offered for which there are no current workarounds/alternatives:
 * Generating interactive maps/charts/graphs
 * Where "interactive" in this context refers to functionality/features like: enabling individual people viewing a graph to adjust the underlying parameters that determine the data they see and how it is visualized.
 * Generating maps/charts/graphs that update automatically
 * This is most often done by pulling in data from Wikidata.
 * Updating maps/charts/graphs on-wiki
 * The Graph Extension enables people to generate graphs using data stored on-wiki. This means, people can update graphs by editing the data on wiki, similar to how they might edit any other page/piece of information on the wikis.
 * In addition to the "distinct features" the Graph Extension offered, it also enabled people to:
 * Specify how this data is represented (e.g. chart type, colors, legend, mouseover events/interactions, etc.)
 * Store data within templates on pages separate from articles
 * Embed/transclude graphs within articles using templates
 * Write Lua modules that:
 * Convert JSON data into a well formatted wiki table
 * Extract data needed for graph and outputs as JSON
 * Create graph templates that can consume data and plot it
 * Pull in data from external sources via URLs
 * From generating article page view graphs to showing air traffic volume, volunteers depended on the Graph Extension in a variety of reader- and editor-focused contexts.
 * PPelberg (WMF) (talk) 18:57, 11 August 2023 (UTC)
 * Impressions: what seems like it could be true? 
 * Absent accessible and easy-to-use features that enable people to store, collaborate on, and present data on-wiki:
 * The data we host/present on our projects is more vulnerable to being outdated/inaccurate
 * The content we offer is more likely to lack corresponding data visualizations
 * In order for the Movement as a whole, and the various communities that comprise it, to evaluate the impact of the work it's doing, they need ways of generating, updating, and visualizing data.
 * To use the Graph Extension, you need/needed a relatively high degree of technical expertise on top of a foundation of data literacy.
 * Related to "3.", the Graph Extension has remained a relatively niche feature that one could, theoretically, never interact with (directly or indirectly) throughout your tenure as an editor.
 * PPelberg (WMF) (talk) 18:57, 11 August 2023 (UTC)
 * @PPelberg I'm interesting in Template:Graph:PageViews: we really need this mechanism working in ruwiki. MBH (talk) 05:07, 19 August 2023 (UTC)
 * "Template:Graph:PageViews: we really need this mechanism working in ruwiki."
 * @MBH: this is helpful to know – I'm glad you said something.
 * To help me more fully understand the extent to which you, and other volunteers at ru.wiki, depend on Template:Graph:PageViews, can you please share what has been made difficult or not possible as a result of it no longer functioning? [i]
 * i. E.g. Are there particular actions decisions that are more difficult, or perhaps not possible, to make because you do not have easy access to information about how views of a particular page vary over time? PPelberg (WMF) (talk) 22:06, 23 August 2023 (UTC)
 * @PPelberg 1) ruwiki extensively uses this template on articles' talk pages, 2) I own a bot for ru- and ukwiki, that generates articles' peak popularity (number of readers) monthly and yearly stats, see ru:File:Просмотры статей об айти-разработке.png, ru:ВП:Пики интереса к статьям/За год, uk:Вікіпедія:Спалахи інтересу до статей/За рік, this stats pages uses this template and now they are broken. MBH (talk) 18:05, 24 August 2023 (UTC)
 * @PPelberg 1) ruwiki extensively uses this template on articles' talk pages, 2) I own a bot for ru- and ukwiki, that generates articles' peak popularity (number of readers) monthly and yearly stats, see ru:File:Просмотры статей об айти-разработке.png, ru:ВП:Пики интереса к статьям/За год, uk:Вікіпедія:Спалахи інтересу до статей/За рік, this stats pages uses this template and now they are broken. MBH (talk) 18:05, 24 August 2023 (UTC)


 * I could really use a mechanism for generating graphs like Our World in Data; something requested at the start of the pandemic and many times since. "we should aim to have a world map graph template just like [OWID], regardless of what we have to do with the graph extension or services behind it." Sj (talk) 07:20, 30 August 2023 (UTC)

Underlying purpose inspiring the extension
Improving graphs and interactive content and Yurik's Dream of Content seem as relevant as ever! This isn't quite responsive to Peter's questions above which were specifically about use of / need for the Graph extension as currently implemented, so I'm putting these points in a separate section; feel free to refactor or merge.


 * Data visualization is an important part of a more beautiful and richly sourced references: helpful to readers, and to the identity of the projects.
 * The existence of good data viz adds incentive to get the underlying data on the projects (and to get it right)
 * The ability to do good data viz attracts people who care about that facet of knowledge production and appreciate an impactful place to contribute that to the commons. This is where having tools that let us make interactive results leads to a new ecosystem of curators and readers
 * Many of our most common use cases might instead use the same visualization libraries as OWID: their context and audience and need for customizable compact summaries from shared data sources is quite similar to ours.
 * Continuity and archival quality: if needed we can downgrade interactives to static visuals, but should not remove data visualizations without replacement. If the extension is going away, there should be a way to convert removed graphs to a static image [that can be run at scale]
 * Uncertainty dissipates community. It helps to have a maintainer and roadmap/plan of record for "how we visualize data on the projects". This is always going to be a range of uses, with a power law, some of which have many alternatives. Currently people who might help develop alternatives are still thinking about Vega migrations (cf the talk page) and may be wasting their time.
 * Losing previous content without replacement for 6 months is weird. We should at least render all previous graphs and embed the resulting images.  Not having a maintainer to do this or explain why it's not being done is weird.  An extended conversation about what we may or may not do in the future (without first doing the above as a stopgap, to avoid losing content people spent thousands of hours making) is very weird. And teaches people not to trust the wikis as a place to make and share visualizations. (Part of the challenge is that it's not obvious who should do it, or if it will be a waste of time b/c a more official solution is about to drop.)
 * We all benefit from building a shared vision for how and where data visualization happens on the projects, and how that fits into the overall development of reading and editing. This is broader and simpler than the questions about this extension.
 * Any sort of roadmap is fine. "eventually reenable a comparable extension" is clear. "we can't secure this, extension is deprecated + tasks closed, please migrate off w/ these tools; we'll find another solution" would be clear. "tools to visualize data should be maintained by community and not depend on staff or need hefty security review" would be clear. We could still debate changing the roadmap but at any point in time the current roadmap should be legible. Sj (talk) 07:20, 30 August 2023 (UTC) 13:18, 6 September 2023 (UTC)


 * @Sj responses/follow up questions to the points you raised below.
 * Before that, thank you for starting this conversation.
 * I think you are spot-on in naming the need for exploring the broader themes/capabilities the Graph Extension is an expression of and is crucial context to consider as we propose a set of next steps.[i]
 * Note: I'm going to post responses in separate comments so that we can [hopefully] explore multiple threads in parallel a bit more easily.
 * i. Propose next steps will come via the update I'll share next Friday, 15 Sep. PPelberg (WMF) (talk) 23:16, 8 September 2023 (UTC)
 * Data visualization is an important part of a more beautiful and richly sourced references: helpful to readers, and to the identity of the projects.
 * I share this assumption that readers value data visualizations.
 * Have you seen/read any research that substantiates this? Asked another way: are you able to share what you've noticed that contribute to you thinking the above is true?
 * For context, the most compelling research I've come across to-date related to the above being true is a study from the Research Team. This research found illustrated Wikipedia pages receive 4x more pageviews than those without images. That same research also found the ratio of image clicks to pageviews to be 1:30 compared to 1:340 for citations and 1:110 for links. PPelberg (WMF) (talk) 23:20, 8 September 2023 (UTC)
 * I would argue that video is a better model for data visualization than an image. It is relatively easy to improve an article with an image. Even a bad image is often better than no image if that's all that is available. Video is much harder - it can improve an article, but it is much more important that the media is of high quality and tells a "story". It is much harder to create a good video, and a bad video really does nothing for an article. I think this is what data visualization is like - it can significantly improve an article, but only if good, plot is more important, and it is much more difficult to create a good visualization regardless of the tools. Bawolff (talk) 02:37, 9 September 2023 (UTC)
 * There are all sorts of readers. Areas where this is not in question: Scholarly journals and research have prized visualizations for over a century. So have statistical or mathematical topics such as budgets, economic histories, epidemiologies, &c. For general-purpose descriptions to a lay audience, textbooks for many decades, encyclopedias since Encarta, and more recently with data journalism (in news) and Wolfram Alpha (in search), subfields or major enterprises have been built on the premise + promise of reliable, legible visualization. Good viz brings readers who better understand topics visualized that way. For current readers on an average article that includes some graphable data: I don't have specific research, but infoboxes and summary tables seem quite popular and widely referenced / screenshotted.  Sj (talk) 23:27, 11 September 2023 (UTC)
 * I would argue that video is a better model for data visualization than an image. It is relatively easy to improve an article with an image. Even a bad image is often better than no image if that's all that is available. Video is much harder - it can improve an article, but it is much more important that the media is of high quality and tells a "story". It is much harder to create a good video, and a bad video really does nothing for an article. I think this is what data visualization is like - it can significantly improve an article, but only if good, plot is more important, and it is much more difficult to create a good visualization regardless of the tools. Bawolff (talk) 02:37, 9 September 2023 (UTC)
 * There are all sorts of readers. Areas where this is not in question: Scholarly journals and research have prized visualizations for over a century. So have statistical or mathematical topics such as budgets, economic histories, epidemiologies, &c. For general-purpose descriptions to a lay audience, textbooks for many decades, encyclopedias since Encarta, and more recently with data journalism (in news) and Wolfram Alpha (in search), subfields or major enterprises have been built on the premise + promise of reliable, legible visualization. Good viz brings readers who better understand topics visualized that way. For current readers on an average article that includes some graphable data: I don't have specific research, but infoboxes and summary tables seem quite popular and widely referenced / screenshotted.  Sj (talk) 23:27, 11 September 2023 (UTC)


 * The existence of good data viz adds incentive to get the underlying data on the projects (and to get it right)
 * Mmm. The relationship you're naming between the usefulness of a knowledge format/representation and peoples' likelihood to contribute the knowledge needed to produce them seems insightful and accurate to me.
 * To the "accuracy" bit, are you able to share a bit about how you seem to have come to think that this relationship holds specifically true for data visualizations in our ecosystem?
 * No worries if this is more of an intuition at this point. Although, if there are things you can point to that cause you to feel confident in this statement, I'd value seeing them. PPelberg (WMF) (talk) 23:21, 8 September 2023 (UTC)
 * Historically across a lot of formats, first one is tried as a style (dot maps; nav templates; infoboxes; short WD descriptions), then it gets incorporated into various editing scripts and bots, then at some point one or two people dedicate themselves to populating and cleaning up the format across whole categories of relevant pages. We have scattered and inhomogenous ways to store tabular data and time series data; having a central use case like this will lead to scripts and other tools standardizing elements that make the data more findable, better updated, editable and more often corrected. Having a quick way to visualize data will over time lead to identifying and fixing mistake and outliers (visible across dozens of covid-tracking projects). Sj (talk) 23:27, 11 September 2023 (UTC)


 * The ability to do good data viz attracts people who care about that facet of knowledge production and appreciate an impactful place to contribute that to the commons. This is where having tools that let us make interactive results leads to a new ecosystem of curators and readers
 * The potential you're naming here is inspiring to me and it leads me to wonder the following…
 * Assuming it's accurate to think the interactive graphs the Graph Extension enabled volunteers to produce did NOT produce the "new ecosystem of curators and readers" you're referring to above, do you have a sense for what might have contributed to this? PPelberg (WMF) (talk) 23:22, 8 September 2023 (UTC)
 * That would not be accurate. It needs an easier onramp, and a place to store versioned tabular data that can be used as the input to such graphs, but the tens of thousands of pages using it, and the handful of specific projects + maintainers on those projects that used it extensively (and commented immediately when it went away) were members of that ecosystem. These things often have a slow takeoff before there is a critical mass of people learning from and borrowing from one another, filling in the various gaps. OWID illustrates the point nicely: we have an obvious source of freely-licensed data on hundreds of topic areas, hundreds of editors spending thousands of hours making equivalent data into static images on Wikipedia (like the 1500 revisions of this graph of COVID cases in Canada), no easy way to do that with Graph (despite a dozen people specifically trying and failing to make that happen). There was also a WikiGraphers User Group created recently, but not pointed to this extension as the natural place to start graphing. Sj (talk) 23:27, 11 September 2023 (UTC)


 * Many of our most common use cases might instead use the same visualization libraries as OWID: their context and audience and need for customizable compact summaries from shared data sources is quite similar to ours.
 * Great spot. I'm thinking this information will come in handy if/when we arrive in a place where we're exploring the possibility of building a successor to the Graph Extension. In the meantime, I've documented this idea on T345962. PPelberg (WMF) (talk) 23:25, 8 September 2023 (UTC)
 * Continuity and archival quality: if needed we can downgrade interactives to static visuals, but should not remove data visualizations without replacement.
 * +1; excellently put. PPelberg (WMF) (talk) 23:26, 8 September 2023 (UTC)
 * If the extension is going away, there should be a way to convert removed graphs to a static image [that can be run at scale].
 * This sounds right to me too. I'm checking with engineering to understand the viability of what you're proposing. I'll report back what I hear here and on T334940 (thank you for raising this there). PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)
 * Uncertainty dissipates community.
 * Agreed and I appreciate you naming this potential.
 * Related to the above: can you think of ways we could improve how we're communicating about the Graph Extension and the plans surrounding it?
 * Perhaps the question above is better suited for a "retro" of sorts once a plan is in place. Tho, I thought I'd seed the question now in case immediate ideas/suggestions came to mind. PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)
 * Currently people who might help develop alternatives are still thinking about Vega migrations (cf the talk page) and may be wasting their time.
 * I hadn't seen this comment; thank you for drawing my attention to it. I'll follow up with this person to try and offer some clarity about what we're thinking. PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)
 * We all benefit from building a shared vision for how and where data visualization happens on the projects, and how that fits into the overall development of reading and editing. This is broader and simpler than the questions about this extension.
 * +1 and I think you investing the time to articulate the above is starting the conversation necessary to help us do what you're describing…thank you!
 * In terms of what's next, here's what I'm thinking:
 * Between now and next Friday, I'll be preparing an update that will include:
 * A proposal for what we do with Graph Extension and the visualizations that have been rendered inaccessible by it being disabled
 * A broader snapshot of how we're currently thinking about the strategic relevance of supporting the creation of, and collaboration on, on-wiki data visualizations.
 * How does that sound? What – if anything – about "1." and "2." would you suggest we adjust and/or add to provide the clarity and direction I think you aptly named us all as collectively needing.
 * Note: Please know the questions I'm asking above are in service of me being newer to this context and motivated to understand how you've come to arrive at these points of view so that we can reason about this space together ^ _ ^ PPelberg (WMF) (talk) 23:30, 8 September 2023 (UTC)
 * I'd just like to echo the sub-bullet that was not quoted here, about having a legible roadmap. A proposal and strategic snapshot will be nice, but I hope that if these do not represent a firm decision that's been made, an ETA can be communicated on when such a decision will be made. For people who are vaguely aware of how software development works, "we don't know when this will be fixed" is an understandable position. "We don't know when this will be fixed, or if it will be fixed, or what will replace it or when it will be replaced" is not very encouraging. Even a decision not to support a graphing module at the core level and kicking it back to the community is a decision that will give volunteers the green light to start working on a replacement. Folly Mox (talk) 01:56, 11 September 2023 (UTC)
 * Seconded! I keep running to graphs all over en.Wikipedia that are broken, and for some articles, it has a really significant impact. We need a clear plan and communication one way or another. Ganesha811 (talk) 14:04, 11 September 2023 (UTC)
 * @Folly Mox: I hope that if these do not represent a firm decision that's been made, an ETA can be communicated on when such a decision will be made.
 * @Ganesha811: We need a clear plan and communication one way or another.
 * +1 to both of the points you both are raising which I understand to be something like: "''While a broader strategic conversation is important, we first need clear direction from you all, the WMF, about what – if anything – you're going to do to improve the current state of things."
 * Assuming I've understood y'all accurately, you can expect me to offer this clarity via an update to this page and T334940 before this Friday, (15 Sep) is over. More context here. PPelberg (WMF) (talk) 19:20, 11 September 2023 (UTC)
 * That sounds good, and yes, I think you've summarized the central point correctly. Every day that graphs are broken is a day that tens of thousands of Wikipedia articles are incomplete (often in significant ways), and therefore are not serving readers well. Ganesha811 (talk) 19:23, 11 September 2023 (UTC)
 * Excellent. Thank you for reviewing, @Ganesha811. And thank you for acting on the instinct to come here to speak up. PPelberg (WMF) (talk) 19:26, 11 September 2023 (UTC)
 * Thank you for the expectation of clarity. I think that's what people are seeking in general. The before / after the strategic conversation chronology is less meaningful to me than the "after five months" temporal milestone, but it sounds like you've got some bullet points prepared for Friday that will indicate what the overall vibe is regarding the issue, so I consider my concerns allayed. Folly Mox (talk) 20:06, 11 September 2023 (UTC)
 * Thank you for the expectation of clarity. I think that's what people are seeking in general. The before / after the strategic conversation chronology is less meaningful to me than the "after five months" temporal milestone, but it sounds like you've got some bullet points prepared for Friday that will indicate what the overall vibe is regarding the issue, so I consider my concerns allayed. Folly Mox (talk) 20:06, 11 September 2023 (UTC)

Update: 15 September
Hi y'all – this update is an effort to arrive at the clarity we need to ensure volunteers around the Movement regain access to the information and capabilities disabling the Graph Extension has left them without.

Within the next two weeks, you can expect another update about how we are thinking about the broader strategic importance of graphs.

With the above in mind, here's what we need your help with to move forward: Can you please review the "Proposal" and "Vega 2 → Vega 5 transition" sections below and share what seems unclear, missing, and/or misguided?

---

Notes: the "Proposal" that follows is an outgrowth of the discussions and information gathering that's been taking place, in large part, on this page and Extension:Graph/Plans over the past three months.

''Note: exploratory work to redeploy the Graph Extension in a sandboxed iframe has started. See T222807.'' ''Note: We estimate this converter currently works for ~80% of graphs, with diminishing returns on additional engineering effort to cover more. We do not plan on continuing to invest significant additional engineering resources here, but instead to simply repurpose the existing codebase as an aid to manual porting.''
 * Proposal
 * To safely restore access to the information and capabilities disabling the Graph Extension has left people without and promote the volunteer+staff collaboration needed to do so, we (the WMF) are committing to:
 * Re-enable the Graph Extension in a sandboxed iFrame with a restrictive content security policy.
 * Once the Graph Extension is reenabled, it will continue to work with Vega 2 for a yet-to-be defined period of time. Note: we'll need to define this window together.
 * After that "yet-to-be defined period of time," Vega 2 support will be discontinued and use of the Graph Extension will require volunteers to make graphs with Vega 5.
 * As soon as possible, make the sandboxed Graph extension available on the beta cluster for testing. See: T346292.
 * Investigate the viability of adding logging to increase our awareness of instances where people are exploiting the security vulnerabilities inherent with restoring support for Vega on our platform. See T346414.
 * Publish the technical documentation needed for developers across the Movement to understand how we implemented the sandboxed CSP approach
 * Publish a clear timeline for when you all can expect all of the above to happen
 * Share regular updates about the progress we're making on the commitments named above on Phabricator and MediaWiki.
 * Support volunteers with code and processes that will ease the transition from Vega 2 to Vega 5 when the time for this transition comes.
 * In support of the above, we'd need to depend on ya'll (volunteers) to:
 * Spread awareness of this proposal and the updates that will come as we start implementation, assuming this proposal moves forward.
 * Manually migrate some proportion of Vega 2-based graphs to be compatible with Vega 5. See the "Vega 2 → Vega 5 transition" section below.
 * Potentially, fix/port graphs that attempted to fetch live data using methods that the sandboxing approach inhibits.
 * Note: the need for the above will become clear once we decide on whether we will restore the pseudo-protocols that were used to fetch data live from the action API, the REST API, WDQS etc, and the precise sandbox parameters we select (domains/ports/http methods allowed). This decision will be made in T346291.
 *  Vega 2 → Vega 5 Transition 
 * Why do we think it's worthwhile to migrate from Vega 2 to Vega 5?
 * Vega 2 has been superseded by Vega 3, 4, then 5 upstream.  Upstream and third-party documentation exclusively refers to syntax in “Vega version 3.0 and later”, and it is difficult for new contributors to find documentation relevant to Vega 2.  The last upstream release (bugfix or security) of Vega 2.x was in January 2017.  Vega 5 was released in March 2019 and is still under active maintenance and development, with the latest 5.25.0 release in April 2023.
 * Volunteers have reported issues with Vega 2's accessibility, syntax, and overall functionality, per this 2023 wish.
 * Vega 5 has made improvements to the library's expression layer that harden it from a security perspective compared to Vega 2.  It is not perfect, but by introducing a parsed expression grammar it offers a more robust foundation for additional security hardening in the future if it proves necessary.
 * Maintaining multiple versions of Vega concurrently is unsustainable in the long run. The wiki community is taxed in the attempt to independently support software which is not being maintained upstream. Our efforts are best spent working in cooperation with upstream and third-party developers, and to do this we need to be working from the upstream Vega 5 code base.
 * What might be required to migrate graphs from Vega 2 to Vega 5?
 * Create a converter that would migrate Vega 2-based graphs to be compatible with Vega 5. @Jdlrobson started work on an initial approach in T335048#8794138.  The initial work needs to be restructured slightly to refocus it on being an aid to manual porting, instead of the automatic translator which was its original goal.
 * Volunteers would need to update  syntax on a case-by-case basis, aided by (1) the ability to run the existing Vega 2 and new Vega 5 specification side-by-side, (2) the partial Vega2-to-5 porting tool which handles 80% of the “obvious” keyword changes and other mechanical conversions, (3) the upstream Vega2 porting guide, and (4) additional documentation or tools which might be created by the wiki community.
 * Update the limited number of Scribunto templates on-wiki which generate  output in Vega 2 format to instead output Vega 5.  This requires both lua and Vega expertise, but fixes a larger number of Vega 2 uses on wiki at once.
 * PPelberg (WMF) (talk) 23:52, 15 September 2023 (UTC)


 * The open question over pseudo-protocols sounds like a pretty big sticking point. I totally understand the reluctance, even just from an operational perspective this couples a bunch of not-really-production things to production in a pretty uncontrolled way, but that would also seriously alter the applicable usecases for the extension. Bawolff (talk) 00:51, 16 September 2023 (UTC)
 * ...that would also seriously alter the applicable usecases for the extension.
 * @Bawolff: can you please say a bit more about the above? Are you most concerned about specific graphs no longer functioning were we to not restore the pseudo-protocols that were used to fetch data live from the action API, the REST API, WDQS etc.? Are you most concerned that the extension would be generally less useful? Something else? PPelberg (WMF) (talk) 23:44, 27 September 2023 (UTC)
 * i'm concerned its a pretty big decision to leave as a question mark Bawolff (talk) 08:38, 28 September 2023 (UTC)
 * I share this concern. I think that without access to data (which is really what the pseudo protocols are about), that the graphs themselves will loose a lot of their potential. Who’s writing json data arrays in wikitext or Lua modules ? —Th e DJ (Not WMF) (talk • contribs) 21:33, 28 September 2023 (UTC)
 * ''"Affording people the ability to fetch live data is core to what makes the Graph Extension uniquely valuable and difficult to find a replacement for."
 * @Bawolff + @TheDJ: assuming it's accurate for me to understand what y'all are sharing as what I've written above, then: understood. Based on what we talked about internally today, offering pseudo-protocol support sounds doable in this new sandboxed approach.
 * The question that remains is what exactly that support looks like. I expect for us (staff + volunteers) to be in a place to shape this together in the next few weeks.
 * Thank you for raising this. You doing so held us accountable to prioritizing talking about this during today's meeting.
 * On our end, we're processing the feedback everyone here has been sharing and working towards sharing an updated proposal in the coming weeks. PPelberg (WMF) (talk) 21:48, 28 September 2023 (UTC)
 * Answering the question of how much time it takes for graphs to become vega 5 is a math equation. There is an missing variable in that equation of how the pseudo-protocols get implemented. In the current gerrit master, pageviews works, commons is mostly there - it connects and there was an fix for an fetching issue, but with Graph disabled I was never able to verify it.
 * Wikidata is not supported at all, there is next to no code for it in gerrit. Wikidata support is not needed, volunteers (users) can use lua wikibase access (https://doc.wikimedia.org/Wikibase/master/php/docs_topics_lua.html), but rewriting from the current Wikidata SQL to lua wikibase is going to take three times as long as converting from vega2 to vega5, without touching the protocol bit. Snævar (talk) 09:54, 16 September 2023 (UTC)
 * Wikidata sparql and wikidata lua aren't really equivalent. There may be some things that can be converted but i would be very surprised if that was everything. Bawolff (talk) 10:02, 16 September 2023 (UTC)
 * As a Wikidata editor, I completely disagree. Wikibase's Lua functions are extremely limited in functionality and do not come close to what SPARQL can do. We mainly use graphs to display statistics (e.g. how many items are instances of something). We can't even fetch a list of inverse links using Lua, we definitely can't use it to create graphs like that. - Nikki (talk) 11:57, 19 September 2023 (UTC)
 * I agree with Nux. Support for Vega 2 should be dropped with automatic translation to Vega 5 as temporary solution. There need to be clear message that graph is automatically converted and needs attention. It will motivate templates maintainers and graph users that directly use Vega syntax to migrate to Vega 5 instead of using static image as replacement for graph in article.
 * I interact with extensions mostly with Template:Graph:Chart template and in my experience this was most common use on Wikipedia articles for charts. Update on underlying Module:Graph is almost done (see |Module:Graph/sandbox, here  and here ). Test cases of template  show how well automatic translation works.
 * I'm trying to update map part of module but without wikiraw:// protocol (T335325) enabled for beta it's not possible to debug it efficiently. Same thing for other templates and modules relying on data fetch with "url".
 * Pietrasagh (talk) 16:28, 17 September 2023 (UTC)
 * Nux raises a compelling point. I think people will be OK with immediately transitioning to Vega 5 so long as it'll work "forever", as opposed to a temporary solution that'll stop working at some arbitrary time. Enterprisey (talk) 20:16, 17 September 2023 (UTC)
 * I agree with Nux, making the process larger may be interesting from a security point of view, but graphs have been broken for months, so just migrating to Vega5 would make the process better for readers, as they don't have two separate time windows with potential failure. Theklan (talk) 13:50, 18 September 2023 (UTC)
 * It sounds to me as though automatic conversion may not quite work, and people may need to see V2 and V5 side by side to test conversion / understand what's not yet working. But as long as the messaging is "V2 graphs no longer work" and v2 is only available in some side-by-side testing mode, it can be clear that we are immediately in the V5 migration period. Sj (talk) 19:38, 20 September 2023 (UTC)
 * I don't know much about the details of this (I'm only here because I was trying to find out when graphs will work again). A converter sounds like a "nice to have" feature, but I hope the priority is in getting graphs working again at all - having graphs which work if we fix them manually would be an improvement over having no graphs. - Nikki (talk) 12:12, 19 September 2023 (UTC)
 * Depending on the distribution of the Vega 2 usage, would it make sense to offer a Wikimedia/Wikipedia "simplified" graph template which would act as a front-end to Vega 5 or whatever graph module is used in the future? Is what the "pseudo-protocols" was referring to?
 * My point being: if there are a large set of simple, static (not dynamically-generated) graphs (e.g. basic pie charts, one-variable bar charts, and one-variable linear/log line graphs), can these be automatically converted to a simplified chart template ? As someone who has worked on just one stacked bar chart, I found the Vega 2 quite difficult; I imagine a less-technical user would like something they could, say, dump from a spreadsheet to make a simple graph. Jason Olshefsky (talk) 12:32, 19 September 2023 (UTC)
 * My point being: if there are a large set of simple, static (not dynamically-generated) graphs (e.g. basic pie charts, one-variable bar charts, and one-variable linear/log line graphs), can these be automatically converted to a simplified chart template ? As someone who has worked on just one stacked bar chart, I found the Vega 2 quite difficult; I imagine a less-technical user would like something they could, say, dump from a spreadsheet to make a simple graph. Jason Olshefsky (talk) 12:32, 19 September 2023 (UTC)

Why not just start with v5?
You ask for what time should the v2 be available. How about no time at all? It has been quite a long time when graphs were not working. Testing v2 and then doing migration seems like a waste of resources to me. As I understand using v2 and then v5 would mean that after a period of graphs working there would be again a period of graphs not working. This work-don't work cycle would not look good (would make Wikipedia look less reliable). So let's just make the new version work... Unless maybe both versions could work at the same time? Nux (talk) 12:58, 16 September 2023 (UTC)
 * Unless maybe both versions could work at the same time? Yes, at least that's what it implies to me. Edit: missed auto-migration part. I support your proposal to just auto migrate everything and just use vega 5. Edit 2: PPelberg (WMF) just thanked this reply so I think yes, both versions would work at the same time Aaron Liu (talk) 19:48, 18 September 2023 (UTC)
 * +1. If there's a compelling reason to do this in two steps rather than in one, I don't see it in what PPelberg posted above. Nardog (talk) 22:03, 18 September 2023 (UTC)

I could see both at the same time, with all V2 renders tagged as "needs conversion". Once most have been converted a scripted update could note on all [talk]pages using an unconverted graph that it's going to stop rendering again soon, then shift to no longer rendering V2, leaving the "needs conversion" tag. Sj (talk) 19:41, 20 September 2023 (UTC)
 * could note on all [talk]pages just do some hatnote in the template like a TfD hatnote, like the tag will output info that it's legacy both at the same time is indeed the graphs team (or whatever it's called)'s plan, Nux's just proposing that we just auto-migrate all graphs without doing anything else when graphs are reenabled and template editors would port the other 20%, with the fact that vega2 doesn't work being an incentive to port to vega5. Aaron Liu (talk) 20:33, 20 September 2023 (UTC)
 * Makes sense. Sj (talk) 21:59, 20 September 2023‎ (UTC)

Add step for archiving unconvertible graphs as images?
Assuming some graphs won't be converted to V5, at some point it would be good to compile a list of graphs that need to be converted to images, and do the conversion [on a server that allows sandboxed v2 rendering]. I'd love to see that included in the timeline, so they don't slip through the cracks. Sj (talk) 21:58, 20 September 2023 (UTC)


 * Somewhere to re-create the V2 graph or map would be great. In-place V2 would have the benefit of a rapid return for broken items. eg the '#tag:graph' -> 'Graph:Street map with marks' -> 'OSM Location map' -> 'final map' chain of conversion could easily be a long process of re-thinking and re-working. If so, working maps during that period would be a benefit of itself. But any method of seeing the V2 output would be a huge help in checking if the new version is matching the original features. Similarly, editors working at hand-converting a graph they added years ago, would doubtless welcome some method of comparing the new with what it is supposed to look like. RobinLeicester (talk) 17:16, 26 September 2023 (UTC)