Extension talk:Graph/Plans

Table of usage
I'm not sure how accurate using "Pages with disabled graphs" is to get a count. Graphs have been disabled for a while now, presumably people have been removing them from pages, and thus also removing the category. Maybe looking at an older dump of the category table might be better but probably quite a bit of work. Bawolff (talk) 23:18, 14 July 2023 (UTC)


 * I'm not sure how accurate using "Pages with disabled graphs" is to get a count. Graphs have been disabled for a while now, presumably people have been removing them from pages, and thus also removing the category.
 * Mmm, great spot, @Bawolff. It looks like @TheDJ spotted a similar issue.
 * As a next step, I'm going to ask some folks internally about how we might go about generating a more accurate measure of usage. Of course, if new ideas emerge in your mind for how we might go about doing this and/or if volunteers who you think would be well equipped to help out with this, please let me know. PPelberg (WMF) (talk) 16:58, 28 July 2023 (UTC)

Including personal opinions
@Bawolff: I appreciate you both being bold in adding this statement and asking whether it's appropriate to do so.

What if you prefaced the bit beginning with "The elephant in the room is that graph is a high maintenance extension... with your username so as to show this is a personal opinion that may or may not be something the information we end up gathering substantiates? How does this sound to you? PPelberg (WMF) (talk) 23:54, 14 July 2023 (UTC)


 * SGTM. Bawolff (talk) 23:59, 14 July 2023 (UTC)
 * Awesome. PPelberg (WMF) (talk) 00:03, 15 July 2023 (UTC)

Generating graphs from Wikidata
hi – two things!


 * 1) Thank you for contributing helpful information to the Extension:Graph/Plans page  🙏🏼
 * 2) Related to "1.", are you able to add an example or perhaps describe in a bit more detail what an up-to-date graph from Wikidata data generated using the Graph Extension looked like? I ask as this is the first time I'm hearing of the extension being used in this way.

Ok, thank you ^ _ ^ PPelberg (WMF) (talk) 21:37, 28 July 2023 (UTC)


 * Hi you can find one example here (see the wikicode). In short, when the graph extension was working, this template get the neutron lifetime from Wikidata and then plot these values as a function of the year.
 * So that, when I add a new value on Wikidata, the graph is automatically updated. Pamputt (talk) 21:46, 28 July 2023 (UTC)
 * Ah, I see – thank you for following up with this added context and for doing so so promptly, @Pamputt!
 * Sure enough, just after I posted the above, I also saw that @Nux helpfully added the example of en:Template:Airport-Statistics.
 * In any event, @Pamputt can you please have a quick look at the below and let me know if there is anything missing, unnecessary, and/or inaccurate about how I'm currently understanding how Wikidata, the Graph Extension, and on-wiki templates can be used to generate auto-updating graphs?
 * How Peter (me) currently understands the relationships between Wikidata, the Graph Extension, and on-wiki templates
 * 1. Use SPARQL to retrieve the information you're seeking from Wikidata. ''[https://query.wikidata.org/#%23%20Scroll%20down%20and%20hit%20blue%20arrow%20down%20to%20run%20and%20see%20the%20results%20%2B%20the%20sources%0ASELECT%20%3Fyear%20%3Fitem%20%3Fshortname%20%28MAX%28%3Fnumber%29%20AS%20%3Fpassengers%29%20%20%20%28SAMPLE%28COALESCE%28%3Freference_URL%2C%20%3Fmonthly_reference_URL2%29%29%20AS%20%3Fsample_reference_URL%29%0AWITH%0A%7B%20%20SELECT%20%3Fitem%20%3Fstatement%20%3Fdate%20%3Fyear%20%3Ftimevalue%20%3Fnumberperperiod%20%3Freference_URL%0A%20%20WHERE%20%20%7B%20%20%20%20%3Fitem%20wdt%3A%20%3Fairport_code%0A%20%20%20%20VALUES%20%3Fairport_code%20%20%20%20%20%7B%20%22%22%20%20%20%20%7D%0A%20%20%20%20%3Fitem%20p%3AP3872%20%3Fstatement.%0A%20%20%20%20%3Fstatement%20pqv%3AP585%20%3Ftimevalue%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20ps%3AP3872%20%3Fnumberperperiod.%0A%20%20%20%20%3Ftimevalue%20wikibase%3AtimeValue%20%3Fdate.%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20pq%3AP518%20%3Fapplies.%20%7D%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20prov%3AwasDerivedFrom%20%2F%20%28pr%3AP854%7Cpr%3AP4656%29%20%3Freference_URL.%20%7D%0A%20%20%20%20FILTER%20%28BOUND%28%3Fapplies%29%3Dfalse%20%7C%7C%20%3Fapplies%20%3D%20wd%3AQ2165236%20%29%0A%20%20%20%20MINUS%20%7B%20%3Fstatement%20wikibase%3Arank%20wikibase%3ADeprecatedRank%20%7D%0A%20%20%20%20BIND%20%28YEAR%28%3Fdate%29%20AS%20%3Fyear%29%0A%20%20%20%20FILTER%20%28%3Fyear%20%3E1949%29.%20%20%20%20FILTER%20%28%3Fyear%20%3C%20YEAR%28NOW%28%29%29%29%0A%20%20%7D%20%7D%20AS%20%25airport%0AWHERE%0A%7B%20%20%20%7B%20%20%20%20%23%20Get%20the%20sum%20of%20monthly%20values%20within%20a%20year%0A%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28SUM%28%3Fmax_numberperperiod%29%20AS%20%3Fnumber%29%20%28SAMPLE%28%3Fmonthly_reference_URL%29%20AS%20%3Fmonthly_reference_URL2%29%0A%20%20%20%20WHERE%0A%20%20%20%20%7B%20%20%20%20%20%20%23%20Get%20the%20maximal%20value%20and%20a%20sample%20reference%20URL%20for%20each%20unique%20month%0A%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28MAX%28%3Fnumberperperiod%29%20AS%20%3Fmax_numberperperiod%29%20%28SAMPLE%28%3Freference_URL%29%20AS%20%3Fmonthly_reference_URL%29%0A%20%20%20%20%20%20%20%20WHERE%20%20%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20%20%20INCLUDE%20%25airport%0A%20%20%20%20%20%20%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%20%3Fprec.%0A%20%20%20%20%20%20%20%20%20%20FILTER%20%28%3Fprec%20%3E%209%29%23%20precision%20more%20precise%20or%20equal%20to%20month%0A%20%20%20%20%20%20%20%20%7D%20%20%20%20%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%20%3Fdate%0A%20%20%20%20%20%20%7D%20%20%20%20%7D%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%0A%20%20%7D%20%20UNION%20%20%7B%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%209%20.%20%20%20%20BIND%20%28%3Fnumberperperiod%20AS%20%3Fnumber%29%20%20%20%20BIND%20%28%3Freference_URL%20AS%20%3Fsample_reference_URL%29%0A%20%20%20%20INCLUDE%20%25airport%0A%20%20%7D%0A%20%20OPTIONAL%20%7B%3Fitem%20wdt%3AP1813%20%3Fthis.%20%20%20%20%20%20%20%20%20%20%20%20%23%20has%20shortname%0A%20%20%20%20FILTER%28LANG%28%3Fthis%29%3D%22en%22%29%20%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%2Cen%22.%20%3Fitem%20rdfs%3Alabel%20%3FitemLabel.%7D%0ABIND%28COALESCE%28%3Fthis%2C%3FitemLabel%29%20as%20%3Fshortname%29%0A%7D%20GROUP%20BY%20%3Fitem%20%3Fshortname%20%3Fyear%20ORDER%20BY%20%3Fitem%20DESC%20%28%3Fyear%29 Example SPARQL query]''
 * 2. Use a template to ingest the data the query you wrote in "1." will have returned and convert that data into a format (e.g. JSON) that the Graph extension can understand
 * 3. Generate a graph using the Graph Extension by "calling" the data you generated in "2." by adding the  notation to the page that you'd like the graph to appear on PPelberg (WMF) (talk) 21:59, 28 July 2023 (UTC)
 * Yes, this is how it worked. The SPARLQ query directly returns the data in the JSON format. From this JSON, the key are the ones given in the SPARQL query (i.e. "main_val", "lower", etc.), and so we can call them in the graph code to format the data as we wish. Pamputt (talk) 04:00, 29 July 2023 (UTC)
 * Thank you for following up with this additional clarity, @Pamputt.
 * To be doubly certain, it sounds like it's two steps (listed below) rather than three steps (as I'd thought above), does that sound accurate?
 * Use SPARQL to retrieve the information you're seeking from Wikidata. ''[https://query.wikidata.org/#%23%20Scroll%20down%20and%20hit%20blue%20arrow%20down%20to%20run%20and%20see%20the%20results%20%2B%20the%20sources%0ASELECT%20%3Fyear%20%3Fitem%20%3Fshortname%20%28MAX%28%3Fnumber%29%20AS%20%3Fpassengers%29%20%20%20%28SAMPLE%28COALESCE%28%3Freference_URL%2C%20%3Fmonthly_reference_URL2%29%29%20AS%20%3Fsample_reference_URL%29%0AWITH%0A%7B%20%20SELECT%20%3Fitem%20%3Fstatement%20%3Fdate%20%3Fyear%20%3Ftimevalue%20%3Fnumberperperiod%20%3Freference_URL%0A%20%20WHERE%20%20%7B%20%20%20%20%3Fitem%20wdt%3A%20%3Fairport_code%0A%20%20%20%20VALUES%20%3Fairport_code%20%20%20%20%20%7B%20%22%22%20%20%20%20%7D%0A%20%20%20%20%3Fitem%20p%3AP3872%20%3Fstatement.%0A%20%20%20%20%3Fstatement%20pqv%3AP585%20%3Ftimevalue%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20ps%3AP3872%20%3Fnumberperperiod.%0A%20%20%20%20%3Ftimevalue%20wikibase%3AtimeValue%20%3Fdate.%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20pq%3AP518%20%3Fapplies.%20%7D%0A%20%20%20%20OPTIONAL%20%7B%20%3Fstatement%20prov%3AwasDerivedFrom%20%2F%20%28pr%3AP854%7Cpr%3AP4656%29%20%3Freference_URL.%20%7D%0A%20%20%20%20FILTER%20%28BOUND%28%3Fapplies%29%3Dfalse%20%7C%7C%20%3Fapplies%20%3D%20wd%3AQ2165236%20%29%0A%20%20%20%20MINUS%20%7B%20%3Fstatement%20wikibase%3Arank%20wikibase%3ADeprecatedRank%20%7D%0A%20%20%20%20BIND%20%28YEAR%28%3Fdate%29%20AS%20%3Fyear%29%0A%20%20%20%20FILTER%20%28%3Fyear%20%3E1949%29.%20%20%20%20FILTER%20%28%3Fyear%20%3C%20YEAR%28NOW%28%29%29%29%0A%20%20%7D%20%7D%20AS%20%25airport%0AWHERE%0A%7B%20%20%20%7B%20%20%20%20%23%20Get%20the%20sum%20of%20monthly%20values%20within%20a%20year%0A%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28SUM%28%3Fmax_numberperperiod%29%20AS%20%3Fnumber%29%20%28SAMPLE%28%3Fmonthly_reference_URL%29%20AS%20%3Fmonthly_reference_URL2%29%0A%20%20%20%20WHERE%0A%20%20%20%20%7B%20%20%20%20%20%20%23%20Get%20the%20maximal%20value%20and%20a%20sample%20reference%20URL%20for%20each%20unique%20month%0A%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20SELECT%20%3Fitem%20%3Fyear%20%28MAX%28%3Fnumberperperiod%29%20AS%20%3Fmax_numberperperiod%29%20%28SAMPLE%28%3Freference_URL%29%20AS%20%3Fmonthly_reference_URL%29%0A%20%20%20%20%20%20%20%20WHERE%20%20%20%20%20%20%20%20%7B%20%20%20%20%20%20%20%20%20%20INCLUDE%20%25airport%0A%20%20%20%20%20%20%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%20%3Fprec.%0A%20%20%20%20%20%20%20%20%20%20FILTER%20%28%3Fprec%20%3E%209%29%23%20precision%20more%20precise%20or%20equal%20to%20month%0A%20%20%20%20%20%20%20%20%7D%20%20%20%20%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%20%3Fdate%0A%20%20%20%20%20%20%7D%20%20%20%20%7D%20%20%20%20GROUP%20BY%20%3Fitem%20%3Fyear%0A%20%20%7D%20%20UNION%20%20%7B%20%20%20%20%3Ftimevalue%20wikibase%3AtimePrecision%209%20.%20%20%20%20BIND%20%28%3Fnumberperperiod%20AS%20%3Fnumber%29%20%20%20%20BIND%20%28%3Freference_URL%20AS%20%3Fsample_reference_URL%29%0A%20%20%20%20INCLUDE%20%25airport%0A%20%20%7D%0A%20%20OPTIONAL%20%7B%3Fitem%20wdt%3AP1813%20%3Fthis.%20%20%20%20%20%20%20%20%20%20%20%20%23%20has%20shortname%0A%20%20%20%20FILTER%28LANG%28%3Fthis%29%3D%22en%22%29%20%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%2Cen%22.%20%3Fitem%20rdfs%3Alabel%20%3FitemLabel.%7D%0ABIND%28COALESCE%28%3Fthis%2C%3FitemLabel%29%20as%20%3Fshortname%29%0A%7D%20GROUP%20BY%20%3Fitem%20%3Fshortname%20%3Fyear%20ORDER%20BY%20%3Fitem%20DESC%20%28%3Fyear%29 Example SPARQL query]''
 * Generate a graph using the Graph Extension by "calling" the data you generated in "1." by adding the  notation to the page that you'd like the graph to appear on
 * PPelberg (WMF) (talk) 00:14, 1 August 2023 (UTC)
 * Yes, that's it :) Pamputt (talk) 04:48, 1 August 2023 (UTC)
 * Yes, that's it :) Pamputt (talk) 04:48, 1 August 2023 (UTC)

Update: 11 August
hey y'all – this update is an effort to help us:
 * 1) Align on what we've come to know and think about the Graph Extension
 * 2) Work together to find the information needed to address the remaining questions/uncertainties

I'm thinking the above will help us arrive at the requirements needed to distinguish viable from non-viable solutions and for the WMF (by way of me) to propose some paths forward for us to consider in the next update, planned for 15 September.

With this in mind, if you see something below that you disagree with, think could benefit from additional clarity/specificity, etc., please say as much by commenting below!

...this update is meant to identify any gaps and inconsistencies in what we're collectively thinking so that we can move forward together.

Now, before getting into what's become more clear since the 14 July update and what new uncertainties/questions surfaced: thank you!

Thank you to @Ahecht, @Bawolff, @Bouzinac, @Edu!, @Matma Rex, @Msz2001, @Novem Linguae, @Nux, @Pamputt, @Pppery, @Quiddity, @RobinLeicester, @User:SBassett (WMF), @Snævar, @Tgr, @TheDJ, and @User:VPoundstone-WMF for patiently and generously sharing the information and resources I've needed to understand and orient myself within the decisions we find ourselves needing to make.

Now onto the update… PPelberg (WMF) (talk) 18:55, 11 August 2023 (UTC)


 * Help needed: what question do we need help answering?
 * How might we go about inviting non-English speakers into this conversation and making them feel welcomed to contribute?
 * How might we go about making more people aware that this conversation is happening? It's important that we increase the likelihood that people who have experiences with and information about the Graph Extension be aware of, and ideally, a part of this conversation.
 * How might we go about forming a more accurate understanding of how widely used the Graph Extension was in relation to features that offer similar functionality?
 * PPelberg (WMF) (talk) 18:56, 11 August 2023 (UTC)
 * For #3, @Bawolff's old write-up is still probably the most comprehensive existing analysis and/or starting point for that question: https://www.mediawiki.org/wiki/User:Bawolff/Reflections_on_graphs SBassett (WMF) (talk) 18:42, 22 August 2023 (UTC)
 * Understood – thank you for pointing me to this, @SBassett (WMF); I've added this to the main page. PPelberg (WMF) (talk) 22:10, 23 August 2023 (UTC)
 * Clarity: what do we agree to be true about the Graph extension and the needs it was meant to meet?
 * There seem to be three distinct features the Graph Extension offered for which there are no current workarounds/alternatives:
 * Generating interactive maps/charts/graphs
 * Where "interactive" in this context refers to functionality/features like: enabling individual people viewing a graph to adjust the underlying parameters that determine the data they see and how it is visualized.
 * Generating maps/charts/graphs that update automatically
 * This is most often done by pulling in data from Wikidata.
 * Updating maps/charts/graphs on-wiki
 * The Graph Extension enables people to generate graphs using data stored on-wiki. This means, people can update graphs by editing the data on wiki, similar to how they might edit any other page/piece of information on the wikis.
 * In addition to the "distinct features" the Graph Extension offered, it also enabled people to:
 * Specify how this data is represented (e.g. chart type, colors, legend, mouseover events/interactions, etc.)
 * Store data within templates on pages separate from articles
 * Embed/transclude graphs within articles using templates
 * Write Lua modules that:
 * Convert JSON data into a well formatted wiki table
 * Extract data needed for graph and outputs as JSON
 * Create graph templates that can consume data and plot it
 * Pull in data from external sources via URLs
 * From generating article page view graphs to showing air traffic volume, volunteers depended on the Graph Extension in a variety of reader- and editor-focused contexts.
 * PPelberg (WMF) (talk) 18:57, 11 August 2023 (UTC)
 * Impressions: what seems like it could be true? 
 * Absent accessible and easy-to-use features that enable people to store, collaborate on, and present data on-wiki:
 * The data we host/present on our projects is more vulnerable to being outdated/inaccurate
 * The content we offer is more likely to lack corresponding data visualizations
 * In order for the Movement as a whole, and the various communities that comprise it, to evaluate the impact of the work it's doing, they need ways of generating, updating, and visualizing data.
 * To use the Graph Extension, you need/needed a relatively high degree of technical expertise on top of a foundation of data literacy.
 * Related to "3.", the Graph Extension has remained a relatively niche feature that one could, theoretically, never interact with (directly or indirectly) throughout your tenure as an editor.
 * PPelberg (WMF) (talk) 18:57, 11 August 2023 (UTC)
 * @PPelberg I'm interesting in Template:Graph:PageViews: we really need this mechanism working in ruwiki. MBH (talk) 05:07, 19 August 2023 (UTC)
 * "Template:Graph:PageViews: we really need this mechanism working in ruwiki."
 * @MBH: this is helpful to know – I'm glad you said something.
 * To help me more fully understand the extent to which you, and other volunteers at ru.wiki, depend on Template:Graph:PageViews, can you please share what has been made difficult or not possible as a result of it no longer functioning? [i]
 * i. E.g. Are there particular actions decisions that are more difficult, or perhaps not possible, to make because you do not have easy access to information about how views of a particular page vary over time? PPelberg (WMF) (talk) 22:06, 23 August 2023 (UTC)
 * @PPelberg 1) ruwiki extensively uses this template on articles' talk pages, 2) I own a bot for ru- and ukwiki, that generates articles' peak popularity (number of readers) monthly and yearly stats, see ru:File:Просмотры статей об айти-разработке.png, ru:ВП:Пики интереса к статьям/За год, uk:Вікіпедія:Спалахи інтересу до статей/За рік, this stats pages uses this template and now they are broken. MBH (talk) 18:05, 24 August 2023 (UTC)
 * @PPelberg 1) ruwiki extensively uses this template on articles' talk pages, 2) I own a bot for ru- and ukwiki, that generates articles' peak popularity (number of readers) monthly and yearly stats, see ru:File:Просмотры статей об айти-разработке.png, ru:ВП:Пики интереса к статьям/За год, uk:Вікіпедія:Спалахи інтересу до статей/За рік, this stats pages uses this template and now they are broken. MBH (talk) 18:05, 24 August 2023 (UTC)


 * I could really use a mechanism for generating graphs like Our World in Data; something requested at the start of the pandemic and many times since. "we should aim to have a world map graph template just like [OWID], regardless of what we have to do with the graph extension or services behind it." Sj (talk) 07:20, 30 August 2023 (UTC)

Underlying purpose inspiring the extension
Improving graphs and interactive content and Yurik's Dream of Content seem as relevant as ever! This isn't quite responsive to Peter's questions above which were specifically about use of / need for the Graph extension as currently implemented, so I'm putting these points in a separate section; feel free to refactor or merge.


 * Data visualization is an important part of a more beautiful and richly sourced references: helpful to readers, and to the identity of the projects.
 * The existence of good data viz adds incentive to get the underlying data on the projects (and to get it right)
 * The ability to do good data viz attracts people who care about that face to knowledge production and appreciate an impactful place to contribute that to the commons. This is where having tools that let us make interactive results leads to a new ecosystem of curators and readers
 * Many of our most common use cases might instead use the same visualization libraries as OWID: their context and audience and need for customizable compact summaries from shared data sources is quite similar to ours.
 * Continuity and archival quality: if needed we can downgrade interactives to static visuals, but should not remove data visualizations without replacement. If the extension is going away, there should be a way to convert removed graphs to a static image [that can be run at scale]
 * Uncertainty dissipates community. It helps to have a maintainer and roadmap/plan of record for "how we visualize data on the projects". This is always going to be a range of uses, with a power law, some of which have many alternatives. Currently people who might help develop alternatives are still thinking about Vega migrations (cf the talk page) and may be wasting their time.
 * Losing previous content without replacement for 6 months is weird. We should at least render all previous graphs and embed the resulting images.  Not having a maintainer to do this or explain why it's not being done is weird.  An extended conversation about what we may or may not do in the future (without first doing the above as a stopgap, to avoid losing content people spent thousands of hours making) is very weird. And teaches people not to trust the wikis as a place to make and share visualizations. (Part of the challenge is that it's not obvious who should do it, or if it will be a waste of time b/c a more official solution is about to drop.)
 * We all benefit from building a shared vision for how and where data visualization happens on the projects, and how that fits into the overall development of reading and editing. This is broader and simpler than the questions about this extension.
 * Any sort of roadmap is fine. "eventually reenable a comparable extension" is clear. "we can't secure this, extension is deprecated + tasks closed, please migrate off w/ these tools; we'll find another solution" would be clear. "tools to visualize data should be maintained by community and not depend on staff or need hefty security review" would be clear. We could still debate changing the roadmap but at any point in time the current roadmap should be legible. Sj (talk) 07:20, 30 August 2023 (UTC) 13:18, 6 September 2023 (UTC)