Extension talk:Graph/Plans/Archive 1

Table of usage

I'm not sure how accurate using "Pages with disabled graphs" is to get a count. Graphs have been disabled for a while now, presumably people have been removing them from pages, and thus also removing the category. Maybe looking at an older dump of the category table might be better but probably quite a bit of work. Bawolff (talk) 23:18, 14 July 2023 (UTC)

I'm not sure how accurate using "Pages with disabled graphs" is to get a count. Graphs have been disabled for a while now, presumably people have been removing them from pages, and thus also removing the category.

Mmm, great spot, @Bawolff. It looks like @TheDJ spotted a similar issue.

As a next step, I'm going to ask some folks internally about how we might go about generating a more accurate measure of usage. Of course, if new ideas emerge in your mind for how we might go about doing this and/or if volunteers who you think would be well equipped to help out with this, please let me know. PPelberg (WMF) (talk) 16:58, 28 July 2023 (UTC)

Including personal opinions

@Bawolff: I appreciate you both being bold in adding this statement and asking whether it's appropriate to do so.

What if you prefaced the bit beginning with "The elephant in the room is that graph is a high maintenance extension... with your username so as to show this is a personal opinion that may or may not be something the information we end up gathering substantiates? How does this sound to you? PPelberg (WMF) (talk) 23:54, 14 July 2023 (UTC)

SGTM. Bawolff (talk) 23:59, 14 July 2023 (UTC)

Awesome. PPelberg (WMF) (talk) 00:03, 15 July 2023 (UTC)

Generating graphs from Wikidata

hi @Pamputt – two things!

1. Thank you for contributing helpful information to the Extension:Graph/Plans page 🙏🏼
2. Related to "1.", are you able to add an example or perhaps describe in a bit more detail what an up-to-date graph from Wikidata data generated using the Graph Extension looked like? I ask as this is the first time I'm hearing of the extension being used in this way.

Ok, thank you ^ _ ^ PPelberg (WMF) (talk) 21:37, 28 July 2023 (UTC)

Hi PPelberg (WMF) you can find one example here (see the wikicode). In short, when the graph extension was working, this template get the neutron lifetime from Wikidata and then plot these values as a function of the year.

So that, when I add a new value on Wikidata , the graph is automatically updated. Pamputt (talk) 21:46, 28 July 2023 (UTC)

Ah, I see – thank you for following up with this added context and for doing so so promptly, @Pamputt!

Sure enough, just after I posted the above, I also saw that @Nux helpfully added the example of en:Template:Airport-Statistics.

In any event, @Pamputt can you please have a quick look at the below and let me know if there is anything missing, unnecessary, and/or inaccurate about how I'm currently understanding how Wikidata, the Graph Extension, and on-wiki templates can be used to generate auto-updating graphs?

How Peter (me) currently understands the relationships between Wikidata, the Graph Extension, and on-wiki templates

1. Use SPARQL to retrieve the information you're seeking from Wikidata. Example SPARQL query

2. Use a template to ingest the data the query you wrote in "1." will have returned and convert that data into a format (e.g. JSON) that the Graph extension can understand

3. Generate a graph using the Graph Extension by "calling" the data you generated in "2." by adding the {{Graph...}} notation to the page that you'd like the graph to appear on PPelberg (WMF) (talk) 21:59, 28 July 2023 (UTC)

Yes, this is how it worked. The SPARLQ query directly returns the data in the JSON format. From this JSON, the key are the ones given in the SPARQL query (i.e. "main_val", "lower", etc.), and so we can call them in the graph code to format the data as we wish. Pamputt (talk) 04:00, 29 July 2023 (UTC)

Thank you for following up with this additional clarity, @Pamputt.

To be doubly certain, it sounds like it's two steps (listed below) rather than three steps (as I'd thought above), does that sound accurate?

1. Use SPARQL to retrieve the information you're seeking from Wikidata. Example SPARQL query
2. Generate a graph using the Graph Extension by "calling" the data you generated in "1." by adding the {{Graph...}} notation to the page that you'd like the graph to appear on

PPelberg (WMF) (talk) 00:14, 1 August 2023 (UTC)

Yes, that's it :) Pamputt (talk) 04:48, 1 August 2023 (UTC)

Update: 11 August

hey y'all – this update is an effort to help us:

1. Align on what we've come to know and think about the Graph Extension
2. Work together to find the information needed to address the remaining questions/uncertainties

I'm thinking the above will help us arrive at the requirements needed to distinguish viable from non-viable solutions and for the WMF (by way of me) to propose some paths forward for us to consider in the next update, planned for 15 September.

With this in mind, if you see something below that you disagree with, think could benefit from additional clarity/specificity, etc., please say as much by commenting below!

...this update is meant to identify any gaps and inconsistencies in what we're collectively thinking so that we can move forward together.

Now, before getting into what's become more clear since the 14 July update and what new uncertainties/questions surfaced: thank you!

Thank you to @Ahecht, @Bawolff, @Bouzinac, @Edu!, @Matma Rex, @Msz2001, @Novem Linguae, @Nux, @Pamputt, @Pppery, @Quiddity, @RobinLeicester, @User:SBassett (WMF), @Snævar, @Tgr, @TheDJ, and @User:VPoundstone-WMF for patiently and generously sharing the information and resources I've needed to understand and orient myself within the decisions we find ourselves needing to make.

Now onto the update… PPelberg (WMF) (talk) 18:55, 11 August 2023 (UTC)

Help needed: what question do we need help answering?

1. How might we go about inviting non-English speakers into this conversation and making them feel welcomed to contribute?
2. How might we go about making more people aware that this conversation is happening? It's important that we increase the likelihood that people who have experiences with and information about the Graph Extension be aware of, and ideally, a part of this conversation.
3. How might we go about forming a more accurate understanding of how widely used the Graph Extension was in relation to features that offer similar functionality?

PPelberg (WMF) (talk) 18:56, 11 August 2023 (UTC)

For #3, @Bawolff's old write-up is still probably the most comprehensive existing analysis and/or starting point for that question: https://www.mediawiki.org/wiki/User:Bawolff/Reflections_on_graphs SBassett (WMF) (talk) 18:42, 22 August 2023 (UTC)

Understood – thank you for pointing me to this, @SBassett (WMF); I've added this to the main page. PPelberg (WMF) (talk) 22:10, 23 August 2023 (UTC)

Clarity: what do we agree to be true about the Graph extension and the needs it was meant to meet?

1. There seem to be three distinct features the Graph Extension offered for which there are no current workarounds/alternatives:
   1. Generating interactive maps/charts/graphs
      1. Where "interactive" in this context refers to functionality/features like: enabling individual people viewing a graph to adjust the underlying parameters that determine the data they see and how it is visualized.
   2. Generating maps/charts/graphs that update automatically
      1. This is most often done by pulling in data from Wikidata.
   3. Updating maps/charts/graphs on-wiki
      1. The Graph Extension enables people to generate graphs using data stored on-wiki. This means, people can update graphs by editing the data on wiki, similar to how they might edit any other page/piece of information on the wikis.
2. In addition to the "distinct features" the Graph Extension offered, it also enabled people to:
   1. Specify how this data is represented (e.g. chart type, colors, legend, mouseover events/interactions, etc.)
   2. Store data within templates on pages separate from articles
   3. Embed/transclude graphs within articles using templates
   4. Write Lua modules that:
      1. Convert JSON data into a well formatted wiki table
      2. Extract data needed for graph and outputs as JSON
   5. Create graph templates that can consume data and plot it
   6. Pull in data from external sources via URLs
3. From generating article page view graphs to showing air traffic volume, volunteers depended on the Graph Extension in a variety of reader- and editor-focused contexts.

PPelberg (WMF) (talk) 18:57, 11 August 2023 (UTC)

Impressions: what seems like it could be true?

1. Absent accessible and easy-to-use features that enable people to store, collaborate on, and present data on-wiki:
   1. The data we host/present on our projects is more vulnerable to being outdated/inaccurate
   2. The content we offer is more likely to lack corresponding data visualizations
2. In order for the Movement as a whole, and the various communities that comprise it, to evaluate the impact of the work it's doing, they need ways of generating, updating, and visualizing data.
3. To use the Graph Extension, you need/needed a relatively high degree of technical expertise on top of a foundation of data literacy.
4. Related to "3.", the Graph Extension has remained a relatively niche feature that one could, theoretically, never interact with (directly or indirectly) throughout your tenure as an editor.

PPelberg (WMF) (talk) 18:57, 11 August 2023 (UTC)

@PPelberg I'm interesting in Template:Graph:PageViews: we really need this mechanism working in ruwiki. MBH (talk) 05:07, 19 August 2023 (UTC)

" Template:Graph:PageViews: we really need this mechanism working in ruwiki. "

@MBH: this is helpful to know – I'm glad you said something.

To help me more fully understand the extent to which you, and other volunteers at ru.wiki, depend on Template:Graph:PageViews, can you please share what has been made difficult or not possible as a result of it no longer functioning? [i]

---

i. E.g. Are there particular actions decisions that are more difficult, or perhaps not possible, to make because you do not have easy access to information about how views of a particular page vary over time? PPelberg (WMF) (talk) 22:06, 23 August 2023 (UTC)

@PPelberg 1) ruwiki extensively uses this template on articles' talk pages, 2) I own a bot for ru- and ukwiki, that generates articles' peak popularity (number of readers) monthly and yearly stats, see ru:File:Просмотры статей об айти-разработке.png, ru:ВП:Пики интереса к статьям/За год, uk:Вікіпедія:Спалахи інтересу до статей/За рік, this stats pages uses this template and now they are broken. MBH (talk) 18:05, 24 August 2023 (UTC)

I could really use a mechanism for generating graphs like Our World in Data; something requested at the start of the pandemic and many times since. "we should aim to have a world map graph template just like [OWID], regardless of what we have to do with the graph extension or services behind it." Sj (talk) 07:20, 30 August 2023 (UTC)

Underlying purpose inspiring the extension

Improving graphs and interactive content and Yurik's Dream of Content seem as relevant as ever! This isn't quite responsive to Peter's questions above which were specifically about use of / need for the Graph extension as currently implemented, so I'm putting these points in a separate section; feel free to refactor or merge.

• Data visualization is an important part of a more beautiful and richly sourced references: helpful to readers, and to the identity of the projects.
• The existence of good data viz adds incentive to get the underlying data on the projects (and to get it right)
• The ability to do good data viz attracts people who care about that facet of knowledge production and appreciate an impactful place to contribute that to the commons. This is where having tools that let us make interactive results leads to a new ecosystem of curators and readers
• Many of our most common use cases might instead use the same visualization libraries as OWID: their context and audience and need for customizable compact summaries from shared data sources is quite similar to ours.
• Continuity and archival quality: if needed we can downgrade interactives to static visuals, but should not remove data visualizations without replacement. If the extension is going away, there should be a way to convert removed graphs to a static image [that can be run at scale]
• Uncertainty dissipates community. It helps to have a maintainer and roadmap/plan of record for "how we visualize data on the projects". This is always going to be a range of uses, with a power law, some of which have many alternatives. Currently people who might help develop alternatives are still thinking about Vega migrations (cf the talk page) and may be wasting their time.
  • Losing previous content without replacement for 6 months is weird. We should at least render all previous graphs and embed the resulting images. Not having a maintainer to do this or explain why it's not being done is weird. An extended conversation about what we may or may not do in the future (without first doing the above as a stopgap, to avoid losing content people spent thousands of hours making) is very weird. And teaches people not to trust the wikis as a place to make and share visualizations. (Part of the challenge is that it's not obvious who should do it, or if it will be a waste of time b/c a more official solution is about to drop.)
• We all benefit from building a shared vision for how and where data visualization happens on the projects, and how that fits into the overall development of reading and editing. This is broader and simpler than the questions about this extension.
  • Any sort of roadmap is fine. "eventually reenable a comparable extension" is clear. "we can't secure this, extension is deprecated + tasks closed, please migrate off w/ these tools; we'll find another solution" would be clear. "tools to visualize data should be maintained by community and not depend on staff or need hefty security review" would be clear. We could still debate changing the roadmap but at any point in time the current roadmap should be legible. Sj (talk) 07:20, 30 August 2023 (UTC) 13:18, 6 September 2023 (UTC)

@Sj responses/follow up questions to the points you raised below.

Before that, thank you for starting this conversation.

I think you are spot-on in naming the need for exploring the broader themes/capabilities the Graph Extension is an expression of and is crucial context to consider as we propose a set of next steps.^[i]

Note: I'm going to post responses in separate comments so that we can [hopefully] explore multiple threads in parallel a bit more easily.

---

i. Propose next steps will come via the update I'll share next Friday, 15 Sep. PPelberg (WMF) (talk) 23:16, 8 September 2023 (UTC)

Data visualization is an important part of a more beautiful and richly sourced references: helpful to readers, and to the identity of the projects.

I share this assumption that readers value data visualizations.

Have you seen/read any research that substantiates this? Asked another way: are you able to share what you've noticed that contribute to you thinking the above is true?

For context, the most compelling research I've come across to-date related to the above being true is a study from the Research Team. This research found illustrated Wikipedia pages receive 4x more pageviews than those without images. That same research also found the ratio of image clicks to pageviews to be 1:30 compared to 1:340 for citations and 1:110 for links. PPelberg (WMF) (talk) 23:20, 8 September 2023 (UTC)

I would argue that video is a better model for data visualization than an image. It is relatively easy to improve an article with an image. Even a bad image is often better than no image if that's all that is available. Video is much harder - it can improve an article, but it is much more important that the media is of high quality and tells a "story". It is much harder to create a good video, and a bad video really does nothing for an article. I think this is what data visualization is like - it can significantly improve an article, but only if good, plot is more important, and it is much more difficult to create a good visualization regardless of the tools. Bawolff (talk) 02:37, 9 September 2023 (UTC)

There are all sorts of readers. Areas where this is not in question: Scholarly journals and research have prized visualizations for over a century. So have statistical or mathematical topics such as budgets, economic histories, epidemiologies, &c. For general-purpose descriptions to a lay audience, textbooks for many decades, encyclopedias since Encarta, and more recently with data journalism (in news) and Wolfram Alpha (in search), subfields or major enterprises have been built on the premise + promise of reliable, legible visualization. Good viz brings readers who better understand topics visualized that way. For current readers on an average article that includes some graphable data: I don't have specific research, but infoboxes and summary tables seem quite popular and widely referenced / screenshotted. Sj (talk) 23:27, 11 September 2023 (UTC)

The existence of good data viz adds incentive to get the underlying data on the projects (and to get it right)

Mmm. The relationship you're naming between the usefulness of a knowledge format/representation and peoples' likelihood to contribute the knowledge needed to produce them seems insightful and accurate to me.

To the "accuracy" bit, are you able to share a bit about how you seem to have come to think that this relationship holds specifically true for data visualizations in our ecosystem?

No worries if this is more of an intuition at this point. Although, if there are things you can point to that cause you to feel confident in this statement, I'd value seeing them. PPelberg (WMF) (talk) 23:21, 8 September 2023 (UTC)

Historically across a lot of formats, first one is tried as a style (dot maps; nav templates; infoboxes; short WD descriptions), then it gets incorporated into various editing scripts and bots, then at some point one or two people dedicate themselves to populating and cleaning up the format across whole categories of relevant pages. We have scattered and inhomogenous ways to store tabular data and time series data; having a central use case like this will lead to scripts and other tools standardizing elements that make the data more findable, better updated, editable and more often corrected. Having a quick way to visualize data will over time lead to identifying and fixing mistake and outliers (visible across dozens of covid-tracking projects). Sj (talk) 23:27, 11 September 2023 (UTC)

The ability to do good data viz attracts people who care about that facet of knowledge production and appreciate an impactful place to contribute that to the commons. This is where having tools that let us make interactive results leads to a new ecosystem of curators and readers

The potential you're naming here is inspiring to me and it leads me to wonder the following…

Assuming it's accurate to think the interactive graphs the Graph Extension enabled volunteers to produce did NOT produce the "new ecosystem of curators and readers" you're referring to above, do you have a sense for what might have contributed to this? PPelberg (WMF) (talk) 23:22, 8 September 2023 (UTC)

That would not be accurate. It needs an easier onramp, and a place to store versioned tabular data that can be used as the input to such graphs, but the tens of thousands of pages using it, and the handful of specific projects + maintainers on those projects that used it extensively (and commented immediately when it went away) were members of that ecosystem. These things often have a slow takeoff before there is a critical mass of people learning from and borrowing from one another, filling in the various gaps. OWID illustrates the point nicely: we have an obvious source of freely-licensed data on hundreds of topic areas, hundreds of editors spending thousands of hours making equivalent data into static images on Wikipedia (like the 1500 revisions of this graph of COVID cases in Canada), no easy way to do that with Graph (despite a dozen people specifically trying and failing to make that happen). There was also a WikiGraphers User Group created recently, but not pointed to this extension as the natural place to start graphing. Sj (talk) 23:27, 11 September 2023 (UTC)

Many of our most common use cases might instead use the same visualization libraries as OWID: their context and audience and need for customizable compact summaries from shared data sources is quite similar to ours.

Great spot. I'm thinking this information will come in handy if/when we arrive in a place where we're exploring the possibility of building a successor to the Graph Extension. In the meantime, I've documented this idea on T345962. PPelberg (WMF) (talk) 23:25, 8 September 2023 (UTC)

Continuity and archival quality: if needed we can downgrade interactives to static visuals, but should not remove data visualizations without replacement.

+1; excellently put. PPelberg (WMF) (talk) 23:26, 8 September 2023 (UTC)

If the extension is going away, there should be a way to convert removed graphs to a static image [that can be run at scale].

This sounds right to me too. I'm checking with engineering to understand the viability of what you're proposing. I'll report back what I hear here and on T334940 (thank you for raising this there). PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)

Uncertainty dissipates community.

Agreed and I appreciate you naming this potential.

Related to the above: can you think of ways we could improve how we're communicating about the Graph Extension and the plans surrounding it?

Perhaps the question above is better suited for a "retro" of sorts once a plan is in place. Tho, I thought I'd seed the question now in case immediate ideas/suggestions came to mind. PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)

Currently people who might help develop alternatives are still thinking about Vega migrations (cf the talk page) and may be wasting their time.

I hadn't seen this comment; thank you for drawing my attention to it. I'll follow up with this person to try and offer some clarity about what we're thinking. PPelberg (WMF) (talk) 23:27, 8 September 2023 (UTC)

We all benefit from building a shared vision for how and where data visualization happens on the projects, and how that fits into the overall development of reading and editing. This is broader and simpler than the questions about this extension.

+1 and I think you investing the time to articulate the above is starting the conversation necessary to help us do what you're describing…thank you!

In terms of what's next, here's what I'm thinking:

Between now and next Friday, I'll be preparing an update that will include:

1. A proposal for what we do with Graph Extension and the visualizations that have been rendered inaccessible by it being disabled
2. A broader snapshot of how we're currently thinking about the strategic relevance of supporting the creation of, and collaboration on, on-wiki data visualizations.

How does that sound? What – if anything – about "1." and "2." would you suggest we adjust and/or add to provide the clarity and direction I think you aptly named us all as collectively needing.

Note: Please know the questions I'm asking above are in service of me being newer to this context and motivated to understand how you've come to arrive at these points of view so that we can reason about this space together ^ _ ^ PPelberg (WMF) (talk) 23:30, 8 September 2023 (UTC)

I'd just like to echo the sub-bullet that was not quoted here, about having a legible roadmap. A proposal and strategic snapshot will be nice, but I hope that if these do not represent a firm decision that's been made, an ETA can be communicated on when such a decision will be made. For people who are vaguely aware of how software development works, "we don't know when this will be fixed" is an understandable position. "We don't know when this will be fixed, or if it will be fixed, or what will replace it or when it will be replaced" is not very encouraging. Even a decision not to support a graphing module at the core level and kicking it back to the community is a decision that will give volunteers the green light to start working on a replacement. Folly Mox (talk) 01:56, 11 September 2023 (UTC)

Seconded! I keep running to graphs all over en.Wikipedia that are broken, and for some articles, it has a really significant impact. We need a clear plan and communication one way or another. Ganesha811 (talk) 14:04, 11 September 2023 (UTC)

@Folly Mox: I hope that if these do not represent a firm decision that's been made, an ETA can be communicated on when such a decision will be made.

@Ganesha811: We need a clear plan and communication one way or another.

+1 to both of the points you both are raising which I understand to be something like: "While a broader strategic conversation is important, we first need clear direction from you all, the WMF, about what – if anything – you're going to do to improve the current state of things."

Assuming I've understood y'all accurately, you can expect me to offer this clarity via an update to this page and T334940 before this Friday, (15 Sep) is over. More context here. PPelberg (WMF) (talk) 19:20, 11 September 2023 (UTC)

That sounds good, and yes, I think you've summarized the central point correctly. Every day that graphs are broken is a day that tens of thousands of Wikipedia articles are incomplete (often in significant ways), and therefore are not serving readers well. Ganesha811 (talk) 19:23, 11 September 2023 (UTC)

Excellent. Thank you for reviewing, @Ganesha811. And thank you for acting on the instinct to come here to speak up. PPelberg (WMF) (talk) 19:26, 11 September 2023 (UTC)

Thank you for the expectation of clarity. I think that's what people are seeking in general. The before / after the strategic conversation chronology is less meaningful to me than the "after five months" temporal milestone, but it sounds like you've got some bullet points prepared for Friday that will indicate what the overall vibe is regarding the issue, so I consider my concerns allayed. Folly Mox (talk) 20:06, 11 September 2023 (UTC)

Update: 15 September

Hi y'all – this update is an effort to arrive at the clarity we need to ensure volunteers around the Movement regain access to the information and capabilities disabling the Graph Extension has left them without.

Within the next two weeks, you can expect another update about how we are thinking about the broader strategic importance of graphs.

With the above in mind, here's what we need your help with to move forward: Can you please review the "Proposal" and "Vega 2 → Vega 5 transition" sections below and share what seems unclear, missing, and/or misguided?

---

Notes: the "Proposal" that follows is an outgrowth of the discussions and information gathering that's been taking place, in large part, on this page and Extension:Graph/Plans over the past three months.

Proposal

To safely restore access to the information and capabilities disabling the Graph Extension has left people without and promote the volunteer+staff collaboration needed to do so, we (the WMF) are committing to:

1. Re-enable the Graph Extension in a sandboxed iFrame with a restrictive content security policy.
   1. Once the Graph Extension is reenabled, it will continue to work with Vega 2 for a yet-to-be defined period of time. Note: we'll need to define this window together.
   2. After that "yet-to-be defined period of time," Vega 2 support will be discontinued and use of the Graph Extension will require volunteers to make graphs with Vega 5.
2. As soon as possible, make the sandboxed Graph extension available on the beta cluster for testing. See: T346292.
3. Investigate the viability of adding logging to increase our awareness of instances where people are exploiting the security vulnerabilities inherent with restoring support for Vega on our platform. See T346414.
4. Publish the technical documentation needed for developers across the Movement to understand how we implemented the sandboxed CSP approach
5. Publish a clear timeline for when you all can expect all of the above to happen
   Note: exploratory work to redeploy the Graph Extension in a sandboxed iframe has started. See T222807.
6. Share regular updates about the progress we're making on the commitments named above on Phabricator and MediaWiki.
7. Support volunteers with code and processes that will ease the transition from Vega 2 to Vega 5 when the time for this transition comes.

In support of the above, we'd need to depend on ya'll (volunteers) to:

1. Spread awareness of this proposal and the updates that will come as we start implementation, assuming this proposal moves forward.
2. Manually migrate some proportion of Vega 2-based graphs to be compatible with Vega 5. See the "Vega 2 → Vega 5 transition" section below.
3. Potentially, fix/port graphs that attempted to fetch live data using methods that the sandboxing approach inhibits.
   1. Note: the need for the above will become clear once we decide on whether we will restore the pseudo-protocols that were used to fetch data live from the action API, the REST API, WDQS etc, and the precise sandbox parameters we select (domains/ports/http methods allowed). This decision will be made in T346291.

Vega 2 → Vega 5 Transition

Why do we think it's worthwhile to migrate from Vega 2 to Vega 5?

1. Vega 2 has been superseded by Vega 3, 4, then 5 upstream.  Upstream and third-party documentation exclusively refers to syntax in “Vega version 3.0 and later”, and it is difficult for new contributors to find documentation relevant to Vega 2.  The last upstream release (bugfix or security) of Vega 2.x was in January 2017.  Vega 5 was released in March 2019 and is still under active maintenance and development, with the latest 5.25.0 release in April 2023.
2. Volunteers have reported issues with Vega 2's accessibility, syntax, and overall functionality, per this 2023 wish.
3. Vega 5 has made improvements to the library's expression layer that harden it from a security perspective compared to Vega 2.  It is not perfect, but by introducing a parsed expression grammar it offers a more robust foundation for additional security hardening in the future if it proves necessary.
4. Maintaining multiple versions of Vega concurrently is unsustainable in the long run. The wiki community is taxed in the attempt to independently support software which is not being maintained upstream. Our efforts are best spent working in cooperation with upstream and third-party developers, and to do this we need to be working from the upstream Vega 5 code base.

What might be required to migrate graphs from Vega 2 to Vega 5?

1. Create a converter that would migrate Vega 2-based graphs to be compatible with Vega 5. @Jdlrobson started work on an initial approach in T335048#8794138.  The initial work needs to be restructured slightly to refocus it on being an aid to manual porting, instead of the automatic translator which was its original goal.
   Note: We estimate this converter currently works for ~80% of graphs, with diminishing returns on additional engineering effort to cover more.  We do not plan on continuing to invest significant additional engineering resources here, but instead to simply repurpose the existing codebase as an aid to manual porting.
2. Volunteers would need to update <graph> syntax on a case-by-case basis, aided by (1) the ability to run the existing Vega 2 and new Vega 5 specification side-by-side, (2) the partial Vega2-to-5 porting tool which handles 80% of the “obvious” keyword changes and other mechanical conversions, (3) the upstream Vega2 porting guide, and (4) additional documentation or tools which might be created by the wiki community.
3. Update the limited number of Scribunto templates on-wiki which generate <graph> output in Vega 2 format to instead output Vega 5.  This requires both lua and Vega expertise, but fixes a larger number of Vega 2 uses on wiki at once.

PPelberg (WMF) (talk) 23:52, 15 September 2023 (UTC)

The open question over pseudo-protocols sounds like a pretty big sticking point. I totally understand the reluctance, even just from an operational perspective this couples a bunch of not-really-production things to production in a pretty uncontrolled way, but that would also seriously alter the applicable usecases for the extension. Bawolff (talk) 00:51, 16 September 2023 (UTC)

...that would also seriously alter the applicable usecases for the extension.

@Bawolff: can you please say a bit more about the above? Are you most concerned about specific graphs no longer functioning were we to not restore the pseudo-protocols that were used to fetch data live from the action API, the REST API, WDQS etc.? Are you most concerned that the extension would be generally less useful? Something else? PPelberg (WMF) (talk) 23:44, 27 September 2023 (UTC)

i'm concerned its a pretty big decision to leave as a question mark Bawolff (talk) 08:38, 28 September 2023 (UTC)

I share this concern. I think that without access to data (which is really what the pseudo protocols are about), that the graphs themselves will loose a lot of their potential. Who’s writing json data arrays in wikitext or Lua modules ? —TheDJ (Not WMF) (talk • contribs) 21:33, 28 September 2023 (UTC)

"Affording people the ability to fetch live data is core to what makes the Graph Extension uniquely valuable and difficult to find a replacement for."

@Bawolff + @TheDJ: assuming it's accurate for me to understand what y'all are sharing as what I've written above, then: understood. Based on what we talked about internally today, offering pseudo-protocol support sounds doable in this new sandboxed approach.

The question that remains is what exactly that support looks like. I expect for us (staff + volunteers) to be in a place to shape this together in the next few weeks.

Thank you for raising this. You doing so held us accountable to prioritizing talking about this during today's meeting.

On our end, we're processing the feedback everyone here has been sharing and working towards sharing an updated proposal in the coming weeks. PPelberg (WMF) (talk) 21:48, 28 September 2023 (UTC)

Answering the question of how much time it takes for graphs to become vega 5 is a math equation. There is an missing variable in that equation of how the pseudo-protocols get implemented. In the current gerrit master, pageviews works, commons is mostly there - it connects and there was an fix for an fetching issue, but with Graph disabled I was never able to verify it.

Wikidata is not supported at all, there is next to no code for it in gerrit. Wikidata support is not needed, volunteers (users) can use lua wikibase access (https://doc.wikimedia.org/Wikibase/master/php/docs_topics_lua.html), but rewriting from the current Wikidata SQL to lua wikibase is going to take three times as long as converting from vega2 to vega5, without touching the protocol bit. Snævar (talk) 09:54, 16 September 2023 (UTC)

Wikidata sparql and wikidata lua aren't really equivalent. There may be some things that can be converted but i would be very surprised if that was everything. Bawolff (talk) 10:02, 16 September 2023 (UTC)

As a Wikidata editor, I completely disagree. Wikibase's Lua functions are extremely limited in functionality and do not come close to what SPARQL can do. We mainly use graphs to display statistics (e.g. how many items are instances of something). We can't even fetch a list of inverse links using Lua, we definitely can't use it to create graphs like that. - Nikki (talk) 11:57, 19 September 2023 (UTC)

True, the pages using the template would need to be changed to say what item should be used. For example, lets look at a page that uses Template:Airport-Statistics, instead of specifying the airport code of the airport the graph is supposed to be of, it would specify the item id. Snævar (talk) 00:28, 9 October 2023 (UTC)

I do not see any signs of you reading the comments of the commits of when the protocols where being considered to be added in the former round in April. It only helps you doing so, giving you the chance to make a counter-argument. Snævar (talk) 00:50, 9 October 2023 (UTC)

I agree with Nux. Support for Vega 2 should be dropped with automatic translation to Vega 5 as temporary solution. There need to be clear message that graph is automatically converted and needs attention. It will motivate templates maintainers and graph users that directly use Vega <graph> syntax to migrate to Vega 5 instead of using static image as replacement for graph in article.

I interact with extensions mostly with Template:Graph:Chart template and in my experience this was most common use on Wikipedia articles for charts. Update on underlying Module:Graph is almost done (see [1], here [2] and here [3]). Test cases of template [4] show how well automatic translation works.

I'm trying to update map part of module but without wikiraw:// protocol (T335325) enabled for beta it's not possible to debug it efficiently. Same thing for other templates and modules relying on data fetch with "url".

Pietrasagh (talk) 16:28, 17 September 2023 (UTC)

Nux raises a compelling point. I think people will be OK with immediately transitioning to Vega 5 so long as it'll work "forever", as opposed to a temporary solution that'll stop working at some arbitrary time. Enterprisey (talk) 20:16, 17 September 2023 (UTC)

I agree with Nux, making the process larger may be interesting from a security point of view, but graphs have been broken for months, so just migrating to Vega5 would make the process better for readers, as they don't have two separate time windows with potential failure. Theklan (talk) 13:50, 18 September 2023 (UTC)

It sounds to me as though automatic conversion may not quite work, and people may need to see V2 and V5 side by side to test conversion / understand what's not yet working. But as long as the messaging is "V2 graphs no longer work" and v2 is only available in some side-by-side testing mode, it can be clear that we are immediately in the V5 migration period. Sj (talk) 19:38, 20 September 2023 (UTC)

I don't know much about the details of this (I'm only here because I was trying to find out when graphs will work again). A converter sounds like a "nice to have" feature, but I hope the priority is in getting graphs working again at all - having graphs which work if we fix them manually would be an improvement over having no graphs. - Nikki (talk) 12:12, 19 September 2023 (UTC)

Depending on the distribution of the Vega 2 usage, would it make sense to offer a Wikimedia/Wikipedia "simplified" graph template which would act as a front-end to Vega 5 or whatever graph module is used in the future? Is what the "pseudo-protocols" was referring to?

My point being: if there are a large set of simple, static (not dynamically-generated) graphs (e.g. basic pie charts, one-variable bar charts, and one-variable linear/log line graphs), can these be automatically converted to a simplified chart template ? As someone who has worked on just one stacked bar chart, I found the Vega 2 quite difficult; I imagine a less-technical user would like something they could, say, dump from a spreadsheet to make a simple graph.~~~ Jason Olshefsky (talk) 12:32, 19 September 2023 (UTC)

Module:Graph has served as an front-end for a lot of Vega graphs. In it's simplest form, it can be told some positions on the x-axis, some positions on the y-axis and the type of graph (f.x. line graph) and it gives the result.

pseudo-protocols are not a front-end. They are essentially (in a oversimplified way) shortened versions of an domain. After an protocol we specify the location of what we want. For example, "tabular://" is a protocol, pointing to commons.wikimedia.org/wiki/Data: and after that we specify what data namespace page we want. The data from that page is then used as data-points in the graph, or less commonly, as text in the graph.

The most simple graphs are likely to be auto-converted by the V2 to V5 converter. I do not think there is an list over the most simple graphs. Snævar (talk) 00:07, 9 October 2023 (UTC)

Pseudo-protocol stuff is useful info. Are these now enabled on the beta.wmflabs site? and are there ones that can access commons images and the OSM maps? RobinLeicester (talk) 11:36, 9 October 2023 (UTC)

No, you need to use full urls like the conversion from wikiraw in https://en.wikipedia.beta.wmflabs.org/wiki/Template:Graph:PageViews .

Connection to Wikidata Queries are nonexistant.

Tabular I have not got to work, a delevoper gave me this link https://commons.wikimedia.org/w/api.php?action=jsondata&formatversion=2&title={{urlencode:{{{table}}}}}&format=json&origin=* but there is still some work to be done. Maybe beta does not have access to commons.wikimedia, only the beta version, IDK. Snævar (talk) 18:34, 10 October 2023 (UTC)

See also https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Graph/+/910613 Snævar (talk) 18:50, 10 October 2023 (UTC)

Thanks - with this info I have got commons images working in the beta. (hadn't thought to try the upload file name!) I have still no idea where to look for the maps that were available via mapsnapshot://, but then, I have not got any sort of handle on vega projections/transforms either, so still plenty to do. RobinLeicester (talk) 15:52, 11 October 2023 (UTC)

Why not just start with v5?

You ask for what time should the v2 be available. How about no time at all? It has been quite a long time when graphs were not working. Testing v2 and then doing migration seems like a waste of resources to me. As I understand using v2 and then v5 would mean that after a period of graphs working there would be again a period of graphs not working. This work-don't work cycle would not look good (would make Wikipedia look less reliable). So let's just make the new version work... Unless maybe both versions could work at the same time? Nux (talk) 12:58, 16 September 2023 (UTC)

Unless maybe both versions could work at the same time? Yes, at least that's what it implies to me.
Edit: missed auto-migration part. I support your proposal to just auto migrate everything and just use vega 5.
Edit 2: PPelberg (WMF) just thanked this reply so I think yes, both versions would work at the same time Aaron Liu (talk) 19:48, 18 September 2023 (UTC)

+1. If there's a compelling reason to do this in two steps rather than in one, I don't see it in what PPelberg posted above. Nardog (talk) 22:03, 18 September 2023 (UTC)

I could see both at the same time, with all V2 renders tagged as "needs conversion". Once most have been converted a scripted update could note on all [talk]pages using an unconverted graph that it's going to stop rendering again soon, then shift to no longer rendering V2, leaving the "needs conversion" tag. Sj (talk) 19:41, 20 September 2023 (UTC)

could note on all [talk]pages just do some hatnote in the template like a TfD hatnote, like the tag will output info that it's legacy
both at the same time is indeed the graphs team (or whatever it's called)'s plan, Nux's just proposing that we just auto-migrate all graphs without doing anything else when graphs are reenabled and template editors would port the other 20%, with the fact that vega2 doesn't work being an incentive to port to vega5. Aaron Liu (talk) 20:33, 20 September 2023 (UTC)

Makes sense. Sj (talk) 21:59, 20 September 2023‎ (UTC)

Add step for archiving unconvertible graphs as images?

Assuming some graphs won't be converted to V5, at some point it would be good to compile a list of graphs that need to be converted to images, and do the conversion [on a server that allows sandboxed v2 rendering]. I'd love to see that included in the timeline, so they don't slip through the cracks. Sj (talk) 21:58, 20 September 2023 (UTC)

Somewhere to re-create the V2 graph or map would be great. In-place V2 would have the benefit of a rapid return for broken items. eg the '#tag:graph' -> 'Graph:Street map with marks' -> 'OSM Location map' -> 'final map' chain of conversion could easily be a long process of re-thinking and re-working. If so, working maps during that period would be a benefit of itself. But any method of seeing the V2 output would be a huge help in checking if the new version is matching the original features. Similarly, editors working at hand-converting a graph they added years ago, would doubtless welcome some method of comparing the new with what it is supposed to look like. RobinLeicester (talk) 17:16, 26 September 2023 (UTC)

There is an Vega V2 to V5 converter in the graph extension, so simple V2 graphs are shown. There has been work from Pietrasagh on Module:Graph, which covers a lot of graphs. The best way to convert the graphs in my opinion is to use Vegas own Vega2 to Vega5 transition guide, use their own editor on their website - which tells you if and what errors there are. Any protocol functionality needs to be tested on WMF wikis - beta is the only way currently. As for enabling Vega2 two out of the five security issues have available code to hack the computer viewing the graph - so yeah, if that gets enabled, I am installing NoScript and blocking all graphs displayed on my own computer. Snævar (talk) 07:33, 30 September 2023 (UTC)

Achievement horizon

Hello, just a simple question : When should we expect it to work again, within a confidence interval? Will we be waiting for weeks/months ? Is it useful to indicate that would-be timeframe in the warning message ? If it is about months to go, then shouldn't we remove graph templates in wikis ? Bouzinac (talk) 09:23, 29 September 2023 (UTC)

Good question. We shouldn't, but we should instead modify the uncomplete and misleading current public error message, by adding the safest (longest) imaginable delay . Their is huge need for a (even too cautiously oversized) repair time. Valmontin (talk) 01:14, 2 October 2023 (UTC)

When should we expect it to work again, within a confidence interval? Will we be waiting for weeks/months ? Is it useful to indicate that would-be timeframe in the warning message ?

@Bouzinac: in short, I expect us to be able to share an estimate for when we all can plan for graphs to be working again before October is over.

The above is built on thinking:

1) Before next week is over, we'll invite y'all to review the roadmap/plan we're proposing we use to guide the work of reenabling the extension

2) Through "1)", we (staff + volunteers) will arrive at a roadmap/plan we are both satisfied and we (staff) can use that updated plan to create an estimate of the sort you're understandably seeking

I recognize that in answering as I have the question you asked remains largely unaddressed. Although, I hope this bit of certainty is of some use. PPelberg (WMF) (talk) 23:31, 3 October 2023 (UTC)

It would be immensely useful to update the warning message. In the short term, PPelberg, can that message point people here for more detail?

Personally I find months-long processes of getting clarity to be somewhat draining on collective effort, and much prefer pessimistic-but-always-accurate messaging of the form "We don't currently have a plan to fix and restore this extension. Discussions of how this might happen in the future are happening here. Discussions of how to export graphs to other formats are happening here." (An internal process that can commit to outcomes w/ large confidence intervals and update iteratively, is as good, but extended uncertainty that can't even make such commitments is not.) Sj (talk) 21:22, 9 October 2023 (UTC)

Before next week is over, we'll invite y'all to review the roadmap/plan we're proposing we use to guide the work of reenabling the extension.

Update: the roadmap I committed us to delivering before today was over is delayed. You can expect this next week.

In the meantime, I've updated Graph/Plans and created Extension:Graph/Plans/Research to create space for it.

Thank you for being patient with us, y'all. I recognize every delay adds to the frustration of being without functionality you are depending on to make decisions and deliver people who come to visit the wikis with the information they're seeking. PPelberg (WMF) (talk) 00:23, 14 October 2023 (UTC)

We are also going to need better documentation on the protocols (the old doc is at Extension:Graph/Guide#External data). Please keep the old doc for now, do not overwrite it, it is still useful for converting purposes.

I also thought, on the user side, that using the user tool Synchronizer would make the rollout time pretty quick. Basically, once an graph is ready, it could be rolled out pretty much immediately. Snævar (talk) 15:45, 14 October 2023 (UTC)

How to help migrate from Vega 2 to Vega 5

Concrete help please?

I came here as an expert programmer, hoping to help, but I very quickly got lost. Can anyone please explain how editors who don't (yet?) understand the difference between Vega 2 and 5 can help devs speed the repairs? Is there a reading list to get up to speed and a step-by-step concrete set of instructions for updating broken graphs? Sandizer (talk) 09:11, 3 October 2023 (UTC)

There is a guide at https://vega.github.io/vega/docs/porting-guide/ on what needs to be changed. On top of that, all of the protocols are have changed (wikidiff, wikiapi, wikifile, etc.), that part needs better documentation. Protocols should be changed to full urls. Also, use the vega editor at https://vega.github.io/editor/#/custom/vega , it will tell you what is wrong with your graph, except for protocol issues. Also, if you are working on Module:Graph or Template:Graph:Lines, then please do not start from scratch on those, some progress on those has allready been done at https://en.wikipedia.beta.wmflabs.org. --Snævar (talk) 15:05, 5 October 2023 (UTC)

I think the Vega guide is probably just showing the port from Vega2 to Vega3. There are several times I have added Vega5 things that where not in the migration guide.--Snævar (talk) 01:53, 21 October 2023 (UTC)

Made a new guide for Vega2 to 5 at Extension:Graph/Vega 2to5 list. Added a few things that are not in the official vega porting guide. Snævar (talk) 18:14, 22 October 2023 (UTC)

Other notes

Some links to other helpful writeups on how people are working on this:

• RobinLeicester's notes
• Jdlrobson on phab: phab:120319, phab:T335048#8794138

added by Sj (talk)

• https://en.wikipedia.beta.wmflabs.org/wiki/Template:Graph:PageViews for an example how to convert "wikiraw" links.
• phab:T335325 on how to avoid CORS issues with full urls

added by Snævar (talk)

Pseudo-protocols and CORS workaround doesn't work on beta.wmflabs -> Error: The host wikimedia.org is not in the list of trusted domains for the protocol https: Pietrasagh (talk) 08:47, 28 October 2023 (UTC)

Yeah, a lot of those full url hints are invalid now, since WMF is now going to make the protocol support work again. This is mainly a developer problem. Personally, I tend to add the missing data from the urls into the graphs and test them that way. If you are expecting something else than a notice, then a dev should answer. Snævar (talk) 10:14, 28 October 2023 (UTC)

Update: 20 October

Hi y'all – this update is meant to:

1. Address the questions that emerged in response to the 15 September update
2. Make clear the roadmap we're proposing to follow to safely and securely re-enable the Graph Extension

With the above in mind, there are two things we need your help reviewing:

1. Roadmap for re-enabling the Graph Extension
2. Responses to questions raised after 15 September update

We are specifically curious to know:

• What – if anything – about Roadmap and/or FAQ is unclear/could benefit from clarification?
• What – if anything – did you expect to see mentioned within the Roadmap or FAQ that you are not seeing?
• What – if anything – about the Roadmap do you think ought to be reconsidered?

Thank you all for being patient as we prioritized the conversations and research needed to arrive at this proposal. PPelberg (WMF) (talk) 22:16, 20 October 2023 (UTC)

Having done a considerable number of Vega 2->5, i would say it will be wasteful to create a complex conversion tool. Most of the conversion is very straightforward and can be done by hand. It is also possible to write a simple tool using Jupyter notebook or observablehq page with a simple JSON->JSON conversion. My recommendation -- roll out Vega5 asap, preferably implementing one addition simial to Elasticsearch's Kibana -- where data.url can be an object instead of just a string - and that object could point to other data sources without any URL escaping. This way authors can do url: {"type": "data", "title": "Ncei.noaa.gov/weather/New York City.tab"} - this would use data from commons:Data:Ncei.noaa.gov/weather/New York City.tab Yurik (talk) 00:52, 22 October 2023 (UTC)

Having done a considerable number of Vega 2->5, i would say it will be wasteful to create a complex conversion tool. Most of the conversion is very straightforward and can be done by hand. It is also possible to write a simple tool using Jupyter notebook or observablehq page with a simple JSON->JSON conversion.

Good spot. It sounds like we're aligned in wanting to avoid work that we expect to be marginally impactful.

With the above in mind, can you think of any questions we ought to consider asking at the end of Phase 1 – before work on the conversion tool is scheduled to start – that could help us verify the extent to which conversion is as a straightforward as you think it might be?

My recommendation -- roll out Vega5 asap…

Understood. Is there something you saw [or didn't see!] in the roadmap that is leading you to think Vega 5 implementation is happening later than you think it ought to?

I ask the above thinking we're aligned in rolling out Vega 5 ASAP. Although, you raising this point is causing me to want to double check :)

…preferably implementing one addition similar to Elasticsearch's Kibana -- where data.url can be an object instead of just a string - and that object could point to other data sources without any URL escaping. This way authors can do url: {"type": "data", "title": "Ncei.noaa.gov/weather/New York City.tab"} - this would use data from commons:Data:Ncei.noaa.gov/weather/New York City.tab

Regarding the technical implementation you described above, I wonder: @Cscott: how does what @Yurik shared relate to what you've had in mind?

All of this aside, thank you reviewing the proposal so promptly and all the work and thinking you've done/contributed to this space.

^ _ ^ PPelberg (WMF) (talk) 23:01, 26 October 2023 (UTC)

There are some moderately complex renames/relocations which our existing vega2->vega5 tool handles which seem to be useful, but 100% agreed that we don't plan to spend any more time on it other than make what we have already done a usable tool for assisting manual porting. That's in the roadmap, I think. We would *love* community help building those porting tools, and if folks want to get started by taking the code we wrote go right ahead!

I'm not 100% certain about your suggestion about data.url, it's something I think we can look at more closely as we implement protocol support for vega 5. If you wanted to open a phab task (or upstream ticket at vega) and write up your proposal fully that would probably be helpful. Protocol support definitely includes grabbing data from sister projects, and we will definitely be working w/ the community to develop a guide on best practices on how to do that, which would be a logical place to include advice like "if you use this object-style syntax for url you won't need to manually url escape the source". cscott (talk) 16:29, 27 October 2023 (UTC)

I am hugely encouraged to see the Roadmap. I am working towards resolving the OSM Location Map conversion, and have some working prototypes on the beta site. Unless I am missing something, Vega5 seems to not like Transforms of inline data declared as "values". I can find no useable output. My solution is to do the mercator projection stuff within a wikipedia template, and send x,y plots to a new replacement for 'Graph:Street map with marks'. That might still be usefully updated for use by wdqs and tabular file sources, but am I right that it is not useable for inline lat and lon transforms? RobinLeicester (talk) 20:27, 23 October 2023 (UTC)

Understood, @RobinLeicester! I've asked members of the engineering team whether there is guidance they can offer about the OSM Location Map conversion you're exploring.

Two things in the meantime…

1. Is this the page where we can see the work you've been doing on the OSM Location Map conversion?
2. We're glad to hear you're encouraged by the roadmap – thank you for sharing as much with us and for continuing to be a helpful voice as we find our way back to graphs being re-enabled ^ _ ^

PPelberg (WMF) (talk) 23:12, 26 October 2023 (UTC)

Yes, of the various subpages of https://en.wikipedia.beta.wmflabs.org/wiki/Special:PrefixIndex/Template:OSM_Location_map/ 'core' processes the parameter blocks, 'Getxy' turns the mercator coords into pixel xy, 'Labelitem' generates the Vega5 inline data items and 'Graphmap' is the replacement for 'Graph:Street map with marks' plus various new options from v5. It is now all working (as seen in /doc) except that the BetaCluster 'Template:if empty' seems to not work, and I have only stripped it out from the first few parameter blocks, so it onlys shows mark1 and mark2 items. RobinLeicester (talk) 23:05, 30 October 2023 (UTC)

I'm not enough of a vega expert to answer that, but it does seem like transforms in vega 5 are pretty powerful. That said, map projections are pretty esoteric. Maybe ask this question to vega upstream? cscott (talk) 16:30, 27 October 2023 (UTC)

Vega5 can defiantly use longitudes and latitudes. In line 63 and 64 in https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Template:Graph:Street_map_with_marks we use the name of the data, here "data", so "datum.coord[0]" is "data.coord[0]". There is an example on Vega, more complex than you need, at https://vega.github.io/vega/examples/airport-connections/ . It uses https://vega.github.io/vega/data/airports.csv which has latitudes and longitudes of USA airports. How did I get that file? Well, since Vega is run client-side your computer has to get the data-points anyway. I watch the networking tab in my browser and pull the hidden data files from there. Not sure why they do not just have a link to it.

Also take a look at https://vega.github.io/vega/docs/projections/ for your mercator projection. Snævar (talk) 17:20, 28 October 2023 (UTC)

Hmm, this should’ve been in tech news. Aaron Liu (talk) 14:41, 24 October 2023 (UTC)

I think adding mention of this in Tech/News is a great idea. Thank you for raising this, @Aaron Liu. Done with help from @Quiddity (WMF). PPelberg (WMF) (talk) 02:46, 26 October 2023 (UTC)

I don’t think we should backport Vega 2 support.
Do we have any timeframes for the roadmap? It’s been almost half a year Aaron Liu (talk) 15:33, 24 October 2023 (UTC)

I don’t think we should backport Vega 2 support.

@Aaron Liu, can you please say a bit more? What have you noticed/experienced/etc. that's causing you to question the value of backporting Vega 2 support?

Do we have any timeframes for the roadmap? It’s been almost half a year.

Good question. I'm thinking there will be a timeline we can share during the first or second week of November. By that time, it will have been 1-2 weeks since mention of the roadmap will have appeared in Tech/News and as a result, I anticipate we'll know whether people see any fundamental concerns with the roadmap. PPelberg (WMF) (talk) 23:06, 26 October 2023 (UTC)

Thanks. On Vega 2, I think that maintaining things with two versions at once would impact the timeline quite a bit and having Vega 2 templates break (or publicly display a warning incl. outside previews) would encourage editors to fix them Aaron Liu (talk) 00:23, 27 October 2023 (UTC)

This. Volunteers are very good at fixing things insofar as they are capable. So let them. Nardog (talk) 01:00, 27 October 2023 (UTC)

Does "first or second week of November" resolve to Nov. 3, 10, 17, or some other date? Tfmorris1 (talk) 16:38, 10 November 2023 (UTC)

@Tfmorris1, hi: you can expect to see an update sometime next week.

I appreciate you following up to ask...you doing so was the prompt I needed to revise the guidance I shared above. PPelberg (WMF) (talk) 17:42, 10 November 2023 (UTC)

@PPelberg (WMF): I feel there is no real willingness to address the problem. The proposed roadmap comes 6 months after graphs were disabled, and we still don't have a clear timeline for reactivating them. A roadmap without a clear timeline is pointless in this case. Even a partial reactivation of the graph template on the English wikipedia, which would be easy to implement by porting it to Vega 5, would be a huge step forward. This apparent unwillingness to invest and solve this is deeply frustrating for all the volunteers that invested huge amounts of time into developing and maintaining charts that are now being deleted and forgotten after 6 months of outage. Also, the communication regarding this incident has been a disgrace. There should be a centralized page with information about the background of what happened, the status of the work to reactivate the extension, and a clear plan of action. This information should be clearly linked wherever there is a message about disabled charts. Ita140188 (talk) 01:36, 25 October 2023 (UTC)

The Roadmap is missing an associated schedule. :) Without it, communication about the update will be mostly of interest to technical users. Strainu (talk) 16:44, 26 October 2023 (UTC)

I hear you, @Strainu.

I'm thinking there will be a timeline we can share during the first or second week of November. I shared some context about this timing above. Although, please let me know if what you see there brings up any new questions for you.

Without it, communication about the update will be mostly of interest to technical users.

This is actually by design. Before committing to a timeline, we've been wanting to ensure the volunteers who are experienced with developing Graphs think the roadmap is viable.

With the above said, I appreciate what I shared does little to address the question you arrived here seeking help answering. Even so, I'm glad you asked…you doing so reminds me and us of how this outage continues to impact people. PPelberg (WMF) (talk) 23:08, 26 October 2023 (UTC)

Well, the only concern I have with the roadmap is it seems to block the deployment to a decision (and implementation) on whether to support Vega2. Can't we do that in parallel to deploying Vega5 so we unblock people who are willing to change the existing graphs? Strainu (talk) 07:22, 27 October 2023 (UTC)

I'm in total agreement Strainu, Aaron Liu and Nardog on this.

In my opinion, bringing Vega 2 support back, even temporarily, is a bad idea principally because it reduces the incentive to port old graphs to the new syntax. If 80% of graphs can be auto-converted, then providing a tool to do that, combined with documentation (with examples) for editors who want to manually convert old graphs to the new format, that's the way forward; I think you'll be surprised how fast the change can be.

It also reduces the engineering effort required to put back support for a dead technology that still constitutes a potential security risk, even with mitigations in place; the sandbox is a very good idea, but it should be used in addition to transitioning to new, more secure software. The rationale for re-supporting Vega 2 is, alas, equivalent to a rationale for supporting it forever, and the situation will only get worse over time. Please bite the bullet, and go directly for Vega 5 and only Vega 5, which will not only get graphs working again faster, but will also avoid analysis paralysis and the many costs of supporting two versions at once. The Anome (talk) 07:31, 31 October 2023 (UTC)

It will take like 3 months to move vega2 graphs to vega5 across the whole WMF sphere, once the protocols have been re-implemented. Enwiki projects a month, small wikis (as in few articles on the wiki) are usually slower. Snævar (talk) 21:52, 6 November 2023 (UTC)

This feels like a satire of the excesses of consultative process. I am (literally) in this picture and I don't like it, so this will be my last comment here. I don't know whether any of the community members participating here feels good about this, or effectively heard/supported. I recognize everyone involved are trying to be as considerate and as effective as possible, but this is an antipattern.

• Why are we spending a thousand person-hours this way?
• Let's not spend a thousand person-hours this way.
• Please instead when something breaks, link to a description of the incident and what's being done about it from the point of breakage, decide on a quick fix, implement it in a way that fixes the most obvious problems right away, publish a draft plan of record for implementation and timelines for it, start in on that plan, link to rolling discussion of that implementation and timeline, regularly update it based on that discussion. If you need a quick sanity check while implementing a first pass, grab any single community dev/editor invested in the problem and talk things out with them, preferably on a wiki page; this does not need a multi-week consult and can happen in parallel to a longer-term feedback thread. Please don't hold off on fixes pending a consultation, please don't hold off on a consultation pending an internal private development of a plan to present for consultation, please don't hold a consultation about the next stage of consultation, and when there are consults at all please have them be a small % of total work done, and done in parallel with fixing the technical problem. (For instance, people on this page have been saying "please deploy vega5 to unblock people" for months, which could have been started while waiting for clarity on the rest.)
• (updating for clarity from feedback I received :) Thoughtful attention to and prompt responses to feedback are helpful. Particularly as complements to clear + confident plans/implementations. Consultation as a substitute for a plan is confusing. Consultation as an obstacle to implementation is harmful. Speed is a feature that is more valuable than all of those other things combined; it's in our name, and almost everything under discussion is reversible.

❤️‍🔥❤️‍🔥, Sj (talk) 15:25, 12 November 2023 (UTC)

hi @Sj. We share the sense that it's time to get moving and I can report that work is underway.

More broadly, I appreciate you inviting us to consider how we might accelerate the speed with which we can move from "issue" to "fix." As you alluded to, discussing this might be better suited for the time after Graphs are back working.

In the meantime, I think it will be important for our future selves to remember that this consultation was driven, in part, by a need for us (staff and volunteers) to align on a plan considering our efforts depend on one another. PPelberg (WMF) (talk) 02:40, 16 November 2023 (UTC)

My general view is that 6 months has been too long an outage for such a general feature. If re-enabling vega2 in a controlled way is the fastest way to get this back then go for it.

I suggest that some appropriate person such as PPelberg should undertake to present a report on this project at Wikimedia Hackathon 2024. Ideally the focus of this would be lessons-learnt and having a reporting deadline might help in providing a sense of urgency to get the work completed before another 6 months has slipped by. And, as such security and upgrade issues are common in IT work, the topic may stimulate useful thinking in other areas.

Andrew Davidson (talk) 15:46, 27 November 2023 (UTC)

Stats

I looked over some graphs and found out that once Module:Graph is complete, 40-50% of pages that use Vega graphs on the English and Spanish Wikipedias will have some graphs converted. I say some, because Module:Graph is not the only way Vega graphs are displayed on those pages. (Spanish Wikipedia and English Wikipedia have functionally the same Module:Graph code) I predict that the conversion to vega5, especially on English and Spanish Wikipedia, will happen quickly at first, and then progressively get slower. As graphs are converted in templates with fewer and fewer transclusions and then graphs used on individual pages, it is inevitable that the progress will get slower.

Also, it is falsely intimidating looking at the numbers in Category:Pages_with_disabled_graphs, with thousands of transclusions on the largest wikipedias, but the actual number of graphs in each project is in the hundreds.

A lot of analytics will need to be done before any number can be given of how long the Vega 2 to 5 conversion takes. Snævar (talk) 01:52, 21 October 2023 (UTC)

Templates are sorted into a group, all wikis in group1 (grp1) have the same version or an outdated one. Group other are all other wikis. enwiki = english wikipedia, enwikisource = english wikisource.

Global usage

Template	Direct transclusions	Most used	wikis in group
Graph stacked grp1	53	en.wp (18), eu.wp (17)	enwiki, bewiki, cswiki, dawiki, elwiki, euwiki, fiwiki, frwiki, itwiki, kywiki, mkwiki, nlwiki, ruwiki, skwiki, slwiki, srwiki, svwiki, trwiki, ttwiki
Graph stacked other	49	de.wp (13)
Graph lines grp1	4731	commons (4707)	commons, eowiki, eswiki, ptwiki, ruwiki, trwiki, zhwiki
Graph lines other	255	en.wp (52)
Graph PageViews	9759	N/A
Graph PageHistory	10	ar.wp (3), en.wp (4)	note: no groups found
Graph pie chart grp1	86	wikidata (85)	wikidata, astwiki
Graph pie chart grp2	1	arwiki (1)	note: only one wiki here

Graph Chart - too many to count (over 10000)

Suggested PageViews group (all the same code or outdated): frwiki, commonswiki, ruwikinews, cawiktionary, ttwiki, mywiki, orwiki, srwiki, suwiki, tlwiki, zh_yuewiki, bnwikisource, enwikisource, huwiki, jawiki, kkwiki, lvwiki, rowiki, wikidatawiki.--Snævar (talk) 19:07, 23 October 2023 (UTC)

On ruwiki, the vast majority of graphs were generated by ru:Module:Statistical, which is already updated to Vega 5 (although the graphs are currently disabled in the module so the transclusions are not in the tracking category for graphs). The significant portion of the rest is Graph PageViews, Module:Graph etc. Colt browning (talk) 15:50, 27 October 2023 (UTC)

Interesting. A lot of the modules sitelinked to the Russian Module:Statistical are in fact different modules. The only wiki to update from that Russian module is krcwiki and it is way behind (63 revisions), maybe some template parameters where removed and added since then. I appreciate the addition, I am mainly figuring out where wikis can collaborate and prevent multiple wikis from working on the same thing on their own. Snævar (talk) 11:19, 28 October 2023 (UTC)

Actually, the differences don't matter for our purposes. The update from ruwiki for the part which generates the graphs will Just Work in all wikis (assuming that the module worked in those wikis before the graphs became broken, which is not always the case). --Colt browning (talk) 16:47, 9 November 2023 (UTC)

Module:Graph counts, Only Wikipedias counted. Total transclusions 30712.

• Group1: total usage 6838, top users: mk (1746), es (1594), members: alswiki, arywiki, astwiki, aswiki, bawiki, bewiki, cswiki, eowiki, eswiki, fiwiki, frwiki, hewiki, hrwiki, hywiki, ltwiki, mkwiki, nowiki, skwiki, urwiki, uzwiki, vecwiki.
• Group2: total usage: 16304, top users: en (10219), ms (1813), members: arwiki, arzwiki, aswiki, azbwiki, azwiki, banwiki, bhwiki, bswiki, bxrwiki, cawiki, cebwiki, ckbwiki, dawiki, enwiki, elwiki, etwiki, euwiki, fawiki, fowiki, guwiki, hiwiki, idwiki, kkwiki, knwiki, kuwiki, lldwiki, lvwiki, mdfwiki, mlwiki, mrwiki, mswiki, mywiki, newiki, nnwiki, orwiki, papwiki, shnwiki, slwiki, sqwiki, srwiki, svwiki, tawiki, tcywiki, tlwiki, yowiki, zhwiki.
• Others total usage: 7570.--Snævar (talk) 12:06, 28 October 2023 (UTC)

Scope

I'm puzzled as to why at [5] some sort of graph (bar? column? line?) of "historical population" data is affected, but a pie chart of religious affiliations (using the {{Pie chart}} template) is unaffected. Care to explain? —DIV (1.129.104.79 13:24, 28 October 2023 (UTC))

w:Template:Pie chart makes a graph using ad-hoc HTML. w:Template:Historical populations eventually wraps w:Module:Graph, which uses the disabled Graph extension. There's no good reason other that history for this to be the case. * Pppery * _{it has begun} 14:34, 28 October 2023 (UTC)

OK. So are you implying that in an ideal world they would both be using the same mechanism: i.e. either both using ad hoc HTML, or both using a dedicated graphing module? —DIV (1.129.111.250 00:14, 30 October 2023 (UTC))

Probably. * Pppery * _{it has begun} 01:40, 30 October 2023 (UTC)

Request tracking with category, not linter flag

In Phase 5, I see Deprecate, and then remove support for Vega 2 from the codebase, creating categories or linter tags to mark any remaining Vega 2 graphs. I came here to request that a category, rather than Linter, be used for this tracking. Linter is for syntax errors, not the sorts of conditions that are tracked at Special:TrackingCategories. We already have Linter being misused for tables that are "too wide" (see task T334528), which has resulted in editors who work on actual Linter errors having to exclude that tracking flag from reports and advise people new to Linter-error-removal to disregard the "error". This "outdated graph format" or whatever the right name is matches very nicely with Special:TrackingCategories and not with Linter. Thanks. Jonesey95 (talk) 20:06, 6 November 2023 (UTC)

Update: 15 November

Hi y'all – a couple of updates:

1. Work is underway: work on implementing sandboxing for the Graph:Extension is happening. You can follow along with this work in T169027 and the patchsets linked within.
2. Support for Vega 2: rather than deferring the question of whether we will support Vega 2 to Phase 2B, we're making the decision to NOT support Vega 2.
   If/when we come to learn Vega 2 support is likely to accelerate/ease porting efforts to Vega 5, we will revisit this position. Thank you to @Aaron Liu, @Nardog, @Nux, @Pietrasagh, @The Anome, and @Theklan for helping us to see the value in this approach.
3. Roll Phase 2B into Phase 4: following on from deciding NOT to support Vega 2 at the outset, we’ll remove Phase 2b incorporate it into Phase 4 by expanding the objectives of the latter to including the following: "Develop the resources (documentation + tooling) necessary to support volunteer porting efforts."
4. Protocol support: to expedite the timeline, to start, we’ll deploy the Graph extension with support for the protocols T335325 already implemented. If/when we come to learn additional protocol support is needed to accelerate/ease porting efforts, we will consider expanding support.Note: we would value hearing why you think the protocols listed in T335325 may or may not be a useful starting point.

I've updated the roadmap to reflect these changes and added a progress tracker to make it easier to see where we are in the process.

Of course, please let us know if anything here brings new questions/thoughts to mind. PPelberg (WMF) (talk) 02:27, 16 November 2023 (UTC)

Thanks! Do we have a timeline with approximate dates? We were promised one that would arrive in the first weeks of November. Aaron Liu (talk) 02:57, 16 November 2023 (UTC)

@Aaron Liu: before this Wednesday (22 November) is over you can expect an approximate date for when we expect to have an initial implementation of the sandboxing approach deployed to the beta cluster.

In the meantime, I've added a "projected completion" column to the roadmap. PPelberg (WMF) (talk) 01:02, 21 November 2023 (UTC)

@Aaron Liu: I've added the projected completion date of Phase 1 to Extension:Graph/Plans#Progress and I'll be posting an update with a bit more context in a new section shortly so that more people are aware.

Note: I appreciate the desire to have time estimates for the latter stages of this project as well. Although in service of speed, we're trying to find the right balance between allocating time to resolving technical challenges and allocating time to estimating how long the former would take. PPelberg (WMF) (talk) 01:52, 23 November 2023 (UTC)

Thanks. That makes a lot more sense. Regular updates on progress are much appreciated. 2A00:23C8:728A:2501:3446:9548:3693:5E54 08:22, 16 November 2023 (UTC)

On protocols, there are 4 modules I am aware of that are going or have gone trough vega2to5 changes. Module:Statistical on ruwiki did not use protocols prior to the change and does not do so now. The other three Module:Graph, Template:Graph:Lines and "Template:OSM Location map" are all waiting on the protocols. Not sure how much clearer that can be. I would also like to withdraw my statement of doing this without protocol support, it was speculative and not meant as an final answer. I do appreciate not supporting Vega2. Snævar (talk) 17:57, 16 November 2023 (UTC)

I have to confess ignorance on whether the protocols are for user convenience or essential for safety. OSM Location map is working fine on the Beta site by using direct file addressing at the moment, but if that is not suitable for final release, it will be looking for access to maps.wikimedia.org/img/ and to commons images (currently accessed using filepath:). RobinLeicester (talk) 16:30, 17 November 2023 (UTC)

Thank you for the update! Re. protocol support: it would be great to eventually enable map:// and tabular://. We use them in ru:Template:SkyMap (8000+ transclusions). Colt browning (talk) 12:34, 18 November 2023 (UTC)

@Colt browning, @RobinLeicester, @Snævar: thank you for raising these considerations about protocol support.

We're going to defer any decisions about expanding the protocol support beyond what phab:T335325 has implemented already for now. We will revisit this decision in Phase 2.

In the meantime, I've added the comments you shared above to phab:T335325 to hold us accountable to considering them in the context of the decision our future selves will need to make about protocol support.

How does this sound to you? PPelberg (WMF) (talk) 01:47, 23 November 2023 (UTC)

Thank you, sounds reasonable. --Colt browning (talk) 12:01, 24 November 2023 (UTC)

Update: 22 November

Hi y'all – a shorter update this week:

1. Timeline: we anticipate Phase 1 to be completed by 22 December 2023. This timing is now reflected in Extension:Graph/Plans#Progress. As the details of latter phases becomes more clear, we will update the the page with their anticipated completion dates.^[i]
2. Performance: in the process of implementing iFrame sandboxing, we discovered that current approach has some performance issues. Namely, the approach blocks caching. The team is actively looking into solutions for this.

---

i. Note: I appreciate some may desire to have time estimates for the latter stages of this project as well. Although in service of speed, we're trying to find the right balance between allocating time to resolving technical challenges and allocating time to estimating how long the former would take. If at any point you find the absence of projected completion dates blocking you, please say as much. PPelberg (WMF) (talk) 02:02, 23 November 2023 (UTC)

Wow. That…seems like a long time but I do not develop for MediaWiki so idk. I really hope the other phases won’t take as long Aaron Liu (talk) 02:23, 23 November 2023 (UTC)

Welcome to the world of large organizations and large codebases. This is lightning-fast compared to some corporate environments. Alas, gone are the days when Magnus Manske could whip something like this up in an afternoon. The Anome (talk) 18:06, 23 November 2023 (UTC)

On my edit to phase 1: The current heading implies to me that all is needed to complete the phase is to test it, when in reality the implementation hasn’t finished yet. Aaron Liu (talk) 00:35, 24 November 2023 (UTC)

Rough estimates ok -- roughly what Q (or year?) will editors be able to update graphs on WP? [Phases 2 / 4]

As noted before, if this takes too long, graphs can be replaced w/ static images from a one-time render. Sj (talk) 19:55, 11 December 2023 (UTC)

If it goes that way, such one-time renders should ideally be served like current SVGs. Aaron Liu (talk) 20:28, 11 December 2023 (UTC)

Update: 22 December

Hello everyone -- I'm Marshall Miller, a director of product at WMF.  Peter is out on leave for a couple months, so I'm picking up this conversation in his absence.  In his last update, Peter said that Phase 1 of validating the iFrame sandboxing approach would be complete around now.  In working on that phase over the last couple weeks, engineering teams determined that the approach would cause performance degradation at a level that won’t be acceptable for the website (the performance issues were first mentioned here in the last update; and you can read the latest discussions amongst the engineers in this Phab task and the ones linked from it).  As part of this work, the team also identified that the approach will cause issues with graph content availability – e.g. with this approach, graphs still won’t be able to be displayed in our mobile apps.  We've been discussing and investigating how to get past and around those issues, but it is unfortunately starting to look like the iFrame sandboxing approach will continue to present technical problems and continue larger architectural issues regarding how the Graphs extension operates differently than the rest of our Mediawiki architecture.  In short, it may be an approach in which we solve one issue only to uncover the next unexpected one, without making it to the outcome we want.

In the couple weeks after the holidays, I'm going to return with more information, and whether we think we need to pick an alternative route.  I know this is not good news to hear given how long we have all been working in this conversation toward a solution that we thought was going to work.  And I know that graphs have been inoperable for much longer than anyone wanted.  As many of you have pointed out, this is a surprisingly thorny space in our architecture with the many security, performance, and product concerns it implicates. I can say that we have been discussing this at the leadership level, and we’re going to keep it up until we find the right path forward together.  Thank you for staying in the conversation -- I can't promise I'll be able to answer questions in the coming couple weeks as I'll be focused on family over the holidays, but I wanted to share what we knew as soon as we could. Feel free to reply, and let's continue discussing in the new year. MMiller (WMF) (talk) 20:49, 22 December 2023 (UTC)

Thank you for honest update.

Since there's no way for graph extension to work in reasonable timeframe (few weeks) we have to focus on other solution that will, at least partially, replace it's functionally.

Simplest solution would be to create tool/gadget that:

• easily create static graphs externally (eg. here) based on graph definition in Vega JSON and wikitext of frequently used templates;
• it would be great if pseudo protocols links could be automatically "translated" to standard urls so external tool could use snapshot of data to create static graph
• scrap svc file and upload to Commons chart image category;
• next step could be to use bots to replace existing graphs definition in articles with static images;
• maybe add link to graph to edit/update in external tool (like one mentioned above).

Other solution would be to delegate sufficient amount of WMF developers workforce to implement what was sketched out in the plan in few weeks (maybe except iFrame sandboxing). I was very happy to see high speed of extension development after vulnerabilities were discovered. After months of stagnation it is disappointing it turned out to be a flash in the pan. I understand complexity of required changes but I lost my hope to see this extension working again in foreseeable future. Feel free to disagree. Pietrasagh (talk) 15:30, 23 December 2023 (UTC)

Pietrasagh - Yea, let's create static graphs for everything. Bots to speed this up will be the most useful tooling. Having such a snapshot will be useful for the archives even if the extension comes back online in the future. Sj (talk) 22:13, 23 December 2023 (UTC)

So it took three months to figure out that iframes cause performance degradation and would not work on mobile? I thought the WMF would have considered these basic issues before proposing the solution in the first place Ita140188 (talk) 18:19, 23 December 2023 (UTC)

No, it took two weeks. It took 3 months for someone to be available to investigate and research the proposed solution. —TheDJ (Not WMF) (talk • contribs) 21:01, 23 December 2023 (UTC)

This is dissapointing news, after looking quite hopeful. I have detected quite a strong vibe of whatever the problem was with Graphoid as a static image solution, but it does feel like an 'on-page edit with static image for viewing' of some sort ought to be possible. It won't solve uses that need up-to-the-minute data and/or user-selected options, but would certainly get a lot of broken elements back again, if the alternative is wholesale abandonment. RobinLeicester (talk) 15:32, 24 December 2023 (UTC)

We can also use beta-cluster to generate static images directly from graphs definitions. From my perspective biggest problem to use beta is that "normal" wiki domains are not allowed for direct url [1] and pseudo-protocols are disabled (e.g. wikiraw) in extensions config on beta-cluster. Workaround with direct url's not always work from my cases. Maybe some pseudo-protocols were pre-processing something in the data on the fly - I don't know how to check it. It's reported on phabricator but nobody picked it up. If someone know where to push this issue please help. /edit: typo/

1. w/extensions/Graph/modules/ext.graph.render/domains.json

::{"GraphAllowedDomains": {
::        "http": [
::            "wmflabs.org"
::        ],
::        "https": [
::            "beta.wmflabs.org"
::        ],
::        "wikirawupload": [
::            "upload.beta.wmflabs.org",
::            "upload.wikimedia.beta.wmflabs.org",
::            "upload.wikimedia.org"
::        ],
::        "wikidatasparql": [
::            "wdqs-test.wmflabs.org",
::            "query.wikidata.org"
::        ],
::        "geoshape": [
::            "maps.wikimedia.org"
::        ]
::    },
::    "GraphAllowHttp": false
::}

Pietrasagh (talk) 07:11, 25 December 2023 (UTC)

The master branch of the Graph extension is in a bit of a halfway state right now (e.g. most custom protocols don't work). If someone is interested in static image generation, it would be pretty straightforward to set up a MediaWiki 1.40 wiki with the pre-disabling version of Graph on Cloud VPS. But depending on what happens with the extension, it might be wasted effort, and it doesn't seem like a particularly urgent problem, so IMO it would make more sense to wait it out. Tgr (WMF) (talk) 22:32, 26 December 2023 (UTC)

Will graphs come back before communities abandon using them completely? At the moment graphs are disabled for eight months. We have been told that they will be back soon™. Now, after that time, we've seen that many efforts to restore them failed. Upgrading Vega 2 to 5 is not sufficient. Limiting the ability to edit graph definition won't be implemented. Sandboxing turns out to be not performant. Eight months is so much time that one engineer could write a library for drawing static graphs from scratch with no overhead, legacy code nor security flaws leaking here and there. Is Vega really what we need and what can be supported by WMF in the long run? Msz2001 (talk) 20:05, 29 December 2023 (UTC)

Wikipedia is one of the most important websites. For far smaller and less important websites, providers talk about availabilities of 99.x%. If graphs do not work until 18 April 2024 we had 0% availability in 12 months.

Owners of websites spend a lot of money for a high availability. In a situation like this, a commercial oorganisation would take some of their money and send several teams out in parallel for different approaches to get any solution working. If a second team returns a bit later with a better solution, their solution would go life.

Please do the same here: WMF is not poor. Send several teams out to find solutions. Independent if volunteers or commercials. Wikipedia is too important to loose more time! Pakwesi (talk) 17:59, 21 January 2024 (UTC)

In an Greek study, https://lnu.diva-portal.org/smash/record.jsf?pid=diva2%3A1568457&dswid=-2876, readers were asked how much interactive features affected their experience on Wikipedia. At page 40, 34.4% thought that interactive graphics would add to their experience. It was the second highest percentage, after Augmented reality and way higher than Artificial Intelligence at 11.5%.

The graph extension is the closest we have to interactive graphics. If I have understood the programmers correctly, the issue is not really Vega itself. Most users cannot really answer questions about exactly how to solve the issue, the necessary technological knowledge is not there.

The users have already figured out how graphs are going to be moved from Vega2 to Vega5, now the programmers need to figure out how the extension is going to work. Snævar (talk) 20:45, 23 January 2024 (UTC)

This is an urgent problem and I am appalled to see @Tgr (WMF) advocating total inaction. Graphs are desperately needed to reenable a wide range of templates, not just obvious graphs. For example, Template:OSM_Location_map which appears on 5500 pages and is often essential to understanding their content has been on heavily reduced functionality ever since this problem arose. Furius (talk) 08:49, 13 February 2024 (UTC)

I very much look forward to the eventual repair of this issue and the repair of tools I use at the English Wikipedia. Fourthords (talk) 18:57, 3 March 2024 (UTC)

@MMiller (WMF): what news? Sj (talk) 22:46, 26 March 2024 (UTC)

It seems (based on comments to phabricator ticket) that WMF has silently abandoned this project. I think that local Wikipedias should focus on solutions with other techniques than graph extension, like CSS charts with a nice example of a new pie chart template/module mentioned in this discussion.

"Plans" article is now very missleading. It should be updated or deleted.

Based on current "progress" it may take a few years for WMF to figure out how to sanitize/sandbox client-side JS or SVGs.

It seems that WMF lacks honesty for volunteers to admit that all our work to make new graph extension work again was/is for nothing.

Further discusion here makes no sense and just waste of time. Pietrasagh (talk) 16:35, 27 March 2024 (UTC)

Hello @Sj and @Pietrasagh -- thanks for checking in, and I'm sorry that we have not been updating regularly. It's not our intention to silently abandon the project, and I appreciate all the time that volunteers have spent thinking about solutions. I've talked with several volunteers in the last few weeks about how they see the future for graphs and interactive content, which has been quite helpful. And I am working in a focused manner with engineers to propose a path forward. As we've all talked about here, this is a particularly thorny challenge and is taking time to have the right conversations. I know it's been a long time, though. I intend to bring more clarity to the WMF side of this effort in the coming couple weeks. MMiller (WMF) (talk) 17:51, 27 March 2024 (UTC)

Replacement for piecharts

Hi. I've created a replacement for piecharts. It doesn't require any JS and it should accessible (as in a11y) so maybe even better then Vega graphs/charts. I've published my module on English and Polish Wikipedia. Feel free to copy it to other wikis.

As a side note: If you want to translate examples then ChatGPT should be able to help... or at least it worked well for PL→EN translation (very few changes were needed).

• Module:Piechart (en)
• Module:Piechart (pl)

See also more info and examples on the Village pump (miscellaneous).

Cheers, Nux (talk) 17:12, 6 January 2024 (UTC)

Awesome work! I've used a different pie chart for now and yours looks way better. Is it possible to make it a donut chart as well? Rasputin 93 (talk) 17:20, 6 January 2024 (UTC)

Thanks. Not sure if I would be able to do donuts in a stable way with CSS (would be much easier with SVG, but that is sadly not possible on Mediawiki). Any example on where would you need a donut instead of pie? Nux (talk) 23:05, 6 January 2024 (UTC)

I currently use this one:

https://rammwiki.net/wiki/Module:Graph

https://rammwiki.net/wiki/Template:Graph:PieChart

I just prefer the donut look for such when using them on amount of songs played within a setlist, how many from each release.

Like here: https://rammwiki.net/wiki/05.08.2023_(concert)#Setlist

Also, is there a way to show only values without percentages? Rasputin 93 (talk) 04:06, 7 January 2024 (UTC)

The automatic scaling is great. A promising sign for piechart supremacy. Thanks Nux :) Sj (talk) 17:17, 24 January 2024 (UTC)

OSM Location map using CSS

I was inspired by @Nux' use of CSS graphics to see if that could be a solution for OSM Location map. (A 'rat deserting the sinking ship'?) The result is a non-vega version which is now at 'close to complete' stage at https://en.wikipedia.org/wiki/Template:OSM_Location_map/sandbox . It jumps through some very ungainly hoops, as it uses the Maplink overlay, but only seems to work if an en:overlay template also adds an invisible square. By re-using the mercator calculations developed to get vega5 working, it can add inline CSS graphics and text instructions on top of the map. (Betraying my ignorance, I had no idea CSS could be used like this). So far as I can tell, it appears to have a lower performace hit than Vega did.

There are a selection of examples at https://en.wikipedia.org/wiki/Template:OSM_Location_map/examples which also showcase some new features not possible with the old graph template. Any thoughts on the stability, performace, sustainability, portability and 'security safety' of this approach would be welcome. So far it only does 10 map-items. I am doing a few more compatibility/bug-find tests with existing map examples, and all being well will then ramp it up to the original 60 and go live in the next few days. RobinLeicester (talk) 12:13, 6 March 2024 (UTC)

Nice work, though certain things sometimes aren't centered and the top-right button has a weird blue link overlaid with it. In the case of the photo-panel demo, going fullscreen takes you to null island. Aaron Liu (talk) 12:42, 6 March 2024 (UTC)

Signpost article summarizing the developments of the last 11 months

I wrote an article for the Signpost trying to provide an overview of the discussions and attempted solutions for this issue since it occurred in April 2023, and to give a sense of where things might stand currently: w:Wikipedia:Wikipedia Signpost/2024-03-29/Technology report

Regards, HaeB (talk) 06:46, 31 March 2024 (UTC)