Extension talk:Chart/Project

How would you like to be notified about project updates?

I think I'm going to create a newsletter similar to Newsletter:Web team's projects which would send an Echo notification with a link to the updates page. It seems to be simpler, less spammy, global by default (since Echo is global), and thus better than mass message on user talk pages. So let me know if you strongly prefer to get mass message on your user talk page anyway. I need to pick one method, can't promise both. (Posting messages about major milestones on village pumps is a different topic, we will be posting those too.) Thanks! SGrabarczuk (WMF) (talk) 02:30, 3 July 2024 (UTC)

Echo notification is good to me. Thanks Pamputt (talk) 20:57, 7 July 2024 (UTC)

Data source

Requiring data to be in the Commons Data namespace is a quite effective way to ensure Charts will not be used. I don’t have figures, but I’m pretty sure many, if not most, uses of Graph don’t use the Data namespace, but rather inline data, template or module subpages, or Wikidata (via SPARQL or via the Wikibase Scribunto interface). If the only data source would be inline, all of these except for SPARQL could be used (including the Commons Data namespace, via modules using its own Scribunto interface); if the only data source is the Commons Data namespace, anything but Data namespace pages becomes unusable, and the project will be a failure. —Tacsipacsi (talk) 09:25, 6 July 2024 (UTC)

That's the same concern I have. If I had to create a graph with the data namespace on Commons as middle man, I'd simply not bother. That's too much effort. I'd rather see it take in-line data first, also considering I have never seen a graph using the data namespace "in the wild" before. DarkShadowTNT (talk) 21:48, 10 July 2024 (UTC)

@Tacsipacsi @DarkShadowTNT this feedback has been noted and we're in the process of analyzing how different data sources were used in Graphs. For the purposes of our upcoming prototype, we will pursue the Commons tabular data approach. While sourcing data from MediaWiki APIs or Wikidata Query Service is not the focus for this initial project, we want to ensure the extension is designed in a way that those sources can be supported in the future without requiring any Lua code. CCiufo-WMF (talk) 19:23, 26 July 2024 (UTC)

The problem is exactly that: not having a purpose. When the project fails, no one won't blame it on a poor scope, we will read how "editors are not using it". Theklan (talk) 21:04, 26 July 2024 (UTC)

I can only repeat myself: this predestines this project to fail. You can spend a lot of money and work hours on this project, but if you build something people don’t need, it’s just throwing out that money and time. No problem, I’ve mostly given up the hope anyway that we’ll have a usable graph implementation ever again. —Tacsipacsi (talk) 09:38, 27 July 2024 (UTC)

Re-reading your reply once again, I noticed that there may be a misunderstanding. I didn’t ask for the ability to get data from the APIs or Query Service directly. I asked for the ability to specify data inline, where the parser tag is placed. That inline data could come wherever the wiki editor wants. This would make the graphs at least not less powerful than the current CSS-based hacks. —Tacsipacsi (talk) 09:43, 27 July 2024 (UTC)

Inline is very relevant, indeed. But having data from the Wikidata Query Service is a must, because up-to-date data can be found there for a lot of things. Anyway, the team decided to make something sub-optimal from the start, so it can be justified on the low usage not improving it in the future. Theklan (talk) 10:23, 27 July 2024 (UTC)

Many (though not all) WDQS queries can be replaced by data queried via the Wikibase Lua interface – and data queried via the Wikibase Lua interface can be specified inline. Lua code may not be as expressive as SPARQL, but it gives you more freedom (e.g. you can add a summarizing table below the graph), and has built-in change propagation, which also means it’s easier on the servers (data isn’t re-queried from Wikidata on every page view) while it’s still guaranteed that if data does change, it propagates within a reasonable time. (WDQS is at its limits, so I’d avoid using it wherever a reasonable alternative exists.) And being able to specify data inline allows not only Wikidata data to be displayed, but also data stored in templates or actually inline in the article text. —Tacsipacsi (talk) 16:41, 28 July 2024 (UTC)

Thanks for the clarification @Tacsipacsi, I hear you on the flexibility that inlining data provides. We are trying to avoid depending on Lua and Templates this time around which is why I was suggesting a path to being able to use WDQS or APIs directly in the future instead. We're going to think about this a bit more and will keep you updated. Do you have specific examples of the types of charts you'd want to be able to create? That would be really helpful. I'd like to explore options that don't make you lose hope in the project! We're open to adjusting our approach as we learn more :) CCiufo-WMF (talk) 00:36, 7 August 2024 (UTC)

I also agree. It may seem harsh to say that a chart without data-points given in the transclusion would be a deal breaker, but that is just the reality of it. There should be an option to give data-points like {{#chart:format=1993 Canadian federal elections.chart|x=1,2,3|y=1,2,3}}. Having the data-points in a json is not simpler. I can see how a programmer would think, "I and my programmer friends know json, so that should be an easy way of doing it", but it is not. A layperson does not know much at all about json. Snævar (talk) 12:38, 3 August 2024 (UTC)

I agree the tabular data pages are not the easiest to work with right now. Making that experience better is something we're considering. Like I mentioned in the thread above, we're going to think about this a bit more and will keep you updated. It would be helpful for us if you could share some examples of the types of charts you'd like to create. Thank you! CCiufo-WMF (talk) 00:45, 7 August 2024 (UTC)

It should be pretty easy to see examples of use by just accessing the tracking category for the graph extension. A few of the charts I created over the years:

en:Nuclear_power#Fukushima_accident

en:Growth_of_photovoltaics#Solar_overcapacity_(2009–2013)

en:Growth_of_photovoltaics#Growth_by_year

en:High-speed_rail_in_Italy#History

Many more have since been deleted because of the broken extension Ita140188 (talk) 01:52, 30 August 2024 (UTC)

Not much to say other than +1. This decision pretty much ensures Chart extension will be useless in many circumstances it gets used, or harder to track vandalism in (since most Commons data pages are not actually protected, even high-use ones). If at least a way to create charts from Lua isn’t provided, this pretty much ensures that any usage of the new extension would be limited to a few cases where people might hack around the unnecessarily rigid extension requirements.
I also think it would’ve been much more courteous to announce these plans properly, so people who might be interested in Graph extension replacement would know that this extension was shot in the foot like this by its developers. stjn^[ru] 11:31, 16 October 2024 (UTC)

Just wanted to echo the concern with not having more ways to source data for graphs. I understand that convenience for developers the present setup affords by giving them total visibility into usage of the feature, and potentially allowing for easily mass-editing the graph data to accommodate breaking changes in the JSON grammar. But I feel that requirements of users need to be kept first.

Some of the priorities of this project seem rather ill-conceived. we want to ensure the extension is designed in a way that those sources can be supported in the future without requiring any Lua code. @CCiufo-WMF But why? Especially when that comes at the cost of making the extension highly inflexible for the short term? It may well be the case that you are planning to add more ways in the "future" – but WMF products have a long history of being developed for a limited time and then effectively abandoned, so that "future" may never come.

Being able to provide graph data inline opens up the ability to templatize them – so that multiple places which use the same graph data with minor variations can specify just those variations as parameters to a template. But for some reason, use of templates seems to have been considered undesirable without any reasons being provided. SD0001 (talk) 12:19, 30 November 2024 (UTC)

So my worries about being unable to use it on my wiki (to which I got no reply here) has been confirmed with this comment. I'll be unable to use it in the same way, by simply calling for data parameters that will fill in a chart according to it by the help of a template.

That would be beyond disappointing. Rasputin 93 (talk) 13:43, 30 November 2024 (UTC)

• Agreeing with all others in this thread. I didn't even realize the fact that this means data can't be templated before the above comment. If I'm understanding it right, just to create a simple bar graph, you will have to actually create two extra pages (the json chart definition and the data itself), which is a very convoluted process that none bar the most dedicated editors will go through, when previously it could all be edited from one page. The only advantage to this approach is it is usable on all languages/wikis, which is good, but not necessary in many cases. There should be the option for both. There also seems to be no updates about how, whether and when existing graphs, missing totally for 1.5 years now, can show anything and make users not have to resort to the source code or web archive. It seems like a herculean effort would be required to create a json and data page for every graph on English Wikipedia. This entire process seems misguided to me: I feel like it should have been, first, create something that requires little adjustment or bots to fit within the current system so that existing graphs can be restored, then begin creating this new system. I was previously hesistant to begin my own personal project to create SVGs for articles, especially high view articles, such as en:World War II or en:2020 United States presidential election, given the redundancy concerns, but I think I am now going to look into this, given I have no confidence this project will be completed any time in the near future for existing graphs. MarkiPoli (talk) 14:05, 30 November 2024 (UTC)

Dream usage: mapping plane destinations

A similar example of ideal Chart extension capabilities

It is not a top priority but you did ask "If you have ideas on what criteria we should consider"...

Every English Wikipedia article has an "Airlines and destinations" section packed with useful data: Example.

Will the new Chart extension be able to map this?

Vega can do something like it, see Mapping Airport Connections Tutorial. Commander Keane (talk) 23:57, 8 July 2024 (UTC)

Thanks for the suggestion @Commander Keane. While this specific example might not be one of the first chart types available, it should be possible within the architecture we are envisioning. CCiufo-WMF (talk) 19:10, 26 July 2024 (UTC)

Usage on private wiki

Hello, I've been following the evolution of this ever since I've added the Graph extension to my wiki only to discover it has been shut down few weeks prior and therefore rendered unusable. I'm now lost on whether the future Chart extension will be usable on a private wiki in a simple way. I've read here and there about data to be stored in Commons, but I'm unsure if this is compatible with the way I envision potential charts to work, since I have regularly updating data stemming from templates. Meaning I change some value in an article's infobox, which will therefore change another value on a different article through templates using the DynamicPageList3 extension and this would be reflected on a chart (or multiple across the wiki). It's a very dependent automation that updates my entire wiki by just changing one value on one page. Currently, I use the same workflow with some very basic pie chart replacement that some user here wrote, but it's just a temporary solution. I don't know if my use-case is too niché or not, hence my question here so I can finally rest my thoughts about this once I get an answer. Rasputin 93 (talk) 02:17, 30 August 2024 (UTC)

• Rasputin1493 how was it rendered unusable? Why can you not still choose to have it on your private wiki... Sj (talk) 16:25, 30 August 2024 (UTC)

  The Graph extension just won't display any graphs whatsoever. Rasputin 93 (talk) 16:29, 30 August 2024 (UTC)

  Maybe I don't know what you mean by 'private'. A private wiki on Wikimedia servers? Or on your own home server? Sj (talk) 16:37, 30 August 2024 (UTC)

  I manage these wikis, they are hosted elsewhere.

  https://rammwiki.net

  https://linkinpedia.com Rasputin 93 (talk) 16:38, 30 August 2024 (UTC)

We should have a complete-ish table of Graphs and Charts

Ita mentioned above that some of his charts made over the years have been deleted since the extension was shut down; that may be a common experience for people who were devoted and expert curators of the genre. I would find a complete list of past graphs/charts to be more useful than the anecdata resulting from surveys and discussions. and it would let us run an archival script to render and save snapshots of them.

This involves:

1. a query across the full wiki dump as of the day Graphs were shut down, for Graph usage specifically
2. a query across the current wiki dump, for any of a range of graphs and charts rendered in various ways (images w/ a caption describing the graph, easytimelines, pie charts, &c)

CCiufo-WMF: is that possible on your end, as part of the background research? Sj (talk) 16:37, 30 August 2024 (UTC)

I don't have this list readily available to share right now, but it is something we're going to look into again as part of preparing to support editors with migrating from existing graphs usage. From what I remember, getting the archival data (from the day graphs were turned off) was more of a challenge. I will get back to you on this, to inform your request below as well. Getting a current dump (or at least at the time we're ready to migrate) should be much easier. CCiufo-WMF (talk) 14:06, 24 September 2024 (UTC)

Archiving old Graphs

Since noone is monitoring the old Graphs pages, posting this here:

• I'd like to make snapshots of all Graphs from last April. Ideally with the output of a query of that database dump (above) to know how many to expect :)
• Tgr I think you mentioned that you had a test server set up that could render this? but that you didn't think we would need it if the extension was fixed soon. Since it's not being fixed, and the Chart: does not have a timeline for identifying and restoring large swaths of old graphs, let's just make the snapshots. Do you still have a server that could be used for this purpose? (If not, can we do this in the wmf cloud / does it need to be on our own server somewhere else?)

Cheers, Sj (talk) 16:41, 30 August 2024 (UTC)

I don't have one but AFAICS it should be simple to set one up on Wikimedia Cloud. Tgr (talk) 22:48, 30 August 2024 (UTC)

I know this was a while ago, but I would strongly agree with doing this. There seems to be no real timeline about when the old graphs will be available, so the snapshots should be made and for lets say all articles with more than X amount of views per day they should be bot replaced, with, obviously, the numerical data either kept and commented out, or deleted in the article and saved to a master server somewhere, ideally both, ready to be re-uploaded to commons as table files (if that's the only way it is able to be done in the future). This shouldn't be impossibly hard for simple charts such as bar, line and pie. Sure, if you're bot replacing, probably not all can be checked for being correct and not malformed, but even a malformed chart is better than a totally non existent one, as it is currently. MarkiPoli (talk) 14:16, 30 November 2024 (UTC)

Sidenote

(Here is how it is covered in the current project plan: Once it's possible to create charts that can replace graph uses, we will work with volunteers to start converting them so that readers can start to see them again... For graphs that cannot be converted, it may be more beneficial to either find an alternative tool to recreate the graph, convert the graph to a static image, or remove the graph altogether. This sort of open-ended timeline, with multiple possible options and uncertainty about what will even be possible, does not work well with distributed community planning. It is much better to take a simple, completable, uniform step, such as making static images, and divide & conquer that.

Labels on hover or tap-click

Graph label example

Seems like there are no labels for data (tooltips shown on hover and tap). Is this planned or should I file a request for that? Labels should contain something like x-y values for simple charts. Maybe configurable for other types of charts (see some examples of labels on my css piechart module).

I can see on beta that there a currently some click regions on the graph near data points, but they seem to do nothing. Not sure if the click-regions are a placeholder for future function or just something that came out of the library being used? Nux (talk) 12:32, 4 September 2024 (UTC)

The way we're rendering the charts right now on the server don't add the type of hover tooltip you're describing. The library we're using to render does support this though, among other interactive features we're hoping can be added in later. Feel free to file a task -- at least investigating how we can add support for this is something we plan to do. CCiufo-WMF (talk) 14:10, 24 September 2024 (UTC)

@CCiufo-WMF: Hi. The addition of tooltips is necessary for the readability of graphs, especially line graphs, particularly when there are several lines. And especially when there is a lot of data. A graph should not just give an idea of how the data are changing, but should also provide direct access to the data. I see that the task could be planned, but is there a deadline?Roland45 (talk) 14:40, 26 October 2024 (UTC)

@Roland45 We are working on adding interactive features like hover tooltips right now, and I hope it will be available on the beta cluster soon. I can ping back here once it is. CCiufo-WMF (talk) 15:21, 1 November 2024 (UTC)

Will .chart edition be limited?

I have seen that the translations are hard-coded inside the .chart at (Beta) Commons. I don't think this is the best approach, as it will have lot of redundant translations, but, anyway... will any user have the ability to edit any given .chart page or will it be limited to certain kind of users? If the later is true, is there a workaround for translations? Theklan (talk) 16:42, 4 September 2024 (UTC)

We're planning to keep charts editing open to everyone, especially to support the translation edits you're describing. We're open to suggestions on how we can improve the editing workflow for multilingual support. I agree the hardcoded translation in the chart definitions could get a bit messy, but it's the best option available to us right now to get things fully working. CCiufo-WMF (talk) 14:17, 24 September 2024 (UTC)

Stacked lines

For this chart i would like to use stacked lines. It looks like it could be possible.

Ainali (talk) 08:22, 11 October 2024 (UTC)

Hey @Ainali, yes this is possible, I've updated your example to use the "area" chart type, which is stacked by default. Let me know what you think! CCiufo-WMF (talk) 20:43, 15 October 2024 (UTC)

That is excellent, thanks! Ainali (talk) 09:41, 17 October 2024 (UTC)

Can't find any September update for this project

@SGrabarczuk (WMF): Is there any later update for this project than the one from August 30, 2024, i.e. almost seven weeks ago?

If so, where can I find it? Isn't Extension:Chart/Project/Updates the correct place to look for updates?

If not, when is the next update/newsletter planned to be available?

Where is the current time plan for this project, stating what and when for the planned project deliverables, published? Larske (talk) 10:33, 15 October 2024 (UTC)

Hey@Larske, these are all excellent questions! We are working on a project update. Normally I'd say it'd be published this week. Given the circumstances (Monday was a holiday for many of us, and I'm taking Friday off) I'm not 100% sure, but I believe this week is doable. There will be a new section on the /Updates page, I will send a notification via the newsletter, and the update will also be mentioned in Tech News. SGrabarczuk (WMF) (talk) 14:25, 15 October 2024 (UTC)

Possibility to have timeline support

The old EasyTimeline have several problems including not rendered in SVG, require font set up for non-Latin script. It would be good to have the functionality in Chart. -- 122.100.88.150 06:47, 26 October 2024 (UTC)

Hey there, replacing EasyTimeline isn't in scope for this project, but it may be a potential future enhancement (see phab:T137291). CCiufo-WMF (talk) 15:15, 1 November 2024 (UTC)

Letting you know :)

"Let us know if you'd like your wiki to be one of the first to receive the new extension." I would like it enabled on Swedish Wikipedia, please. Ainali (talk) 09:08, 26 October 2024 (UTC)

@Ainali Thanks for letting us know! In the meantime feel free to continue testing in the beta cluster, and then on test wiki once we've deployed there. CCiufo-WMF (talk) 15:16, 1 November 2024 (UTC)

Fr-Wikipedia application

I can confirm what was said above, namely that accessing json is not as easy as all that, especially when there is a huge amount of data to transpose. You need suitable tools, which not everyone has. However, this is the route I had envisaged in 2020 to internationalise country demography data, by creating tables in Commons such as this one. A module written in lua would then have processed the data. But the project failed. In fact, the main advantage of having the tables in Commons is precisely the possibility for any wikipedia to use the data.

That's why I'm applying to have the tool developed on the FR-Wiki as a test.Roland45 (talk) 14:53, 26 October 2024 (UTC)

@Roland45 thank you for the historical context, and I'm glad to hear you'd like to see Charts on FR-wiki. Being able to easily reuse Charts in any wiki is exactly why we chose Commons as the central store. And yes I agree, there can be more done to make it easier to work with the Data namespace. CCiufo-WMF (talk) 15:19, 1 November 2024 (UTC)

it.wiki

I would like to point out that Italian Wikipedia is one of the projects in the forefront with testing features like Charts, being one of the group 1 wikis regarding MediaWiki releases. we will be very interested to get the feature (even in beta) on our project. Valepert (talk) 11:14, 3 November 2024 (UTC)

@Valepert good to know, thanks! CCiufo-WMF (talk) 17:15, 7 November 2024 (UTC)

wuu.wiki

It will be nice if wuu.wiki is involved in first sites to receive the new extension. Lt2818 (talk) 14:27, 4 November 2024 (UTC)

Thanks for letting us know @Lt2818 CCiufo-WMF (talk) 17:15, 7 November 2024 (UTC)

Rationale behind Data: on commons

I'm excited for charts being back on pilot wikis! I get that it's kind of late now, but I wonder why Commons was chosen as the source for data instead of Wikidata, the existing wiki for collating data? Aaron Liu (talk) 20:14, 29 November 2024 (UTC)

It's a fair question why we don't turn on the Data: namespace on Wikidata. But for transclusions that look like media, we currently store all of the associated structured data on Commons. We definitely have two major different use cases for tabular data: structured metadata and chart data. Both have been stored on Commons since SDC, so changing that would be a more abstract discussion not specific to this extension :) Sj (talk) 15:54, 30 November 2024 (UTC)

Meta

Hoping to see this on Meta-wiki as well. Sj (talk) 15:56, 30 November 2024 (UTC)