Map improvements 2018

This page is about the work of the Collaboration team in early 2018. For ongoing map support see Wikimedia Maps

During the first half of 2018, the WMF Collaboration team made improvements to Kartographer and associated map functions. The team’s engagement with maps was prompted in part by the overwhelming support the maps community gave the 2017 Community Wishlist proposal Kartographer Improvements, though the work was not restricted to items named there. The project concluded as planned at the end of June 2018.

Map improvements 2018 Group: Collaboration Start: 2018-02-01 End: 2018-06-30 Team members: Collaboration team: Stephane Bisson, Pau Giner, Kosta Harlan, Moriel Schottlender, Elena Tonkovidova Reading Infrastructure and other Maps colleagues: Chris Koerner, Corey Floyd, Michael Holloway, Guillaume Lederreay, Paul Norman, Deb Tankersley Management: Ryan Kaldari (engineering manager), Roan Kattouw (lead engineer), Joe Matazzoni (product manager) ← Wikimedia Maps/2015-2017 Wikimedia Maps →

Goals

The immediate goals of the Map Improvements project are:

• To ensure that Kartographer and the associated maps technology stack are stable and can be easily maintained as maps gain a wider audience.
• To accomplish the two “main wishes” named in the Community Wishlist proposal, along with as many of the other wishes as are possible in the time provided. The main wishes are T112948, “All map location names should be shown in the user's language” (an extremely challenging but crucial job), and T180907, “Add zoom level 19.”

Goals that relate to extending maps to a wider audience—such as bringing mapframe functionality to English Wikipedia (as requested by RfC) and making sure that mapframe works properly on wikis that use Flagged Revisions (T151665)—are also under serious consideration.

Background and approach

Maps are a valuable form of visual data that can improve reader understanding of many topics in our encyclopedias and other free-knowledge projects. Yet currently, only 14 Wikipedias, of almost 300, have the mapframe function that puts an actual map—as opposed to just a link to a map—on article pages. Meanwhile, basic improvements are necessary to bring the Kartographer technology and user experience up to software-development and general wiki best practices—and to make the system easier to maintain. On the user-experience front, for example, Kartographer currently displays maps only in the language of the territory mapped, rather than in the language of the wiki where the map exists.

While advanced interactive and annotation features for maps can be useful, especially on Wikivoyage, it’s the judgement of our engineering and design teams that supporting such features—and making them technically stable and usable by a broad audience—would require significant investment in technical staffing extending over many years to come. The Map Improvements 2018 project will, therefore, not emphasize these types of features, prioritizing instead fixes to the system that make it easier to maintain, especially at scale, while improving the user experience for the broader and more basic use case one might call “locator” maps—maps which show the position of events or entities, along with basic geographic features. (Just to be clear, advanced features that exist now will continue; they just won't be the focus of our upgrades.)

The decision not to build out interactive and advanced annotation features at this time may disappoint some who voted for the Kartographer Improvements Wishlist proposal. To put that decision in perspective, however, it might be worth noting that the investment represented by the Map Improvements 2018 project, which will absorb the full energies of the Collaboration team over five-plus months, is already much larger than the scope usually allowed for Wishlist tasks.

Feedback

We want to hear from anyone with an interest in maps. If you have thoughts about any of the tasks on our board, please leave them in the task discussions; or use the Maps Improvements 2018 talk page for comments and questions about the project in general.

If there’s a Phabricator task you’re interested in—and if you feel the task is consistent with the Map Improvement 2018 goals described above—then give it this Phabricator tag: “Collaboration-Feature-Rollouts (Collaboration-Maps).”  That will bring it to the team’s attention, and we’ll consider whether it can be prioritized for action as part of this project.

Project Updates

To see a list of the tasks associated with this project, go to the Collaboration team Phabricator board and sort for the tag “Collaboration-Feature-Rollouts (Collaboration-Maps).”

July 11, 2018: Map Improvements project has concluded, Wikimedia Maps page is new location for maps talk

The Map Improvements 2018 was always meant to have a hard stop at the end of June, 2018.  Now, having accomplished the project’s main goals, the Collaboration team has moved on to other work (and taken a new name, Growth). This project has concluded and this project page will no longer be updated or monitored.

Members of the Reading Engineering team will handle map bugs and maintenance going forward. To read about and discuss their activities, please see the Wikimedia Maps project page.

May 11, 2018: Should your wiki change its map fallback language?

As detailed in previous posts, how well a given map displays in a given language will vary, because our maps get their data from the open-source mapping project OpenStreetMap. So which names are available in what language depends on what the volunteers have entered. The chart at right gives you an idea of the relative name data depth of various languages in OSM.

To deal with this issue, the map internationalization system falls back as gracefully as it can: first to languages configured as fallbacks in MediaWiki for the requested language, then to any transliterated names in the same script, and finally to the language of the territory mapped.

This system will work better in some situations than others. There are various things you can do to make specific maps more readable in your language: You can add more names to OSM in your language. You can use lang="local" to specify the language for a particular map (as described below). Or you might want to change your fallback languages.

A map of China displayed with the languages set to Catalan (at left) versus English demonstrates the potential value of making a more widely translated language part of a language's fallback scheme.

Why would you want a different fallback?

The image at right compares a map of China as it appears with the language set to Catalan (left) versus English. The Catalan version shows only three major cities in Latin script; the rest of the map has fallen back to Chinese. The English map is nearly all in English.

The first reason for this discrepancy is clear from looking at the table above: While OSM includes almost 3 million names in English, in Catalan it offers fewer than 80,000. The second reason, though, has to do with fallbacks: As of now, Catalan’s only fallback language is Oc, a language with even fewer names in OSM than Catalan.

Language choices can be personal and political. So I would never recommend what languages Catalan or any other wikis should include as fallbacks. But, were Catalan, just as an example, to add a second language after Oc that a) uses the Latin script and b) has a large collection of name data, then Catalan readers would see more readable maps for many foreign places.  

Now, it’s possible that adding a popular language like English to your fallbacks will solve some problems while creating others—especially for maps of your local territory in your local language. It was for such situations that we invented the parameter lang="local" (read all about the uses of this new parameter).

How to change your fallback languages

We added some information to the Kartographer Help page about changing your fallback language. As it says there, you’ll want to demonstrate consensus. Before you consult your community, though, it’s important to understand two things about the fallback language lists.

1. The general language fallback list was create for a different purpose than maps. These general fallbacks govern messaging on the wiki: if a user-interface message isn’t available in language a, it displays in language b, and so on.
2. We based the maps fallback list on the general one, but it is a separate list. That means you can change your map fallback languages without changing your general wiki language fallbacks.

The fact that the you can change your map fallbacks without changing your general fallbacks should make discussions around this simpler. The help page above includes a link to a partially-populated Phabricator form designed to help get your request to the right team once you’ve consulted with your community.

May 9, 2018: Map internationalization launched everywhere—and embedded maps now live on 276 Wikipedias

As of today, interactive (Kartographer) maps no longer display in the language of the territory mapped; instead, you’ll read them in the content language of the wiki where they appear—or in the language their authors specify (subject to availability of multilingual data). In addition, mapframe, the feature that automatically embeds dynamic maps right on a wiki page, is now live on most Wikipedias that lacked the feature. (Not included in the mapframe launch are nine Wikipedias that use the stricter version of Flagged Revisions).

If you you’re new to mapframe, this Kartographer help page shows how to get started putting dynamic maps on your pages.  If you’d like to read more about map internationalization: this Special Update explains the feature and its limitations; this post and this one describe the uses of the new parameter, lang="xx", which lets you specify a map’s language. And here are some example maps, to illustrate the new capabilities.

These features could not have been created without the generous programming contributions and advice of map-loving volunteers, including Yurik, framawiki, Naveenpf, TheDJ, Milu92, Atsirlin, Evad37, Pigsonthewing, Mike Peel, eranroz, Gareth and Abbe98. My apologies to anyone I’ve missed.

The Collaboration team's Map Improvements 2018 project wraps up at the end of June, so please give internationalized maps and mapframe a try soon and give us your feedback on the project talk page. We’re listening.

April 30, 2018: Mapframe maps now work on English Wikipedia!

You can now use the mapframe function to embed maps on pages in English Wikipedia. The map at right (which shows the location of this year’s Wikimania conference) demonstrates the functionality.  Mapframe was requested by the English community via RfC last year and the request was reconfirmed this spring.   If you you’re new to mapframe, this Kartographer help page shows how to use it.

To be crystal clear: mapframe maps don’t yet have internationalization (read all about this coming capability below).  Until we're able to release these long-awaited features—some time in May, 2018, if all goes well—maps will continue to display in the language of the territory mapped. You can, however, experiment with internationalized maps on testwiki now—here’s a page of examples.

Please try mapframe out on English Wikipedia and let us know what you think on the project talk page. We’re listening.

April 26, 2018: OSM name data quirks and the uses of lang="local"

A screen from OpenStreetMap shows data for the city of Los Angeles. You can see a local "name," with no language code, as well as several multilingual names that identify the languages using two-letter codes—ar (Arabic),  ba (Bashkir), bn (Bengali), etc.

In our last post we introduced the new parameter and value, lang="local", which causes a map to display in the language of the territory mapped—essentially turning off internationalization and reverting to the previous behavior, which displays maps in the local language(s).  Today we’ll dive a bit into why we created this new capability and when you might want to use it. First the “why.”

A dive into OSM name data

Objects, or “features,” in OpenStreetMap (OSM) can have a local “name,” for which no language code is specified, and “multilingual names,” which include a short code that tells the software what language a given multilingual name is in. Click to expand the screenshot from OSM on the right to see an example for the city of Los Angeles.

By default our internationalization software tries to display names in the language of the wiki, searching out names that have the desired language code. For each label, If it doesn’t find the language it seeks, it falls back as gracefully as possible: first to languages configured as fallbacks in MediaWiki for that language, then to any transliterated names in the same script, and finally to the language of the territory mapped (for a more technical description of the fallback scheme, see T192701).   In our example, then, when an English-language wiki hunts for a name for Los Angeles in English, labeled with “en”, no such multilingual name exists. But it’s not an issue in this case, since no fallback language is defined for English. So the software falls back to the local name, “Los Angeles.” Problem solved.

The lang="local" option

But what happens when a fallback language (or a transliterated name) is available?  The Chechen language, for example, falls back to Russian. So when displaying a map of Chechnya in Chechen, it’s possible that some names that are actually available in Chechen might display in Russian—in cases where no duplicate, multilingual Chechen names (with the code “ce” attached) exist. This is the time to try lang="local". If you create a map that should show names in the local language but you see names of prominent features appearing in another language, try lang="local". Doing so will give precedence to the local names. (Alternatively, you might wish to go to OSM and add multilingual names, with the language code, for those features.)

Maps of Bellgium with a specified language (English, at left) versus with lang="local". The localized version respects the fact that some Belgian names are actually entered into OSM's local "name" field in multiple languages.

Please note, by the way, that I couldn’t find a Chechen example to illustrate this in OSM, because it looks like OSM volunteers who work on Chechnya make a point of entering both a local name and a multilingual name for Chechen features—which is a good practice. Unfortunately, not all languages are consistent about providing such duplicate data.

Another situation where this issue might pop up is with multilingual countries like Belgium, where some local names are entered into OSM in multiple official languages, like so: België - Belgique - Belgien. So, if a person creating a map of Belgium wanted, for some reason, to preserve the local practice of displaying all official languages,  she could use lang="local" (see the illustration).

Please give us your feedback

Are you seeing the problem described above as you explore internationalized maps? Are you seeing different problems? Does the lang="local" solution help? Or would you be more likely, personally, to go to OSM and add multilingual names? Please try the new features out on testwiki or testwiki2 and leave feedback on the Map Improvements 2018 talk page. We’re listening!

April 25, 2018, You can now try out internationalization (on testwiki)

You can now show maps in languages of your choice on our testwikis. I made these two pages to demonstrate the new features: map examples testwiki2 and  map examples testwiki. By default, internationalized maps display in the language of the wiki (which is English for the testwikis). So to experiment with this, you’ll want to use the two new mapframe parameters we’ve added (more on how to use these below):

• lang="xx"  Shows map labels in the language you specify.
• lang="local" Shows map labels in the languages of the territory mapped (essentially opting out of internationalization).

Right now, internationalization works only with mapframe, not maplink (which should be working some time next week). If you don’t have experience putting mapframe maps on a wiki page, this Kartographer help page will get you started with some examples.

Project status and plans

Using the parameters above in your mapframe code, you can try out internationalized maps on the test wikis. The two test wikis,  test and test2 have slightly different configurations: On test, the maps embedded on a page are static graphics (clicking on the graphic launches a dynamic version); on test2, embedded maps are dynamic and can be scrolled and zoomed in place.  

Our plan is to wait a week or two and assess user comments about the feature. At that point, we’ll decide whether to move forward with a general release or keep making fixes.

How to use the new parameter

To use the new language parameter, just insert it into your mapframe code like so (the example sets the map of Europe to Chinese, or “zh”):

<mapframe latitude="48.45" longitude="9.58" text="Europe, in Chinese" zoom="4" width="500" height="400" align="left" lang="zh" />

• lang="xx" Use this whenever you want to override the language of the wiki on which you’re publishing your map. Specify the language using the short (usually two- or three-letter)  language codes used for each wiki.
• lang="local" Adding the “local” value will cause your map to display without internationalization—i.e., in the language(s) of the territory mapped. This command can help you deal with circumstances that may come up because of the way OpenStreetMap data is coded. We’ll post an update soon that goes into more detail about when and why you might use lang="local".  

Please give us your feedback

Internationalization is a big change for Kartographer maps ; we know there will be uses for—and consequences of—this change we haven’t considered.  We'd like to ask you, the maps community, to help bring some of these to light. How will you use these new capabilities? What works well now? What, doesn’t? What could be improved?  Please try the new features out on testwiki or testwiki2. You can just make an article page and start mapping; or you may prefer to work in your user space. In either case, please then leave feedback on the Map Improvements 2018 talk page. We’re listening!

April 18, 2018, Special Update on Map Internationalization

A map of Asia before internationalization, with labels in a jumble of languages and alphabets.

One of the “main wishes” from the winning 2017 Community Wishlist proposal  Kartographer Improvements was that maps should be able to display label data in the wiki's language instead of only in the language of the territory mapped, as currently (T112948).  We’re on schedule to  see the first real results for this challenging goal very soon, so it seems like a good time to check in with our map users on the project.  We want to hear from you about how well this new feature will work for you and what we can do (with the resources we have) to make it work better for everyone.

Project status and plans

A map of Asia as it will show on English-language wikis after internationalization (click to enlarge).

We’d wanted to release mapframe with internationalization to wikis that have mapframe enabled by the end of April, 2018. But as we’re increasingly able to see the feature in action—and to see how complex the font, language and user experience issues can be—we’ve decided to release internationalization first  on testwiki, to give us time to clean up any issues that users are likely to discover.

The testwiki release is now scheduled for the week of April 23–27. The date for a full production release will depend on the type of feedback we get from testers.

How the feature works

Currently, mapframe maps display labels in the language of the territory mapped. The first image at right shows how the existing software displays a map of Asia.  As you can see, a wide variety of different languages and alphabets display simultaneously (click on the image to get the full effect). The second image shows how that same map will display on an English-language wikis after internationalization.

• Where do the map labels come from? The Wikimedia mapping service gets its map data from the open-source mapping project OpenStreetMap. Those labels are created by OpenStreetMap volunteers. Which means that what labels are available for which languages varies from location to location and language to language, depending on what the volunteers have entered (more about this under Known Limitations, below.).

Adding labels in a new language to OpenStreetMap. Here, the user enters an English name for a Greek village.

• How are multilingual labels displayed? Once internationalization is implemented, each wiki will try to display map labels in the language of that wiki. For each label, if the desired language isn’t available, the system falls back gracefully:  first to the languages configured as fallbacks in MediaWiki for that language, then to transliterated labels if available (e.g., for a map of Japan in English, Japanese names spelled out in the latin alphabet), and finally to the language of the territory mapped.
• What can you do if labels are missing? If a particular label isn’t available in the language you need, you can add the label in OpenStreetMap. See the screenshot, which shows the interface for the process: you find the map feature you want to label, log in, enter Edit mode, select the feature, select your language from a menu, add the label, and upload it.

This map of Paris as it will appear after internationalization on a Russian wiki shows a mix of Latin and Cyrillic labels. The name “Paris” is in Russian, and you can see Cyrillic transliterations for most neighborhoods and some streets. The rest "falls back" to French.

Known limitations

• Availability of multilingual labeling is variable. As noted above, the amount of label data available from OSM varies from location to location and language to language. In general, country, state, province and prominent city and town names are present in many languages, while smaller villages and other features may not be.  The table at left shows the relative quantity of translated label data for the 25 most commonly translated languages in OpenStreetMap. You might also like to examine the map, at left, of Paris as it will appear on a Russian-language wiki.
• Changes in OpenStreetMap take time to propagate. It's not hard to add labels in the language of your choice to OpenStreetMap, as described above. Once uploaded, your new label data will convey from OSM to our map system. But such additions don’t show up on the wiki immediately; servers have to talk to each other, new map images have to be generated, etc.  Right now, the lag can be from 1 to 48 hours. We’re working on some changes that will cut that range down to anywhere from minutes (if the stars align) to 24 hours. That speedup should be completed in May.
• Only some types of objects are labeled. To make Wikimedia maps readable and uncluttered, our map styles are configured to show only a subset of the available data on OpenStreetMap, with a focus on features that are useful for what we’re calling “locator” maps: place names, streets, transit features, parks, and some geographic features, like lakes. Among the many things you won’t see labels for are restaurants, stores, houses of worship, archeological sites, tourist attractions…. If our map styles don’t show a category of object, like restaurants, then entering a restaurant name in  OpenStreetMap in your language won’t change the map on your wiki—though it will enrich OSM's data generally.

Use cases for this feature

Locate a city in a foreign country, or foreign country in the world (basic locator maps)

For an article about Antwerp, an English Wikipedia editor wants to show the city’s location within Belgium.

• Process for a mapframe map: the user enters the city coordinates, the zoom level, and data for a marker position into mapframe code in a page or template.

A static locator map of Antwerp as seen on English Wikipedia today (left) is compared with a dynamic mapframe map. The green marker is optional.

• Results: a map displays with a slightly higher level of detail to what exists on many typical Wikipedia city entries today (e.g., see the comparison at right, or the Antwerp page).

• Advantages of using mapframe: The mapframe  map shows a similar level of detail on the page as does the static map style in use today. But it can be popped up and zoomed in and out to show more detail or more context.  The process of adding the map is also liable to be more streamlined. And, of course, the dynamic map is less likely to become outdated as borders or place names change.

• Disadvantages of using mapframe: Depending on the size of the town and the desired language, a label in that language may not be available (though it can be entered at OpenStreetMap.org). In general, users creating a static graphic may have more control over what is or isn’t shown and the precise framing of the map.

Give us your feedback on this feature

So, what do you think? What might you do with this new capability? What other use cases can you see for dynamic, internationalized maps?  Do you think you will enter new label data in OpenStreetMap to create more detailed maps in your language? Does the process of creating mapframe maps seem easier than what you do now or harder? What might make this new internationalization capability more attractive—understanding that resources for the Map Improvements 2018 project are finite (Collaboration team rolls off at the end of June, 2018)? Please post your comments on the project talk page. We’re listening.

April 11, 2018

We’ve made great progress on rationalizing the overall Kartotherian code. This has enabled us to complete some of the desired user features, including these Community Wishlist items:

• T180907: Add zoom level 19 (one of two “main wishes” from the Wishlist proposal)
• T140209: Don't show "hand" mouse icon on unclickable objects
• T141750: <maplink> and/or <mapframe>: clickable marker area too big

In the coming days, some version of the following note will go out to most Wikipedia communities proposing to release mapframe on all but nine Wikipedias that currently lack the feature.

Time to bring embedded maps (‘mapframe’) to most Wikipedias

Mapframe is a feature that enables users to easily display interactive maps right on wiki pages. Currently, most Wikipedias don’t have mapframe. But fifteen Wikipedias, along with literally all the other wiki projects, are using mapframe today to display maps on hundreds of thousands of pages.

A little background: over the last few months, the Foundation’s Collaboration team has been working to improve the stability and user experience of the maps service. In addition, a question about long-term support for the maps service was recently settled, and a small team has been assigned for routine maintenance. Given these developments, bringing the benefits of mapframe to Wikipedias that lack the feature seems both safe and supportable. (Nine Wikipedias that use a stricter version of Flagged Revisions will not get mapframe in this release.)

Maps are a valuable form of visual data that can improve readers’ understanding across a wide range of topics. If you know of any reasons why mapframe shouldn’t be implemented on your Wikipedia, let us know on the project talk page. Unless we hear from you, we plan to release mapframe to most Wikipedias in May, 2018. So, if you foresee an issue, let us hear from you. Otherwise, happy mapping!

March 29, 2018

We consulted with English Wikipedia on Village Pump as to whether community members are still in favor of the proposal put forward in an October RfC requesting that mapframe be turned on. Broad support remains for the idea, so we are moving forward with a plan to release mapframe on English Wikipedia soon after we launch the internationalization feature (T112948) described above. Expect a release by the end of April, 2018.

Awards

Project was awarded Innovation Award (OpenStreetMap Awards) . The 2018 award ceremony took place on 30th of July at the State of the Map 2018 conference.