Recommendations to support language onboarding are formulated through a series of conversations with various stakeholders from WMF staff and the volunteer community. See below.
Next hypothesis - 2.2.6: If we document the pre-incubator, incubator, and post-incubator journeys for the five pilot wikis with quantitative and qualitative data, we will be able to better support new languages in the future.
For the qualitative analysis plan with active editors, alongside the quantitative analysis, please get in touch with Wikimedia_Research.
As an onboarding resource, the Starter_kit can be used and and support from the Language Diversity Hub and Telegram can help us stay in contact with the new wiki communities throughout the pilot.
The meeting focused on preparations for setting up wikis for the five newly approved languages, discussing timelines, necessary features, and monitoring strategies. The team also highlighted the importance of documenting the wikis' journeys and maintaining communication with new wiki communities.
Minimum 2 active editors in the last three months sounds good. Exclude editors editing across more than 5 languages.
What would be considered a success to make this intervention "worth it"?
Does it meet the Language Committee's criteria for graduation? If not, how does it compare to the control group?
Establish a percentage range to distinguish between 'average' and 'above average' outcomes.
Explore developing a campaign toolkit for how-to use the features/tools that are not yet on Incubator and list of articles new wikis should have using the following resource as a reference.
In the meeting, the measurement plan was reviewed along with selection criteria in the Language Committee proposal and the success criteria for the experiment.
Experiment Hypothesis: If we provide production wiki access to incubating language projects, this will result in at least X% additional editor productivity, leading to X% additional content generation than usually on incubator wiki.
Baselines:
Look at averages.
Examine activity when projects "graduate" from the Incubator to a production wiki.
Compare similar wikis to avoid comparing dissimilar ones.
Set Percentages: Abandon a set percentage, OR consult SRE for successful intervention criteria.
Effect Measurement:
Project Level:
Net content.
Pages (content, template, and rest).
Bytes (content).
Average number of monthly active editors (active: at least 5 constructive edits).
Editor Level (Productivity):
Average number of constructive edits.
Editing session completion rate.
Tool usage.
Content Translation: Determine the time required to enable content translation for a new language project (TODO: Consult Language team).
Visual editor: Available on Incubator but still buggy.
Citoid, Wikidata.
Extensions: Complete list of tools/extensions/features not on Incubator available here.
Interlanguage links and other additions not available on Incubator.
Future Hypothesis: How to support tool usage if there is no uptick.
Timeline:
Likely 1 month for editing in this pilot.
Initial analysis and follow-up analysis.
Evaluate in 1 month, but unlikely to prove or disprove anything.
Will not stop intervention unless there is vandalism or similar issues.
Determining Success:
Success if compelling evidence shows it's working, with input from LangCom.
If no compelling case, extend time and create additional hypothesis next quarter.
Continue study based on learnings.
Onboarding Support:
General onboarding support should be aligned with the Language Inclusion Support Plan.
Minimal onboarding to reduce bias in measurement.
Provide how-to resources equally to all 5 wikis for new features on a production wiki. Resources:
How to use Content Translation, Visual editor, citation tool, etc.
Link to existing resources on wiki.
Action items
Look at one-month and three-month averages for editing baselines
Look in to which features/tools the production wiki will offer that aren’t available on Incubator
Look into what resources exist for how-to use these ^ features/tools
Find out from SRE and others what would be considered successful to make this intervention "worth it"
The team brainstormed criteria for measuring the success of the incubation pilot. The hypothesis is that providing production wiki access to incubating language projects will increase editor productivity and content generation by at least X%. Baselines will be identified by examining averages and activity during project graduations from the Incubator to a production wiki. The discussion included measuring effects at both project and editor levels, and providing equitable onboarding support. Action items include reviewing editing baselines, identifying additional tools not available in Incubator, finding related resources, and consulting SRE on success criteria.
Machine translation: Rather than translatability with top languages based on Wikipedia size, it should be top languages because on most often used for content translation (see ContentTranslationStats - Note: long time to load)
Machine translation: are we only considering languages with machine translation capabilities?
Under current criteria, yes, because otherwise we would be testing two different interventions (having MT access vs. not having MT access). Hypothesis rationale: machine translation is a modern feature for editing within production wikis.
We might not want to exclude languages that don’t have MT capablities. Concerns about exclusion of languages without MT capabilities, especially since these are marginalized languages.
TODO: Discuss more options about how to incorporate the non-MT languages if possible
During the meeting, the criteria for including wikis and stratifying them in the incubation experiment were brainstormed. It was decided to prioritize languages most often used for content translation rather than just the top languages by Wikipedia size. Currently, only languages with machine translation capabilities are considered to avoid testing two different interventions simultaneously. However, there were concerns about excluding marginalized languages that lack machine translation capabilities. Options to incorporate these non-MT languages will be explored. Additionally, criteria that exclude closed projects were discussed for removal.
Apart from content translation and Wikidata, there isn’t much difference. The big issue in the incubator is the template issue.
The incubator was set up before the Language Committee existed, making it difficult for people without wiki experience.
We should have a step-by-step guide showing people what they need to do for their wiki to be approved, for example, the newcomer homepage.
The intention was to start with an experiment, with very minimal technical interventions. We are trying to see if we can justify further changes based on how the experiment goes.
Looking at the larger picture, we need to figure out the role different stakeholders play in supporting the languages across various projects.
Interested in the details of the pilot. It’s not clear how the pilot is supposed to work.
How does the Language Committee process proposals such as these? How long does it take to make such decisions?
All our discussions are done on our mailing list, either public or private. We consider the proposal, whether it is a change of Language policy, and see what other members say. We come to a conclusion and possibly have a vote if there is a disagreement.
There is no request for a policy change, but there might be a recommendation to the policy.
Long waiting time for creation was a big problem a couple of years ago, but nowadays it's not really an issue anymore, thanks to work done by User:Ladsgroup and probably more people, unless Wikiscript is broken.
We don’t have good tools to notice which projects are active and which are not, to help us discover which projects are getting ready for approval. Incubator Dashboard is currently not working.
Maybe it’s not practical to approve languages with no content on the incubator. We could be a bit flexible with the activity as part of this experiment.
Maybe Wikitongues can help with their contacts in languages not represented in Wikipedia.
WMF Assumptions on the Pilot
Only approved requests
Maximum of 5 wikis for ease of logistics (based on random selection within buckets)
Wiki projects with prospects of community growth and volume of activity suitable for analysis
Publicly stated details about the pilot to set clear expectations
Thoughts from the Language Committee
Open Questions about the Pilot:
Developing criteria for the selection of wikis
5 wiki cap
Wikis without test projects
Possible ways to bucket/stratify for random selection: edits, editors, pages, region, MT availability, internet users of the language, etc.
Time window for editing before concluding the pilot
Future of the wikis that will be part of the pilot:
After one year, the language must have approximately 100 articles and no fewer than 5 active editors, or it gets reverted back to the Incubator.
The meeting focused on discussing the language incubation pilot project and outlining the next steps for seeking approval from the Language Committee. Key issues highlighted included the need for better templates, step-by-step guidance for new users, and improved tools to track project activity. The group emphasized the importance of minimal technical interventions in the experiment to justify future changes and considered potential collaboration with Wikitongues. The Language Committee asked questions about criteria for selecting wikis, the pilot's success criteria, and the future of pilot wikis, emphasizing the need for clear guidelines and expectations.
To define the measurement and selection criteria for the experiment using this opening question: What can we learn in three months or less with a limited amount of groundwork?
Which wikis should be selected? (e.g., creole languages for which there is no Wikipedia yet, maybe those that are currently doing translation testing)
Something less strict than the Language Committee requirement but with at least one editor.
It can be a language that is in the incubator or not; we can do both, maybe. The important thing is having active people.
Wondering if bringing in non-existing wikis would be complicated with regard to existing policies, and if there are existing communities yet
Yes, we would be bypassing requirements like expert review (that the language people claim to be writing in, is actually that very language)
Will the interface be translated before languages get a new wiki?
If we don’t ask them to translate first, it will be part of the success criteria later.
Can we give some a different URL or dark logo or some other type of beta space? (to keep the thrill of incubating process)
SRE wants to minimize work, and therefore dark logo is more possible than a new type of URL
From a product analytics standpoint, randomized selection is more appropriate for an A/B test. What is our success measurement going to be? Is it faster editing? Is it other types of benefits?
Because of limited time, we need to pick one project type (e.g., Wikipedias only, since Wikipedias have the most support). Regarding selection, we could stratify test wikis into buckets based on characteristics (number of edits, number of editors, number of pages, Machine Translation available), and then randomly selection. We can do a random selection once we have done the stratification.
What do we mean by faster editing? If we think about translation tools, these are the types of tools that are available, or that they will have available?
What we hope to see is increased editing activity.
Random selection make sense and is fair. But we want to make sure that we have a filtering out of projects that don’t make sense to include in this experiment (e.g., a project with 0 edits for more than a year probably doesn’t make sense to include)
Where were the pain-points? Content? Interface? Translating?
Regarding only looking at Wikipedias, there is an argument for looking at other projects like Wikisource, because there are many languages that would benefit from this (e.g., Urdu has a large Wikipedia, but no WikiSource); we also don’t know if Wikipedia is going to work for all communities; are we perpetuating the idea that Wikipedia is the way to get into our community.
Will it be less engineering work to set up a new wiki for a language if the language already has another hosted wiki (like a hosted Wikipedia or Wiktionary)?
Re. Wikisource, there are a lot of potential extra challenges that arise like finding the sources, copyright issues, relicensing, digitization, etc.
Things to consider
Workload on SRE
Existing community
We need buy-in from SRE now that if we find this intervention to be successful, we will want to expand this intervention to all languages ← Because this will involve migrating ~1000 projects
What will we consider as success for this pilot or a threshold for a community to have its own wiki?
Statistical testing
Number of editors, edits, new editors per language from the same group (test wiki vs still in incubator)
Qualitative testing
Threshold
After 1 one year, the language must have ~100 articles and < 5 active editors or it gets reverted back to the incubator
Action items
Develop Criteria/Stratification Buckets from Analysts and Research perspective
Develop minimum criteria for wiki selection from a community growth perspective
In this meeting, we brainstormed key questions with the goal of defining the measurement and selection criteria for the experiment, using this opening question: "What can we learn in three months or less with a limited amount of groundwork?" We discussed various criteria for selecting a wiki for the pilot project, considering analysis, research, and community growth perspectives. As the next step, we will develop initial criteria based on this discussion, followed by several more brainstorming meetings, including with the Language Committee.
In this meeting, participants received an update on the Incubator conversations, the reasons the Wikimedia Foundation is prioritizing this area, the recommendations gathered through the process, the chosen recommendation for a pilot for the Foundation's annual plan for the next fiscal year (2024-25), the next steps involved, and the project timeline. These updates were shared during the Language Community Meeting (May).
A fundamental limitation arises when one item links to another project that doesn’t work with Incubator because there are many pages on the same topic. For example, the item for Berlin has five articles that would need to be linked to the item.
Splitting Incubator into multiple wikis is difficult. When wikis graduate from Incubator, not all of their articles are connected yet. This is something that can still be addressed by improving the import process.
Infoboxes, if they take data from Wikidata, are default connected to the item. Perhaps there can be ways to inform the infobox about this by adding something to the article.
Templates and modules - Data box needs to be investigated if it can work with Incubator.
These issues can be worked around but not fundamentally fixed. It will be very difficult to fix some of these issues. All these Wikidata issues would not exist if they were proper wikis with one language.
There is something to not let every random person start a wiki. A workaround - giving them separate wikis but marking them somehow special, making it very clear that this is an incubating wiki, putting them into a special domain all that would solve at least Wikidata as well as CX problems.
Although Incubator helps to build a community, if they spend too much time there, it may kill a community. Those wikis are the ones that should be supported by Wikidata.
In the meeting, it was highlighted that a significant challenge arises when a Wikidata item links to another project incompatible with Incubator, resulting in multiple pages on the same topic. For instance, the Berlin item has five articles requiring linkage. Splitting Incubator into multiple wikis poses difficulties, as not all articles are connected upon graduation. Improving the import process could address this issue. Infoboxes connected to Wikidata items by default may be informed about this through article additions. The compatibility of Data box with Incubator requires investigation. While workarounds exist, some issues may remain fundamentally unresolved. Restricting the creation of random wikis and marking them as incubating could alleviate problems with Wikidata and CX.
Idea: Automating New Language Addition and Approval in Incubator
A better notification system for the Language Committee to promptly review requests and track project progress would aid in improving graduation rates. The potential number of requests created in the future could probably be easily measured.
By examining how requests remain in discussion status or on hold before incubation on the "Request for new languages" page, we could query the date of page creation for each request.
Relevant discussions occur on language committee mailing lists. The approval of the Montenegro Wikipedia may not happen anytime soon; there is currently significant discussion on the lists. Occasionally, communities may not utilize the requests page for the creation of a language.
The hypothesis involves combining the two steps. The time to resolution could be a key metric here - determining whether the request was accepted or rejected. Will there be planned outreach to smaller communities as part of this effort?
A better direct measurement, rather than increasing requests from smaller languages, would involve reducing the time taken for request resolution. An indirect result would likely be an increase in requests from smaller language communities and quicker turnaround times.
The Language Diversity Hub paper provides qualitative (interview-based) data about experiences during each part of the language onboarding process. We could compare interview data after the changes are implemented.
Idea: Providing Access to Modern Wiki Features Beyond Incubator
Ideally, every wiki should have the same start and end time. Otherwise, measuring this would prove difficult. For the top 15-20 languages already in the Incubator, we could provide them with a wiki and assess their edit activity before and after Incubator to determine if there's an increase.
Conducting a randomized control trial to select wikis may encounter resistance from volunteers. If we pursue this approach, extensive consultation with various stakeholders would be necessary.
Typically, in an A/B test, the final outcome is identical for both groups, and the test is time-limited. For instance, if a new feature is tested, and it succeeds, everyone receives the feature; otherwise, everyone reverts to the previous version. Hence, the crucial question to ponder is, if the 5 wikis from the top 20 we test show success (e.g., increased editing activity), what will happen to the remaining 15? Ideally, they should also receive new wikis. Conversely, if the test fails, will the 5 wikis be closed and reverted? Thus, we must consider the ultimate outcome for all wikis, not just the selected ones, based on the test's success or failure.
Idea: Improving Editing Experience Within the Incubator
At least it takes two years to graduate. What can be measured in two quarters? Bytes, pages, or the number of edits added, etc.
Anything related to edit activity can be easily observed.
The number of editors would be really important to look at, especially if this number increases with the user-friendliness of new features. I didn’t start editing until the visual editor became available.
If we examine various tests conducted before and after the visual editor was implemented on wikis, we can observe what types of increases occurred and use that data to make predictions here.
The most important metric is completion rate. Out of 100 sessions initiated, how many people were successful in completing them?
Are there any similar interventions from which we can draw numbers to support our hypothesis?
Edit completion rate can be measured for all editing-related improvements. We can use the edit attempt step for any editing activity to understand the edit completion rate.
Idea: Automate Backend Site Creation
It will be easy to track the number of sites created in terms of actual tracking for the amount of time it takes for the process to happen. The "Last meaningful edit date" that occurred on the project can be tracked right now, along with the duration between the last edits and site creation, so we can examine the decrease in the amount of time.
One missing variable is the date of the request for site creation.
This idea might not improve editors' experience; rather, it impacts the people who create the sites - both staff and volunteers. You can measure how much time they invest in this. There may be more work involved later to close wikis aggressively.
This meeting focused on identifying key variables and potential metrics to track the impact of proposed recommendations on the contributor experience and subsequent activity within the Incubator.
Integrating CX with Incubator is not worth the effort, experience will be subpar for various reasons.
Incubator's specific language naming rules necessitate introducing a new concept to MediaWiki, potentially competing with categories and namespaces. It's crucial to monitor for vandalism and invest in investigation.
Emphasizing that experimental approach isn't about a single tool like CX, but rather a comprehensive effort, is essential. Adding CX will not be the same experience without Wikidata, templates, etc.
Comparing growth approaches through experimental idea is a positive step.
Suggestion to seeing Incubator as a status, not just a place through experimental approach to simplify transitions and avoid template complexities.
Implementation involves seeking permission and considering the perspective of SRE in creating new wikis.
AddWiki's impact is seen as secondary to the idea, though it falls outside the language team's scope.
The meeting emphasized that it’s not about one tool like CX being available on Incubator, but the entire experience. A suggestion was made to rename the experimental idea to emphasize that it’s not merely about skipping the Incubator, but rather about granting these languages access to modern features. There can be different approaches to this. Another suggestion was made to investigate the risks associated with the idea and how to mitigate them, while also viewing the Incubator more as a status than a physical place, aiming to simplify transitions for new wikis and avoid unnecessary complexities.
Gather feedback on the experiment proposal to provide production wiki access to XX wikis for monitoring over one year.
Understanding necessary resources, support, technical complexities involved and collaboration efforts required from tech teams to develop and test this approach.
AddWiki is currently broken. There are still 10 wikis waiting to be created. Security considerations need to be made. Renaming a wiki is impossible. Why skip Incubator given the current challenges with wiki creation?
To try something simple and see how things work without putting in a lot of effort. This would require approval from the language committee. Selecting languages carefully is necessary.
This is part of the annual plan process. Significant changes are desired in this area, but not heavy investment initially. Experimenting in a controlled environment is essential, and decisions about further investment will be made based on the learnings.
There isn’t good infrastructure to clean up wikis. Wikis are not deleted; they remain in the database forever. It might be acceptable for a language project to skip Incubator if it has a preceding project in Incubator because there won’t be any requests for renaming language, at least. What should automated wiki creation look like? One idea to fix the broken AddWiki script is to move it to Core.
Considering language wikis with a preceding project in Incubator for the study - risks might be lower. We will have to see if we can find such wikis. And they may not require expert approval and more relaxed editing activity requirements. There are already many wikis that probably need to be closed, so adding 5 more to it should not be a big deal. Is it really impossible to delete a database for a wiki?
For closed wikis, we still have to update their schema. Completely deleting a wiki, we don’t do that as it is complex. Closing is when there is no activity or severe content problems.
One big problem in Incubator - just the wiki which is pretending to be hundreds of wikis. A wiki-farm-like system could be built - why is it not being considered? Tim Starling has some good ideas for how to make it better. Who in the long term can own it? Maybe MediaWiki Platform.
One wiki is created per month on average. Creating a wiki would take a couple of days unless a wiki breaks. If it is only 3-5 wikis, it would be a good opportunity to onboard some people in the Language team to create wikis using the existing process to avoid a bus factor, etc: https://wikitech.wikimedia.org/wiki/Add_a_wiki.
AddWiki's current malfunction and backlog of pending wikis raise concerns about security and the feasibility of renaming. Investigating why AddWiki is broken, potential fixes, responsible parties, and integration with MediaWiki Core with Tim Starling are suggested. Discussion revolves around the challenges of skipping Incubator and proposes a trial period with careful language selection. Additionally, considerations for language projects with prior Incubator presence and the complexities of permanent wiki deletion are discussed. Furthermore, an implementation plan for the idea is proposed, outlining prerequisites such as recruiting languages with preceding projects in Incubator and obtaining approval from the language committee.
Gather feedback on the proposed idea to implement a self-serve system for new wikis: automate site creation and provide production wiki access to XX wikis for monitoring over one year.
Wanted to clarify if the interest lies in bringing a new wiki into production on languagecode.wikipedia.org. On the technical level, we know what needs to happen but lack the expertise to execute it. The incubator isn't ideal for use, and there's currently no team available to address these challenges.
We're considering not only Wikipedia versions of languages but also other projects.
Automation will take 48 months.
It's a matter of priorities. Configurations are challenging to automate, and there may be resistance to changing the system. While there are engineering hurdles, how high of a priority is this for the organization?
Creating more wikis that aren't utilized leads to additional technical debt.
Making small changes is a good start, but it's essential to decide whether to improve or bypass the incubator. Making small changes to the incubator wont make you learn how having what it means to have no incubator.
Stewards monitor new wikis and can be relevant stakeholders.
Improving the incubator seems sensible from an annual perspective.
The meeting emphasized that automating the way configuration is managed is very challenging. With decision-making and commitment, it can be possible, but it will take longer. A suggestion was made to stick with a decision on what to test and learn. Improving the incubator won’t help understand what it means not to have it. If wanting to test this, it can be done manually without automation. In terms of annual planning, improving the incubator makes sense given the limited time. However, if aiming for a longer-term goal of automation, it can be split into shorter goals. It's also possible to contact stewards and keep them in the loop to get perspectives on new wiki creation and patrolling.
Experiment proposal to provide production wiki access to XX wikis for monitoring over one year.
Future approach to develop a self-serve automated system for creating new language wikis.
Understanding necessary resources, support, and collaboration efforts required from tech teams to develop and test both experimental and permanent approaches.
We already have a lot of wikis that are not very active. We need a way to transition them from a full-fledged wiki back to a limited wiki.
Or should we remove them entirely?
Wikis are occasionally deleted, but this is very rare.
What are we trying to learn? There are benefits to both incubator and standalone wikis. What do we hope to learn within a year?
Graduation from the incubator is often a strong incentive for people to write articles in the incubator, but after they graduate that incentive is gone. If there’s a way to demote a wiki, then that would add an incentive we don’t currently have.
Some features like content translation and Wikidata are unavailable in the incubator but could drive engagement.
What aspects of our approach are experimental versus permanent. What defines the end of the experiment?
What are the requirements here technically?
Little concerned about the “self-service” part. This is not necessarily an easy-to-fail/cheap experiment. What can we improve about wiki creation that would be a win regardless? Lots of wikis may end up being costly in the long run.
The Language team didn't want to define the scope themselves. They want to define it together.
There is research for the rate, we could do something partial to see the effect on the process.
Which parts can be automated? Self-serve for whom? What is full-feldged vs limited?
Self-serve for two kinds of people—people who want to start a new wiki in the language (currently the process is convoluted editing of talk pages/templates). The other group is the volunteer language committee—the committee who decides whether the language graduates or not. The technical process of creating the wiki would then happen post-approval in the background.
What stops incubator from having these features?
Content translation works across wikis, and incubator is just a single wiki. Same for Wikidata. It’s probably not impossible, but it is required for all features.
Community process is owned by language committee vs technical process, the technical process (e.g., using addWiki.php) is broken a lot and could use some improvement.
Possible next steps could be exploring if improvements to the incubator are viable and integrating extensions to enhance functionality.
The meeting focused on brainstorming two initiatives: an experiment proposing production wiki access for monitoring over a year and the future development of a self-serve automated system for creating new language wikis. Participants discussed the potential sunset of inactive wikis, considering transitions to limited status or complete removal. They highlighted the need to understand the technical requirements and potential costs associated with self-service automation. Concerns were raised about the experimental nature of the proposals and the distinction between temporary and permanent approaches. Key points included incentives for incubator graduates, the absence of certain features in the incubator, and challenges with the technical process. Possible next steps involve exploring improvements to the incubator and integrating extensions to enhance functionality.
Wondering if the mass-contribution model that made Wikipedia what it is today is still viable, considering the evolving landscape of the Internet. While some projects retain traction, there's a noticeable downward trend. How can an encyclopedia be created from scratch in 2024? If hypotheses exist, the incubator, or the incubation phase more broadly, may offer the right platform to test them.
Facts are language-agnostic and should be recorded as such. There's already an extensive collection of facts within wiki projects, making it inefficient for small language communities to duplicate research efforts. New language communities should contribute their unique cultural knowledge to the collective human knowledge, accessible to all regardless of language.
The prose of a Wikipedia article serves as a presentation layer that can be dynamically generated for different languages, lengths, and levels of detail tailored to the audience and context. This line of thought led to the concept of a "Multilingual Wikidata powered Wikipedia."
For new wikis, the workflow involves:
Identifying culturally relevant topics for documentation.
Building a registry of local reliable sources.
Creating and populating Wikidata entities and properties for these topics.
Writing Wikipedia articles based on the data in Wikidata.
This approach ensures:
Existence of articles for readers.
Creation of a training dataset for language-specific Wikipedia text rendering.
Utilization of Wikidata by other language communities.
Additionally, a separate process extracts factual information from existing Wikipedia articles and integrates it into Wikidata to benefit other languages.
Is there a way to use Wikidata more to show meaningful content?
Some examples: Q405 (Moon) on Wikidata, Monumental on Toolforge
Two hypothesis
Can we have an article rendering from wikidata for new communities?
Is it more welcoming to contribute to an outline or start on a blank page?
Abstract Wikipedia is trying to build something similar.
Extension:ArticlePlaceholder: it might be a good idea to revive it, improve it and deploy it to many more languages.
Resonator shows the same information as the Wikidata article page.
Here is a paper to understand how this is different from Abstract Wikipedia.
Language onboarding is to target communities for which there is no Wikipedia yet and to streamline the repetitive process around this.
Consider addressing design question 3 to implement a solution that eliminates the repetitive process involved in request creation and approval on wikis. Even if alternative approaches are made for Incubator in the future, this solution would still remain valid; backend adjustments may be necessary, but the frontend would remain the same.
Related to design question 1, some ideas for improving interlanguage links discoverability; these links currently display languages in which there are fully-fledged articles:
If an article exists, display it more prominently and not hide it behind a button (such as the language selector).
If there is no article in a language, still indicate if Wikipedia exists in that language (with an invitation to create the article using CX).
If there is no Wikipedia available in a language, we can still present it to potential readers/editors of that language. This could be approached as a carefully handpicked experiment.
During the meeting, User:SBisson (WMF) raised concerns about the viability of Wikipedia's mass-contribution model in the evolving internet landscape. They suggested using the incubator phase to test hypotheses for creating new encyclopedias. It was proposed that facts should be language-agnostic, reducing duplication of research efforts across wiki projects. The concept of a "Multilingual Wikidata powered Wikipedia" was discussed, along with the workflow for new wikis, emphasizing the importance of culturally relevant topics and reliable local sources. Utilizing Wikidata was highlighted for meaningful content presentation. Various hypotheses were presented, including rendering articles from Wikidata and improving language onboarding. The potential revival and enhancement of Extension:ArticlePlaceholder were mentioned. Resonator and Abstract Wikipedia were also referenced. Language onboarding strategies and improving interlanguage link discoverability were discussed, with ideas for making articles and Wikipedia availability more prominent. Addressing design question 3 was suggested to streamline repetitive processes in wiki request creation and approval.
Gather feedback on the proposed idea to implement a self-serve system for new wikis and provide production wiki access to XX wikis for monitoring over one year.
Repetitive tasks like creating new wikis should be handled by humans more effectively. Improving automation is crucial, but if it breaks, fixing it can take days. Configuration for wiki groups of Incubator and graduated wikis is essential. If an Incubator wiki transitions to an actual wiki, how many wikis do we expect in the first year?
If the numbers increase significantly (close to 100), a larger investment will be necessary. It shouldn’t be a one-off intervention. Assuming scripts for automation go to the MediaWiki team, ownership in this area would be required. Monitoring a couple of creation requests every month should be manageable.
Of the 700 wikis in Incubator, less than half show useful activity. Exporting them to the new system when ready would be beneficial.
Identify a few communities interested in the experiment. Create configurations for this set of wikis and provide them as part of the service. Define available extensions and consider translating writing configurations as a first step.
Determine responsibility for wiki creation, sharing, and approach for sustainability. Consider side effects and maintenance associated with automation.
Consult with the Language Committee to confirm the preferred course of action and understand considerations.
The meeting convened to gather feedback on implementing a self-serve system for new wikis and providing production wiki access to a specified number of wikis for one year's monitoring. Discussions emphasized the need for more efficient handling of repetitive tasks by humans and the importance of automation improvement. Key considerations included configuring wiki groups, anticipating the number of wikis in the first year, and the necessity of a larger investment for significant increases in wiki creation numbers. Suggestions were made to identify interested communities for the experiment, create configurations, define available extensions, and determine responsibility for wiki creation and maintenance.
Brainstorming on following design research questions to explore social pathways for language onboarding:
How might we raise awareness about languages not yet featured on Wikipedia, facilitating their inclusion? How can a design intervention guide potential contributors in initiating the process of adding their languages?
How can we streamline the process for new communities to assess their readiness for launching a Wikipedia in their language? Additionally, how might we seamlessly direct them to alternative Wikimedia projects or contribution avenues if they are not yet prepared?
What design strategies can we implement to encourage potential contributors to take necessary actions to add their languages to Wikipedia? Can we simplify the process through a user-friendly form, automating steps where possible, to eliminate the complexity of manual submissions? Note: Currently, the process for creating a new wiki is highly manual and lengthy. Please refer to this 10-page document for a step-by-step description of the process: New wiki creation: step-by-step process description
How can we enhance the user experience for new Wikipedia contributors by providing personalized guidance, tips, and tricks throughout their editing journey, similar to the Wikipedia Adventure and Growth features?
How might we effectively visualize the progress of a language wiki to its contributors, ensuring transparency and motivation for continued contribution? What design elements could be employed to make this progress tracking intuitive and engaging for users?
Seems like 22 wikis per year is the highest number seen; do we know what influenced the increase?
Had to measure how much of this comes from activity from humans and how much comes from technical features. There were not a lot of changes in the recent years. Improved documentation on Translate Wiki to help people understand localization processes much better. In 2021, there were many Wikipedias approved in Taiwan, and this is because there was a university project in Taiwan to improve languages. In 2 or 3 years, they were making videos on the incubator.
In the current context, when trying to find something as a way forward, it becomes difficult if don’t have the baseline context. Need to have some qualitative information to help predict some trends; otherwise, don’t know what is influencing some trends and engagement patterns that can be built upon for the future.
Chicken and egg situation because the incubator there is the way it works. Can go over the history of project approvals and provide some comments on what helped some projects go faster, but if want to measure this, need to start building the solution.
Do we have a sense of numbers actively represented on Wikipedia and those not?
There are 7000 languages and around 300 are documented.
Statistics about this are hard to get, perhaps Ethnologue.
Do we have a strategic future of projects that are not doing us well?
At a strategic level, do have to hold some space on what exactly it means to go forward. There is a thinking on how do we graduate from where we are..merging things..or what does not serve a purpose anymore..there are some plans on how to build SLO’s etc…very early stages for now..nothing concrete yet.
As we think about these recommendations, let’s think about what can be controlled, and what can be influenced further based on expertise and responsibilities. Some recommendations come with legacy thinking for example about language portal. It comes from the thinking of the web as a starting point…but maybe that’s not how people are thinking about things anymore..might have to think about it from another lens on how small languages are coming into our platform, how can we build this in a direction that makes sense..recommendations are good guiding pieces..might be something to sit with and see how we can facilitate the process.
We don’t have to restrict selves with Wikipedias..for example, Wikisource, Wiktionary, have smaller communities…what can be done to have the space more welcoming..experience spaces, attention can be given to.
The meeting discussed the recent increase in new wikis, particularly noting the influence of various factors including human activity and technical features. It was highlighted that despite limited changes in recent years, improved documentation on Translate Wiki has aided understanding of localization processes. The approval of numerous Wikipedias in Taiwan in 2021 was attributed to a university project aimed at language improvement, with videos produced over 2-3 years on the incubator. Concerns were raised about the need for qualitative information to predict trends and engagement patterns for future planning. The meeting also touched upon challenges in measuring project approvals and emphasized the importance of building solutions. Additionally, there was discussion on the representation of languages on Wikipedia and strategic considerations for future projects, including the need to adapt to evolving perspectives. Recommendations were made to consider what can be controlled or influenced, with a focus on fostering a welcoming environment for smaller language communities beyond Wikipedias, such as Wikisource and Wiktionary.
More ideas for outcome variables we could look at:
Time spent in incubator
Time spent on edits
Editor success rates (e.g., successful edit rates vs. abandonment rates) - first, we need to find out if the Incubator has these logs; if it does, it will take 10-12 hours to query this
More activity (e.g., more monthly edits) - if the tech updates improve the editing experience, we would hope/hypothesize that the improvements would result in more editors
Number of editors - if the tech updates improve the editing experience, we would hope/hypothesize that the improvements would result in more editors
Note: We should just look at projects with at least five editors.
Tracking success-vs-abandonment of edits, as well as time taken to perform edit (location).
Looking at abandonment rates of Incubator (e.g., look at ratio of successful vs. abandoned edits)
We could use this to test pre- and post- improvements
It only shows successful graduates. So while successful graduates may show a decrease in time spent in Incubator, this doesn’t capture the hundreds of projects that are still in the Incubator, with many years spent in the Incubator.
As time goes on, next year and the year after this graph may show many longer lines towards the top of the graph as more projects created after 2012 continue to graduate, so in 10 years this graph may actually show many more long lines at the top of the graph. (location).
This meeting focused on identifying key variables and potential metrics to track the impact of initial recommendations on the contributor experience and subsequent activity within the Incubator.
Proposed Experiment: Automate the site creation process, which currently takes several days, and provide production wiki access to XX wikis for monitoring over one year.
Discuss with the language committee and consult with SREs for feedback.
Understand the current effort in wiki site setup and explore the potential for a self-service model.
This meeting aimed to identify collaborative efforts necessary from both WMF’s SRE and Language teams to streamline the process of language onboarding. It discussed one proposed experiment: to automate the site creation process and provide production wiki access to XX wikis for monitoring over one year. Follow-up meeting thoughts included an acknowledgement from the SRE team that self-serve is the way to go. The biggest dependency from the SRE team is the creation of DNS entry for the new language. Currently, User:ASarabadani (WMF) has been supporting mostly in his volunteer capacity. As this is a product officially unowned by any specific team, implementing it would require involvement from a few WMF teams - the MediaWiki dev team, a few SRE teams (Data Persistence, Service Ops, Traffic), and Release Engineering teams.
To discuss specific technical areas that the Wikimedia Foundation should prioritize for the Incubator and identify potential solutions to address challenges within these prioritized technical areas.
Simplifying content creation: The problem of prefixes remains a significant challenge. Limited support is available for users dealing with prefixes through gadgets that hide them during editing. Only one community member is actively addressing this issue. Extension:WikimediaIncubator has lots of bugs on Phabricator. Editing on Incubator should feel like editing on normal wikis.
tet.wiktionary.org redirects you to Incubator, which is what we want.
The link https://bew.wikipedia.org/ does not redirect you to Incubator but it should.
Wikidata support and content translation: A Phabricator task I created in 2013 focuses on improving Wikidata support and content translation. Incubator is disconnected from Wikidata. The Content Translation extension doesn't work, which is a big problem.
Templates and CSS stylesheets: Navigating infoboxes is complicated. You can't copy-paste from other Wikipedias as they rely on other templates and stylesheets. There is always a proposal from User:Aaharoni-WMF about global templates.
Thoughts on possible solutions: Simplify the user experience, discuss the proposal from User:Aaharoni-WMF on creating a new wiki for a new project much faster Phab:T228745.
Affiliates: Establish a forum for affiliates where they can share resources around language onboarding and creation. Some way to help with coordination efforts would be nice. e.g., Wikimedia Indonesia, Taiwan and Nigeria they are doing a lot but everyone has to reinvent the wheel.
Simplifying content creation remains a significant challenge on Incubator, particularly due to issues with prefixes. MediaWiki extension of Incubator also has lots of bugs. The goal is to ensure that editing on Incubator feels like editing on normal wikis. Proposals from User:Aaharoni-WMF regarding global templates and new wiki creation, could offer valuable technical support in resolving these challenges.
Global templates is a huge engineering effort. It may not happen anytime soon and there is another solution for it. Consider meeting with MediaWiki team.
Content translation integration also seems complicated as the team is considering to feature freeze soon. So it is unlikely that Incubator can get CX anytime soon.
Incubator is a shared infrastructure which makes it a complicated resource. Instead, a placeholder production wiki for each community can solve all problems; that is not yet indexed in the search results. SREs are anyways already doing this but after a wiki graduates from Incubator.
If we take the steps to improve onboarding and editing experience in Incubator and create urgency around the problem space, there are chances that the request to create wiki may increase. Also the fact that the infra for the incubator is complex is part of a larger process, we have the production and the test and beta, they degrade as you go down and many engineers have been complaining about it for a while, so moving incubator to production would be a good solution.
Global templates pose a significant engineering challenge, and while meeting with MediaWiki team could offer a solution, it may not happen soon. Content translation integration is complex and unlikely due to an upcoming feature freeze. Creating placeholder production wikis for each community, initially unindexed in search results, could address many issues after a wiki graduates from Incubator. Improving onboarding and editing experiences in Incubator may increase requests to create wikis. Moving Incubator to production could address concerns about its infrastructure complexity.
To discuss specific technical areas that the Wikimedia Foundation should prioritize for the Incubator and identify potential solutions to address challenges within these prioritized technical areas.
What specific technical areas should the Wikimedia Foundation prioritize for the Incubator?
Expediting the language approval process in a faster way
Simplifying template importing process
Allowing for better page search features (currently it is hard to find code errors for templates as the search does not look through the coding for specific words)
Enhancing the efficiency of site set-up process (many new projects currently involve communities whom are often non-tech savvy, making it difficult for them to adjust in contributing in the incubator for their language due to the incubator's very technical nature)
Integrating content translation and Wikidata (The fastest way for contributors to make new pages/articles seems to be through translating existing articles from other languages that they understand. It is quite cumbersome for them to import the pages from other languages into their projects as there is no content translation tool that is available on the Incubator like the one on Wikipedia for existing language Wikipedias)
What potential solutions can address challenges in the technical areas that need to be prioritized?
Providing a progress bar for each test project which lists down specific goals to improve the test projects to improve the approval process. (a progress bar helps tremendously in giving contributors incentive to continue contributing until the project is approved. Currently many do not know the status of their projects due to not having an obvious indicator if things are going well or not)
Providing basic templates that are commonly used built into the basic projects at the start of a project (ie: citation, references, infobox, main page. Communities seem to feel more enthusiastic about contributing when the pages they make resemble existing Wiki projects instead of just being bare bones)
Improving prefix compatibility with template coding (Currently the option to auto insert prefix into the coding is buggy, I have stopped using it for the time being and have not tried it for a few months, so I am unsure if improvements have been done)
Increase number of language committee members that aid in project approval processes. (There seems to be quite a lag between Language Committee members responding to approval requests due to only having one or two members responding for very few days or even weeks)
What aspects of language onboarding can the affiliates/communities concentrate their efforts on?
Template translations such as references, infobox, main page.
Having a more compact and precise list of common pages needed to be available in the project.
Setting weekly/monthly goals for page creation/translation with specific themes.
To prioritize technical areas for the Incubator, we need to focus on expediting the language approval process, simplifying template importing, improving page search features, and enhancing the efficiency of the site set-up process. Integrating content translation and Wikidata can also streamline the process for contributors. Solutions to address these challenges include providing progress bars for test projects, offering basic templates at the start of a project, improving prefix compatibility with template coding, and increasing the number of language committee members. Affiliates and communities can concentrate their efforts on template translations, creating a compact list of common pages, and setting goals for page creation and translation.
To discuss specific technical areas that the Wikimedia Foundation should prioritize for the Incubator and identify potential solutions to address challenges within these prioritized technical areas.
It's a bit frustrating to see how Wikipedia has improved on other wikis like English Wikipedia but when you come to Incubator, you have to open multiple tabs, and navigation is challenging. Training people on how to use it is difficult - how to add words, get the link from another tab, and come back to Incubator to finish editing. With Content Translation, you can look at it side by side. The interface is complicated for a new user.
In translatewiki, you can change the interface language, but you can't do that in Incubator. It's English-centric, and it doesn't tolerate multilingualism. If you're not logged in, you can't change from English to Igbo or impossible. If you want to read in other languages in Incubator, that's not possible; you can't change the interface language. The mobile version of Incubator is really hard.
Why do some projects get approved in no time, but others take a lot of time?
Areas for affiliates: Less than 20 projects approved to graduate from Incubator. The Nigerian wiki community has lots of projects in Incubator. How can I get someone from the Nigerian community or other Wikimedia community organizations to give feedback on certain content areas? Sometimes it can be hard to get people from within the community to recruit such individuals. The process around adding wikis to Incubator and wikis graduating from Incubator can benefit from collaboration between the local community and affiliates and other regional Wikimedia community organizations to get feedback on their language project to speed up the process.
When a wiki is migrated from Incubator to a site of its own, some articles are reduced.
It's frustrating to see the contrast between the improvements in Wikipedia and the challenges faced in Incubator. Navigating Incubator involves cumbersome tasks like juggling multiple tabs, making training difficult. Content Translation offers a more streamlined approach for side-by-side editing. Incubator’s interface is complex, especially for new users. Unlike translatewiki, Incubator lacks multilingual support, making it English-centric. Project approval processes vary, raising questions about efficiency. Collaboration with affiliates and regional Wikimedia organizations could expedite the project approval process. When migrating wikis from Incubator, some articles may be reduced.
To discuss specific technical areas that the Wikimedia Foundation should prioritize for the Incubator and identify potential solutions to address challenges within these prioritized technical areas.
The most important aspect, advocated by User:Aaharoni-WMF for years, is the implementation of global templates. When copying an infobox from one wiki to another, it currently requires copying modules and templates, which involve other modules and templates, resulting in a substantial workload. When starting a test wiki on Incubator, you begin fresh. Some building blocks for the technical infrastructure for new wikis would be ideal.
Implementing a basic set of guidelines for the wikis, including principles such as neutral point of view, verifiability, and basic project policies, would make it easier for new wikis to get started.
Establishing good on-wiki communication with the Village Pump page, etc., is essential. Incubator initially catered to individuals accustomed to editing Wikipedia. However, most people who edit now lack editing experience from other wikis. Thus, we must teach them how wikis work.
Areas for affiliates: Outreach to their areas and language communities and helping them connect with people doing stuff. For example, a partnership with the University of Toronto and the Language Diversity Hub, who have experience developing keyboards and apps for phones, tablets, and computers.
The primary focus, advocated by User:Aaharoni-WMF over the years, is the adoption of global templates. Establishing foundational building blocks for the technical infrastructure of new wikis would be advantageous. Implementing basic guidelines for wikis, such as principles of neutrality and verifiability and effective on-wiki communication would facilitate their initiation. Initially tailored to Wikipedia editors, Incubator serves individuals with varying levels of wiki experience, necessitating educational efforts. Outreach to language communities and potential partners holds promise for collaborative solutions in developing keyboards and apps.
To discuss specific technical areas that the Wikimedia Foundation should prioritize for the Incubator and identify potential solutions to address challenges within these prioritized technical areas.
Started working on Incubator in 2017. Not so friendly, compared to main language wikis. Why this is a new place to start a new language if it is so complicated? When you want to add a reference you need to use templates - how many newbies would understand this workflow? No resources available to navigate through Incubator myself.
Technical side of Incubator should be made the focus. Complicated templates that are currently being used, improve on features so it is easy for people to navigate around without need of technical training. Citation template - you have to type it right and then popups and there are different templates around citation. One can easily make a mistake in picking a right template while citing a resource. Data box installation on Incubator from folks like User:Jon Harald Søby seems promising - this allows picking data from Wikidata easily. This is big improvement for Incubator. When I started it, this was not available. Formating features such as bold, italics these features are also not available. If there can be even alternative to data box, which is going to be less technical for users and reduces the installation process, perhaps an improved tool or a built in feature that would be ideal.
Most of us do translation in Translatewiki.net, and see it reflect on Incubator interface. You need some level of expertise to translate on Translatewiki.net. Maybe a glossary of technical terms in Incubator for folks who do the translation. When they say "talk page" or "account" or "file", what do they mean?
Possible solutions could be considering alternative methods to starting a wiki if Incubator is too complex, focusing on improving documentation for Incubator and identifying areas for enhancement and exploring the possibility of direct translation from Incubator rather than Translatewiki.net to streamline the process.
The discussion brought to light concerns about the usability and technical intricacies of the Incubator platform compared to main language wikis. There were frustrations expressed regarding the need for technical proficiency to navigate Incubator, particularly with templates and citation formats. Suggestions included simplifying features and enhancing documentation to increase accessibility. Additionally, there was a suggestion to explore alternative methods for initiating a wiki if Incubator's complexities persist. Another point raised was the necessity for a glossary of technical terms to aid users, especially in translation endeavors. Overall, the emphasis was on finding solutions to address these technical challenges and enhance the user experience on Incubator.
To discuss specific technical areas that the Wikimedia Foundation should prioritize for the Incubator and identify potential solutions to address challenges within these prioritized technical areas.
The Incubator isn't the sole method for language onboarding; it must be considered from a social perspective as well.
Pathways for languages entering Wikimedia haven't been clearly defined yet.
Communities creating a Wikipedia in their language rely on encyclopedic references and knowledge bases, while other wiki projects like Wiktionary may be more relevant for some.
Consider establishing a welcoming page asking questions such as "What Wikimedia activities are happening in your language community?" to orient communities to various forms of contributions available in Wikimedia.
Explore how people currently discover and navigate to the Incubator, aiming to make the process more discoverable, welcoming, and less complex.
Multilingual Wikisource faces similar challenges, with new wikis being created recently with Indonesian languages, yet the creation process after approval is time-consuming and tracking contributions proves difficult.
Beta Wikiversity is also worth examining.
Even communities with their own language wiki may not know how to proceed if they wish to create a new project.
Many new communities lacking resources or the need for a Wikipedia turn to the Incubator, warranting orientation to available pathways during the initial phases.
Testing how Wikidata/Content Translation integration in the Incubator can support languages by working with a few language communities may be beneficial.
The discussion highlighted the need to view language onboarding beyond just the Incubator, emphasizing a holistic approach with a social perspective. The lack of defined pathways for languages entering Wikimedia was acknowledged, and the suggestion of creating a welcoming page to orient communities to the available contributions was made. Additionally, challenges faced by multilingual Wikisource and Beta Wikiversity were explored, along with the need to assist communities in navigating the project creation process. Lastly, the potential benefits of integrating Wikidata/Content Translation into the Incubator to support language communities were considered.
23 wikis got created last year which is a record. This maybe influenced by many factors, especially by the language committee. We weren't able to track and provide support to languages and projects when they are active. Now, we have the Incubator dashboard which is helpful.
Sometimes feels disconnected. Better to be collaborative. Some solutions may require some sacrifice from the community. Creation of wiki requires a team to be responsible for it...depending on a single individual is not sustainable. We have to first have the ownership.
We should also consider the social infrastructure. Example of social infrastructure needed: Understanding what articles need to be written. Also people need help from more experienced editors on how to create articles amongst other things. Some of these things can be automated away by technical solutions such as providing suggestions for articles to write.
Communities have a preference about the type of topics that they want to write but they don't understand that they have to write about diverse set of topics. They don't necessarily understand "why" they need to have a diverse set of topics. Maybe documentation needs to be improved or made more accessible? The approval process is not transparent. An improvement in this process is needed.
Currently ownership for this whole process is missing. Technical expertise is difficult to find within the community. There will be a collaborative effort to identify teams responsible for this process within the Foundation. For example: the Language team can lead the expertise part, the community growth team can lead the social aspect of things.
We want to keep the focus for now on the technical aspect since we want to have the Language team work on this and this is where their experience will help.
Technical question: thinking about the importance of multilingual wikisource and the importance of that for new incubator projects because it provides sources for those folks to use.
Haven't thought about how we will measure impact, but hoping that the researchers / analysts can help us with this. Measurement criteria will vary based on the recommendations that come up.
If we look back over 10 years of innovations, some were accepted very positively, some were criticised. Some features that were accepted positively were related to talk pages were Echo or mentioning user names in talk pages. Even recent talk page improvements such as Reply button (DiscussionTools). This is similar to what we are doing with language incubator because it involves discussing this with relevant stakeholders.
Lot can be learnt from DiscussionTools. There was a big community consultation. Both the consultation and the solution were very well received.
Not a huge community consultation. There have been a lot of community consultation, but a much more focused one. We also have a lot of feedback from various stakeholders. We don't have feedback from "the small wiki monitoring" team. Very quiet but active bunch of groups. Our work will probably affect them.
They monitor for vandalism, and if the way wikis are created might affect them.
Thinking about the timeline. Can we make the whole process more collaborative? Can we publish a draft? If there is feedback, make changes and go through this process a few times.
During the meeting, the objective was to initiate planning discussions and kickoff for the languages onboarding project. Several key points were highlighted, including the record creation of 23 wikis last year, potentially influenced by the language committee. Suggestions were made for collaborative efforts, emphasizing the necessity of shared ownership for sustainable outcomes. Social infrastructure considerations, such as facilitating article creation and improving documentation accessibility, were discussed, along with the importance of diverse topic coverage. Additionally, questions regarding the measurement of impact and lessons learned from past projects were addressed, highlighting the need for research and community consultation. The meeting concluded with a discussion on the timeline and the potential for iterative collaboration to refine the project's approach.
