Parsoid/Parser Unification/Confidence Framework

The Parsoid Usage Confidence Framework is aimed at evaluating the readiness and efficacy of enabling Parsoid on diverse wikis. This guide delineates the steps and methodologies for operating the framework to ensure thorough evaluations and informed decisions regarding Parsoid deployments.

The output of this framework should not force us to use "cold numbers" to deploy Parsoid for the sake of achieving the goal. We should use to enquiry more profound product questions.

Metrics

These are the current metrics we can realistically use for decision making:

Supported Extensions Coverage

Visual diff coverage

Qualitative Reports

As the framework evolves and gaps are discovered we might want to look at the following metrics:

Linting errors
- Linting errors might be a good indicator to triage low risk wikis to explore but it's still debatable whether it would be useful for confidence assessment

Number of bugs on the critical path

Parsoid metrics

Performance and Parsoid missing functionality in affected pages
- Performance and missing functionalities are moving targets. Current parsoid metrics should be evaluated across the board, it's not clear what would be the value if we can differentiate per wiki so this will be out of scope in the initial experiments.

Confidence Score

Based on the combination of the metrics we should always have an answer for enabling Parsoid Read Views for every wiki:

Not confident at all

Not confident

Somewhat confident
- That means this wiki can be considered the next target and we should investigate the metrics

Confident

Extremely confident

With that we will be able to look at the reasons for not being confident with Parsoid and focus on its characteristics and architecture with a Product Strategy lens to mitigate the gaps that affect our confidence to deploy Parsoid for Read Views.

Goals

Turn on parsoid on self-contained and safe places to develop the framework for "Parsoid Usage Readiness".

Immediate Goals

Deploy Parsoid in controlled and safe environments to enable learning and development of the "Parsoid Usage Readiness Framework"
- Globally for specific wikis
  - First candidates are Wikitech DiscussionTools and Officewiki Read Views
  - Next English Wikivoyage DiscussionTools
  - Later Hebrew Wikivoyage DiscussionTools
- Allow users to opt-in to Parsoid in any wiki by using the ParserMigration extension

Next wikis

Wiki Name

Visual Diff Coverage (%)

Supported Extensions Coverage (%)

Confidence

Reason

Status

Office Wiki

N/A (Visual Diff can't run for private wikis)

100%

Confident

Deployed

Wikitech DiscussionTools

98.55% pixel perfect; 99.87% px perfect and minor vertical whitespace shifts; 0.096% contains known differences we won't fix; 0.032% contain real diffs to be fixed before roll-out

Extensions critical for Wikitech DiscussionTools are presumed to be identified in the visual diff process

Extremely Confident

Report

Deployed

English Wikivoyage

99.7% pixel perfect or vertical whitespace shifts.

Extensions critical to wikivoyage are presumed to be identified in the visual diff process.

Extremely Confident

Report

Deployed

Hebrew Wikivoyage

99.4% pixel perfect or vertical whitespace shifts.

Extensions critical to wikivoyage are presumed to be identified in the visual diff process.

Very confident

Report

Deployed

Romanian Wikivoyage

98.7% pixel perfect or vertical whitespace shifts.

Extensions critical to wikivoyage are presumed to be identified in the visual diff process.

Very confident

Report

Deployed

Czech Wikivoyage

99.5% pixel perfect or vertical whitespace shifts.

Extensions critical to wikivoyage are presumed to be identified in the visual diff process.

Very confident

Report

Deployed

Communication

Open questions

Decide how many features we need to turn on to avoid community discontent

This is not about having 100% of wikitext features done, but does coverage matter?

What will be the target metrics to get parsoid read views on testwiki?

What is the amount of wikitext that users focus on DiscussionTools?