Parsoid
Parsoid — PHP-библиотека, входящая в состав MediaWiki (начиная с версии 1.35), которая используется для преобразования между вики-текстом и HTML. Она разрабатывается с 2012 года, изначально была написана на JavaScript и создана для поддержки VisualEditor . В конечном итоге цель состоит в том, чтобы полностью заменить текущий собственный парсер MediaWiki на Parsoid.
Parsoid
![]() Двухсторонний парсер вики-текста Преобразует в обе стороны вики-текст и HTML/XML DOM с помощью RDFa.
|

- О более старой версии Parsoid, написанной на JavaScript (Node.js), см. Parsoid/JS.
Устаревший парсер всё ещё поддерживается в MediaWiki 1.43 (LTS), но, скорее всего, не будет поддерживаться в следующей LTS.
Технические детали
Parsoid — это приложение, которое может переводить туда и обратно синтаксис вики-текста MediaWiki и эквивалентную модель документа HTML/RDFa с расширенной поддержкой автоматической обработки и богатого редактирования.
Он разрабатывается командой Фонда Викимедиа с 2012 года. В настоящее время он широко используется в VisualEditor , Перевод содержимого и других инструментах.
Parsoid призван обеспечить безупречное прямое и обратное преобразование, то есть избежать потери информации, а также предотвратить «грязные различия».
В вики-проектах Викимедиа для нескольких приложений Parsoid в настоящее время в качестве прокси-сервера используется RESTBase , в котором хранится HTML-код, переведённый Parsoid. Ожидается, что RESTBase со временем будет заменена на кэш, более тесно интегрированный с MediaWiki.
Подробнее о проекте читайте в в этом посте от марта 2013 года. Чтобы узнать об используемой модели HTML, смотрите MediaWiki DOM spec.
Изначально Parsoid был структурирован как веб-сервис и написан на JavaScript с использованием Node.js
.
A tech talk from February 2019 (slides) and blog post describes the process of porting it to PHP.
В настоящее время API расширения Parsoid находится в активной разработке; дневник разработчика от августа 2020 года описывает эту работу.
Репозиторий GitHub: https://github.com/wikimedia/parsoid
Использование
- Parsoid/Releases - Список релизов, выпущенных для Parsoid
- Parsoid/API - для веб-интерфейса
- MediaWiki DOM spec - для осмысления HTML, который вы получаете от API, разработанный для использования в качестве будущего формата хранения данных
- Parsoid/LanguageConverter - Примечания о реализации LanguageConverter Парсоидом
- Parsoid/Known differences with Core Parser output
Установка
Parsoid включен в MediaWiki начиная с версии 1.35. Для включения не требуется никакой настройки.
Parsoid exports an internal REST API which was historically used by RESTBase and not accessible outside the WMF internal cluster. This is no longer required for Visual Editor or core read views, and the internal API is being deprecated and is planned for removal in MW 1.43.
Parsoid is nominally a composer library used by mediawiki core.
Если по каким-то причинам вам все ещё требуется внутренний API, вы можете явно загрузить Parsoid «как расширение», добавив в LocalSettings.php
:
wfLoadExtension( 'Parsoid', "$IP/vendor/wikimedia/parsoid/extension.json" );
Всем остальным сторонним пользователям внутреннего API Parsoid настоятельно рекомендуется перейти на основную конечную точку REST HTML-страницы, которая обеспечивает эквивалентную функциональность.
Разработка
Разработка ведется в Parsoid Git repository. Рецензирование кода происходит в Gerrit. See Gerrit/Getting started to set up an account for yourself.
If you use the MediaWiki-Vagrant development environment using a virtual machine, you can simply add the role visualeditor
to it and it will set up a working Parsoid along with Extension:VisualEditor .
The instructions below are for MediaWiki 1.35 or later. Check Parsoid/JS if you are running the old version of Parsoid written in JavaScript, and used for MW 1.34 and earlier.
Linking a developer checkout of Parsoid
In a standard MediaWiki installation, Parsoid is included from MediaWiki as a composer library, wikimedia/parsoid
.
For development purposes you usually want to use a git checkout of Parsoid, and not the version bundled in MediaWiki core as a composer library. The following lines added to LocalSettings.php allow use of a git checkout of Parsoid (optionally), load the Parsoid REST API with wfLoadExtension (rather than using the version bundled in VisualEditor) and manually do the Parsoid configuration which is usually done by VisualEditor:
$parsoidInstallDir = 'vendor/wikimedia/parsoid'; # bundled copy
#$parsoidInstallDir = '/my/path/to/git/checkout/of/Parsoid';
// For developers: ensure Parsoid is executed from $parsoidInstallDir,
// (not the version included in mediawiki-core by default)
// Must occur *before* wfLoadExtension()
if ( $parsoidInstallDir !== 'vendor/wikimedia/parsoid' ) {
function wfInterceptParsoidLoading( $className ) {
// Only intercept Parsoid namespace classes
if ( preg_match( '/(MW|Wikimedia\\\\)Parsoid\\\\/', $className ) ) {
$fileName = Autoloader::find( $className );
if ( $fileName !== null ) {
require $fileName;
}
}
}
spl_autoload_register( 'wfInterceptParsoidLoading', true, true );
// AutoLoader::registerNamespaces was added in MW 1.39
AutoLoader::registerNamespaces( [
// Keep this in sync with the "autoload" clause in
// $parsoidInstallDir/composer.json
'Wikimedia\\Parsoid\\' => "$parsoidInstallDir/src",
] );
}
wfLoadExtension( 'Parsoid', "$parsoidInstallDir/extension.json" );
# Manually configure Parsoid
$wgVisualEditorParsoidAutoConfig = false;
$wgParsoidSettings = [
'useSelser' => true,
'rtTestMode' => false,
'linting' => false,
];
$wgVirtualRestConfig['modules']['parsoid'] = [
// URL to the Parsoid instance.
// If Parsoid is not running locally, you should change $wgServer to match the non-local host
// While using Docker in macOS, you may need to replace $wgServer with http://host.docker.internal:8080
// While using Docker in linux, you may need to replace $wgServer with http://172.17.0.1:8080
'url' => $wgServer . $wgScriptPath . '/rest.php',
// Parsoid "domain", see below (optional, rarely needed)
// 'domain' => 'localhost',
];
unset( $parsoidInstallDir );
These lines are not necessary for most users of VisualEditor, who can use VisualEditor's auto-configuration and the bundled Parsoid code included in MediaWiki, but they will be required for most developers.
If you're serving MediaWiki with Nginx, you'll need to also add something like this in your server block (Assuming your MediaWiki setup has its files residing in /w/
):
location /w/rest.php/ {
try_files $uri $uri/ /w/rest.php?$query_string;
}
To test proper configuration, visit {$wgScriptPath}/rest.php/{$domain}/v3/page/html/Main%20Page
where $domain
is the hostname in your $wgCanonicalServer
.
(Note that production WMF servers do not expose the Parsoid REST API to the external network.)
Запуск тестов
To run all parser tests and mocha tests:
$ composer test
The parser tests have quite a few options now which can be listed using php bin/parserTests.php --help
.
If you have the environment variable MW_INSTALL_DIR
pointing to a configured MediaWiki installation, you can run some additional tests with:
$ composer phan-integrated
Преобразование простого вики-текста
You can convert simple wikitext snippets from the command line using the parse.php
script in the bin/
directory:
$ echo '[[Foo]]' | php bin/parse.php
The parse script has a lot of options.
php bin/parse.php --help
gives you information about this.
Отладка Parsoid (для разработчиков)
See Parsoid/Debugging for debugging tips.
Непрерывная интеграция
As of October 2021
Parsoid is always available as a library since it is a composer dependency of MediaWiki core. But two pieces are not enabled:
- Parsoid ServiceWiring
- Parsoid's external REST api
The test runner Quibble would enable it if it detects mediawiki/services/parsoid.git
has been cloned as part of the build.
In which case it:
- points the autoloader for
Wikimedia\Parsoid
to the cloned code (effectively replacing the version installed by composer) - Load the extension
wfLoadExtension( 'Parsoid', '/path/to/cloned/repo' );
The ServiceWiring should be enabled in MediaWiki starting with 1.38.
The REST API would theorically never get merged in MediaWiki: a) it has never been exposed to the public in production, it is an internal API used by RESTBase which is going away; b) it never has been security audited and c) it is redundant with the enterprise MediaWiki API. The solution will be for VisualEditor to invoke Parsoid directly via the VisualEditor Action API which would save a round trip through the REST API.
Loading the extension is thus a hack which enables using interfaces subject to change and which we don't really want people to use yet.
For most purposes, parsoid should thus not be added as a CI dependency, the only exception as of October 2021 is the Disambiguator MediaWiki extension.
Loading parsoid as an extension let us run MediaWiki integration test jobs against mediawiki/services/parsoid.git
(such as Quibble, apitesting) and ensure Parsoid and MediaWiki work together.
An extension may be able to write tests with Parsoid even when the repository has not been cloned.
Since it is a composer dependency of MediaWiki core the MediaWiki\Parsoid
namespace is available, but the service wiring part is not (it is extension/src
in the Parsoid repository and exposed as the \MWParsoid
namespace).
The ParsoidTestFileSuite.php
code would only run the parser tests if Parsoid has been loaded (which should be the default with MediaWiki 1.38).
For CI, Parsoid is tested against the tip of mediawiki, whereas mediawiki is tested with the composer dependency. In case of a breaking change, the Parsoid change get merged first (which breaks its CI but not MediaWiki one) and MediaWiki get adjusted when Parsoid is updated. It is thus a one way change.
Release build
For MediaWiki release builds, we have an integration of Parsoid ServiceWiring into VisualEditor in order to have VisualEditor work without further configuration (beside a wfLoadExtension( 'VisualEditor' )
).
The release build also enables the REST API and hook everything us so that parsoid works out of the box.
This is done by copying a bit of parsoid code into VisualEditor which is not in the master branch of VisualEditor since that would be obsolete as soon as Parsoid is updated.
Instead the code is maintained in two places.
Техническая документация
- Parsoid/Internals: documentation about Parsoid internals with links to other details.
- PHP Porting notes and help-wanted tasks
- Parsoid deployment agenda on Wikimedia cluster (code normally deployed every Monday and Wednesday between 1pm - 1:30pm PST)
- Parsoid/Round-trip testing: The round-trip testing setup we are using to test the wikitext -> HTML DOM -> wikitext round-trip on actual Wikipedia content.
- Parsoid/Visual Diffs Testing: Info about visual diff testing for comparing Parsoid's html rendering with php parser's html rendering + a testreduce setup for doing mass visual diff tests.
- Parsoid/limitations: Limitations in Parsoid, mainly contrived templating (ab)uses that don't matter in practice. Could be extended to be similar to the preprocessor upgrade notes (Might need updating)
- Parsoid/Bibliography: Bibliography of related literature
Ссылки для разработчиков Parsoid
- See Parsoid/Debugging for debugging tips.
- Upgrading or adding packages to Parsoid
- See these instructions for syncing Parsoid's copy of parser tests to/from core
- Parsoid has a limited library interface for invoking it programatically.
- Tech Talk about Retargeting extensions to work with Parsoid
- So you want your extension to work with Parsoid
- Parsoid HTML Specification Versioning
- So you are going to change Parsoid output
Ссылки для разработчиков Parsoid (в кластере Викимедиа)
- Parsoid/Deployments
- RT testing commits (useful to check regressions and fixes)
- Deployment instructions for Parsoid
- Kibana dashboard
- Grafana dashboard for wt2html metrics
- Grafana dashboard for html2wt metrics
- Prometheus breakdown for the Parsoid cluster on eqiad
- Prometheus breakdown for the Parsoid cluster on codfw
- Jenkins Job Builder docs for updating jenkins jobs
История
Оригинальное приложение было написано на JavaScript (с использованием Node.js) и начало работать на кластере Викимедиа в декабре 2012 года. В 2019 году Parsoid был портирован на PHP, и в декабре 2019 года PHP-версия заменила JS-версию на кластере Викимедиа. Parsoid в настоящее время интегрируется в ядро MediaWiki, с целью в конечном итоге заменить текущий встроенный парсер MediaWiki. In early 2024, Parsoid began to be used on some production wikis of the Wikimedia Foundation as the default parser for read views.
Parsoid (the PHP version) has been natively bundled with MediaWiki since version 1.35, released in September 2020. For non-Wikimedia installations, Parsoid/JS was supported until the end-of-life of MediaWiki 1.31 (LTS) in September 2021.
См. также
- API
- RESTBase: a caching / storing API proxy for page HTML translated by Parsoid
- Quarterly review meetings of the Parsoid team: April 2015, January 2015 (earlier)
- Parser 2011/Parser plan: Early (now relatively old) design ideas and issues
- Special:PrefixIndex/Parsoid/: Parsoid-related pages on this wiki
- Extension:ParsoidBatchAPI (archived)
- parsoid-jsapi: a high-level interface for extraction and transformation of wikitext, similar to the mwparserfromhell API.
- Alternative parsers
- Parsoid/Parser Unification
Внешние ссылки
- Source code (GitHub mirror)
- JS Documentation (old version of Parsoid)
- PHP Documentation
- Parsoid on the Wikimedia Commons
Контакты
If you need help or have questions/feedback, you can contact us in #mediawiki-parsoid подключиться or the wikitext-l mailing list.
If all that fails, you can also contact us by email at content-transform-team
at the wikimedia.org
domain.
Parsoid is maintained by the Content Transform Team.
Получить помощь
|