Parsoid/Deployments/2022

Dec 13 - Dec 15:  V0.17.0-a9 as part of 1.40.0-wmf.14

• Fix wikilink parsing issues on redlinks, T309024
• TOCData: Extract code from Parser.php::finalizeHeadings
• Protect against long match length in CHAR_REFS_REGEX, T322892
• Update wikimedia/idle-dom to 1.0.0
• html2wt debugging: Improve debug tracing to clarify stages
• Test and comments tweakings, T309024
• Improve test coverage for round trip testing.
• build: Updating qs to 6.11.0
• Fix red link handling with url that contains a fragment, T309024
• Improve error feedback in RoundTrip.js

Dec 6 - Dec 8:  V0.17.0-a8 as part of 1.40.0-wmf.13

• Bump Parsoid HTML version to 2.7.0
• Add a new TOCData class to hold an array of SectionMetadata objects
• Add method to check if variant conversion is implemented for language, T318401
• Bump ecmaVersion to 2018
• build: Updating composer dependencies
• build: Updating mediawiki/mediawiki-phan-config to 0.12.0
• Make Transformhandler accept all Etags.
• Avoid self:: keyword as string in callables
• ests: Use more precise PHPUnit assertions
• Allow captions ending in px, T276675
• Remove references to 'mw-headline-anchor', T323773

Nov 29 - Dec 1:  V0.17.0-a7 as part of 1.40.0-wmf.12

• Rename SectionMetaData -> SectionMetadata; other improvements
• Add a test for red links in templated references, T323479
• Remove redlink URL parameters, T323479
• Fixes data-id for "follow" refs
• tests: Replace assertEmpty with assertSame
• Don't require charset parameter in content-type matcher
• Fix bad assumption in AttributeExpander:splitTokens, T322557
• build: Migrate PHPUnit from 8.5 to 9.5, T320245
• Unbreak Parsoid CI: Fix lang variant test with changed expectation
• Utils::clone: Support limited DOMNode cloning during debugging, T323297
• Remove redlink normalization from tests, T309024
• Make tests compatible with redlinks, T309024
• Align redlink representation between Parsoid and the legacy parser, T309024
• Revert use of BCP 47 codes in Parsoid's Content-Language header, T323052
• Update e2e tests after Parsoid REST API changes in core, T322715
• Split methods to take/return BCP-47 language codes
• Add methods to convert to/from BCP-47 and MediaWiki-internal language codes
• Allow use of wikipeg 3.0.0 and PHP 8.1
• DataAccess: Fix string interpolation, T314096
• PageHandler: Fix string interpolation, T314096
• Document where Parsoid is expecting/exposing mediawiki-internal language codes, T320662
• Turn on phan warnings about the use of deprecated methods
• Split more language-related tests
• Remove Dodo from composer.json, T307618
• Document ordering info for the MigrateTempateMetas DOM pass.
• Followup on 9145f812: Add test case to prevent future regressions

Nov 10:  V0.17.0-a6

In order to break a depdency cycle and fix CI, a branch was made from the v0.17.0-a5 release and a single patch added on top, which allows the use of the wikimedia/wikipeg 3.0.0 release (which in turn adds support for PHP 8.1). No other changes from v0.17.0-a5.

Nov 1 - Nov 3:  V0.17.0-a5 for 1.40.0-wmf.8

• Revert 1e4bfa13: P-wrapping & MigrateTemplateMetas interact badly
• Followup to d934a6ee: Improve handling of mixed syntax quote tags
• ParserTests syncing: Update directions for integrated mode
• MigrateMeta Followups: Deal with foster box metas
• Clean up unreachable code in WTUtils.php
• Ensure DSR computation is accurate if an unclosed comment is present, T284693
• Move MigrateTemplateMeta pass higher up in the pipeline
• Remove comments about WTUtils::fosterCommentData
• Improve hoisting of template meta tags, T306003

Oct 25 - Oct 27:  V0.17.0-a4 for 1.40.0-wmf.7

• ParserTests: Split off some more test files
• DOMNormalizer: Add missing diff-mark on merged node, T320319
• Remove "href missing in a" warning, T321281
• Sync parserTests with core
• More splitting of the parserTests file
• Enable wt2wt in integrated mode for core repo parser tests
• html2wt: Handle malformed data-mw part without crashing, T319079
• Add parentheses to clarify logical tests
• api-testing: e2e test for roundtrip-test in CI, T314244
• Sync parserTests with core
• Start splitting parserTests file into multiple files, T111604
• sync-parserTests.js: Support file renaming via 'oldPath' parameter
• Sync parserTests with core and all extensions
• Overhaul parser test file syncing
• Minor: Convert vars to consts in parserTest syncing scripts
• Sync *parserTests.text with core
• Enable integrated mode for media parser test files
• Followup to d790e6ff: Get rid of invalid check
• Drop php pre-7.4 compat code
• Fix PHP_VERSION typo
• Fix whitespace in Grammar.php
• Tokens: Rename dataAttribs to clearer dataParsoid name
• Update regression-testing.php to php 7.4

Oct 11 - Oct 13:  V0.17.0-a3 for 1.40.0-wmf.5

• Do not generate annotation tokens on </> in templates, T319078
• Make extractPageBundle inner content to be arrays
• composer.json: Allow psr/container 2, T319055

Oct 4 - Oct 6:  V0.17.0-a2 for 1.40.0-wmf.4

• Improvements to SectionMetaData class
• Revert "Disable the GC if running under PHP 7.2", T230861
• Bump minimum required PHP version to 7.4 in phan
• Bump minimum required PHP version to 7.4; minimum required MW to 1.40
• Mark extensions as core dom spec markup compatible
• Add 'outputHasCoreMwDomSpecMarkup' as option in extension registration, T309024
• Declare a first-class object for section metadata, T296025
• extension.json: Add license-name
• Fix review comments from previous patch, T186241

Sep 13 - Sep 15:  V0.17.0-a1 for 1.40.0-wmf.1

• Add attributes to Parsoid external links, T186241
• Enable scalar_implicit_cast phan checks, T316715

Sep 12:  V0.16.0-a21 for 1.39 RC

• Sync media parserTests with core
• Update extension API to let extensions add temp data on nodes

Aug 30 - Sept 1: V0.16.0-a20 as part of 1.39.0-wmf.27

• Restore cleanupFormattingTagFixup pass, T314059
• Improve handling of "empty" nodes
• TransformHandler: do not ignore ETags., T310710, T311819
• Add nofollow support to SiteConfig & disable test checks for nofollow, T186241
• Ensure that broken parser functions are still recognized as parser functions, T314524
• Switch uses of Assert:unreachable() to throwing an UnreachableException, T247868

Aug 16 - 18:  V0.16.0-a19 as part of 1.39.0-wmf.25

• ParserTests/TestMode: move this class from core to Parsoid
• ParserTests/Test.php: don't warn if 'selser' undefined in test runner options
• Internationalization support for Parsoid, T309024
• Bump content version from 2.5.0 -> 2.6.0
• Add config support for nowiki stripping for ext args in #tag calls, T299103
• Update documentation for empty patches
• TestRunner: actually run html/parsoid+standalone tests
• rt-testing: Don't post original wikitext for non-selser html2wt
• Use class property to build trace prefixes in DOMRangeBuilder
• Don't verify template wrapping expectations in annotation wrapping, T307172
• Fix dumping feature: write to logger, not error_log
• Update CLI logging features to log to file
• Prevent traversing too far up the tree in computeListEltWidth, T313105
• Log when active formatting elements are reopened inside figures, T314059
• Remove unnecessary element assertions in ComputeDSR
• Add loop and muted attributes to media elements, T308230
• Update baseconfigs

Jul 26 - Jul 28:  V0.16.0-a17 as part of 1.39.0-wmf.22

• Followup on d28f22a2: Additional fixes
• Additional normalization of HTML to improve diff classification
• parserTests.txt: Add test case for piped link using |, T290526
• e2e test: cover etag handling with structure test, T311819
• Allow multivalues in rel attributes, T186241
• Remove isExtendedAnnotationWrapperTag utility
• Generalize DOMTraversing with state
• Implement T2529 sol-nl hack in Parsoid's native template expansion path, T241150

Jul 19 - Jul 21:  V0.16.0-a16 as part of 1.39.0-wmf.21

• Implement relative redirects in parsoid extension., T311867
• composer.json: Allow wikimedia/scoped-callback v4.0.0
• Add type hints to match built-in interface for php 8.1, T289879
• Fix bug in Parsoid's Poem implementation, T311953
• Omit DisplaySpace processing inside pre nodes, T305658
• Use core's TransformHandler and ParsoidHandler, T310467

Jul 5 - Jul 7:  V0.16.0-a15 as part of 1.39.0-wmf.19

• Revert "Only disable nested template expansion when not native"
• ParserTests: Update <tag> hook implementation to match core
• Fix edge case bug in TableFixups exposed by integrated parsertest run
• ParserTests: Redlink missing templates. Don't output empty spans.
• Fix bug in expanding and encapsulating params
• Only disable nested template expansion when not native
• TemplateHandler: Document decoupled template processing model
• Minor: Fix comment by removing stray JS code line

Jun 28 - Jun 30:  V0.16.0-a14 as part of 1.39.0-wmf.18

• Bump output content version to 2.5.0, T273505
• Account for format being in gallery media options, T214601
• Use caption as alt on imgs when not present and caption isn't visible, T297443, T162360, T63566
• Consolidate media types to mw:File, T273505

Jun 21 - Jun 23:  V0.16.0-a13 as part of 1.39.0-wmf.17

• Change DOMTraverser to stay within the subtree of the passed in node
• ParserTests: Make <indicator> mocks match output in integraged mode
• Avoid a call to get file info if there are no valid containers
• TestRunner.php: Update comments w/r/t interwiki setup
• TemplateHandler: Check explicitly for an empty string, T310667
• Stop recognizing noicon and noplayer media options, T134880
• Copy changes to core api tests to soften schema version check
• Make end-to-end API test more robust

Jun 14 - Jun 16:  V0.16.0-a12 as part of 1.39.0-wmf.16

• Followup on e71d5197: Check for nulls before calling traverse
• Followup to 0042a88e: Fix broken usage of DOMTraverser, T310220
• Add forward compatibility to avoid serializing alt from caption, T297443, T162360, T63566
• Strip parser function prefix from content, T310025
• Get rid of nonsensical complexity in TemplateHandler, T303368
• Colon escape is already stripped in getWikiLinkTargetInfo
• Allow colon escape after interwiki prefix, T286425
• Assure we have a figure if $forceBlock is set when rendering media, T309901

Jun 7 - Jun 9:  V0.16.0-a11 as part of 1.39.0-wmf.15

• WTS: Minor cleanup for code readability
• WTS: Remove a few unnecessary PORT-FIXME comments
• Only adjust for parser func safesubst in tpl target resolution, T309405
• Remove priority hints and element timing from test normalization, T308621, T308707
• Account for multiple active formatting elts reopened in media

May 31 - June 2:  V0.16.0-a10 as part of 1.39.0-wmf.14

• StubMetadataCollector: fix "already set" test for MERGE_STRATEGY_WRITE_ONCE
• AutoLoader: use registerNamespaces instead of $psr4Namespaces
• Add forward compatibility to serialize mw:File, T273505
• Set title for tooltips if caption isn't visible, T297443, T108380
• SiteConfig: Clear extension cache on extension registration

May 24-26:  V0.16.0-a9 as part of 1.39.0-wmf.13

• Improve robustness of tagging misnested link-in-link content, T307412

May 17-19:  V0.16.0-a8 as part of 1.39.0-wmf.12

• Remove guard against missing figcaption
• Set a title attribute for files linking to pages
• Revert "Lint captions on inline media without an explicit alt options", T297443
• Hoisting references outside of links, T301293
• Resolve linter test port-fixme, T237463
• Use isset to eliminate warnings
• Cleanup variable naming and documentation for clarity
• Cleanup confusing halign setting
• Don't escape extension tags shadowing html5 tags if found in source, T300436
• Lint html stashed in data-mw of mw:Extension/references, T214994
• Only lint content defined by a specific ref, T301293, T214994
• Prioritize body->html over body->id when diffing refs
• Fix using sort order from data-mw when serializing template params, T304730
• Move escapeNowikiTags to WTSUtils
• Escape dom elements with extension tag names when serializing, T300436

April 26-28:  V0.16.0-a7 as part of 1.39.0-wmf.9

• Use DOMRangeBuilder "top level range" algorithm for nested ranges, T306186

Not deployed separately: v0.16.0-a6

• Lint defined width for framed or manualthumb formats
• Add lint test for multiple media formats defined, T305641
• Drop sizing when serializing frame or manualthumb, T64938
• Manualthumb does not do automatic scaling, T305759, T64938
• Manualthumb is a format, like the others, T305641, T305759
• Stop using ParsoidServices
• Add check for bad files, T75581
• Apply manualthumb consistently across media types, T302114
• Don't rely on the link element to roundtrip the |page= option, T259931

April 12-14:  V0.16.0-a5 as part of 1.39.0-wmf.7

• ParserTests: Don't normalize style attributes
• ParserTests: Bunch of minor tweaks / refactoring
• Test.php: Remove Env references + make a function public
• Run `npm audit fix`
• ParserTests: Add a normalizeWT helper to Test.php
• Followup to 11cde591: Update rt-testing node.js scripts as well
• Remove ServiceWiring and Config from Parsoid extension
• Don't strip mw:IndentPre meta tags till the very end, T305597
• Making Translate extended ranges uneditable - fixing bugs, T305066
• Revert "Adjust whiteline handling around table for extended annotation ranges"
• ParserTests: Extract HTML normalization to the Test.php class
• Lint invalid upright options, T216003
• Drop ip-utils 3.x compatibility, core is on 4.0.0 now
• Making Translate extended ranges uneditable, T305066
• Adjust whiteline handling around table for extended annotation ranges

April 9: v0.15.0-a27 on REL1_38 branch

Cherry-picked changes since v0.15.0-a26:

• Making Translate extended ranges uneditable - fixing bugs, T305066
• Making Translate extended ranges uneditable, T305066
• Followup to 29a92a69: Fix missing unwrap invocation in PWrap
• PWrap: Improve p-wrapping behavior to minimize DOM range expansion
• PWrap: Improve readability by renaming and documenting a helper
• Make the `null` value clearer in PWrap
• Revert "Add temporary ParsoidSiteConfigInit hook", T303029

April 5-7:  V0.16.0-a4 as part of 1.39.0-wmf.6

• Add a class for unwrapping of optional nodes in dom pwrapper
• Additional simplification of PreHandler
• Fix off-by-1 DSR error by cleaning up and simplifying PreHandler, T277415
• Followup to 29a92a69: Fix missing unwrap invocation in PWrap
• PWrap: Improve p-wrapping behavior to minimize DOM range expansion
• Add base constructors for abstract Config types
• PWrap: Improve readability by renaming and documenting a helper
• Make the `null` value clearer in PWrap
• Log extra info if we're unable to get link source when bailing, T302092
• Use in_array instead of array_search, where possible
• Followup of bb14815b: Refine TemplateHandler::convertToString

March 30:  V0.15.0-a26 on REL1_38 branch

Cherry-picked changes since v0.15.0-a25:

• Add a class on the span representing the broken media element, T304010, T270150
• Fix complaint that test did not perform any assertions
• Document known failures testing functionality
• Drawings are derived and therefore always need the thumbnail treatment
• Add html/parsoid sections for svg rendering by page language tests, T208620
• Add baseconfig for ruwiki and refresh existing baseconfigs
• Support rendering SVG based on page language, T208620
• Use the check from RevisionRecord::getContent, T273111

Mar 29-31:  V0.16.0-a3 as part of 1.39.0-wmf.5

• Improve performance of getAutoUrlTerminatingChars
• Add badFile to standalone response, T75581
• Sync mediaParserTests with core
• Document timing metrics we collect

Mar 22-23:  V0.16.0-a2 as part of 1.39.0-wmf.4

• Collect html2wt metrics on non-selser path as well
• Sync parser tests files with core, T295187
• Add a class on the span representing the broken media element, T304010, T270150

Mar 22:  V0.16.0-a1 as part of 1.39.0-wmf.3

• Fix complaint that test did not perform any assertions
• Document known failures testing functionality
• Drawings are derived and therefore always need the thumbnail treatment
• Add html/parsoid sections for svg rendering by page language tests, T208620
• Add baseconfig for ruwiki and refresh existing baseconfigs
• Support rendering SVG based on page language, T208620
• Use the check from RevisionRecord::getContent, T273111

Also, the following patches, which were tagged in v0.15.0-a25 but not released:

• Create abstract marker interface for the PageConfigFactory type
• [take 2] Expose ParserOutput/ContentMetadataCollector via ParsoidExtensionAPI

Unreleased: v0.15.0-a25

This release was tagged in order to allow CI to test patches which moved portions of Parsoid to core.

• Create abstract marker interface for the PageConfigFactory type
• [take 2] Expose ParserOutput/ContentMetadataCollector via ParsoidExtensionAPI

Mar 15-17:  V0.15.0-a24 as part of 1.38.0-wmf.26

• Allow wikimedia/object-factory ^4.0.0
• Env isn't needed to dump diff marks
• Fix dumping diff marks
• Remove seemingly unnecessary template target resolution
• Add a class for the return value of the template expand call
• Construct TemplateEncapsulator earlier and pass it around
• Split template expansion / preprocessing from encapsulation

Mar 8-10:  V0.15.0-a23 as part of 1.38.0-wmf.25

• Rename frame method s/fetchArg/expandArg/
• PreHandler: s/MULTILINE_PRE/SOL_AFTER_PRE to better reflect state
• Run ParserBeforePreprocess hook in DataAccess::preprocessWikitext
• Fix Grammar to not assume that a Token is returned, T303057
• Do not break heading detection on annotations, T296412
• html2wt: Use info level for unactionable Cite logspam caused by CX usage
• DOMDiff: Don't attempt to mark DOM Fragments, T302885
• Remove obsolete comment from parserTests SiteConfig
• Make processExtensionModule consistent for ease of reading
• Fix unsetting native extension tags in parserTests
• AttributeExpander: tokensToString invalidates srcOffsets
• AttributeExpander: Update token attributes/tsr correctly after processing, T299374
• Add mw-file-description class on links to the file description page, T292657
• Followup to df3124f9: Fix bad refactoring
• PreHandler: s/popLastNL/pushLastNL
• PreHandler: Get rid of recursion in onAny handler
• Clean up TemplateHandler::resolveTemplateTarget

Mar 1-3:  V0.15.0-a22 as part of 1.38.0-wmf.24

• Account for strtok boolean return type, T302093
• Change Config/* classes to be abstract classes instead of interfaces, T261161
• Move bulk of template arg expansion to the Frame class
• Add logging of nested annotation tag removals
• Add description links consistently
• Don't repeat request for width if file is missing
• Only define our services if they haven't already been moved to core, T302118
• Move sanitization to when we're actually setting an external link

Feb 22-24:  V0.15.0-a21 as part of 1.38.0-wmf.23

• ContentMetadataCollector: Use PHP constants to store merge strategy strings
• Tweak signature of ContentMetadataCollector::addCategory()
• Revert part of 55da3090: Setting up template args exposes other bugs, T301948, T302007, T301986
• Benchmark read view content stripping, T272331
• Cleanup unnecessary data-parsoid="{}" in parser tests.
• Minor cleanup in TableFixups & WrapSectionsState for future DRYing
• Followup to 50461b53: Fixup code updating transclusion parts, T301524
• Tweak parameters for API requests to add more metadata
• Don't add synthetic magic words to the output
• Move ::appendToHead() into DOMUtils
• Move substTopLevelTemplates logic to Core/

Feb 15-17:  V0.15.0-a20 as part of 1.38.0-wmf.22

• Minor improvements to the ContentMetadataCollector interface
• Move ContentModelHandler back to Core/
• Consolidate some files under src/Wikitext/
• Get rid of the ExtensionContentModelHandler
• Move ContentModelHandler stuff from Core/ to Config/
• SiteConfig handler caching: Use an instance variable vs a static
• Cache extension tag handlers
• Fix old wikitext escaping bug, T277592
• Add preprocessWikitext to ParsoidExtensionAPI
• Introduce Parsoid\Core\Wikitext to hold wikitext concepts / operations
• Don't add synthetic categories from template expansion either
• Ensure frame args are properly set up in the pipeline
• Get rid of unused noDataAccess() mode
• Reorganize code in TemplateHandler::onTemplate for clarity
• Don't add synthetic categories to output HTML
• Get rid of optionalNewlines production from the grammar added in 6f4cd078
• build: Updating dependencies
• Moving Translate annotation declaration to the Translate repository, T295170

Feb 8-10:  V0.15.0-a19 as part of 1.38.0-wmf.21

• Update Parsoid's CoreMetadataCollector to reflect patches made to core
• If we aren't expanding templates don't bother with the template name
• Move convertToString outside of call to encapTokens
• Simplify TemplateHandler::convertAttribsToString
• Assert that we aren't expanding an already expanded token, T300393
• Minor tweak to function name in ParsoidExtensionAPI
• TemplateHandler: Simplify resolveTemplateTarget
• Augment the CompoundTemplateInfo to indicate if it came from a param
• Get rid of retry signal from TokenHandlerResult
• Don't retry AttributeExpansion on templated attribute tokens, T300393
• Minor tweaks to lookupArg
• Set pipeline opts for attribute expander used in TemplateHandler
• Improve output of regression-testing.php script
• Simplify bailTokens

Feb 1-3:  V0.15.0-a18 as part of 1.38.0-wmf.20

• Add temporary ParsoidSiteConfigInit hook
• Allow wikimedia/ip-utils ^4.0.0
• Support lang= option for SVGs, T208620
• Better handling of templated annotation attributes, T299375
• Initialize 'inTemplate' pipeline option always and fix usages, T226437
• ParserPipelineFactory: Remove duplicate call to defaultOptions

Jan 25-27:  V0.15.0-a17 as part of 1.38.0-wmf.19

• WikiLinkHandler: Reprocess incorrect tokens to the right stage, T299185, T299454, T299457
• Use MediaWiki's object factory when instantiating extensions
• Shuttle tokens to the end of the stage

Jan 20:  V0.15.0-a16 to beta

Parsoid v0.15.0-16 was tagged in order to push out CI changes for phab:T240775/gerrit:753824 It should be replaced with a thoroughly tested release build before the next train deploy.

• Simplify DOM class aliases., T240775

Jan 18-20:  V0.15.0-a15 as part of 1.38.0-wmf.18

• Clarify a comment around special case for {{!}}, T299311
• Revert "Remove special case for {{!}} magic word", T299311
• Stop emitting "inline-media-caption" lints, T297443, T299302
• Fix parsing of interlanguage links with spaces, T156308
• WikitextSerializer: Fix production crasher, T299196
• Don't log wt2html metrics for linter requests + other tweaks
• Merge the first two token transform stages

Jan 11-14:  V0.15.0-a14 as part of 1.38.0-wmf.17

Big wt2html changes

• Detect auto-insertion and tag deletion in RemexPipeline, T221145
• Followup to 819630e5: Match handling of mixed-syntax tags
• Followup to 819630e5: Make endTSR behave like tsr on mw:EndTag metas
• Followup to 819630e5: Handle autoInsertedFlags on tokens
• Partially mimic impact of mw:TSRMarker metas on MigrateTrailingNLs

Other notable changes

• Remove special case for {{!}} magic word
• TSP: Process retokenized tokens through TTM stages 1 & 2, T297922
• SiteConfig: Address old FIXME related to computing function synonyms
• TableFixups: Minor perf tweaks and cleanup, T298665
• TableFixups: Log error on reparse failure, don't crash
• Logging: Add pages with wt2html timePerKB > 500 ms to slow-parsoid log
• Add metrics to track time spent linting
• Add strict_type declaration to a number of files missing them
• Resolve a few more stale PORT-FIXMEs + update method docs
• Followup to 78e60013: Eliminate spurious semantic diff reports
• Replace usages of deprecated wfWikiID(), T298059
• Use ParserOptions::newFromAnon instead of ParserOptions::newCanonical
• Remove scrubWikitext as an option, T297298
• Lint captions on inline media without an explicit alt options, T297443