Parsoid/Deployments/2022

Dec 13 - Dec 15: ✅

 * Fix wikilink parsing issues on redlinks, T309024
 * TOCData: Extract code from Parser.php::finalizeHeadings
 * Protect against long match length in CHAR_REFS_REGEX, T322892
 * Update wikimedia/idle-dom to 1.0.0
 * html2wt debugging: Improve debug tracing to clarify stages
 * Test and comments tweakings, T309024
 * Improve test coverage for round trip testing.
 * build: Updating qs to 6.11.0
 * Fix red link handling with url that contains a fragment, T309024
 * Improve error feedback in RoundTrip.js

Dec 6 - Dec 8: ✅

 * Bump Parsoid HTML version to 2.7.0
 * Add a new TOCData class to hold an array of SectionMetadata objects
 * Add method to check if variant conversion is implemented for language, T318401
 * Bump ecmaVersion to 2018
 * build: Updating composer dependencies
 * build: Updating mediawiki/mediawiki-phan-config to 0.12.0
 * Make Transformhandler accept all Etags.
 * Avoid self:: keyword as string in callables
 * ests: Use more precise PHPUnit assertions
 * Allow captions ending in px, T276675
 * Remove references to 'mw-headline-anchor', T323773

Nov 29 - Dec 1: ✅

 * Rename SectionMetaData -> SectionMetadata; other improvements
 * Add a test for red links in templated references, T323479
 * Remove redlink URL parameters, T323479
 * Fixes data-id for "follow" refs
 * tests: Replace assertEmpty with assertSame
 * Don't require charset parameter in content-type matcher
 * Fix bad assumption in AttributeExpander:splitTokens, T322557
 * build: Migrate PHPUnit from 8.5 to 9.5, T320245
 * Unbreak Parsoid CI: Fix lang variant test with changed expectation
 * Utils::clone: Support limited DOMNode cloning during debugging, T323297
 * Remove redlink normalization from tests, T309024
 * Make tests compatible with redlinks, T309024
 * Align redlink representation between Parsoid and the legacy parser, T309024
 * Revert use of BCP 47 codes in Parsoid's Content-Language header, T323052
 * Update e2e tests after Parsoid REST API changes in core, T322715
 * Split methods to take/return BCP-47 language codes
 * Add methods to convert to/from BCP-47 and MediaWiki-internal language codes
 * Allow use of wikipeg 3.0.0 and PHP 8.1
 * DataAccess: Fix string interpolation, T314096
 * PageHandler: Fix string interpolation, T314096
 * Document where Parsoid is expecting/exposing mediawiki-internal language codes, T320662
 * Turn on phan warnings about the use of deprecated methods
 * Split more language-related tests
 * Remove Dodo from composer.json, T307618
 * Document ordering info for the MigrateTempateMetas DOM pass.
 * Followup on 9145f812: Add test case to prevent future regressions

Nov 10: ✅
In order to break a depdency cycle and fix CI, a branch was made from the v0.17.0-a5 release and a single patch added on top, which allows the use of the wikimedia/wikipeg 3.0.0 release (which in turn adds support for PHP 8.1). No other changes from v0.17.0-a5.

Nov 1 - Nov 3: ✅

 * Revert 1e4bfa13: P-wrapping & MigrateTemplateMetas interact badly
 * Followup to d934a6ee: Improve handling of mixed syntax quote tags
 * ParserTests syncing: Update directions for integrated mode
 * MigrateMeta Followups: Deal with foster box metas
 * Clean up unreachable code in WTUtils.php
 * Ensure DSR computation is accurate if an unclosed comment is present, T284693
 * Move MigrateTemplateMeta pass higher up in the pipeline
 * Remove comments about WTUtils::fosterCommentData
 * Improve hoisting of template meta tags, T306003

Oct 25 - Oct 27: ✅

 * ParserTests: Split off some more test files
 * DOMNormalizer: Add missing diff-mark on merged node, T320319
 * Remove "href missing in a" warning, T321281
 * Sync parserTests with core
 * More splitting of the parserTests file
 * Enable wt2wt in integrated mode for core repo parser tests
 * html2wt: Handle malformed data-mw part without crashing, T319079
 * Add parentheses to clarify logical tests
 * api-testing: e2e test for roundtrip-test in CI, T314244
 * Sync parserTests with core
 * Start splitting parserTests file into multiple files, T111604
 * sync-parserTests.js: Support file renaming via 'oldPath' parameter
 * Sync parserTests with core and all extensions
 * Overhaul parser test file syncing
 * Minor: Convert vars to consts in parserTest syncing scripts
 * Sync *parserTests.text with core
 * Enable integrated mode for media parser test files
 * Followup to d790e6ff: Get rid of invalid check
 * Drop php pre-7.4 compat code
 * Fix PHP_VERSION typo
 * Fix whitespace in Grammar.php
 * Tokens: Rename dataAttribs to clearer dataParsoid name
 * Update regression-testing.php to php 7.4

Oct 11 - Oct 13: ✅

 * Do not generate annotation tokens on  in templates, T319078
 * Make extractPageBundle inner content to be arrays
 * composer.json: Allow psr/container 2, T319055

Oct 4 - Oct 6: ✅

 * Improvements to SectionMetaData class
 * Revert "Disable the GC if running under PHP 7.2", T230861
 * Bump minimum required PHP version to 7.4 in phan
 * Bump minimum required PHP version to 7.4; minimum required MW to 1.40
 * Mark extensions as core dom spec markup compatible
 * Add 'outputHasCoreMwDomSpecMarkup' as option in extension registration, T309024
 * Declare a first-class object for section metadata, T296025
 * extension.json: Add license-name
 * Fix review comments from previous patch, T186241

Sep 13 - Sep 15: ✅

 * Add attributes to Parsoid external links, T186241
 * Enable scalar_implicit_cast phan checks, T316715

Sep 12: ✅

 * Sync media parserTests with core
 * Update extension API to let extensions add temp data on nodes

Aug 30 - Sept 1:✅

 * Restore cleanupFormattingTagFixup pass, T314059
 * Improve handling of "empty" nodes
 * TransformHandler: do not ignore ETags., T310710,  T311819
 * Add nofollow support to SiteConfig & disable test checks for nofollow, T186241
 * Ensure that broken parser functions are still recognized as parser functions, T314524
 * Switch uses of Assert:unreachable to throwing an UnreachableException, T247868

Aug 16 - 18: ✅

 * ParserTests/TestMode: move this class from core to Parsoid
 * ParserTests/Test.php: don't warn if 'selser' undefined in test runner options
 * Internationalization support for Parsoid, T309024
 * Bump content version from 2.5.0 -> 2.6.0
 * Add config support for nowiki stripping for ext args in #tag calls, T299103
 * Update documentation for empty patches
 * TestRunner: actually run html/parsoid+standalone tests
 * rt-testing: Don't post original wikitext for non-selser html2wt
 * Use class property to build trace prefixes in DOMRangeBuilder
 * Don't verify template wrapping expectations in annotation wrapping, T307172
 * Fix dumping feature: write to logger, not error_log
 * Update CLI logging features to log to file
 * Prevent traversing too far up the tree in computeListEltWidth, T313105
 * Log when active formatting elements are reopened inside figures, T314059
 * Remove unnecessary element assertions in ComputeDSR
 * Add loop and muted attributes to media elements, T308230
 * Update baseconfigs

Jul 26 - Jul 28: ✅

 * Followup on d28f22a2: Additional fixes
 * Additional normalization of HTML to improve diff classification
 * parserTests.txt: Add test case for piped link using |, T290526
 * e2e test: cover etag handling with structure test, T311819
 * Allow multivalues in rel attributes, T186241
 * Remove isExtendedAnnotationWrapperTag utility
 * Generalize DOMTraversing with state
 * Implement T2529 sol-nl hack in Parsoid's native template expansion path, T241150

Jul 19 - Jul 21: ✅

 * Implement relative redirects in parsoid extension., T311867
 * composer.json: Allow wikimedia/scoped-callback v4.0.0
 * Add type hints to match built-in interface for php 8.1, T289879
 * Fix bug in Parsoid's Poem implementation, T311953
 * Omit DisplaySpace processing inside pre nodes, T305658
 * Use core's TransformHandler and ParsoidHandler, T310467

Jul 5 - Jul 7: ✅

 * Revert "Only disable nested template expansion when not native"
 * ParserTests: Update hook implementation to match core
 * Fix edge case bug in TableFixups exposed by integrated parsertest run
 * ParserTests: Redlink missing templates. Don't output empty spans.
 * Fix bug in expanding and encapsulating params
 * Only disable nested template expansion when not native
 * TemplateHandler: Document decoupled template processing model
 * Minor: Fix comment by removing stray JS code line

Jun 28 - Jun 30: ✅

 * Bump output content version to 2.5.0, T273505
 * Account for format being in gallery media options, T214601
 * Use caption as alt on imgs when not present and caption isn't visible, T297443,  T162360,  T63566
 * Consolidate media types to mw:File, T273505

Jun 21 - Jun 23: ✅

 * Change DOMTraverser to stay within the subtree of the passed in node
 * ParserTests: Make mocks match output in integraged mode
 * Avoid a call to get file info if there are no valid containers
 * TestRunner.php: Update comments w/r/t interwiki setup
 * TemplateHandler: Check explicitly for an empty string, T310667
 * Stop recognizing noicon and noplayer media options, T134880
 * Copy changes to core api tests to soften schema version check
 * Make end-to-end API test more robust

Jun 14 - Jun 16: ✅

 * Followup on e71d5197: Check for nulls before calling traverse
 * Followup to 0042a88e: Fix broken usage of DOMTraverser, T310220
 * Add forward compatibility to avoid serializing alt from caption, T297443,  T162360,  T63566
 * Strip parser function prefix from content, T310025
 * Get rid of nonsensical complexity in TemplateHandler, T303368
 * Colon escape is already stripped in getWikiLinkTargetInfo
 * Allow colon escape after interwiki prefix, T286425
 * Assure we have a figure if $forceBlock is set when rendering media, T309901

Jun 7 - Jun 9: ✅

 * WTS: Minor cleanup for code readability
 * WTS: Remove a few unnecessary PORT-FIXME comments
 * Only adjust for parser func safesubst in tpl target resolution, T309405
 * Remove priority hints and element timing from test normalization, T308621, T308707
 * Account for multiple active formatting elts reopened in media

May 31 - June 2: ✅

 * StubMetadataCollector: fix "already set" test for MERGE_STRATEGY_WRITE_ONCE
 * AutoLoader: use registerNamespaces instead of $psr4Namespaces
 * Add forward compatibility to serialize mw:File, T273505
 * Set title for tooltips if caption isn't visible, T297443,  T108380
 * SiteConfig: Clear extension cache on extension registration

May 24-26: ✅

 * Improve robustness of tagging misnested link-in-link content, T307412

May 17-19: ✅

 * Remove guard against missing figcaption
 * Set a title attribute for files linking to pages
 * Revert "Lint captions on inline media without an explicit alt options", T297443
 * Hoisting references outside of links, T301293
 * Resolve linter test port-fixme, T237463
 * Use isset to eliminate warnings
 * Cleanup variable naming and documentation for clarity
 * Cleanup confusing halign setting
 * Don't escape extension tags shadowing html5 tags if found in source, T300436
 * Lint html stashed in data-mw of mw:Extension/references, T214994
 * Only lint content defined by a specific ref, T301293,  T214994
 * Prioritize body->html over body->id when diffing refs
 * Fix using sort order from data-mw when serializing template params, T304730
 * Move escapeNowikiTags to WTSUtils
 * Escape dom elements with extension tag names when serializing, T300436

April 26-28: ✅

 * Use DOMRangeBuilder "top level range" algorithm for nested ranges, T306186

Not deployed separately: v0.16.0-a6

 * Lint defined width for framed or manualthumb formats
 * Add lint test for multiple media formats defined, T305641
 * Drop sizing when serializing frame or manualthumb, T64938
 * Manualthumb does not do automatic scaling, T305759,  T64938
 * Manualthumb is a format, like the others, T305641,  T305759
 * Stop using ParsoidServices
 * Add check for bad files, T75581
 * Apply manualthumb consistently across media types, T302114
 * Don't rely on the link element to roundtrip the |page= option, T259931

April 12-14: ✅

 * ParserTests: Don't normalize style attributes
 * ParserTests: Bunch of minor tweaks / refactoring
 * Test.php: Remove Env references + make a function public
 * Run `npm audit fix`
 * ParserTests: Add a normalizeWT helper to Test.php
 * Followup to 11cde591: Update rt-testing node.js scripts as well
 * Remove ServiceWiring and Config from Parsoid extension
 * Don't strip mw:IndentPre meta tags till the very end, T305597
 * Making Translate extended ranges uneditable - fixing bugs, T305066
 * Revert "Adjust whiteline handling around table for extended annotation ranges"
 * ParserTests: Extract HTML normalization to the Test.php class
 * Lint invalid upright options, T216003
 * Drop ip-utils 3.x compatibility, core is on 4.0.0 now
 * Making Translate extended ranges uneditable, T305066
 * Adjust whiteline handling around table for extended annotation ranges

April 9:
Cherry-picked changes since v0.15.0-a26:
 * Making Translate extended ranges uneditable - fixing bugs, T305066
 * Making Translate extended ranges uneditable, T305066
 * Followup to 29a92a69: Fix missing unwrap invocation in PWrap
 * PWrap: Improve p-wrapping behavior to minimize DOM range expansion
 * PWrap: Improve readability by renaming and documenting a helper
 * Make the `null` value clearer in PWrap
 * Revert "Add temporary ParsoidSiteConfigInit hook", T303029

April 5-7: ✅

 * Add a class for unwrapping of optional nodes in dom pwrapper
 * Additional simplification of PreHandler
 * Fix off-by-1 DSR error by cleaning up and simplifying PreHandler, T277415
 * Followup to 29a92a69: Fix missing unwrap invocation in PWrap
 * PWrap: Improve p-wrapping behavior to minimize DOM range expansion
 * Add base constructors for abstract Config types
 * PWrap: Improve readability by renaming and documenting a helper
 * Make the `null` value clearer in PWrap
 * Log extra info if we're unable to get link source when bailing, T302092
 * Use in_array instead of array_search, where possible
 * Followup of bb14815b: Refine TemplateHandler::convertToString

March 30: ✅
Cherry-picked changes since v0.15.0-a25:
 * Add a class on the span representing the broken media element, T304010, T270150
 * Fix complaint that test did not perform any assertions
 * Document known failures testing functionality
 * Drawings are derived and therefore always need the thumbnail treatment
 * Add html/parsoid sections for svg rendering by page language tests, T208620
 * Add baseconfig for ruwiki and refresh existing baseconfigs
 * Support rendering SVG based on page language, T208620
 * Use the check from RevisionRecord::getContent, T273111

Mar 29-31: ✅

 * Improve performance of getAutoUrlTerminatingChars
 * Add badFile to standalone response, T75581
 * Sync mediaParserTests with core
 * Document timing metrics we collect

Mar 22-23: ✅

 * Collect html2wt metrics on non-selser path as well
 * Sync parser tests files with core, T295187
 * Add a class on the span representing the broken media element, T304010, T270150

Mar 22: ✅
Also, the following patches, which were tagged in v0.15.0-a25 but not released:
 * Fix complaint that test did not perform any assertions
 * Document known failures testing functionality
 * Drawings are derived and therefore always need the thumbnail treatment
 * Add html/parsoid sections for svg rendering by page language tests, T208620
 * Add baseconfig for ruwiki and refresh existing baseconfigs
 * Support rendering SVG based on page language, T208620
 * Use the check from RevisionRecord::getContent, T273111
 * Create abstract marker interface for the PageConfigFactory type
 * [take 2] Expose ParserOutput/ContentMetadataCollector via ParsoidExtensionAPI

Unreleased: v0.15.0-a25
This release was tagged in order to allow CI to test patches which moved portions of Parsoid to core.
 * Create abstract marker interface for the PageConfigFactory type
 * [take 2] Expose ParserOutput/ContentMetadataCollector via ParsoidExtensionAPI

Mar 15-17: ✅

 * Allow wikimedia/object-factory ^4.0.0
 * Env isn't needed to dump diff marks
 * Fix dumping diff marks
 * Remove seemingly unnecessary template target resolution
 * Add a class for the return value of the template expand call
 * Construct TemplateEncapsulator earlier and pass it around
 * Split template expansion / preprocessing from encapsulation

Mar 8-10: ✅

 * Rename frame method s/fetchArg/expandArg/
 * PreHandler: s/MULTILINE_PRE/SOL_AFTER_PRE to better reflect state
 * Run ParserBeforePreprocess hook in DataAccess::preprocessWikitext
 * Fix Grammar to not assume that a Token is returned, T303057
 * Do not break heading detection on annotations, T296412
 * html2wt: Use info level for unactionable Cite logspam caused by CX usage
 * DOMDiff: Don't attempt to mark DOM Fragments, T302885
 * Remove obsolete comment from parserTests SiteConfig
 * Make processExtensionModule consistent for ease of reading
 * Fix unsetting native extension tags in parserTests
 * AttributeExpander: tokensToString invalidates srcOffsets
 * AttributeExpander: Update token attributes/tsr correctly after processing, T299374
 * Add mw-file-description class on links to the file description page, T292657
 * Followup to df3124f9: Fix bad refactoring
 * PreHandler: s/popLastNL/pushLastNL
 * PreHandler: Get rid of recursion in onAny handler
 * Clean up TemplateHandler::resolveTemplateTarget

Mar 1-3: ✅

 * Account for strtok boolean return type, T302093
 * Change Config/* classes to be abstract classes instead of interfaces, T261161
 * Move bulk of template arg expansion to the Frame class
 * Add logging of nested annotation tag removals
 * Add description links consistently
 * Don't repeat request for width if file is missing
 * Only define our services if they haven't already been moved to core, T302118
 * Move sanitization to when we're actually setting an external link

Feb 22-24: ✅

 * ContentMetadataCollector: Use PHP constants to store merge strategy strings
 * Tweak signature of ContentMetadataCollector::addCategory
 * Revert part of 55da3090: Setting up template args exposes other bugs, T301948, T302007, T301986
 * Benchmark read view content stripping, T272331
 * Cleanup unnecessary data-parsoid="{}" in parser tests.
 * Minor cleanup in TableFixups & WrapSectionsState for future DRYing
 * Followup to 50461b53: Fixup code updating transclusion parts, T301524
 * Tweak parameters for API requests to add more metadata
 * Don't add synthetic magic words to the output
 * Move ::appendToHead into DOMUtils
 * Move substTopLevelTemplates logic to Core/

Feb 15-17: ✅

 * Minor improvements to the ContentMetadataCollector interface
 * Move ContentModelHandler back to Core/
 * Consolidate some files under src/Wikitext/
 * Get rid of the ExtensionContentModelHandler
 * Move ContentModelHandler stuff from Core/ to Config/
 * SiteConfig handler caching: Use an instance variable vs a static
 * Cache extension tag handlers
 * Fix old wikitext escaping bug, T277592
 * Add preprocessWikitext to ParsoidExtensionAPI
 * Introduce Parsoid\Core\Wikitext to hold wikitext concepts / operations
 * Don't add synthetic categories from template expansion either
 * Ensure frame args are properly set up in the pipeline
 * Get rid of unused noDataAccess mode
 * Reorganize code in TemplateHandler::onTemplate for clarity
 * Don't add synthetic categories to output HTML
 * Get rid of optionalNewlines production from the grammar added in 6f4cd078
 * build: Updating dependencies
 * Moving Translate annotation declaration to the Translate repository, T295170

Feb 8-10: ✅

 * Update Parsoid's CoreMetadataCollector to reflect patches made to core
 * If we aren't expanding templates don't bother with the template name
 * Move convertToString outside of call to encapTokens
 * Simplify TemplateHandler::convertAttribsToString
 * Assert that we aren't expanding an already expanded token, T300393
 * Minor tweak to function name in ParsoidExtensionAPI
 * TemplateHandler: Simplify resolveTemplateTarget
 * Augment the CompoundTemplateInfo to indicate if it came from a param
 * Get rid of retry signal from TokenHandlerResult
 * Don't retry AttributeExpansion on templated attribute tokens, T300393
 * Minor tweaks to lookupArg
 * Set pipeline opts for attribute expander used in TemplateHandler
 * Improve output of regression-testing.php script
 * Simplify bailTokens

Feb 1-3: ✅

 * Add temporary ParsoidSiteConfigInit hook
 * Allow wikimedia/ip-utils ^4.0.0
 * Support lang= option for SVGs, T208620
 * Better handling of templated annotation attributes, T299375
 * Initialize 'inTemplate' pipeline option always and fix usages, T226437
 * ParserPipelineFactory: Remove duplicate call to defaultOptions

Jan 25-27: ✅

 * WikiLinkHandler: Reprocess incorrect tokens to the right stage, T299185,  T299454,  T299457
 * Use MediaWiki's object factory when instantiating extensions
 * Shuttle tokens to the end of the stage

Jan 20: ✅
Parsoid v0.15.0-16 was tagged in order to push out CI changes for T240775/753824 It should be replaced with a thoroughly tested release build before the next train deploy.
 * Simplify DOM class aliases., T240775

Jan 18-20: ✅

 * Clarify a comment around special case for |, T299311
 * Revert "Remove special case for | magic word", T299311
 * Stop emitting "inline-media-caption" lints, T297443, T299302
 * Fix parsing of interlanguage links with spaces, T156308
 * WikitextSerializer: Fix production crasher, T299196
 * Don't log wt2html metrics for linter requests + other tweaks
 * Merge the first two token transform stages

Jan 11-14: ✅
Big wt2html changes
 * Detect auto-insertion and tag deletion in RemexPipeline, T221145
 * Followup to 819630e5: Match handling of mixed-syntax tags
 * Followup to 819630e5: Make endTSR behave like tsr on mw:EndTag metas
 * Followup to 819630e5: Handle autoInsertedFlags on tokens
 * Partially mimic impact of mw:TSRMarker metas on MigrateTrailingNLs

Other notable changes
 * Remove special case for | magic word
 * TSP: Process retokenized tokens through TTM stages 1 & 2, T297922
 * SiteConfig: Address old FIXME related to computing function synonyms
 * TableFixups: Minor perf tweaks and cleanup, T298665
 * TableFixups: Log error on reparse failure, don't crash
 * Logging: Add pages with wt2html timePerKB > 500 ms to slow-parsoid log
 * Add metrics to track time spent linting
 * Add strict_type declaration to a number of files missing them
 * Resolve a few more stale PORT-FIXMEs + update method docs
 * Followup to 78e60013: Eliminate spurious semantic diff reports
 * Replace usages of deprecated wfWikiID, T298059
 * Use ParserOptions::newFromAnon instead of ParserOptions::newCanonical
 * Remove scrubWikitext as an option, T297298
 * Lint captions on inline media without an explicit alt options, T297443