Parsoid/Deployments

Planned deployments, linked from wikitech:Deployments. For a list of past deployments, look for 'parsoid' in wikitech:Server Admin Log.

See wikitech:Parsoid#Deploying changes to learn how to deploy a new version of Parsoid.

Monday, Aug 5, 2019 around 1:28pm PT:  Deployed 7232dff

• Only html p-tag is strong indent pre suppressing
• Rename getMagicPatternMatcher to getParameterizedAliasMatcher
• Remove some dead code + simplify getMagicPatternMatcher in WikiConfig.js
• Use SiteConfig::magicWordCanonicalName in WikiLinkHandler::getOptionInfo
• WikiLinkHandler.php: Fix incorrect port of ambiguous JS check
• Work around Remex's failure to normalize DOM during DOM build
• Use a nonambiguous title for parser function frames
• Fix dsr test to check start against null, not 'truthy'
• Parser functions need a frame title too
• Move cached wiki configs to top level directory
• Make Frame::title a proper Title object, not a string
• T228223: Assert KVs aren't found when calling TokenUtils::tokensToString()
• Fix incorrect types (JS & PHP) in DOMFragmentBuilder and related utils
• Ensure Parsoid native template expansion works on recent MediaWiki
• Add addHTMLTemplateParameters options to bin/parse.php
• Enable stage2 hybrid testing in jenkins (and fix some bugs)

Wednesday, July 24, 2019 2:00pm PT:  Deployed df1af404

• T227216: Set top frame's source text when parsing from a stash (this was cherry-picked into the last deploy)
• Fix regression in ru:Fable Legends (RT testing fix)
• Pass frame instead of env in DOM processors and handlers (RT testing fix)
• Convert extTagOffsets to a DomSourceRange
  • Even though this changes data-parsoid, the extTagOffsets property in data-parsoid isn't actually used by any current html2wt code, so this is safe to deploy without a version bump
• Follow up to Gallery: shift TSRs in the DOM
• Gallery: shift TSRs in the DOM, rather than fibbing about srcOffset
• Set srcText during native template expansion
• WikiLinkHandler fixes (mirrored some formatting changes during PHP port over onto the JS side)
• T226523: Test for pipe before trying to resolve target
• T226451: Fix OOM when parsing template (minor JS tweak to maybeExtensionTag in the tokenizer)

Wednesday, July 3, 2019 5:00 pm PT:  Deployed d355bc90 (deploy-20190703 branch)

• T227216: Fix template corruption when reloading stashed wikitext

Wednesday, June 26, 2019 1:35 - 1:56 pm PT:  Deployed 31d356a5

• Ensure that proper source texts are used when parsing (adds assertions)
• Fix case (typeOf vs typeof) in attributes.
  • This ought to be a no-op in JavaScript, which has a case-insensitive HTML DOM
• Other changes to the PHP port which should not affect the JS service

Thursday, June 20, 2019 around 10:34 am PT:  Deployed 4fa8d01

• Cite lint handler: Use nextSibling instead of nextNode
• WTS: Remove some dead code
• Minor cleanup to Utils/DOMPostOrder
• Improve DSR on <figcaption>
• Followup to 2a2d879: Improve DSR computation precision
• Move the Frame class to its own JS file
• Update baseconfigs and add for formatversion=2
• html2wt: A number of fixes to wikitext escaping and regexps
• Tweaks to TemplateHandler
• Make Nowiki html2wt be like all other extensions + tweak behavior
• DOMFragments: Use sealFragment instead of unwrapFragment
• Remove cite-specific leak from cleanup
• Traverse with env
• Fix for native content
• Remove `manager` from JS Frame object
• WikiLinkHandler: Convert width/height to strings
• Fix tokensToString implementation and use sites
• Assert no nulls in tokensToString
• T211251: Fix crasher when encapsulating empty doc
• WikiLinkHandler.buildLinkAttrs always returns an KV[] for contentKVs
• Add Token::getAttributeKV helper

Monday, June 17, 2019 around 1:45 pm PT:  Deployed 2bf94f0 (deploy-2019-06-10 branch)

• T225217: Revert "Stop generating an old dom when none is provided"

Wednesday, May 29, 2019 around 1:28 pm PT:  Deployed 8546c79

• T219927, T211125: Switch logging to rsyslog
• Don't mutate cached values in tokenizer
• Remove dead code
  • From TokenTransformManager.js
  • From TokenUtils.js
• Rename PipelineUtils::buildDOMFragmentTokens to eliminate confusion
• Port LinkHandler
  • Fixes a regex in the JS version to match MediaWiki's idea of whitespace
• Stop emitting section offsets in the pagebundle
• Stop generating an old dom when none is provided
• Explicitly call yargs.options() instead of passing to .usage()
• Bump to service-runner@2.6.19

Wednesday, May 15, 2019 around 1:12 pm PT:  Deployed 6658cad

• Template Wrapping: Change warning to assertion to fail fast
• Some minor html2wt cleanup: Remove prototype injection
• Move to worker-farm@1.7.0 instead of fork
• Simplify SelectiveSerializer constructor
• Fix Util.decodeURI and .decodeURIComponent
• Fix linkTrailRegex
• Convert `extTagWidths` to `extTagOffsets`
• Refactoring in PreHandler:
  • Fix resetState implementation
  • Handle EOFTk in PreHandler onEnd
  • No need to skipOnAny from onAny handler
  • Reset is always called with enableAnyHandler
  • Prefer resetState to resetting in EOFTk

Thursday, May 2, 2019 around 10:45 am PT:  Deployed 9786781

• Simplify nowiki_text rule
• Back port latest Sanitizer::normalizeCss to Parsoid (JS and PHP)
• Only monkey-patch console.assert on Node >= v10
• Remove dead tsrDelta code from UnpackDOMFragments.*
• Remove dead code from TokenUtils:shiftTokensTSR
• Clarify names of offset fields + remove redundant fields

Monday, Apr. 29, 2019 around 1:19 pm PT:  Deployed c9dab9d

wt -> html:

• Prepare DOM before emitting it from the tree builder
• Back port latest Sanitizer::validateCodepoint from core
• Synchronize Sanitizer::setupAttributeWhitelist with core
• Preserve leading/trailing whitespace on invalid templates
• Fix a missing case in TokenUtils.tokenTrim
• T106578, T113194: Ensure PHP and JS are consistent wrt allowed entities
• Apply urltext optimizations to JavaScript tokenizer

html -> wt:

• Treat data-parsoid-diff like data-parsoid & data-mw wrt load/store
• html2wt: Get rid of invalid / unnecessary isString checks
• WTS: Get rid of unnecessary double-newline normalization
• Fix onSOL buglet in html2wt introduced by refactor
• Get rid of the only `nlConstraints.a` use
• T205338: Create a DOMHandler class to ease porting
• Let div fallback to html element handler

Other changes:

• Use `Wt2HtmlResource` and `Html2WtResource` instead of Parser/Serializer
• Linter: Consistently use lowercase tag names in lintObj params
• Fix console.assert when running under node >= 0.10
• T219072: Support splicing more PHP components into the parse pipeline
• Fix serializing fosterable metas after 83c17fc
• T219938: Port HTML5Treebuilder and its test suite to PHP
• T221384, T219943: Update wikipeg to 2.0.2
• Update to WikiPEG 2.0.3
• Update shrinkwrap for wikipeg@2.0.3

Monday, Apr. 15, 2019 around 1:30 pm PT:  Deployed 83c17fc9

• Convert cite extension to es6 class structure
• Remove DOM level 4 check from DOMPostProcessor
• Make extensions with post-processors return constructors
• Various DOMTraverser fixes
• Port HandleLinkNeighbours handler
• Convert HTML5TreeBuilder to es6 class structure
• Fix DOMDataUtils.loadDataAttribs to accept options instead of bool
• Call domino's HTMLParser.insertToken() directly

Wednesday, Apr. 3, 2019 around 1:20 pm PT:  Deployed 0b3bb10

• T212597: Update lib/config/wmf.sitematrix.json
• Convert handlers to es6 class structure
• Simplify addExtLinkClasses DOM pass + port it to PHP
• Fix bad return in onlyinclude handler
• DOMTraverser cleanup
• Organize DOMPostProcessor constructor
• Changes to JS code while porting to PHP:
  • Transfer pwrap DOM pass from the php-prototype branch
  • T219337: Port tokenizer to PHP
    • Update wikipeg version to 2.0.1
    • Add --trace option to inspectTokenizer.js
    • Resolve superficial token stream differences between JS and PHP
  • Make processors pass phan

Tuesday, Mar. 26, 2019 around 10:28 am PT:  Deployed f58c3d1

• Introduce new DOMUtils.{match,has}TypeOf()/{match,has}NameAndTypeOf() helpers
• T219023: html2wt: Fix 'isSimpleLink' detection
• Miscellaneous fixes to entity encoding
• Use comment encoding for tunnelled fosterable content
• Restrict reinsertable fostered content to internal metas
• Let fosterable nodes remain unfostered across serialization boundaries
• Tokenizer efficiency improvements
• Audit uses of Node#getAttribute() + add missing file to PHP codebase
• Fix the regex used when looking for extension end tags

Thursday, Mar. 14, 2019 around 10:24 am PT:  Deployed f3e2209

• Update isXMLTag and isExtTag predicates to match their names
• Consolidate block_tag_opened and xmlish_tag_opened
• T213950: Fixes external links with special characters roundtrip
• Refactor getWikiLinkTargetInfo to accept strings instead of KV
• Address FIXME comments by cloning cache entries before modification

Wednesday, Mar. 13, 2019 around 1:35 pm PT:  Deployed ea80d1b

• PEG rule parameters; Switch from pegjs to wikipeg
• Make transform test runners quiet by default; fix "" handling in KVs
• Protect data-object-id attribute

Thursday, Mar. 7, 2019 around 10:51 am PT:  Deployed d4e76d5

• T202905: Fix new linter category to enable code work with templates
• Tweak storeDataAttribs to suppress DOM nodes in data-parsoid.tmp
• TokenHandler.processTokensSync: Dont pass strings to onTag handler

Monday, Mar. 4, 2019 around 2:05 pm PT:  Deployed 1660395

• Avoid serialize/parse of data attributes when treebuilding
• T214099: Move language conversion work into lib/parse.js
• T214099: Move redlink updating into lib/parse.js
• T202905: Linter.js: Add new function to detect the use of links in links
• templatedepth is either an int or false
• Remove redundant dataParsoid call

Tuesday, Feb. 26, 2019 around 10:17 am PT:  Deployed e82347d

• T204608: Use a bag-on-the-side implementation for node data
• T214099: Bump num_workers to 3
• T217093: Use env.createDocument in lib/api/apiUtils.js
• T214099: Use fork of worker-farm

Wednesday, Feb. 20, 2019 around 1:28 pm PT:  Deployed 9b204a0

• Bump content version to 2.1.0
• T153080, T169975: Add media info in a post-processing pass
  • Remove false assertion that file tokens wouldn't have data-mw
  • T215824: Fix crashers from file in link scenarios
• Skip separators when looking for the next th/td
• DOMDataUtils: Remove return statements from setData* utils
• Assert that the .dataobject isn't touched after storing attrs on a node
• Add some strategic isElt guards
• Simplify and clean up stops usage

Monday, Feb. 11, 2019 around 1:25 pm PT:  Deployed b4b9603 (deploy-2019-02-11 branch)

• Minor JS fixes to make conversion to PHP better
• T208901: Update pwrap.js wrt templatestyles p-wrapping expectations
• T215537: Reduce the batch size for pageprop requests
• T213468: PHP section numbers are assigned during tokenization
• T215638: ListHandler tokens don't need to be special snowflakes either

Wednesday, Feb. 6, 2019 around 1:05 pm PT:  Deployed fb67a71

• Stop producing content version 1.x
• Move bulk of transformTokens code from SyncTTM to TokenHandler
• Improve TokenHandler flags for readability
• Refactor ConstrainedText to make it easier to port
• Backport some improved comments and function names from PHP port.

Thursday, Jan. 24, 2019 around 3:54 pm PT:  Deployed 4772f44

• T214649, T214648: Revert "Get rid of `nativeGallery` option and enable it by default"
• Set `nativeGallery` to `false`

Thursday, Jan. 24, 2019 around 11:03 am PT:  Deployed f1d717f

• Convert several files to use an ES6 class structure
• Remove unnecessary dependency from WikitextSerializer -> escape handlers
• Handle encoded pipes in link's "alt" option
• T187958: Match php parser gallery caption parsing
• Get rid of `nativeGallery` option and enable it by default
• Eliminate use of prevToken from QuoteTransformer
• Always pass an actual boolean (not "undefined" or "null") as `sol` option
• Get rid of unused prevToken arg from token handler signatures
• T205337: Simplify SyncTTM and handlers
• T214103: Instrument language variant conversions
• Work around aggressive exception handling in the tokenizer

Tuesday, Jan. 8, 2019 around 11:00 am PT:  Deployed 2c5dc7b

• T197616: Add test-commons.wikimedia.org
• Tweak QuoteTransformer code + add edgy test specing prevToken arg
• T205491: QuoteTransformer quote tokens don't need to be special snowflakes
• T209772: Add helpers to ease binding context when load/storing data attribs
• T199926: Remove unnecessary pattern from interwiki checks
• Simplify DOMUtils.visitDOM helper
• No need to close over CleanUp.stripMarkerMetas
• Use escapeIdForExternalInterwiki when rendering interwiki links
• Remove `figureHandlerImpl`
• Convert NodeList to Array in `addRedLinksG`
• Refactor tokenizeSync signature to avoid potentially ignoring args
• Stop leaking manager (an impl. detail) to extensions

Code refactoring

• T209194: Export one class per file for various things
• T204622: Convert various things to use ES6 class syntax
• Migrate handlers out of DOMPostProcessor into their own files
• Rename Normalizer to DOMNormalizer and update file name to match

Deployments from previous years

• Parsoid/Deployments/2013
• Parsoid/Deployments/2014
• Parsoid/Deployments/2015
• Parsoid/Deployments/2016
• Parsoid/Deployments/2017
• Parsoid/Deployments/2018