Parsoid/Deployments

Planned deployments, linked from Deployments. For a list of past deployments, look for 'parsoid' in Server Admin Log.

See Parsoid to learn how to deploy a new version of Parsoid.

Friday, Nov. 30, 2018 around 12 pm PT: ✅

 * T194083: Add 'inserted' diff-marker to the correct node

Wednesday, Nov. 28, 2018 around 1:16 pm PT: ✅

 * html2wt normalization: Fix subtle bug skipping templated content
 * T209236, T210437: Uniformly use "wikitext entities" not "HTML5 entities"
 * Support more link types in file alt/link options
 * T184755: Scrub DOM to convert   sequences to NL-emitting normal forms
 * T187142: Dedupe template style occurrences
 * Fix tracing apirequest
 * Consistently apply data attributes to fragment wrapper
 * Convert pre extension to using toDOM instead of tokenHandler
 * Convert gallery extension to using toDOM instead of tokenHandler
 * Make sanitizing and applying attributes to nodes a helper
 * T206777, T205710, T205546, T204477: Add 4 new wikis
 * T208470: html2wt: Handle } in table cells properly
 * T207286: Added high resolution timer functionality to parsoid

Wednesday, Nov. 7, 2018 around 1:54 pm PT: ✅

 * Use proper `new` and splat syntax for ES6.
 * T206940: Image alt and link options can contain arbitrary wikitext which is stripped
 * Support --useBatchAPI=false option in parserTests, and make tests pass
 * Use formatversion=2 for ImageInfo requests
 * Handle JSON formatversion=2 for parse/preprocess/batch API requests

Monday, Nov. 5, 2018 around 1:41 pm PT: ✅

 * Update sitematrix
 * T205334: Remove some dead code
 * T208360: Split Utils and DOMUtils into smaller chunks based on functionality
 * Cleanup time trace output some more
 * Added dom dumping options for all DOM passes
 * Refactor content version handling and use better names
 * Fix potential content version inconsistency in apiUtils.wt2htmlRes

Monday, Oct. 29, 2018 around 1:35 pm PT: ✅

 * T100841: Allow users to dynamically configure new wikis
 * Stop always using a span representation for inline dom fragments
 * T186965, T167349: Treat template-styles tag specially wrt p-wrapping
 * T186965, T198618: Strip / tags in TOC
 * Time tracing:
 * Trace API requests
 * T206040: Capture async tokenizing time
 * Add some assertions where stops aren't popped
 * T207956: Accept spaces at the end of table_start_tag
 * Account for trailing newlines from parse requests for templatestyles

Wednesday, Oct. 17, 2018 around 1:40 pm PT: ✅

 * T204622: Use ES6 classes and subclasses
 * Update `var` to ES6 `let`/`const`
 * T187848: Fix token transformer return types
 * Tighten up (and document) the constraints on transformer returns
 * ParagraphWrapper: Transformers should always return a tokens array
 * Cleanups to TokenTransformManager
 * TokenHandler: move initialization to constructor instead of init method
 * Remove use of Function#bind in LinkHandler
 * Export extension registration method on WikiConfig
 * Remove deprecated ParsoidConfig#setInterwiki/removeInterwiki methods
 * Remove strictAcceptCheck config
 * Lower language converter warning levels to info
 * T207093: Stop suppressing aborting when recursively tokenizing includes
 * Separate sync tokenizing calls that expect error suppression

Monday, Oct. 15, 2018 around 1:32 pm PT: ✅

 * T205642: Handle callers of parseWikitextToDOM with non-string attribute keys
 * T206003: Set request agent after updating uri with mwApiServer
 * Cite leakage
 * Remove more Cite leakage from html2wt code
 * Rename parsing pipeline options to something more semantic
 * Remove another source of Cite leaking in html2wt code
 * T187848: Fix token transformer return types
 * No special token array handling in AsyncTTM
 * Clean up token handling in SyncTTM
 * Ensure that we always use {tokens:[...]} form in TTM.
 * Ensure ParserFunctions flattens its token arrays.
 * Ensure LinkHandler flattens its token arrays.
 * Always return `tokens` array in token transformers.
 * Ensure LanguageConverter markup flattens token arrays in attributes
 * No need to check for extinct `{token:...}` form in TTM
 * bin/transformTests.js
 * Count failures during transformTests and exit with non-zero status if failed
 * T205455: Emit tests only when the token is transformed
 * Updates to keep up with TTM changes
 * T205743: Replaced token type 'COMMENT' with 'CommentTk'
 * Make bin/transformTests.js executable

Tuesday, Oct. 2, 2018 around 11:01 am PT: ✅

 * T163438: Fix the logic (and comments) when skipping the funnel into mwApiServer
 * ConfigRequest doesn't need a separate proxy argument
 * Remove the possibility of proxy being set on request options
 * T205674: Don't migrate template start markers past auto-inserted table ends
 * T205673: Pass through inTemplate pipeline options when handling language variant

Monday, Oct. 1, 2018 around 1:45 pm PT: ✅

 * Bump content version to 2.0.0 + add downgrades to 1.8.0
 * Set strictAcceptCheck in prod
 * T198504, T133673: Use elements for rendering audio files
 * HTTP API:
 * T202666: Downgrade original content when serializing where possible
 * Add some metrics around downgrading

Thursday, Sep. 27, 2018 around 1:09 pm PT: ✅ (deploy-2018-09-27 branch)

 * Forward compatibility for audio tags
 * Add some mocha tests around serializing 2.0.0 content

Thursday, Sep. 27, 2018 around 11:04 am PT: ✅ (deploy-2018-09-27 branch)

 * Update domino to 2.1.0
 * Reversion the data-mw split as 999.0.0
 * T198511: Use `resource` attribute for Media:.... links
 * T163438: Respect the proxy overriding even when going through the mwApiServer
 * Rename config / file names to clarify wmf-specific configurations
 * Make QuoteTransformer a proper ES6 class
 * Avoid using JS Function#bind in QuoteTransformer
 * Avoid the use of Array.prototype.METHOD
 * Domino 2.0.3+ supports case-insensitive matches in querySelector
 * Remove one source of Cite leakage in generic html2wt code
 * T108776: Update sitematrix and remove special casing for labswiki
 * T205334: Remove unused token toString and associated unused (tracing) code
 * HTTP API:
 * Updates should return the original version
 * Separate updates from reparse / up/downgrades
 * Update the version in the head when downgrading
 * T114413: Ensure that Content-Language and Vary headers are not "undefined"
 * Avoid string concat for function names
 * Use uppercase for HTTP header names

Wednesday, Sep. 5, 2018 around 1:19 pm PT: ✅

 * Effectively revert
 * Only add trailing ws to table cell if preceded by another cell on the row
 * Rewrite include_limits in terms of xmlish_tag
 * T198400, T202819: Update sitematrix.json
 * Collect configs for extensions in a single object in WikiConfig
 * Tweak extension config a bit more

Tuesday, Aug. 28, 2018 around 11:01 am PT: ✅

 * Bump content version to 1.8.0
 * T199246: Expose content inside to editors via data-mw attribute
 * Tweak extension DOM config for extensibility
 * T198221: Support directionality for reference

Monday, Aug. 20, 2018 around 1:26 pm PT: ✅

 * Updates to the Parsoid Extension API
 * Cite: Get rid of tokenHandler that relies on Parsoid internals (and associated bug fixes)
 * T130224: Enable extensions to distinguish between self-closed and empty refs
 * Set data-mw.body to undefined instead of null for extension tags
 * Get rid of unused extTagId parser option
 * Remove Cite-specific flag from parser pipeline options
 * Consolidate meta type regexp in AE and stop matching "Extension"
 * Rename wrapTemplates to expandTemplates to clarify its use
 * T199926: html -> wt: Parsoid sometimes trips up on | chars in hrefs
 * Move uri sanitizing utils to the Sanitizer
 * URI decode interwiki titles before sanitizing

Thursday, Aug. 16, 2018 around 11:47 am PT: ✅ (deploy-2018-08-16 branch)

 * T201115: Prepare for MCR

Monday, Aug. 6, 2018 around 1:15 pm PT: ✅

 * Post-tidy-replacement: Add detection for new lint
 * LanguageConverter: switch to byte-oriented state machines
 * Improve byte machine format
 * T199849: Improve templatedata spec compliance wrt leading and trailing newlines
 * T198400, T199577, T199509: Update sitematrix.json
 * T200403: Add a fast path to avoid unnecessarily retokenizing the extlink href
 * Use .getAttribute instead of Util.lookup where possible
 * T200403: Test for a valid protocol before attempting to tokenize extlink content
 * Convert error classes in ApiRequest to proper ES6 classes
 * Don't permit < in html attribute positions
 * T201054: Ignore solidus (/) in various attribute tokenizing states

Thursday, July 26, 2018 around 10:59 am PT: ✅

 * Remove pre_indent rule from the grammar
 * Update domino and jsdoc-wmf-theme
 * Use DOM-based p-wrapping of text found on same wikitext line as block nodes
 * T194806: Pare down some things in paragraph wrapping
 * p-wrapping code on DOM
 * T110004: Move to DOM based wrapping of bare text
 * Remove meta markers before handling empty elements
 * Close paragraphs when entering indent pre
 * Account for mw:(Start|End)Tags in isRenderingTransparentNode
 * Account for about attributes on empty paragraphs from templates
 * Don't force paragraphs inside blockquotes to serialize on a new line
 * Don't migrate trailing nls from fostered content under any condition
 * Update edge-case handling in migrateTemplateMarkerMetas
 * Quote transformer is tripped up on html table cells
 * Make migrateTemplateMarkerMetas more robust
 * Drop auto-inserted empty elements before p-wrapping
 * Migrate template marker metas past auto-inserted start/end tags
 * pwrap: Update auto-inserted start/end flags during tree splits
 * Remove auto-inserted empty elements after meta marker cleanup
 * T156099, T188478: Updated extension interface + native Parsoid impl of Poem
 * T194083: Followup on : Refine that original check

Monday, July 23, 2018 around 1:22 pm PT: ✅ (deploy-2018-07-23 branch)

 * T199808: Update foundationwiki url
 * T194083: Dump debugging info for nested inserted dom-diff flags error

Monday, June 25, 2018 around 11:20 am PT: ✅

 * Implement pb2pb endpoint for language variant conversion
 * T197949: Ensure Parsoid doesn't crash on unimplemented/invalid language variant conversions
 * T197702: Correct Vary header in pb2pb endpoint for language variant conversion
 * T196799: Crasher fix: Set inTemplate for pipelines where it was missing, add further protection from missing data-mw parts
 * Cleanups:
 * Move _findHeaders to DOMUtils
 * Split ExtensionHandler from TemplateHandler
 * Get rid of unused _startDocumentPipeline and recordArgDict state
 * Convert DU.addRedLinks and TemplateRequest.setPageSrcInfo to Promise.async

Monday, June 11, 2018 around 1:49 pm PT: ✅

 * Set last nltk when encountering a block while pre collecting
 * Stop shimming array includes
 * T191843: TRY #2: Fix inefficiency in SyncTokenTransformManager
 * Get rid of pointless DU.hasNodeName helper

Thu, June 7, 2018, around 11:10 am PT: ✅

 * T183706, T192726, T194879, T196357, T196360: Update sitematrix to activate newer wikis
 * Suppress p-wrapping of fostered content to match the php parser
 * T43716: Basic implementation of Serbian/Kurdish/Pig Latin LanguageConverter support
 * This functionality is not yet fully-compliant with PHP (or supported); it is being deployed in order to unblock the services team work on T159985.

Monday, June 4, 2018 around 1:34 pm PT: ✅
wt2html


 * T157418: Trim whitespace in wikitext headings, lists, tables
 * is a rendering transparent node
 * Fix incorrect pipeline options in AttributeExpander (subtle old bug)
 * T194777: Migrate trailing categories out of dt/dd tags as well
 * T195317: Update migrateTrailingCategories pass to handle comments
 * Add rudimentary tracing to QuoteTransformer
 * Set DSR on mw:FallbackId spans
 * T195174: Protect indented table syntax from indent pre parsing
 * T195174: Protect the pipe variable as well from indent pre
 * T194659: Fix off by one in attribute expander
 * T195227: Check for own properties when using object as a dict
 * T195171: Fix confusion when attribute expander is used for template targets

html2wt


 * T194763: Whitespace scrubbing should be sensitive to SOL-sensitive markup
 * Ensure diff markers aren't lost during normalizations
 * T194658: Empty td and th cells are serialized with a single whitespace char
 * Followup to eae8f0c0: Delete spurious normalization change
 * T195414: html2wt: Strip mw:FallbackId spans before DOM diffs
 * Skip deletion markers while counting children
 * T195486: Minimize whitespace dirty diffs in existing headings, tables, lists
 * T195486, T195414: WS-minimization heuristics only apply to comments & text nodes
 * T195414: Suppress whitespace heuristics for HTML versions < 1.7.0
 * Redo emitting of optional trailing space for  s and  s
 * html2wt: Get rid of useless mw:FallbackId checks

Other


 * Bump HTML version to 1.7.0
 * Upgrade request for node security advisory 664
 * Remove some pre-node-v6 compatibility code

Tuesday, May 29, 2018 around 11:04 am PT: ✅
Again, because of T195414: Whitespace normalized outside of modified tree

Tuesday, May 29, 2018 around 10:36 am PT: ✅ Reverted as above
wt2html


 * T157418: Trim whitespace in wikitext headings, lists, tables
 * is a rendering transparent node
 * Fix incorrect pipeline options in AttributeExpander (subtle old bug)
 * T194777: Migrate trailing categories out of dt/dd tags as well
 * T195317: Update migrateTrailingCategories pass to handle comments
 * Add rudimentary tracing to QuoteTransformer
 * Set DSR on mw:FallbackId spans
 * T195174: Protect indented table syntax from indent pre parsing
 * T195174: Protect the pipe variable as well from indent pre
 * T194659: Fix off by one in attribute expander
 * T195227: Check for own properties when using object as a dict
 * T195171: Fix confusion when attribute expander is used for template targets

html2wt


 * T194763: Whitespace scrubbing should be sensitive to SOL-sensitive markup
 * Ensure diff markers aren't lost during normalizations
 * T194658: Empty td and th cells are serialized with a single whitespace char
 * Followup to eae8f0c0: Delete spurious normalization change
 * T195414: html2wt: Strip mw:FallbackId spans before DOM diffs
 * Skip deletion markers while counting children
 * T195486: Minimize whitespace dirty diffs in existing headings, tables, lists

Other


 * Upgrade request for node security advisory 664
 * Remove some pre-node-v6 compatibility code

Wednesday, May 23, 2018 around 1:56 pm PT: ✅
Reverted because of T195414: Heading whitespace normalized outside of modified tree

Wednesday, May 23, 2018 around 1:18 pm PT: ✅ Reverted as above
wt2html


 * T157418: Trim whitespace in wikitext headings, lists, tables
 * is a rendering transparent node
 * Fix incorrect pipeline options in AttributeExpander (subtle old bug)
 * T194777: Migrate trailing categories out of dt/dd tags as well
 * T195317: Update migrateTrailingCategories pass to handle comments
 * Add rudimentary tracing to QuoteTransformer
 * Set DSR on mw:FallbackId spans
 * T195174: Protect indented table syntax from indent pre parsing

html2wt


 * T194763: Whitespace scrubbing should be sensitive to SOL-sensitive markup
 * Ensure diff markers aren't lost during normalizations
 * T194658: Empty td and th cells are serialized with a single whitespace char
 * Followup to eae8f0c0: Delete spurious normalization change

Other


 * Upgrade request for node security advisory 664
 * Remove some pre-node-v6 compatibility code

Thursday, May 17, 2018 around 10:15 am PT: ✅ (deploy-2018-05-16 branch)

 * T194821: Whitelist wikibase-lexeme content model
 * T194687: s/doNotIgnoreMissingTitles/includeMissingTitles/ in TemplateDataRequests

Monday, May 14, 2018 around 1:54 pm PT: ✅

 * T194082: TRY #2: normalizeDOM: normalize a node after it is merged with a sibling
 * T194083: Distinguish between inserted & deleted diff markers
 * T194084: More fun with non-string template targets
 * Code cleanup and refactoring:
 * Refactor & simplify couple DU helpers
 * Simplify emitsSolTransparentSingleLineWT DOM helper
 * Add DU.isRenderingTransparentNode helper
 * Cleanup normalizeSiblingPair for readability

Wednesday, May 9, 2018 around 1:17 pm PT: ✅

 * T194081: Fix crasher in moveFormatTagOutsideATag
 * T188118: Suppress autoInsertedEnd flags where not required
 * Update some dependencies

Monday, May 7, 2018 around 1:41 pm PT: ✅

 * T192909: Unconditionally strip section tags
 * Cleanup section tags
 * The `html5-legacy` mode of ID generation has been removed from core
 * Return NlTk instead of "\n" in tokenizer almost everywhere
 * PEG: Inline table_line rule & rename table_lines to table_line
 * Parse more block constructs in media captions

Monday, April 30, 2018 around 1:21 pm PT: ✅

 * normalizeDOM: normalize a node after it is merged with a sibling
 * T186358: Increase the default height mw:Audio to 32px
 * Fix newline handling in p-wrapper when # newlines > 2
 * T191700: Avoid piping links for formatting when possible
 * T192909: Sections that expand tpl ranges lose roundtrip data when dropped
 * Update sitematrix

Thursday, April 12, 2018 around 11:02 am PT: ✅

 * Allow DOMTraverser handlers to specify a context (w/o an explicit bind)
 * Sync addExtLinkClasses with new PHP strictly-syntactic rules
 * Stop adding valign classes to block media
 * T191839: Fix crasher in normalizeDOM.js
 * Revert "Fix inefficiency in SyncTokenTransformManager"

Monday, Apr. 9, 2018 around 1:32 pm PT: ✅

 * Fix inefficiency in SyncTokenTransformManager
 * T191281: Serialize gallery image captions in single line context
 * Avoid cloning result token arrays
 * T187386: Protect against data-mw w/o parts when section wrapping
 * T185266: Prevent crashing on foundationwiki pages
 * Make the decodeURI / decodeURIComponent distinction more apparent
 * Code cleanup (without changes to functionality):
 * Rename Util.getArgInfo to Util.getExtArgInfo
 * Get rid of immediately invoked function
 * Make the Sanitizer "static" and decouple it from the parsing pipeline

Wednesday, Apr. 4, 2018 around 1:18 pm PT: ✅

 * Get rid of nested_block_line rule
 * Account for failure to make a major version bump for sections
 * Handle indented tables with content following it
 * Don't unwrap p-wrapping around -style wikitext
 * html2wt:
 * T177102: Add new lines before and after lists in wikitext
 * T189474: Image captions ignore paragraph tags
 * Extra newline now suppressed after a final table tag

Monday, Mar. 12, 2018 around 1:37 pm PT: ✅
when we need to emit that HTML tag
 * T188670: Fix even further crashes from non-string template targets
 * T90902: Remove &amp;nbsp; and similar characters from section IDs
 * Minor: html2wt: Emit

Monday, Mar. 5, 2018 around 1:25 pm PT: ✅

 * The `normalize` post processor isn't needed any more
 * Remove type from _startTokenPipeline
 * Suppress bogus warning about "variant" parameter to action=parse
 * T188591: xmlish tags in wikilink title position are invalid
 * Use the same string for the href in linkToMedia that got us there
 * BODY is not a block tag

Wednesday, Feb. 28, 2018 around 1:26 pm PT: ✅

 * Bump patch level of content version to 1.6.1
 * Bug fix in Async Token Transform Manager
 * T58756: external links class= now setting free, text and autonumber
 * DOM Passes: add option to enable/disable it on nested DOMs
 * Unbreak time tracing on nested DOM passes
 * T169006: Redirect to the right path
 * Handle exception from title-invalid-talk-namespace
 * Don't process token attributes unnecessarily
 * Don't allocate attribute arrays if not required

Monday, Feb. 26, 2018 around 1:42 pm PT: ✅

 * Make Token "a real base class instead of some weird hackish object"
 * Use an ES6 class for wt2html/tt/TokenHandler
 * Use Promise.async/yield in env#getParserEnv / env#switchToConfig
 * Clean up:
 * Minor cleanup in AttributeExpander
 * Eliminate unnecessary property from state in TemplateHandler
 * Fix verifyTokensIntegrity
 * Simplify accumChain code in Async TTM
 * Upgrade eslint
 * Improve jsdoc

Tuesday, Feb. 20, 2018 around 10:34 am PT: ✅

 * Update some dependencies
 * Clean up:
 * remove Batcher#onLegacyResponse
 * Move escapeRegExp from Util to JSUtil
 * Remove wrapperStyles dead code
 * Remove old defensive code
 * Speed up fast path (no logging) of Logger
 * Use Promise.async/yield in lib/ext/Cite
 * Use Promise.async/yield in lib/ext/Nowiki

Thursday, Feb. 15, 2018 around 10:42 am PT: ✅

 * LinkHandler: Inline the invocation in #getWikiLinkHandler
 * WTS: Use Promise.async/yield
 * wt2html/tt/LinkHandler: get rid of setupDataMW
 * Promisify AttributeExpander transformation
 * Bump to domino@2.0.1
 * Fix performance regression caused by

Wednesday, Feb. 14, 2018 around 1:30 pm PT: ✅

 * Hoist Page out of the MWParserEnvironment class
 * Get rid of the handleUnbalancedTables DOM pass
 * Replace `.bind(this)` and `var self = this;` with arrow functions

Thursday, Feb. 8, 2018 around 10:27 am PT: ✅

 * T186630: Don't try to bump parser resource use if setPageSrcInfo is throwing exceptions
 * AttributeTransformManager#process now returns a Promise
 * Replace callbacks with Promises in Logger
 * Remove remaining uses of async module

Tuesday, Feb. 6, 2018 around 10:15 am PT: ✅

 * T183515: Permit extension tags in xmlish attribute values
 * Consolidate bailing out from ext/wiki links
 * T129372: Match core's parsing of gallery dimensions
 * T181408: Rethink responsive references wrappers
 * tsr end is only valid when bailing out if there's no content
 * Use autoGenerated property when deciding on selser reuse
 * Use Promise.async/yield in various places
 * Have expandValuesToDOM return a Promise
 * Add `TokenTransformManager.addTransformP` helper
 * Bunch of minor cleanup in Frame.expand & TTM
 * Allow Frame#expand to return a Promise
 * Fix implementation of padleft/padright

Monday, Jan. 29, 2018 around 1:23 pm PT: ✅ (deploy-2018-01-29 branch)

 * T185643: Strip everything before the first pf token
 * T185346: Stop clearing tplInfo if don't find a table for fostered content
 * T185385: Fix crasher from rendering self-closed html pre tags
 * T185267: Fix infinite recursion from linting named ref cycles
 * Fix a bug linting content from templated extension tags with lint handlers

Thursday, Jan. 18, 2018 around 1:03 pm PT: ✅

 * Enable useWorker

Thursday, Jan. 18, 2018 around 11:17 am PT: ✅

 * T45094: Replace with for references
 * Deep clone expression result before modifying
 * Fix crasher from tokenizing headings

Tuesday, Jan. 16, 2018 around 10:30 am PT: ✅

 * Update sitematrix
 * Assert supported options for the parser pipeline cache key
 * Restrict nested refs not coming from templates
 * Give some time for async loggers to deliver fatal messages
 * Add a function to create the references node uniformly
 * Improve handling of section-wrapping and extcontent boundary conflicts
 * Linter:
 * Fix crasher in Cite - don't assume well-formedness

Monday, Jan. 8, 2018 around 1:22 pm PT: ✅

 * T182349: Account for SOL transparent templates in p-wrapping
 * Only use max depth detection since we aren't doing template expansion
 * Move addRedLinks to DOMPostProcessor; allow async processors
 * T43716, T159985: Create skeleton of language variant support in Parsoid API
 * Add node-pn package
 * Linter:
 * Simplify logic a bit for readability + fix file comment
 * T183893: Linter: Suppress multiple-unclosed-formatting-tags false positives
 * Don't crash while detecting fostered content
 * Don't emit missing-end-tag warnings for builder-inserted tags

Tuesday, Jan. 2, 2018 around 10:21 am PT: ✅

 * T143378: Template syntax w/ empty link target shouldn't be considered broken
 * Use a Set for trace/debug/dump flags
 * Log api warnings from batch requests
 * Pass the page name when parsing extension content
 * Cleanup and refactor so mw api warnings are always logged
 * T110961: Use pure compute workers for the request processing (disabled by default in the config)
 * T123446: Bring back request timeouts (disabled by default in the config)
 * Linter:
 * Suppress false positives for multiline-table-in-list
 * Additional updates to multiline-html-table-in-list
 * Suppress false positives for multiple-unclosed-formatting-tags
 * Move linting pass to the very end
 * Make linter its own self-contained DOM pass
 * Use extension mechanism to provide ext-specific lint handers