Wiki Techstorm/Programme/Regex/Notes

Regex workshop during Techstorm 2019 notes

Regex

Regular expressions
Not a programming language
A way to describe a pattern of text

Example: matching dates of birth

\bg\. .+?\s(\d{1,2}\.\d{1,2}\.)?\d{4}

Matches dates of birth in a book of biographies:

g. <town> dob:(D)D.(M)M.YYYY
g. Utretch 12.4.1878 
g. Utretcht 7.12.1895
g. Maastricht 1867

A more precise version:

\bg\. .+?\s(\d{1,2}\.\d{1,2}\.)?1[89]\d\d

Ensures the year starts with 18 or 19.

End-of-entry pattern

\(Adres:.+\n?.*\)

Escapes parentheses
Note: the dot (.) does not match newlines by default

Structure of entries

<name> <occupation> <place of birth and date of birth> <blurb>

Basic tokens

. = any character
\. = literal dot
\b = word boundary
\s = whitespace
\d = digit
\w = word character
\n = newline

Breakdown of example regex

\b      word boundary
g       literal "g"
\.      literal dot
.+?     any char, non-greedy
\s      whitespace
(\d{1,2}\.\d{1,2}\.)? optional day+month
\d{4}   year

Where regex is used

VisualFileChange.js documentation on Wikimedia Commons

 (requires regex between slashes: /.../)

Google Docs
MediaWiki search & replace
Code
OpenRefine
PHP (uses delimiters like /.../)
Python (no delimiters needed)

Resources:

Techstorm 2019 Wikidata and OpenRefine programme page

Handy tools

Microsoft Excel Fuzzy Lookup add-in

Quantifiers

* = zero or more
? = zero or one
+ = one or more
{} = specific min/max

Groups

() = group
(X)? = optional group
(X|Y|Z) = alternatives
(?!X|Y|Z) = not these
(apple) \1 = backreference

Example:

\bg\. .+?\s((\d{1,2}\.\d{1,2}\.)?\d{4})

Group 1 = full date
Group 2 = day + month

Backreferences

In OpenRefine:

value.replace(/(de|van)\s(.+)/, '$2')

$1, $2 refer to groups
Sometimes \1 is used instead

Commons example

/.+?Foto: (.+)\/re:publica\n\| Source = (.+)\n\| Date = (.+)\n\|Author = /

Replacement:

| Source = $2
| Date = $3
| Author = $1

Character classes

[^A] = not A
[A-Z] = uppercase letters
[a-z] = lowercase letters

Example:

Br[eo][ao]d

Matches: Breod, Bread, Broad, Brood

Anchors

^ = start of line
$ = end of line

Example: surname extraction

((van|de|van de)\s)?\w+$

Greediness

Regex is greedy by default.

Example:

adres\s=.+\

Non-greedy:

adres\s=.+?\

Better:

adres\s=.+?(\d)

Sublime Text tips

Toggle regex: button or Alt+R
Select all matches: Alt+Enter
Multi-cursor editing supported

Examples:

^(.+)\n\1

Removes duplicate consecutive lines.

\s$

Matches trailing whitespace.

Commons API

Wikimedia Commons API documentation

Bookmarklet

WebRef citation bookmarklet Automatically creates cite web references