Extension:AbuseFilter/Rules format

From MediaWiki.org
Jump to navigation Jump to search
Other languages:Project:Language policy English  • 日本語 • 한국어 • मराठी • русский • 中文

The rules are formatted much as conditionals in a C/Java/Perl-like language.

Strings

You can specify a literal by placing it in single or double quotes (for strings), or by typing it in as-is (for numbers, both floating-point and integer). You can get linebreaks with \n, tab characters with \t, and you can also escape the quote character with a backslash.

Use the + (plus) symbol to concatenate two literal strings or the values of two vars with a string value.

Examples
"This is a string"
'This is also a string'
'This string shouldn\'t fail'
"This string\nHas a linebreak"
1234
1.234
-123

User-defined variables

You can define custom variables for ease of understanding with the assign symbol := in a line (closed by ;) within a condition. Such variables may use letters, underscores, and numbers (apart from the first character) and are case sensitive. Example (from w:en:Special:AbuseFilter/79):

(
	line1:="(\{\{(r|R)eflist|\{\{(r|R)efs|<references\s?/>|</references\s?>)";
	rcount(line1, removed_lines)
) > (
	rcount(line1, added_lines)
)

Arrays

AbuseFilter has support for non-associative arrays, which can be used like in the following examples.

my_array := [ 5, 6, 7, 10];
my_array[0] == 5
length(my_array) == 4
string(my_array) == "5\n6\n7\n10\n" //Note: the last linebreak will be removed soon
5 in my_array == true
'5' in my_array == true
'5\n6' in my_array == true //Note: this is due to how arrays are casted to string, i.e. by imploding them with linebreaks
1 in my_array == true //Note: this happens because 'in' casts arguments to strings, so the 1 is catched in '10' and returns true.

Comments

You can specify comments using the following syntax:

/* This is a comment */

Arithmetic

You can use basic arithmetic symbols to do arithmetic on variables and literals with the following syntax:

  • - — Subtract the right-hand operand from the left-hand operand.
  • + — Add the right-hand operand to the left-hand operand.
  • * — Multiply the left-hand operand by the right-hand operand.
  • / — Divide the left-hand operand by the right-hand operand.
  • ** — Raise the left-hand operand to the exponential power specified by the right-hand operand.
  • % — Return the remainder given when the left-hand operand is divided by the right-hand operand.

The type of the returned result is the same that would be returned by PHP, for which a lot of documentation may be found online. More exhaustive examples may be found in this AF parser test.

Example Result
1 + 1 2
2 * 2 4
1 / 2 0.5
9 ** 2 81
6 % 5 1

Boolean operations

You can match if and only if all of a number of conditions are true, one of a number of conditions are true, or one and only one of all conditions are true.

  • x | y — OR – returns true if one or more of the conditions is true.
  • x & y — AND – returns true if both of the conditions are true.
  • x ^ y — XOR – returns true if one, and only one of the two conditions is true.
  • !x — NOT – returns true if the condition is not true.

Examples

Code Result
1 | 1 true
1 | 0 true
0 | 0 false
1 & 1 true
1 & 0 false
0 & 0 false
1 ^ 1 false
1 ^ 0 true
0 ^ 0 false
!1 false

Simple comparisons

You can compare variables with other variables and literals with the following syntax:

  • < and >—Return true if the left-hand operand is less than/greater than the right-hand operand respectively.
  • <= and >=—Return true if the left-hand operand is less than or equal to/greater than or equal to the right-hand operand respectively.
  • == (or =) and !=—Return true if the left-hand operand is equal to/not equal to the right-hand operand respectively.
  • === and !==—Return true if the left-hand operand is equal to/not equal to the right-hand operand AND the left-hand operand is the same/not the same data type to the right-hand operand respectively.
Example Result
1 == 2 false
1 <= 2 true
1 >= 2 false
1 != 2 true
1 < 2 true
1 > 2 false
2 = 2 true
'' == false true
'' === false false
1 == true true
1 === true false
['1','2','3'] == ['1','2','3'] true
[1,2,3] === [1,2,3] true
['1','2','3'] == [1,2,3] true
['1','2','3'] === [1,2,3] false
[1,1,''] == [true, true, false] true
[] == false & [] == null true
['1'] == '1' false[1]

Built-in variables

The abuse filter passes various variables by name into the parser. These variables can be accessed by typing their name in, in a place where a literal would work. You can view the variables associated with each request in the abuse log.

Variables from AbuseFilter

Variables available
Description Name Data type Notes
Action action string One of: edit, move, createaccount, autocreateaccount, delete, upload[2], stashupload[3]
Edit count of the user user_editcount string Empty for unregistered users.
Name of the user account user_name string Note: this is empty for "createaccount" action, use accountname instead.
Time email address was confirmed user_emailconfirm string In the format: YYYYMMDDHHMMSS
Age of the user account user_age integer In seconds; 0 for unregistered users.
Whether the user is blocked user_blocked boolean true for blocked registered users, false for unregistered users.
Groups (including implicit) the user is in user_groups array of strings
Rights that the user has user_rights array of strings
Page ID article_articleid integer Deprecated. Use page_id instead.
Page ID (found in the page's HTML source - search for wgArticleId) page_id integer In theory this is 0 for new pages, but this is unreliable. Instead, use "page_age == 0" to identify new page creation.
Page namespace article_namespace integer Deprecated. Use page_namespace instead.
Page namespace page_namespace integer refers to namespace index
Page age (in seconds) page_age integer the number of seconds since the first edit (or 0 for new pages)
Page title (without namespace) article_text string Deprecated. Use page_title instead.
Page title (without namespace) page_title string
Full page title article_prefixedtext string Deprecated. Use page_prefixedtitle instead.
Full page title page_prefixedtitle string
Edit protection level of the page article_restrictions_edit string Deprecated. Use page_restrictions_edit instead.
Edit protection level of the page page_restrictions_edit array of strings
Move protection level of the page article_restrictions_move string Deprecated. Use page_restrictions_move instead.
Move protection level of the page page_restrictions_move array of strings
Upload protection of the file article_restrictions_upload string Deprecated. Use page_restrictions_upload instead.
Upload protection of the file page_restrictions_upload array of strings
Create protection of the page article_restrictions_create string Deprecated. Use page_restrictions_create instead.
Create protection of the page page_restrictions_create array of strings
Last ten users to contribute to the page article_recent_contributors array of strings Deprecated. Use page_recent_contributors instead.
Last ten users to contribute to the page page_recent_contributors array of strings This tends to be slow. Try to put conditions more likely evaluate to false before this one, to avoid unnecessarily running the query. This value is empty if the user is the only contributor to the page(?), and only scans the last 100 revisions
First user to contribute to the page article_first_contributor string Deprecated. Use page_first_contributor instead.
First user to contribute to the page page_first_contributor string This tends to be slow.[4] Try to put conditions more likely evaluate to false before this one, to avoid unnecessarily running the query.
Variables available for some actions
Description Name Data type Notes
Edit summary/reason summary string Summaries automatically created by MediaWiki ("New section", "Blanked the page", etc.) are created after the filter checks the edit, so they will never actually catch, even if the debugger shows that they should.[5]
Whether or not the edit is marked as minor minor_edit string Deprecated and always set to false[6]
Old page wikitext, before the edit (no more in use) old_wikitext string This variable can be very large. Consider using removed_lines if possible to improve performance.
New page wikitext, after the edit new_wikitext string This variable can be very large. Consider using added_lines if possible to improve performance.
Unified diff of changes made by edit edit_diff string
Unified diff of changes made by edit, pre-save transformed edit_diff_pst string This tends to be slow. Checking both added_lines and removed_lines is probably more efficient.[7]
New page size new_size integer
Old page size old_size integer
Size change in edit edit_delta integer
Lines added in edit, pre-save transformed added_lines_pst array of strings Use added_lines if possible, which is more efficient.
Lines added in edit added_lines array of strings includes all lines in the final diff that begin with +
Lines removed in edit removed_lines array of strings
All external links in the new text all_links array of strings
Links in the page, before the edit old_links array of strings
All external links added in the edit added_links array of strings This tends to be slow. Consider checking against added_lines first, then check added_links so that fewer edits are slowed down. This follows MediaWiki's rules for external links. Only unique links are added to the array. Changing a link will count as 1 added and 1 removed link.
All external links removed in the edit removed_links array of strings This tends to be slow. Consider checking against removed_lines first, then check removed_links so that fewer edits are slowed down. This follows MediaWiki's rules for external links. Only unique links are added to the array. Changing a link will count as 1 added and 1 removed link.
New page wikitext, pre-save transformed new_pst string
Parsed HTML source of the new revision new_html string This variable can be very large. Consider using added_lines if possible to improve performance.
New page wikitext, after the edit new_text string This variable can be very large. Consider using added_lines if possible to improve performance.
Old page wikitext, parsed into HTML (no more in use) old_html string Disabled for performance reasons.
Old page wikitext, before the edit (no more in use) old_text string Disabled for performance reasons.
Unix timestamp of change timestamp string int(timestamp) gives you a number with which you can calculate the date, time, day of week, etc.
SHA1 hash of file contents file_sha1 string [2]
Size of the file in bytes file_size integer The file size in bytes[2]
Width of the file in pixels file_width integer The width in pixels[2]
Height of the file in pixels file_height integer The height in pixels[2]
Page ID of move destination page moved_to_articleid string Deprecated. Use moved_to_id instead.
Page ID of move destination page moved_to_id string
Title of move destination page moved_to_text string Deprecated. Use moved_to_title instead.
Title of move destination page moved_to_title string
Full title of move destination page moved_to_prefixedtext string Deprecated. Use moved_to_prefixedtitle instead.
Full title of move destination page moved_to_prefixedtitle string
Namespace of move destination page moved_to_namespace string
Move destination page age (in seconds) moved_to_age integer
Namespace of move source page moved_from_namespace string
Title of move source page moved_from_text string Deprecated. Use moved_from_title instead.
Title of move source page moved_from_title string
Full title of move source page moved_from_prefixedtext string Deprecated. Use moved_from_prefixedtitle instead.
Full title of move source page moved_from_prefixedtitle string
Page ID of move source page moved_from_articleid string Deprecated. Use moved_from_id instead.
Page ID of move source page moved_from_id string
Move source page age (in seconds) moved_from_age integer
Account name (on account creation) accountname string
Content model of the old revision old_content_model string See Help:ChangeContentModel for information about content model changes
Content model of the new revision new_content_model string See Help:ChangeContentModel for information about content model changes

Variables from other extensions

Description Name Data type Values Added by
Global groups that the user is in global_user_groups array CentralAuth
⧼abusefilter-edit-builder-vars-board-id⧽ board_articleid integer Deprecated. Use board_id instead. StructuredDiscussions
⧼abusefilter-edit-builder-vars-board-id⧽ board_id integer StructuredDiscussions
Namespace of Structured Discussions board board_namespace integer refers to namespace index StructuredDiscussions
⧼abusefilter-edit-builder-vars-board-title⧽ board_text String Deprecated. Use board_title instead. StructuredDiscussions
⧼abusefilter-edit-builder-vars-board-title⧽ board_title string StructuredDiscussions
⧼abusefilter-edit-builder-vars-board-prefixedtitle⧽ board_prefixedtext String Deprecated. Use board_prefixedtitle instead. StructuredDiscussions
⧼abusefilter-edit-builder-vars-board-prefixedtitle⧽ board_prefixedtitle string StructuredDiscussions
Source text of translation unit translate_source_text string Translate
Whether or not the change was made through a Tor exit node tor_exit_node boolean true if the action comes from a tor exit node. TorBlock
Whether or not a user is editing through the mobile interface user_mobile boolean true for mobile users, false otherwise. MobileFrontend
Whether the user is editing from mobile app user_app boolean true if the user is editing from the mobile app, false otherwise. MobileApp
Whether or not a user is editing through a zero-rated carrier via Wikipedia Zero user_wpzero boolean Note: This variable is only valid when filtering an action. When examining a past edit or batch testing, it'll always be null. WikimediaEvents
⧼abusefilter-edit-builder-vars-page-views⧽ article_views integer Deprecated. Use page_views instead. HitCounters
⧼abusefilter-edit-builder-vars-page-views⧽ page_views integer the amount of page views HitCounters
⧼abusefilter-edit-builder-vars-movedfrom-views⧽ moved_from_views integer the amount of page views of the source page HitCounters
⧼abusefilter-edit-builder-vars-movedto-views⧽ moved_to_views integer the amount of page views of the target page HitCounters
⧼abusefilter-edit-builder-vars-is-proxy⧽ is_proxy integer Whether this action was performed through a proxy AutoProxyBlock
⧼abusefilter-edit-builder-vars-sfs-blocked⧽ sfs_blocked boolean Whether the IP address is blocked using the stopforumspam.com list StopForumSpam

Notes

When action='move', only the summary, action, timestamp and user_* variables are available. The page_* variables are also available, but the prefix is replaced by moved_from_ and moved_to_, that represent the values of the original article name and the destination one, respectively. For example, moved_from_title and moved_to_title instead of page_title.

Since MediaWiki 1.28 (https://gerrit.wikimedia.org/r/#/c/295254/), action='upload' is only used when publishing an upload, and not for uploads to stash. A new action='stashupload' is introduced, which is used for all uploads, including uploads to stash. This behaves like action='upload' used to, and only provides file metadata variables (file_*). Variables related to the page edit, including summary, new_wikitext and several others, are now available for action='upload'. For every file upload, filters may be called with action='stashupload' (for uploads to stash), and are always called with action='upload'; they are not called with action='edit'.

Filter authors should use action='stashupload' | action='upload' in filter code when a file can be checked based only on the file contents – for example, to reject low-resolution files – and action='upload' only when the wikitext parts of the edit need to be examined too – for example, to reject files with no description. This will allow tools that separate uploading the file and publishing the file (e.g. UploadWizard or upload dialog) to inform the user of the failure before they spend the time filling in the upload details.

Keywords

Where not specifically stated, keywords cast their operands to strings

The following special keywords are included for often-used functionality:

  • like (or matches) returns true if the left-hand operand matches the glob pattern in the right-hand operand.
  • in returns true if the right-hand operand (a string) contains the left-hand operand. Note: empty strings are not contained in, nor contain, any other string (not even the empty string itself).
  • contains works like in, but with the left and right-hand operands switched. Note: empty strings are not contained in, nor contain, any other string (not even the empty string itself).
  • rlike (or regex) and irlike return true if the left-hand operand matches (contains) the regex pattern in the right-hand operand (irlike is case insensitive). The system uses PCRE. The only PCRE option enabled is PCRE_UTF8 (modifier u in PHP); for irlike both PCRE_CASELESS and PCRE_UTF8 are enabled (modifier iu).
  • if ... then ... else ... end
  • ... ? ... : ...
  • true, false and null


Examples

Code Result Comment
"1234" like "12?4" True
"1234" like "12*" True
"foo" in "foobar" True
"foobar" contains "foo" True
"o" in ["foo", "bar"] True Due to the string cast
"foo" regex "\w+" True
"a\b" regex "a\\\\b" True To look for the escape character backslash using regex you need
to use either four backslashes or two \x5C. (Either works fine.)
"a\b" regex "a\x5C\x5Cb" True

Functions

A number of built-in functions are included to ease some common issues. They are executed in the general format functionName( arg1, arg2, arg3 ), and can be used in place of any literal or variable. Its arguments can be given as literals, variables, or even other functions.

name description
lcase Returns the argument converted to lower case.
ucase Returns the argument converted to upper case.
length Returns the length of the string given as the argument. If the argument is an array, returns its number of elements.
string Casts to string data type. If the argument is an array, implodes it with linebreaks.
int Casts to integer data type.
float Casts to floating-point data type.
bool Casts to boolean data type.
norm Equivalent to rmwhitespace(rmspecials(rmdoubles(ccnorm(arg1)))).
ccnorm Normalises confusable/similar characters in the argument, and returns a canonical form. A list of characters and their replacements can be found on git, eg. ccnorm( "Eeèéëēĕėęě3ƐƷ" ) === "EEEEEEEEEEEEE".[8] The output of this function is always uppercase.
ccnorm_contains_any Normalises confusable/similar characters in the arguments, and returns true if the first string contains any strings from the following arguments (unlimited number of arguments, logic OR mode). A list of characters and their replacements can be found on git.
ccnorm_contains_all Normalises confusable/similar characters in the arguments, and returns true if the first string contains every strings from the following arguments (unlimited number of arguments, logic AND mode). A list of characters and their replacements can be found on git.
specialratio Returns the number of non-alphanumeric characters divided by the total number of characters in the argument.
rmspecials Removes any special characters in the argument, and returns the result. (Equivalent to s/[^\p{L}\p{N}]//g.)
rmdoubles Removes repeated characters in the argument, and returns the result.
rmwhitespace Removes whitespace (spaces, tabs, newlines).
count Returns the number of times the needle (first string) appears in the haystack (second string). If only one argument is given, splits it by commas and returns the number of segments.
rcount Similar to count but the needle uses a regular expression instead. Can be made case-insensitive by letting the regular expression start with "(?i)".
get_matches MW 1.31+ Looks for matches of the regex needle (first string) in the haystack (second string). Returns an array where the 0 element is the whole match and every [n] element is the match of the n'th capturing group of the needle. Can be made case-insensitive by letting the regular expression start with "(?i)". If a capturing group didn't match, that array position will take value of false.
ip_in_range Returns true if user's IP (first string) matches specified IP ranges (second string). Only works for anonymous users. Supports both IPv4 and IPv6 addresses.
contains_any Returns true if the first string contains any strings from the following arguments (unlimited number of arguments in logic OR mode). If the first argument is an array, it gets casted to string.
contains_all Returns true if the first string contains every strings from the following arguments (unlimited number of arguments in logic AND mode). If the first argument is an array, it gets casted to string.
equals_to_any Returns true if the first argument is identical (===) to any of the following ones (unlimited number of arguments). Basically, equals_to_any(a, b, c) is the same as a===b | a===c, but more compact and saves conditions.
substr Returns the portion of the first string, by offset from the second argument (starts at 0) and maximum length from the third argument (optional).
strlen Same as length.
strpos Returns the numeric position of the first occurrence of needle (second string) in the haystack (first string), starting from offset from the third argument (optional, default is 0). This function may return 0 when the needle is found at the begining of the haystack, so it might be misinterpreted as false value by another comparative operator. The better way is to use === or !== for testing whether it is found.
str_replace Replaces all occurrences of the search string with the replacement string. The function takes 3 arguments in the following order: text to perform the search on, text to find, replacement text.
rescape Returns the argument with some characters preceded with the escape character "\", so that the string can be used in a regular expression without those characters having a special meaning.
set Sets a variable (first string) with a given value (second argument) for further use in the filter. Another syntax: name := value.
set_var Same as set.

Examples

Code Result Comment
length( "Wikipedia" ) 9
lcase( "WikiPedia" ) wikipedia
ccnorm( "w1k1p3d14" ) WIKIPEDIA ccnorm output is always uppercase
ccnorm( "ωɨƙɩᑭƐƉ1α" ) WIKIPEDIA
ccnorm_contains_any( "w1k1p3d14", "wiKiP3D1A", "foo", "bar" ) true
ccnorm_contains_any( "w1k1p3d14", "foo", "bar", "baz" ) false
ccnorm_contains_any( "w1k1p3d14 is 4w3s0me", "bar", "baz", "some" ) true
ccnorm( "ìíîïĩїį!ľ₤ĺľḷĿ" ) IIIIIII!LLLLLL
norm( "!!ω..ɨ..ƙ..ɩ..ᑭᑭ..Ɛ.Ɖ@@1%%α!!" ) WIKIPEDAIA
norm( "F00 B@rr" ) FOBAR norm removes whitespace, special characters and duplicates, then uses ccnorm
rmdoubles( "foobybboo" ) fobybo
specialratio( "Wikipedia!" ) 0.1
count( "foo", "foofooboofoo" ) 3
count( "foo,bar,baz" ) 3
rmspecials( "FOOBAR!!1" ) FOOBAR1
rescape( "abc* (def)" ) abc\* \(def\)
str_replace( "foobarbaz", "bar", "-" ) foo-baz
ip_in_range( "127.0.10.0", "127.0.0.0/12" ) true
contains_any( "foobar", "x", "y", "f" ) true
get_matches( "(foo?ba+r) is (so+ good)", "fobaaar is soooo good to eat" ) ['fobaaar is soooo good', 'fobaaar', 'soooo good']

Order of operations

Operations are generally done left-to-right, but there is an order to which they are resolved. As soon as the filter fails one of the conditions, it will stop checking the rest of them (due to short-circuit evaluation) and move on to the next filter (except for phab:T43693). The evaluation order is:

  1. Anything surrounded by parentheses (( and )) is evaluated as a single unit.
  2. Turning variables/literals into their respective data. (i.e., page_namespace to 0)
  3. Function calls (norm, lcase, etc.)
  4. Unary + and - (defining positive or negative value, e.g. -1234, +1234)
  5. Keywords
  6. Boolean inversion (!x)
  7. Exponentiation (2**3 → 8)
  8. Multiplication-related (multiplication, division, modulo)
  9. Addition and subtraction (3-2 → 1)
  10. Comparisons. (<, >, ==)
  11. Boolean operations. (&, |, ^)

Examples

  • A & B | C is equivalent to (A & B) | C, not to A & (B | C). In particular, both false & true | true and false & false | true evaluates to true.
  • A | B & C is equivalent to (A | B) & C, not to A | (B & C). In particular, both true | true & false and true | false & false evaluates to false.

Condition counting

The condition limit is (more or less) tracking the number of comparison operators + number of function calls entered.

Further explanation on how to reduce conditions used can be found at Extension:AbuseFilter/Conditions.

Useful links

Notes

  1. Comparing arrays to other types will always return false, except for the example above
  2. 2.0 2.1 2.2 2.3 2.4 The only variables currently available for file uploads (action='upload') are user_*, page_*, file_sha1, file_size, file_mime, file_mediatype, file_width, file_height, file_bits_per_channel (the last five were only added since the release for MediaWiki 1.27, gerrit:281503). All the file_* variables are unavailable for other actions (including action='edit').
  3. Since MediaWiki 1.28 (https://gerrit.wikimedia.org/r/#/c/295254/)
  4. Several filters (12) that use this variable have showed up in the AbuseFilterSlow Grafana dashboard (requires logstash access to view). Moving this variable to towards the end of the filter seemed to help.
  5. See phabricator:T191722
  6. Since this commit
  7. Some filters using this variable have showed up in the AbuseFilterSlow Grafana dashboard (example, requires logstash access). For instance, instead of using "text" in edit_diff_pst (or even edit_diff), consider something like "text" in added_lines & !("text" in removed_lines)
  8. Be aware of phab:T27619. You can use Special:AbuseFilter/tools to evaluate ccnorm( "your string" ) to see which characters are transformed.