Extension:AbuseFilter/Rules format

From MediaWiki.org
Jump to: navigation, search
Language:Project:Language policy English  • 日本語 • मराठी • русский

The rules are formatted much as conditionals in a C/Java/Perl-like language.

Literals

You can specify a literal by placing it in single or double quotes (for strings), or by typing it in as-is (for numbers, both floating-point and integer). You can get linebreaks with \n, tab characters with \t, and you can also escape the quote character with a backslash.

Examples

"This is a string"
'This is also a string'
'This string shouldn\'t fail'
"This string\nHas a linebreak"
1234
1.234
-123

Comments

You can specify comments using the following syntax:

/* This is a comment */

Variables

The abuse filter passes various variables by name into the parser. These variables can be accessed by typing their name in, in a place where a literal would work. You can view the variables associated with each request in the abuse log.

You can define more variables for ease of understanding with the assign symbol := in a line (closed by ;) within a condition. Example (from w:en:Special:AbuseFilter/79):

(
	line1:="(\{\{(r|R)eflist|\{\{(r|R)efs|<references\s?/>|</references\s?>)";
	rcount(line1, removed_lines)
) > (
	rcount(line1, added_lines)
)

Lists:

a_list := [ 5, 6, 7];

All variables

Variables available
Description Name Data type Values
Action action string edit, move, createaccount, autocreateaccount, delete, upload[1], stashupload[2]
Edit count of the user user_editcount string Empty for unregistered users.
Name of the user account user_name string
Time email address was confirmed user_emailconfirm string YYYYMMDDHHMMSS
Age of the user account user_age in seconds; 0 for unregistered users.
Whether the user is blocked user_blocked boolean is 1 for blocked registered users; null for unregistered users.
Whether or not a user is editing through the mobile interface user_mobile boolean is 1 for mobile users.
Groups (including implicit) the user is in user_groups
Rights that the user has user_rights
Page ID (found in the page's HTML source - search for wgArticleId) article_articleid integer In theory this is 0 for new pages, but this is unreliable. Instead, use "old_size==0" to identify new page creation.
Page namespace article_namespace integer refers to namespace index
Page title (without namespace) article_text string
Full page title article_prefixedtext string
Edit protection level of the page article_restrictions_edit
Move protection level of the page article_restrictions_move
Upload protection of the file article_restrictions_upload
Create protection of the page article_restrictions_create
Last ten users to contribute to the page article_recent_contributors Empty if the user is the only contributor to the page(?), only scans the last 100 revisions
First user to contribute to the page article_first_contributor This tends to be slow.[3] Try to put conditions more likely evaluate to false before this one, to avoid unnecessarily running the query.
Variables available for some actions
Description Name Data type Values
Edit summary/reason summary string
Whether or not the edit is marked as minor minor_edit string [4]
Old page wikitext, before the edit old_wikitext This variable can be very large. Consider using removed_lines if possible to improve performance.
New page wikitext, after the edit new_wikitext This variable can be very large. Consider using added_lines if possible to improve performance.
Unified diff of changes made by edit edit_diff
Unified diff of changes made by edit, pre-save transformed edit_diff_pst This tends to be slow. Checking both added_lines and removed_lines is probably more efficient.[5]
New page size new_size integer
Old page size old_size integer
Size change in edit edit_delta
Lines added in edit, pre-save transformed added_lines_pst string Use added_lines if possible, which is more efficient.
Lines added in edit added_lines array of strings includes all lines in the final diff that begin with +
Lines removed in edit removed_lines
All external links in the new text all_links
Links in the page, before the edit old_links
All external links added in the edit added_links
All external links removed in the edit removed_links
New page wikitext, pre-save transformed new_pst
Parsed HTML source of the new revision new_html This variable can be very large. Consider using added_lines if possible to improve performance.
New page text, stripped of any markup new_text This variable can be very large. Consider using added_lines if possible to improve performance.
Disabled old_html
Disabled old_text
Whether or not the change was made through a tor exit node tor_exit_node boolean 0, 1 (only available if TorBlock is installed)
Unix timestamp of change timestamp string int(timestamp) gives you a number with which you can calculate the date, time, day of week, etc.
SHA1 hash of file contents file_sha1 [1]
Size of the file in bytes file_size integer The file size in bytes[1]
Page ID of move destination page moved_to_articleid
Full title of move destination page moved_to_prefixedtext
Namespace of move destination page moved_to_namespace
Namespace of move source page moved_from_namespace
Full title of move source page moved_from_prefixedtext
Page ID of move source page moved_from_articleid
Account name (on account creation) accountname
Content model of the old revision old_content_model string See Help:ChangeContentModel for information about content model changes
Content model of the new revision new_content_model string See Help:ChangeContentModel for information about content model changes
Variables provided by CentralAuth
Description Name Data type Values
Global groups that the user is in global_user_groups
StructuredDiscussions variables
Description Name Data type Values
Page ID of Structured Discussions board board_articleid
Namespace of Structured Discussions board board_namespace refers to namespace index
Title of Structured Discussions board board_text
Full title of Structured Discussions board board_prefixedtext

Notes

When action='move', only the summary, action, timestamp and user_* variables are available. The article_* variables are also available, but the prefix is replaced by moved_from_ and moved_to_, that represent the values of the original article name and the destination one, respectively. For example, moved_from_text and moved_to_text instead of article_text.

Since MediaWiki 1.28 (https://gerrit.wikimedia.org/r/#/c/295254/), action='upload' is only used when publishing an upload, and not for uploads to stash. A new action='stashupload' is introduced, which is used for all uploads, including uploads to stash. This behaves like action='upload' used to, and only provides file metadata variables (file_*). Variables related to the page edit, including summary, new_wikitext and several others, are now available for action='upload'. For every file upload, filters may be called with action='stashupload' (for uploads to stash), and are always called with action='upload'; they are not called with action='edit'.

Filter authors should use action='stashupload' | action='upload' in filter code when a file can be checked based only on the file contents – for example, to reject low-resolution files – and action='upload' only when the wikitext parts of the edit need to be examined too – for example, to reject files with no description. This will allow tools that separate uploading the file and publishing the file (e.g. UploadWizard or upload dialog) to inform the user of the failure before they spend the time filling in the upload details.

Page/Article namespace

See also Manual:Namespace

English Wikipedia namespaces
Basic namespaces Talk namespaces
0 Main Talk 1
2 User User talk 3
4 Wikipedia Wikipedia talk 5
6 File File talk 7
8 MediaWiki MediaWiki talk 9
10 Template Template talk 11
12 Help Help talk 13
14 Category Category talk 15
100 Portal Portal talk 101
108 Book Book talk 109
Virtual namespaces
-1 Special
-2 Media

Simple comparisons

You can compare variables with other variables and literals with the following syntax:

  • < and >—Return true if the left-hand operand is less than/greater than the right-hand operand respectively.
  • <= and >=—Return true if the left-hand operand is less than or equal to/greater than or equal to the right-hand operand respectively.
  • == (or =) and !=—Return true if the left-hand operand is equal to/not equal to the right-hand operand respectively.
  • === and !==—Return true if the left-hand operand is equal to/not equal to the right-hand operand AND the left-hand operand is the same/not the same data type to the right-hand operand respectively.
Example Result
1 == 2 false
1 <= 2 true
1 >= 2 false
1 != 2 true
1 < 2 true
1 > 2 false
2 = 2 true
'' == false true
'' === false false
1 == true true
1 === true false

Arithmetic

You can use basic arithmetic symbols to do arithmetic on variables and literals with the following syntax:

  • - — Subtract the right-hand operand from the left-hand operand.
  • + — Add the right-hand operand to the left-hand operand.
  • * — Multiply the left-hand operand by the right-hand operand.
  • / — Divide the left-hand operand by the right-hand operand.
  • ** — Raise the left-hand operand to the exponential power specified by the right-hand operand.
  • % — Return the remainder given when the left-hand operand is divided by the right-hand operand.
Example Result
1 + 1 2
2 * 2 4
1 / 2 0.5
9 ** 2 81
6 % 5 1

String concatenation

You can use the + (plus) symbol to concatenate two literal strings or the values of two vars with a string value.

Keywords

The following special keywords are included for often-used functionality:

  • like (or matches) returns true if the left-hand operand matches the glob pattern in the right-hand operand.
  • in returns true if the right-hand operand (a string) contains the left-hand operand.
  • rlike (or regex) and irlike return true if the left-hand operand matches (contains) the regex pattern in the right-hand operand (irlike is case insensitive). The system uses PCRE. The only PCRE option enabled is PCRE_UTF8 (modifier u in PHP); for irlike both PCRE_CASELESS and PCRE_UTF8 are enabled (modifier iu).
  • contains
  • if ... then ... else ... end
  • ... ? ... : ...
  • true, false and null


Examples

Code Result Comment
"1234" like "12?4" True
"1234" like "12*" True
"foo" in "foobar" True
"foo" regex "\w+" True
"a\b" regex "a\\\\b" True To look for the escape character backslash using regex you need
to use either four backslashes or two \x5C. (Either works fine.)
"a\b" regex "a\x5C\x5Cb" True

Functions

A number of built-in functions are included to ease some common issues. They are executed in the general format functionName( arg1, arg2, arg3 ), and can be used in place of any literal or variable. Its arguments can be given as literals, variables, or even other functions.

name description
lcase Returns the argument converted to lower case.
ucase Returns the argument converted to upper case.
length Returns the length of the string given as the argument.
string Casts to string data type.
int Casts to integer data type.
float Casts to floating-point data type.
bool Casts to boolean data type.
norm Equivalent to rmwhitespace(rmspecials(rmdoubles(ccnorm(arg1)))).
ccnorm Normalises confusable/similar characters in the argument, and returns a canonical form. A list of characters and their replacements can be found on git, eg. ccnorm( "Eeèéëēĕėęě3ƐƷ" ) === "EEEEEEEEEEEEE".[6] Note that the extension AntiSpoof is required for this function to have an effect. Without it the string will simply be left unchanged.
ccnorm_contains_any Normalises confusable/similar characters in the arguments, and returns true if the first string contains any strings from the following arguments (unlimited number of arguments). A list of characters and their replacements can be found on git. Note that the extension AntiSpoof is required for this function to have an effect. Without it the arguments will simply be left unchanged.
specialratio Returns the number of non-alphanumeric characters divided by the total number of characters in the argument.
rmspecials Removes any special characters in the argument, and returns the result. (Equivalent to s/[^\p{L}\p{N}]//g.)
rmdoubles Removes repeated characters in the argument, and returns the result.
rmwhitespace Removes whitespace (spaces, tabs, newlines).
count Returns the number of times the needle (first string) appears in the haystack (second string). If only one argument is given, splits it by commas and returns the number of segments.
rcount Similar to count but the needle uses a regular expression instead. Can be made case-insensitive by letting the regular expression start with "(?i)".
get_matches MW 1.31+ Looks for matches of the regex needle (first string) in the haystack (second string). Returns an array where the 0 element is the whole match and every [n] element is the match of the n'th capturing group of the needle. Can be made case-insensitive by letting the regular expression start with "(?i)". If a capturing group didn't match, that array position will take value of false.
ip_in_range Returns true if user's IP (first string) matches specified IP ranges (second string). Only works for anonymous users. Supports both IPv4 and IPv6 addresses.
contains_any Returns true if the first string contains any strings from the following arguments (unlimited number of arguments).
substr Returns the portion of the first string, by offset from the second argument (starts at 0) and maximum length from the third argument (optional).
strlen Same as length.
strpos Returns the numeric position of the first occurrence of needle (second string) in the haystack (first string). This function may return 0 when the needle is found at the begining of the haystack, so it might be misinterpreted as false value by another comparative operator. The better way is to use === or !== for testing whether it is found.
str_replace Replaces all occurrences of the search string with the replacement string. The function takes 3 arguments in the following order: text to perform the search, text to find, replacement text.
rescape Returns the argument with some characters preceded with the escape character "\", so that the string can be used in a regular expression without those characters having a special meaning.
set Sets a variable (first string) with a given value (second argument) for further use in the filter. Another syntax: name := value.
set_var Same as set.

Other

  • convert returns the second argument converted to variant language specified by the first argument. ONLY apply on wikis with LanguageConverter class. (New func added on rev:49399, need support of MediaWiki after rev:49397)

Examples

Code Result Comment
length( "Wikipedia" ) 9
lcase( "WikiPedia" ) wikipedia
ccnorm( "w1k1p3d14" ) WIKIPEDIA ccnorm output is always uppercase
ccnorm( "ωɨƙɩᑭƐƉ1α" ) WIKIPEDIA
ccnorm_contains_any( "w1k1p3d14", "wiKiP3D1A", "foo", "bar" ) true
ccnorm_contains_any( "w1k1p3d14", "foo", "bar", "baz" ) false
ccnorm_contains_any( "w1k1p3d14 is 4w3s0me", "bar", "baz", "some" ) true
ccnorm( "ìíîïĩїį!ľ₤ĺľḷĿ" ) IIIIIII!LLLLLĿ Not all characters get normalized[6]
norm( "!!ω..ɨ..ƙ..ɩ..ᑭᑭ..Ɛ.Ɖ@@1%%α!!" ) WIKIPEDIA
norm( "F00 B@rr" ) FOBR norm removes whitespace, special characters and duplicates, then uses ccnorm
rmdoubles( "foobybboo" ) fobybo
specialratio( "Wikipedia!" ) 0.1
count( "foo", "foofooboofoo" ) 3
count( "foo,bar,baz" ) 3
rmspecials( "FOOBAR!!1" ) FOOBAR1
rescape( "abc* (def)" ) abc\* \(def\)
str_replace( "foobarbaz", "bar", "-" ) foo-baz
ip_in_range( "127.0.10.0", "127.0.0.0/12" ) true
contains_any( "foobar", "x", "y", "f" ) true

Boolean operations

You can match if and only if all of a number of conditions are true, one of a number of conditions are true, or one and only one of all conditions are true.

  • x | y — OR – returns true if one or more of the conditions is true.
  • x & y — AND – returns true if both of the conditions are true.
  • x ^ y — XOR – returns true if one, and only one of the two conditions is true.
  • !x — NOT – returns true if the condition is not true.

Examples

Code Result
1 | 1 true
1 | 0 true
0 | 0 false
1 & 1 true
1 & 0 false
0 & 0 false
1 ^ 1 false
1 ^ 0 true
0 ^ 0 false
!1 false

Order of operations

Operations are generally done left-to-right, but there is an order to which they are resolved. As soon as the filter fails one of the conditions, it will stop checking the rest of them (due to short-circuit evaluation) and move on to the next filter (except for phab:T43693). The evaluation order is:

  1. Anything surrounded by parentheses (( and )) is evaluated as a single unit.
  2. Turning variables/literals into their respective data. (i.e., article_namespace to 0)
  3. Function calls (norm, lcase, etc.)
  4. Unary + and - (defining positive or negative value, e.g. -1234, +1234)
  5. Keywords
  6. Boolean inversion (!x)
  7. Exponentiation (2**3 → 8)
  8. Multiplication-related (multiplication, division, modulo)
  9. Addition and subtraction (3-2 → 1)
  10. Comparisons. (<, >, ==)
  11. Boolean operations. (&, |, ^)

Examples

  • A & B | C is equivalent to (A & B) | C, not to A & (B | C). In particular, both false & true | true and false & false | true evaluates to true.
  • A | B & C is equivalent to (A | B) & C, not to A | (B & C). In particular, both true | true & false and true | false & false evaluates to false.

Condition counting

The condition limit is (more or less) tracking the number of comparison operators + number of function calls entered.

Further explanation on how to reduce conditions used can be found at Extension:AbuseFilter/Conditions.

Useful links

Notes

  1. 1.0 1.1 1.2 The only variables currently available for file uploads (action='upload') are user_*, article_*, file_sha1, file_size, file_mime, file_mediatype, file_width, file_height, file_bits_per_channel (the last five were only added since the release for MediaWiki 1.27, gerrit:281503). All the file_* variables are unavailable for other actions (including action='edit').
  2. Since MediaWiki 1.28 (https://gerrit.wikimedia.org/r/#/c/295254/)
  3. Several filters (12) that use this variable have showed up in the AbuseFilterSlow Grafana dashboard (requires logstash access to view). Moving this variable to towards the end of the filter seemed to help.
  4. Always false since https://gerrit.wikimedia.org/r/#/c/296268/
  5. Some filters using this variable have showed up in the AbuseFilterSlow Grafana dashboard (example, requires logstash access). For instance, instead of using "text" in edit_diff_pst (or even edit_diff), consider something like "text" in added_lines & !("text" in removed_lines)
  6. 6.0 6.1 Be aware of phab:T27619. You can use Special:AbuseFilter/tools to evaluate ccnorm( "your string" ) to see which characters are transformed.