Wikimedia Release Engineering Team/Logspam

Log spam
#Logspam is the term we use to describe noisy error messages that do not represent actual error conditions or the errors are being ignored (purposefully not prioritized) by the responsible parties (if any exist). Specific error messages that have been identified by Release Engineering are tracked in the #Wikimedia-Log-Errors phabricator project.

Why it's a problem
Logspam is a problem because extra noise in logs can make it more difficult for humans to detect real problems with a brief glance at a log dashboard, for example fatalmonitor.

During deployments, deployers need to be able to quickly notice any new problems that are introduced by the newly deployed code. If error messages are drowned out by logspam then we could miss a more serious issue that should have been easily noticed.

Incorrectly categorized log messages
The most common example of this type would be expected conditions being recorded as exceptional conditions, ex. Debug notices or Warnings which are logged as Errors.

Undefined index notices
These are a common occurrence in php code. Whenever you attempt to access an index of an array but the array does not contain the specified key, hhvm will log a notice. These are coding errors and they need to be fixed. If the array index is not always expected to exist then the code needs to check with  or   before referencing the key.