Reading/Web/Team/Setting up alerts with grafana

This page guides. It assumes you have access to Logstash and Grafana.


 * Create a Phabricator ticket to kick things off. You'll need help from SRE for this.
 * Using logstash use the "dev tools" option to construct a query to describe what you want to alert. When finalized add the alert to a new folder in puppet example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/667695
 * In Grafana, you should now be able to access the data stream you added. Access https://grafana-rw.wikimedia.org/ for the admin version of Grafana and construct a graph using the data.
 * Go to the alert tab and define the conditions for the alert. Example (required edit rights). More information on setting up alerts can be found on the grafana docs.
 * Setup the alert. example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/667737
 * To test the alert, lower the conditions. An alert for reading web should go to the mailing list and the #wikimedia-operations IRC channel. Only one alert will be sent when the issue begins and when it resolves.