Reading/Web/Team/Setting up alerts with grafana

This page guides. It assumes you have access to Logstash and Grafana.

• Create a Phabricator ticket to kick things off. You'll need help from SRE for this.
• Using logstash use the "dev tools" option to construct a query to describe what you want to alert. When finalized add the alert to a new folder in puppet example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/667695
• In Grafana, you should now be able to access the data stream you added. Access https://grafana-rw.wikimedia.org/ for the admin version of Grafana and construct a graph using the data.
• Go to the alert tab and define the conditions for the alert. Example (required edit rights). More information on setting up alerts can be found on the grafana docs.
• Setup the alert. example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/667737
• To test the alert, lower the conditions. An alert for reading web should go to the mailing list and the #wikimedia-operations IRC channel. Only one alert will be sent when the issue begins and when it resolves.