This page guides. It assumes you have access to Logstash and Grafana.
- Create a Phabricator ticket to kick things off. You'll need help from SRE for this.
- Using logstash use the "dev tools" option to construct a query to describe what you want to alert. When finalized add the alert to a new folder in puppet example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/667695
- In Grafana, you should now be able to access the data stream you added. Access https://grafana-rw.wikimedia.org/ for the admin version of Grafana and construct a graph using the data.
- Go to the alert tab and define the conditions for the alert. Example (required edit rights). More information on setting up alerts can be found on the grafana docs.
- Setup the alert. example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/667737
- To test the alert, lower the conditions. An alert for reading web should go to the mailing list and the #wikimedia-operations IRC channel. Only one alert will be sent when the issue begins and when it resolves.