What a good record holds
Start and end times and duration, what each location saw (status code, timings, DNS answer, certificate), what changed shortly before, who was told and when, the cause when known and the action taken. That is the evidence for the client message, the status page and the monthly report. Google’s SRE book advises setting “clear conditions for declaring an incident” and keeping a living incident document that several people can edit at once; a monitoring incident record is the machine-made start of that document.
Incident versus alert versus regional issue
An alert is a notification. An incident is the period. A regional issue is a failure seen by one region only, which does not open an incident under a two-region rule but is worth reading.
When an incident closes
An incident closes when checks succeed again. A careful monitor waits for more than one good answer, so a site that flaps is not closed and reopened every minute. The end time in the record should be the time of that recovery, so the duration the client reads is the duration visitors lived through, no shorter and no longer.
An example record
Start 14:07, end 14:19, duration 12 minutes. Regions A and B: timeout on https://www.example.com/. The DNS A record for www changed at 14:02 from 203.0.113.10 to 203.0.113.99. Alert sent to the on-call address at 14:07. Cause: the new address was a parking page that did not answer on port 443. Action: record restored at 14:15 and the TTL lowered before further changes, owner the agency’s lead developer. A record like this answers the client’s first three questions without a call.
What to check
That every incident has a cause or an honest “unknown”, a prevention action with an owner and a mention in the next report.
Sources
Read on . Standards and vendor pages change; confirm the current text before you rely on it.