Common causes
A bad route or hiccup at the monitor’s location, a DNS resolver timing out, a firewall or rate limiter blocking the monitor’s addresses, a timeout shorter than a slow but working page needs, and a restart during a deploy.
Why they are costly
Each one costs a person’s time, and a team that has seen many learns to ignore the next alert, which may be real. Google’s SRE book describes the effect: when pages are too frequent, “employees second-guess, skim, or even ignore incoming alerts”. Its workbook gives the share a name, precision: “the proportion of events detected that were significant”, which is 100 percent only when every alert corresponds to a significant event.
Ways to reduce them
Re-check from the same location, require several consecutive failures or confirm from a second region. Each costs something: retries add little delay and do not help with longer path problems; consecutive-failure rules add delay; second-region confirmation can hide an outage seen by one region only. Counting real and false alerts each month gives the data to tune.
A worked example
A site is checked every minute from two regions. At 14:03 region A gets a timeout while region B gets a 200 in 310 ms. Under a one-region rule that is an alert and a phone call. Under a two-region rule it is a regional issue with no alert, and the timeout and its timing are stored. At 14:04 region A passes again. The monthly report then shows one regional issue and no incident, which is the honest description of what happened.
What to check
The share of last quarter’s alerts that were real, and which cause produced the false ones.
Sources
Read on . Standards and vendor pages change; confirm the current text before you rely on it.