Silent failure is the normal failure
The visible failure, where a job stops and someone gets an alert, is the good case. The expensive case is a job that keeps running and quietly processes nothing, or processes the wrong thing.
The defence is to alert on absence as well as on errors. If a process normally handles a few dozen items a day, zero items is an event, not a quiet success.
Everything you depend on will change
Integrations break because the other side moved. A field gets renamed, a response gains a wrapper, a limit gets introduced. None of that arrives as a notification you will read in time.
Assume drift. Validate the shape of what arrives rather than trusting it, and fail loudly when it does not match.
- A field renamed or removed upstream
- A new rate limit or quota
- Credentials expiring quietly
- A format change in a file somebody else generates
A runbook, not a person who remembers
When something stops at an inconvenient hour, the question is what to do, not who built it. That answer belongs in writing next to the thing itself.
A runbook is short: what this does, what it depends on, how to tell it is healthy, how to restart it safely, and what must never be run twice.
Make reruns safe
The most common recovery is running the job again. If running it twice sends two messages or creates two records, recovery becomes its own incident.
Design so that repeating an operation lands in the same place as doing it once. This single property removes most of the fear from fixing things.