Runbook example: review a failed write before retrying

A recovery runbook records what happened, checks the destination and assigns a person to decide whether a write may be replayed. When the response is missing, treat the side effect as uncertain until read-back evidence resolves it. This runbook example separates “not observed”, “already present” and “duplicated” so a transport failure does not become an automatic second write.

By Ai Flowi · Published 10 October 2026

State the incident boundary and stop condition

The procedure covers one workflow writing enquiry records into a known worksheet. Start with the failed or missing response; finish by pairing the replay decision with destination evidence, or assigning the remaining uncertainty to a responder. This recovery record supports human review without implementing rollback, automatic retries or exactly-once delivery.

Stop new deliveries to the affected test scope through your existing sender controls while collecting evidence. Confirm whether requests are already queued or in flight. If another writer can still change the same records, record that uncertainty and withhold replay approval. This procedure assumes you can identify the affected destination and the submitted event reference.

Copy the incident record

Use role references and redacted evidence locations. Keep credentials, full provider errors and personal submission contents in the team’s restricted incident system, outside the shareable recovery note. Fill unknown fields explicitly; an empty response field must not be interpreted as a rejected request.

Incident reference:
Workflow and affected destination:
Detection time and timezone:
Responsible responder:
Last confirmed accepted event:
Sender pause confirmed / in-flight deliveries unresolved:
Affected event references and request versions:
Observed response status / response unavailable:
Read-back time, range and completeness:
Observed rows per event reference:
Approved field comparison: match / mismatch / unknown:
Side-effect classification: absent / present / duplicated / unresolved:
Stop condition encountered:
Reviewer and decision: hold / no replay / approved single replay:
Decision reason and request version:
Replay attempt reference, if authorized:
Post-decision read-back:
Closure evidence or unresolved assignment:

“Last confirmed accepted event” is a useful investigation anchor, not proof that later events failed. Timestamp order can help correlate records but cannot replace event matching. Keep the request version with the reference so an existing row for an older payload is not mistaken for successful execution of the current one.

Gather evidence before changing the destination

1. Record the sender’s attempt references, submitted event references and available response statuses. Preserve the difference between “no response received” and an explicit rejection.
2. Confirm the exact spreadsheet and worksheet used by the affected flow. A successful read of another tab provides no evidence about this write.
3. Read the full relevant range, including event references and the approved comparison fields. Record when the snapshot was taken and whether pagination, truncation or access restrictions could hide rows.
4. Count matching rows for each reference. Compare the submitted version and fields with each match; the count alone cannot identify stale or overwritten data.
5. Ask the assigned reviewer to classify the outcome and record the permitted next action. Leave uncertain cases on hold.

Before replay, require evidence that the first attempt can no longer complete late. A temporarily absent row is insufficient when a request remains in flight or visibility is delayed. If you cannot establish that condition, keep the delivery unresolved and escalate through the team’s existing process.

Reconcile three synthetic deliveries

The following records are invented solely to demonstrate the decision. They are not production logs or test results. The worksheet snapshot is taken after the sender is paused and the reviewer has confirmed there are no remaining in-flight attempts in this synthetic scenario.

The reviewer marks enquiry-A present, enquiry-B duplicated and enquiry-C absent under the stated snapshot assumptions. If an extra writer or unread range appears, enquiry-C changes to unresolved. The decision depends on evidence quality; “zero matches” is not a universal retry instruction.

For enquiry-B, preserve both row references and the two sender attempt references. Do not delete either row as part of this procedure. A separate authorized correction process must select the surviving row and investigate whether a downstream action also occurred twice. Spreadsheet duplication and external action duplication are separate questions.

Configure a read-only investigation step

Status: documented. The standalone Read Range settings below are documented in the catalogue; not production-tested by Ai Flowi for this recovery procedure.

example configuration, not a recorded test

In your Workflow workspace, connect your Google account with access to the affected sheet through Google connection setup; any charges from that provider remain your responsibility.

Use the Read Range reference for its full interface; set range_name=RecoveryDemo!A:C to cover the synthetic event reference, version and message, identify their spreadsheet with spreadsheet_id, and select its authorized account through google_account.

Keep the returned worksheet snapshot for manual comparison and attach redacted row references to the incident record. A read does not lock the sheet, prevent another writer, establish the completeness of another range or perform a replay. If read-back fails, stop: the original write’s outcome remains unknown.

Decide and verify one replay

If the reviewer authorizes enquiry-C, record the exact original reference, request version and synthetic fields to replay. Repair the known configuration or access issue first, without silently editing the payload. Record a distinct attempt reference for the replay so the two delivery attempts remain distinguishable.

After that single authorized attempt, read back again and compare both count and fields. One correct row establishes the observed destination state at that time. An absent, mismatched or duplicated result remains open; do not keep replaying until a favorable response appears. If this write initiates another system’s action, obtain that system’s evidence before closure as well.

How do I apply the recorded platform limitations in a runbook?

Use the webhook error reference for the recorded downstream-error and concurrency limits that constrain a replay decision.

A useful recovery note for enquiry-C reads: “Original attempt response unavailable; affected range checked in full; no v1 row found; sender paused and late completion ruled out; reviewer authorizes one replay of the unchanged v1 payload.” After replay, attach a new snapshot and record the observed matching row count. This filled note ties the permission to one version and one attempt, rather than leaving a reusable authorization to send the event repeatedly.

If the original response appears while the note is being prepared, add it to the evidence and reconsider the classification. Do not erase the earlier uncertainty from the incident history. If it contradicts the worksheet snapshot, check whether it names the same destination and attempt; a response for another run cannot resolve this incident. The replay decision remains on hold until the conflict is explained.

Related: AIOps: understand alert correlation and its limits.

Close with evidence, or keep an assigned exception

Closure needs a dated read-back, the reviewed request version, the chosen action and the reviewer reference. An unresolved record needs an assigned responder and a clear next evidence request. Do not use this procedure when the destination cannot be inspected, the request cannot be correlated or late completion remains possible; those conditions require escalation before replay.