Batch inference: preserve row identity and reconcile partial results

Batch inference applies an existing model to a collection of inputs rather than serving one immediate request at a time. Keep a stable identity for every input, match each returned result to its exact input version, and reconcile missing or failed rows before declaring the collection complete. Three returned results cannot complete a four-request manifest.

Capability status: documented. Batch Run and Table Operations are documented in the catalogue; not production-tested by Ai Flowi. Their combined configuration below has no recorded runtime test.

By Ai Flowi · Published 10 October 2026

A collection is not a provider batch API

A real-time request needs its individual answer for the next interaction. A batch groups inputs for a shared processing and review task, such as summarising supplied text records. Grouping alone says nothing about whether execution is sequential, parallel or asynchronous.

For this row-based task, Batch Run’s catalogue entry says Runs an LLM on each row of a DataFrame column. If no column is specified, all columns are used. That establishes the processing intent. It does not establish provider asynchronous batch APIs, order preservation, exactly-once execution, discounts or latency. The machine learning pipeline explanation covers the training boundary; this guide starts with an already configured model.

Copyable four-row input manifest

Synthetic planning artifact; no model was run. Copy these records into your controlled source table. The identifiers and ledger fields are your proposed application contract, not built-in Batch Run settings. Use a new request_key for each changed input version, and keep an unchanged key across retries of that same version.

[
  {"request_id":"R-A","input_version":"v1","request_key":"R-A-v1","source_text":"The library opens at noon."},
  {"request_id":"R-B","input_version":"v1","request_key":"R-B-v1","source_text":"The workshop moved to Tuesday."},
  {"request_id":"R-C","input_version":"v1","request_key":"R-C-v1","source_text":"The collection point is the east entrance."},
  {"request_id":"R-D","input_version":"v1","request_key":"R-D-v1","source_text":"The briefing is postponed; no replacement date is given."}
]

Before submission, reject blank identities, duplicate keys and empty source text. Preserve the original text separately from generated output. Record the selected model and instruction revision alongside the manifest so a changed prompt does not silently masquerade as a retry.

A row number is a viewing position, not a durable identity. Sorting the table must not change which text belongs to R-B. Asking a model to repeat an identifier is also insufficient: the output can alter it. Accept identity only from a verified mapping back to the submitted record.

Configure only the documented interfaces

You need a Table-producing source, permission to process its text, and a configured model connection. Obtain the model provider’s account and API key under your own billing arrangement; its usage charges sit outside the workflow subscription. Use connection guidance and the Language Model reference for configuration instead of placing credentials in source rows.

The linked loading, conversion and Language Model interfaces carry the same documented status; this example adds no test evidence for them.

example configuration, not a recorded test

1. Inspect the source Table and confirm the four manifest records and their keys. If your source is CSV, follow Load CSV and Data → DataFrame for loading and conversion; this example begins after that inspection.
2. On Batch Run, connect the Table to df and a configured LanguageModel connection to model. Both fields are required in the export.
3. Set column_name to source_text, output_column_name to draft_summary, and system_message to: “Summarise only the supplied text in one sentence. Preserve stated uncertainty; add no missing dates or facts.” Explicit column selection keeps ledger fields out of the requested text task.
4. Inspect batch_results, whose documented output type is Table. Verify that each result can be traced to its original key and version. The catalogue supplies no result envelope or error schema; if identity cannot be established, stop before merging.
5. Reconcile inspected results against this manifest on request_key using Table Operations’ "Merge" and "left" choices, following the existing join configuration for its inputs.

The customer data enrichment join explanation covers population preservation and duplicate-key effects.

For this manifest, reconcile B1 into the four decisions below before considering B2. R-A’s evidence belongs to the library sentence even though it arrived second; R-C’s belongs to the entrance sentence even though it arrived first. R-D remains an unresolved request with a separately confirmed failure, rather than disappearing from the collection. Keep its original text available for the retry’s later review.

Worked partial-result and retry ledger

The following invented ledger records three returned results from batch B1. A separate outcome inspection confirms that R-D failed transiently, produced no usable result and has no attempt still running. Its failure entry is not inferred merely from absence. Result and error references stand for separately inspected evidence; they are not provider response fields or actual generated summaries.

[
  {"request_key":"R-C-v1","attempt_id":"A-C1","batch_id":"B1","result_state":"returned","result_ref":"held-result-C1","error_ref":null,"review_state":"pending"},
  {"request_key":"R-A-v1","attempt_id":"A-A1","batch_id":"B1","result_state":"returned","result_ref":"held-result-A1","error_ref":null,"review_state":"approved"},
  {"request_key":"R-B-v1","attempt_id":"A-B1","batch_id":"B1","result_state":"returned","result_ref":"held-result-B1","error_ref":null,"review_state":"pending"},
  {"request_key":"R-D-v1","attempt_id":"A-D1","batch_id":"B1","result_state":"failed","result_ref":null,"error_ref":"inspected-transient-error-D1","review_state":"not_ready"},
  {"request_key":"R-D-v1","attempt_id":"A-D2","batch_id":"B2","result_state":"returned","result_ref":"held-result-D2","error_ref":null,"review_state":"pending"}
]

The first three entries deliberately arrive out of manifest order. Matching by position would attach R-C’s output to R-A. Matching by key yields three returned requests and one confirmed failure in B1. B2 contains only R-D, retains its input version and uses a new attempt ID; preserve A-D1 when adding A-D2.

After the hypothetical retry, four distinct requests have results across five attempts. Only R-A has an approved review in this ledger. Execution coverage and review completion therefore remain separate. A reviewer checks each summary against its source, especially R-D’s missing replacement date, before accepting the collection.

Should I hold, retry or review a batch inference result?

• Missing outcome: keep the request unknown while inspecting available evidence. A timeout may leave work running; do not automatically resubmit the whole batch.
• Duplicate or stale result: hold repeated keys, conflicting attempts and results for an earlier input version. Resolve which attempt is accepted before joining.
• Confirmed failed row: retry separately only after checking the cause and unresolved side effects. Correcting the source text creates a new input version, not an unchanged retry.
• Returned but unsupported summary: mark review as rejected and retain the output. A transport success cannot approve a fabricated date.

Limits, usage and when to skip it

The catalogue lists enable_metadata but does not define its fields; it cannot substitute for your identity check. Error handling, row limits, concurrency and retry behavior are also unspecified. Durable ledger storage and automatic reconciliation are not established by these component interfaces.

Use a direct request for an immediate interaction. Use an ordinary table checklist for a small collection you can review manually. Avoid this proposal when you require verified asynchronous provider jobs, measured throughput or automatic acceptance without inspecting outputs.

Plans: Go from USD8/month billed yearly (USD10 monthly), Plus from USD16 (USD20 monthly), Pro 100 from USD80 (USD100 monthly), Pro 200 from USD160 (USD200 monthly) and Pro 500 from USD400 (USD500 monthly), with 5,000 to 250,000 Flowi Tokens/month and top-ups; 1 token per node run; third-party API and model usage is not included.

For usage estimation, follow the existing marketing automations usage discussion; consult LLM observability for request evidence and missing measurements.