LLM observability: traces, metrics and missing evidence
By Ai Flowi · Published
LLM observability is the practice of collecting execution evidence that helps explain what happened during a language-model request. Logs describe events, metrics summarize measurements and traces connect stages of the same request. Evaluation adds a judgment about an answer’s quality; a successful execution status alone does not establish that the answer was supported.
Four evidence types, four questions
| Evidence | Question it helps answer | Boundary |
|---|---|---|
| Log event | What happened at a recorded point? | An isolated error message may lack the request or stage context. |
| Metric | How often or how much across a defined population? | An aggregate can hide the individual failing request. |
| Trace | Which stages belonged to this request? | Missing spans can leave the execution sequence incomplete. |
| Evaluation | Did the output satisfy an explicit quality criterion? | A score applies to the supplied sample and evaluator, not every future answer. |
These evidence types can be related, but they are not interchangeable. An error counter can reveal an increase without locating the input that caused it. A trace can locate a retrieval failure without deciding whether the final answer is factually correct. Keep the quality criterion separate from the transport or node status.
Define an observation contract
For a multi-stage request, a proposed external observation record can include a parent request ID, span ID, parent span ID, stage, input version, status and separately measured start and end times. Add an optional quality result with its evaluation criterion and version. Record whether timing and evaluation are present rather than filling missing values with reassuring defaults.
The request ID connects stages belonging to one request. A span ID identifies one stage attempt, and a parent span ID records its relationship to a containing stage. An input version identifies the reviewed prompt or source snapshot without requiring the complete text in every log. Distinguish a fresh request from an attempted retry.
This is a proposed application schema, not a claim about fields exported by AI Flowi Workflow. The component catalogue does not establish an automatic trace collector. A field called duration still needs a real measurement source.
Related: Data quality metrics: measure defects without guessing missing facts.
A fictional trace with two kinds of failure
Hypothetical external trace; not an exported AI Flowi log and not a recorded test. In this example, all stages have parent request ID request_demo. The numbers are invented elapsed milliseconds from one clock origin. No production latency is being asserted.
| Span / parent | Stage and input version | Start / end | Status and quality |
|---|---|---|---|
| prepare / request_demo | Prepare; prompt_v1 | 0 / 20 | Success; quality not assessed. |
| retrieve / request_demo | Retrieve; corpus_v2 | 20 / unknown | Failure event recorded; end measurement absent. |
| answer / request_demo | Generate; prompt_v1 | 40 / 90 | Success; later review finds an unsupported answer. |
The retrieval stage’s latency is unknown because its end measurement is missing. The answer stage has an illustrative measured interval of 50 milliseconds, but that does not reconstruct retrieval duration or complete request latency. Subtracting the retrieval start from the answer start would incorrectly assume the stages are contiguous and fully observed.
The unsupported answer is a different failure from the retrieval error. Generation returned successfully, yet the quality review found that the answer lacked evidence. The trace suggests a question to investigate: did the generation stage proceed without the required retrieved context? It does not prove that missing retrieval caused the unsupported claim.
Why do metrics need a denominator and a window?
Define whether a success rate counts complete requests, model calls, spans or recorded charges. Retried attempts can increase the call count without increasing the number of user requests. An observation window can also omit a request that began earlier or finished later. Keep the unit and time-window rule with each metric.
For latency, include only records with compatible, valid start and end measurements, and report the excluded or missing population. A missing duration is unknown, not zero. Do not average timestamps from unrelated clock origins. For an interval on one process, a monotonic clock avoids interpreting a wall-clock adjustment as elapsed work.
Model token usage and workflow node charges also need distinct units. Missing token usage cannot be repaired by counting words in the prompt. Record the source of each measurement and mark an estimated value as an estimate.
What AI Flowi Workflow’s Usage page establishes
The Workflow usage and token history guide explains the charge records, whose lack of recorded duration makes separately collected timing necessary for the proposed trace.
For request_demo, first ask whether the missing retrieval end was never measured, was dropped during collection or has not arrived yet. Those possibilities require different repairs. Store a collection cutoff with an investigation snapshot, then revise it if a late event arrives. The absence of an event at that cutoff establishes a gap in the evidence, not a completed or cancelled retrieval stage.
Evaluation and JSON components
The LangWatch integration reference owns the evaluator request contract used to assess the answer stage in this example.
Follow the JSON Object Builder reference when assembling separately supplied observation fields into a record.
example configuration, not a recorded test
With your own LangWatch account/API key and separately billed provider access, select an available LangWatch Evaluator.evaluator_name appropriate to the quality criterion. Set input to the permitted request_demo question and output to its preserved answer. If assessment uses contexts, supply the evidence actually available to that generation attempt; log any later-added reference evidence as a different assessment condition.
In a separate proposed assembly step, configure JSON Object Builder.field_names for request reference, span reference, evaluator version and evaluation result. Connect the result only after inspecting the generated fields and output shape. Correlation IDs are supplied by your external design; the evaluator does not export a trace ID in this catalogue. For this proposed observation-record configuration, LangWatch Evaluator and JSON Object Builder are documented in the catalogue; not production-tested by Ai Flowi. The combined evaluator and record assembly configuration remains untested.
Attach the assessment to the answer attempt, not only to request_demo. If that request has two generation attempts, a parent-only join can attach the first answer’s assessment to its replacement. Keep the exact answer version and assessment time with the span reference. A later successful assessment must not overwrite the earlier unsupported-answer finding or imply that retrieval succeeded.
Related: AIOps: understand alert correlation and its limits.
Minimize recorded content and diagnose missing evidence
Keep an access-controlled source reference for the evaluated answer instead of copying its full prompt into every span. When redaction removes part of the evidence, record which source version the permitted sample represents and that content was withheld. For request_demo, an investigator who cannot inspect the retrieval content cannot settle whether the answer ignored it; preserve that diagnostic limit alongside the quality finding. Credentials never belong in observation payloads.
For a failed request, check which stage observations are present, which measurements are absent and whether an answer has a separate reviewed quality result. Do not treat missing evaluation as a passing evaluation. See recorded use cases for recipe evidence and LLM hallucination for unsupported-output terminology.