Text classification: labels, abstention and evaluation

By Ai Flowi · Published 10 October 2026

Text classification assigns one or more labels to a unit of text using a defined taxonomy. The unit might be a document, an abstract or a passage; the taxonomy determines what the labels mean. A useful classifier also has a way to abstain when the supplied text does not support a label or leaves competing labels unresolved.

Single-label, multi-label and hierarchical tasks

Single-label classification chooses one category for each accepted input. That choice needs a rule when the text covers several subjects. “Choose the main topic” is insufficient if reviewers disagree about whether main means the opening subject, the longest discussion or the document’s stated purpose.

Multi-label classification permits several independently applicable categories. It needs an inclusion rule for each category and a representation for an empty set. A hierarchy organizes labels into parent and child categories, such as Operations followed by Facilities. Decide whether a parent can be accepted without selecting a child, and whether multiple branches are allowed.

These structures are task contracts, not measures of model sophistication. A multi-label response is unsuitable for a destination that accepts one category unless a reviewed rule resolves it. A hierarchy can describe the taxonomy while the actual task still requires one leaf label.

A synthetic taxonomy for document abstracts

Fictional, permissioned internal-document scenario; no classifier run is reported. Imagine a research team sorting abstracts into an index. Taxonomy version topics_v1 uses three single-label categories plus a review state.

Consider three invented abstracts: “Procedure for calibrating temperature measurements” supports Methods; “Inspection schedule for laboratory ventilation” supports Facilities; and “Approval and retention rules for study records” supports Governance. These are author-assigned teaching labels, not predictions.

“Approval requirements for a laboratory renovation” concerns both a facility and an approval rule. Under a single-label scheme without a priority rule, return needs_review and retain both reasons. In a reviewed multi-label scheme, Facilities and Governance could both apply. Changing the label structure changes the correct answer.

What is the difference between classification, entity recognition and keyword matching?

A document can mention a university while its topic is ventilation maintenance. Named entity recognition identifies and types that name’s span; classification assigns a label to the abstract as a whole. Those outputs answer different questions.

A topic keyword is evidence to inspect, not necessarily a classification rule. The word “approval” can occur in an exclusion, a quotation or an incidental sentence. Start with deterministic matching when the category is explicitly supplied by a trusted field. Use interpretation when the topic has to be inferred from permitted text, and retain the taxonomy boundary that supports the inference.

The support ticket classification guide applies labels to ticket triage and routing. This document-index example has no queue, severity or customer-action requirement. The label contract here concerns subject matter and reviewability.

Related: Keyword extraction from text: phrases and evidence.

Abstention is a decision, not a parsing failure

Use an explicit abstention reason: insufficient content, outside taxonomy, conflicting topics or unsupported output. Preserve the original document reference and taxonomy version. A well-formed needs_review record can satisfy the output contract while remaining unsuitable for automatic indexing.

A shortened abstract may omit the sentence that determines its subject. If only “laboratory renovation” survives truncation, Facilities looks plausible even when the missing passage explains approval policy. Record which text version was classified and mark an incomplete abstract for review under topics_v1. A label on a fragment must not silently become the label for an unseen complete document.

For this example, a proposed application result has document_ref, taxonomy_version, label, evidence_excerpt and abstention_reason. A label outside the approved inventory is a contract violation. A permitted label with irrelevant evidence is a semantic failure. Record those separately.

An illustrative confusion matrix

Hand-constructed scoring illustration, not benchmark evidence. Suppose six labelled examples receive the hypothetical assignments below. Rows show reference labels; columns show proposed labels. Each row contains two examples.

The diagonal contains four correct assignments out of six. Methods precision is two correct Methods predictions out of three Methods predictions; its recall is two found Methods examples out of two reference Methods examples. Facilities has one correct prediction out of one proposed Facilities label, but only one of its two reference examples was found. The distinction reveals errors an overall total can conceal.

Record abstentions outside this accepted-assignment matrix. If two additional examples were withheld for review, the evaluation population would be eight and accepted coverage would be six out of eight. Do not present accuracy on accepted assignments without reporting that coverage and examining the reviewed cases.

Held-out evaluation and changing labels

Keep a held-out set separate from the examples used to write the taxonomy or tune the prompt. Include short, mixed-topic and out-of-scope abstracts. Group near-duplicate documents before splitting so a lightly edited copy does not appear in both development and evaluation sets.

Ask reviewers to annotate independently, then inspect disagreements. A vague inclusion rule is a taxonomy problem; a prediction that violates an agreed rule is a model mistake. Preserve unresolved human disagreement instead of pretending every reference is certain.

Taxonomy drift occurs when label meanings or categories change. Version the taxonomy and record how old categories map to new ones before comparing counts. Changing document language or subject mix is a different shift, which can alter error patterns even when the labels stay fixed.

AI Flowi Workflow component boundaries

Use the Structured Output and Batch Run catalogue entries for the interfaces underlying these document-taxonomy trials.

The two trials answer the same subject-label question at different input scales. In the batch trial, keep document references in the review copy but select only the abstract for model interpretation. An administrative folder called Governance must not become evidence for classifying an abstract about measurement procedures. Check the resulting label against the text rather than against the storage location.

example configuration, not a recorded test

For a single abstract, supply the document reference and text through Structured Output.input_value. Set system_prompt to the reviewed topics_v1 inclusion rules and needs_review policy; set schema_name to document_topic and configure an inspected output_schema Table for the proposed result fields. Select a configured LanguageModel-compatible model.

For a separate batch trial, provide a permissioned Table through Batch Run.df, set column_name to abstract and output_column_name to topic_proposal, and place the taxonomy in system_message. Inspect whether source references and result values remain associated. No direct Structured Output-to-Batch Run chain or enforced batch-result schema is asserted.

Both alternatives require your own model-provider account/API key, with provider charges paid separately. Structured Output and Batch Run are documented in the catalogue; not production-tested by Ai Flowi. Use manual review where the collection is small or the labels control consequential decisions. The use-case hub supplies existing workflow examples with their own scope; structured output explains the format boundary.