Machine learning pipeline: training and inference boundaries
A machine learning pipeline is a sequence of stages that prepares data, develops or applies a model and checks the resulting outputs. A training pipeline uses examples to fit a model; an inference pipeline uses an existing model to produce predictions for new inputs. Moving data through a workflow does not, by itself, train a model.
The key boundaries are which data can influence learning, which evidence is reserved for evaluation and which model version receives a live input. Make those boundaries visible before selecting tools. A pipeline can produce a valid-looking prediction while using information that would not exist when the real decision is made.
A training and inference swimlane
Conceptual diagram for a fictional ticket classifier. No model is trained, deployed or scored here. The task is to assign a topic to a ticket from text available when it is first submitted. The classifier’s categories are delivery, product and account; unresolved text needs review.
| Stage | Training lane | Inference lane | Artifact to retain |
|---|---|---|---|
| Source selection | Permitted historical ticket text and reviewed topic labels. | New ticket text available at submission. | Source revision, selection policy and available-field list. |
| Partition and preparation | Separate training, validation and test examples; fit learned preparation on training only. | Apply the approved preparation associated with the selected model. | Partition ledger and preparation revision. |
| Model use | Fit parameters using training examples and labels. | Apply a fixed model version to prepared input. | Model version and compatible input contract. |
| Decision checks | Use validation for development choices; evaluate the final choice on held-out test evidence. | Check output category, uncertainty policy and any later reviewed outcome separately. | Evaluation policy or per-input review record. |
The lanes share an input meaning, not identical source records. Historical training data may include reviewed labels unavailable for a new ticket. Those labels guide learning; they must not accidentally become part of the new-ticket input contract.
Related: Fine tuning and transfer learning: changing weights versus supplying context.
What training, validation and test sets do
The training set supplies examples used to fit learned parameters. Validation data supports development choices, such as selecting among configurations. Test data is held aside to examine the final selected approach. Repeatedly changing the model after inspecting the test results turns that evidence into another development influence.
The split should match the intended use. Related copies of one ticket should stay together so a near-duplicate does not appear on both sides of an evaluation boundary. If the goal concerns future tickets, examine whether a time-based separation better represents the information that will be available later.
There is no universal split percentage for this exercise. Dataset size, related records and the task’s timing matter. Record the partition rule and example identities rather than relying on a fraction alone to establish independence.
The data cleaning explanation covers learned preprocessing across partitions; for this classifier, retain the preparation revision with each model so the submission-time input can be reconstructed.
How does a ticket example expose data leakage?
Invent a ticket T-A whose submitted text is “The parcel has not arrived.” Its reviewed topic is delivery. A later case note says “Replacement authorised after delivery review.” The intended classifier runs at submission, before that later note exists.
| Candidate field | Available at submission? | Decision | Reason |
|---|---|---|---|
| submitted_text | Yes | Allowed input | The task receives the original request. |
| reviewed_topic | No | Reference label only | It is the answer the classifier is supposed to predict. |
| resolution_note | No | Exclude from submission-time input | It records later events and directly names delivery review. |
| final_route | No | Exclude from submission-time input | A routing outcome cannot stand in for an earlier prediction. |
Including resolution_note during training or evaluation lets the model use a clue unavailable at inference time. A strong result on that contaminated input would not establish performance on new submissions. The root problem is the data-availability contract, so removing the field only from the live workflow leaves the evaluation misleading.
A different task could legitimately use the later note, such as summarising resolved cases. That would be another pipeline with another input boundary. Do not reuse its evaluation evidence to claim that the submission-time classifier has been validated.
Model version and preparation version travel together
For this conceptual exercise, call the selected model ticket_model_A and its preparation text_rules_A. These are fictional artifact labels, not provider model identifiers. An inference record should identify both, plus the topic-policy version and the exact input revision.
If preparation changes, the same words may reach the model in a different form. If category meanings change, an unchanged model output may mean something different to the destination. Keep those revisions linked so a reviewer can reconstruct why the same ticket received a different proposal.
Suppose text_rules_A preserves the word “not,” but a replacement preparation revision removes it. T-A would reach the unchanged model without the negation in “has not arrived.” Recording only ticket_model_A would hide the changed input meaning. Compare the prepared text with its source and treat the new preparation as a different candidate pairing to evaluate, rather than assuming an unchanged model version makes the result comparable.
Where AI Flowi Workflow fits
The Language Model component and Prompt Template reference explain provider inference and prompt construction, both documented in the catalogue; not production-tested by Ai Flowi.
A provider call can apply an existing model to T-A, but it does not fit the classifier depicted in the training lane. Changing the prompt’s category instructions changes the inference policy; it supplies no evidence that model parameters were updated. Record that prompt revision separately from ticket_model_A and text_rules_A. If an existing model replaces the fictional trained classifier, describe the resulting design as a different inference experiment with its own review evidence.
The catalogue is the component authority. No native training, deployment, feature-store or monitoring service is claimed here. Provider-backed experiments require a separately paid provider account and any applicable credentials. A workflow diagram cannot substitute for an actual training artifact or an evaluation record tied to the selected model and preparation pair.
Common pipeline mistakes and related concepts
Do not place copies of one case in development and test partitions, feed later outcomes into earlier predictions or compare results after silently changing labels. Check feature availability at the moment of inference before fitting anything. A diagram alone cannot establish that the implemented pipeline preserves those boundaries.
ETL owns data movement and transformation. AI agent evaluation owns task and tool observations, while LLM explains language models. Annotation and synthetic data cover reference labels and invented examples used before learning or testing.
By Ai Flowi · Published
Related: Batch inference: preserve row identity and reconcile partial results.