Video analysis with AI: visual evidence beyond transcripts
AI video analysis examines a clip’s visual content so a reviewer can ask about actions that a transcript does not describe. Start with a permitted video, identify the clip being analysed, ask a narrow question, then compare the answer with the actual footage. Keep visible actions, spoken statements and inferred explanations separate.
A transcript can tell you what someone said while an object was moved; it may not tell you where that object went. The YouTube integration reference covers transcript, comments and metadata retrieval. This guide adds a worksheet for reviewing visual claims about one supplied training clip.
The proposed video workflow is example configuration, not a recorded test. Its components are documented in the catalogue; not production-tested by Ai Flowi for this clip-analysis sequence.
1. Choose a question the video can answer
Ask about a concrete visible event, such as whether a tray was placed beside a shelf or on it. Avoid bundling object movement, identity, intent and procedural compliance into one question. A view of a person moving a tray may support movement and placement; it does not establish the person’s identity, emotion or whether an entire safety procedure was followed.
Use a synthetic training scenario: a presenter says “return the tray to its marked position” while placing it near a shelving unit. The spoken words do not identify the final position. The relevant visual question is “Where is the tray placed, and is the marking visible?” This scenario is an authored review exercise, with no real recording or provider answer claimed.
Before uploading, confirm permission to process the recording and to send it to the selected provider. You need AI Flowi Workflow, a selected video file and your own TwelveLabs account/API key. Refer to connections for credential handling, and confirm the provider and workspace accept your file before proceeding. References: connections
2. Separate loading, indexing and answering
The catalogue gives TwelveLabs Pegasus Index Video this description: Index videos using TwelveLabs and add the video_id to metadata.
TwelveLabs Pegasus is described as Chat with videos using TwelveLabs Pegasus API.
These are different stages with distinct outputs; indexing is not an answer to the question.
The video-review tables scroll sideways on small screens. Focus a table region to inspect its remaining columns with the keyboard.
| Component | Required settings and proposed choices | Output and boundary |
|---|---|---|
| Video File | file_path: select the permitted synthetic training file. | dataframe: Table. Its media-location columns are not documented. |
| TwelveLabs Pegasus Index Video | api_key: your provider credential; videodata: inspected Data/JSON representing the clip. Record your chosen index_id or index_name; listed model_name: "pegasus1.2". | indexed_data: JSON. Inspect the returned metadata rather than assuming a nesting path for video ID. |
| TwelveLabs Pegasus | api_key and message are required. Supply the reviewed question, confirmed video_id and applicable index settings. The exported model_name choice is "pegasus1.2". | response: Message; processed_video_id: Message. Neither declares frame coordinates or timestamp fields. |
These model names belong to the catalogue snapshot, not a statement that the provider currently accepts them for your account. The answer node also exposes videodata and temperature; no default, sampling bounds or quality guarantee is exported. Choose and verify those settings in your configured workspace.
3. Inspect each handoff before submitting a question
- Load the selected source through Video File and inspect its Files Table. Record the local source label and revision in your own worksheet.
- Resolve the Table-to-Data/JSON boundary before indexing:
videodataon the indexing node does not list Table as an accepted type. A matching media reference must be inspected and prepared; no direct wire is established here. - Configure the provider key and intended index. Run a controlled indexing example only after confirming the input shape, then inspect the JSON result and associated video identity.
- Supply that confirmed identity to the question stage. A JSON metadata object is not itself the Message accepted by
video_id; for this staged design, copy the inspected ID into the field. - Set
messageto “Describe the tray’s final visible position. Say if the marking or landing surface is obscured. Separate observed actions from spoken instructions.” - Read the returned response and verify that
processed_video_ididentifies the intended clip before reviewing any content claim.
The manual inspections make the proposed flow reviewable without inventing a payload adapter. They remain necessary until a controlled runtime example establishes the media shape, indexing completion and identifier mapping. The catalogue offers no readiness-polling or indexing-latency contract for this sequence.
4. Review an answer against visible evidence
Prepare the worksheet independently by watching the clip. Record reviewer-observed positions or playback ranges only after inspecting the media; the following entries are authored acceptance criteria. They are not provider timestamps, returned fields or observed analysis results.
| Proposed claim | Evidence needed | Review decision |
|---|---|---|
| The tray moves towards the shelf. | An unobscured view showing that movement. | Accept only the visible action. |
| The tray is on the marked spot. | The landing surface and marking both visible. | Unresolved if a hand or camera angle blocks either. |
| The presenter says to return the tray. | Audible speech or separately checked transcript. | Record as speech, not proof of placement. |
| The procedure was performed safely. | A separate applicable procedure and sufficient evidence of each requirement. | Outside this narrow visual exercise. |
If the model says “on the shelf” but the clip reveals only “beside the shelf,” reject that wording. If the final position is off-camera, keep it unknown. A detailed answer cannot supply visual evidence for an event that the recording does not show.
5. Use ambiguity and wrong-clip checks
Prepare a second permitted clip where the narration is unchanged but the visible action differs. The review target must follow the footage rather than copying the spoken instruction. Also test an obscured final position, a silent version and an unrelated clip with a similar filename. Confirm that an uncertain response remains visibly uncertain.
Keep source identity, question wording, selected model, response text and reviewer correction together. After replacing the source recording, repeat identity checks instead of reusing an old video ID. A result from the previous revision may be internally coherent and still answer the wrong question.
Related: Computer vision: distinguish image evidence, recognition and inference.
What are the scope and limits of video analysis AI?
Use ordinary playback when a person can directly inspect the event without a model call. Do not use this design for surveillance, identity recognition, emotion detection or an unattended compliance judgment. The export does not establish supported codecs, upload limits, transcript alignment, frame sampling or timestamp accuracy.
To locate a relevant clip across several recordings, use the separate video retrieval design before applying this review. Searching a collection and answering about one chosen clip require different acceptance checks.
Plans and workflow capacity. Go: 5,000 Flowi Tokens; Plus: 25,000; Pro 100: 50,000; Pro 200: 100,000; Pro 500: 250,000 per month. Choose the capacity that fits your workflow, connected tools and review process. Book a demo
Count loading, indexing and question executions from actual run history, including attempts repeated after a failed handoff. No visual accuracy, successful combined run or provider processing-time measurement is supplied for this proposal.
By Ai Flowi · Published .