Object detection and scene understanding both score a fixed set of trained classes. A vision-language model (VLM) instead understands free-text vocabulary; “person wearing a hard hat,” “forklift near pedestrians,” “smoke without visible flames”, and explains its reasoning with every result. Use VLM analysis when the thing you're looking for doesn't map cleanly onto a pre-trained class list, or when you want a written explanation alongside each detection.
VIF ships with a ready-to-run VLM sidecar (an OpenAI-compatible vLLM server running Qwen3-VL-4B-Instruct-FP8, commercial-use friendly) so it can run entirely on your own GPU, or you can point it at a hosted OpenAI-compatible endpoint instead.
Analysis modes
VLM analysis works one of three ways, controlled entirely by which configuration fields you set, and there is no separate mode switch. The Manager UI (Engine Manager → VIF → Stream Configuration, with Detector Type set to VLM) presents these as three named modes:
- Detect — Set a list of class names (any short phrase, open vocabulary). VIF returns a per-class verdict with reasoning for each class actually visible in the frame, rather than a fixed score for every class every time. You can optionally attach a hint to a class to disambiguate it (for example, clarifying that “fire” means a visible open flame, not red lighting).
- Describe — Set no class names and no custom prompt. VIF returns a free-text description of each analysis window instead of a class list.
- Custom — Write your own prompt and, optionally, your own output schema. Useful when you need structured output beyond the standard class/reasoning shape (for example, reading a value off a gauge).
Results are delivered through the same event listeners as object and scene detection — ID3 tags, webhooks, log files, and video overlays — with one difference: VLM results don't include bounding boxes, since a VLM doesn't localize objects spatially. Overlays show the detected class names or description text instead of boxes.
Resilience
If the VLM endpoint becomes unreachable, the stream stays live, and VIF emits empty results and resumes analysis automatically once the endpoint recovers. During an outage, the overlay shows an “AI offline” badge (distinguishing a real outage from a genuinely quiet scene), and the stream's status reports a degraded flag in the Manager dashboard. This is separate from the VIS connection status, which remains connected throughout a VLM-endpoint outage; only the VLM endpoint itself is affected.
Deployment
The VLM sidecar is opt-in via the vlm Docker Compose profile; it never starts with a bare docker compose up. Supported topologies include running everything on one machine, splitting the engine from VIS and the VLM sidecar across two machines, running one engine against several VIS/VLM pairs, or pointing several engines at a single shared VIS/VLM deployment.
For exact commands, environment variables (model selection, GPU pinning, memory limits, air-gapped weight pre-seeding), and topology diagrams, see docs/VLM_GUIDE.md in the VIF GitHub repo; this is the source of truth for setup and configuration and is kept in sync with each release.
Configuration reference
VLM settings live in the vlm_analysis block of video-intelligence.json, set globally as defaults or per stream as overrides. For the full field reference including prompt placeholders, response schema behavior, and sidecar tuning variables- see docs/VLM_GUIDE.md.
