Wowza’s Video Intelligence Framework (VIF) turns live video into actionable, real-time insights. It embeds intelligence directly into live streaming workflows, enabling object detection, event identification, and automated response as video is ingested.
By separating the streaming and intelligence layers, VIF lets you deploy, upgrade, or replace computer vision models without redesigning your infrastructure. As models evolve, your pipeline stays intact.
Built on Wowza Streaming Engine, VIF deploys wherever your video operates, cloud, edge, or fully on-premises. It addresses security, compliance, and latency needs and enables advanced analytics at the same time. There’s no need to replace your current video infrastructure. VIF enhances it by adding intelligence, scalability, and future-ready capabilities.
Core components
VIF consists of two components that work together:
- Video Intelligence Controller (VIC) - Runs inside Wowza Streaming Engine as a set of plugin modules. VIC samples frames from live streams, sends them to VIS for analysis, receives detection results, and routes them to output channels (ID3 metadata, overlays, webhooks, logs, and custom Java listeners). VIC is non-blocking, which means it never interrupts the live stream.
- Video Intelligence Service (VIS) - a standalone, GPU-accelerated inference engine that runs as a Docker container. VIS receives frames from VIC and runs the AI models: RF-DETR for object detection, vifi-clip for scene analysis (experimental), and any custom-trained RF-DETR models. VIS returns structured JSON results per frame.
Note: VIC and VIS communicate over WebSocket (ws:// or wss://) and can run on the same host or on separate machines. VIC and VIS can be deployed together on one host (all-in-one) or on separate machines (split-server). Multiple WSE instances can share a single VIS, or one WSE can route to multiple VIS instances. See Deploy Wowza Streaming Engine with VIF for details.
How VIF processes a stream
- A live stream arrives at Wowza Streaming Engine via RTMP, RTSP, SRT, or WebRTC.
- VIC samples frames from the stream at the configured interval.
- VIC sends frames to VIS over WebSocket.
- VIS runs inference (object detection, scene analysis, or a custom model) and returns detection JSON.
- VIC routes results to all configured output channels simultaneously.
Steps 1, 2, 4, and 5 run inside WSE (VIC). Step 3 runs on the GPU host (VIS).
Object detection
Object detection is an integrated capability within the Video Intelligence Framework and operates directly within the Wowza Streaming Engine live streaming pipeline. The system analyzes video frames in real time, converts detections into structured events, displays them in Engine Manager, logs them in JSON format, and delivers them through webhooks.
Wowza VIF supports computer vision models. Wowza provides the infrastructure, orchestration, and integration layer, enabling customers to deploy their preferred models and embed intelligence directly into their streaming workflows.
Object detection is one of several intelligence capabilities available today, with more planned. Customers can expand their video analytics over time without changing their underlying streaming architecture.
Scene understanding
In addition to detecting individual objects, VIF can classify the overall activity or situation visible in a video frame, for example, identifying
a traffic intersection or crowded stadium.
Scene understanding classifies the overall scene by comparing video frames against configurable scene description labels. The default
configuration matches stream names using the scene.* regex; any stream name starting with `scene` triggers scene analysis.
Vision-language model (VLM) analysis
VIF can also route frames to a vision-language model (VLM) for open-vocabulary analysis. Instead of scoring a fixed set of trained classes, you can ask a VLM about any concept in plain language (for example, "person wearing a hard hat" or "smoke without visible flames"), and it returns its reasoning alongside each result. VIF ships with a ready-to-run local VLM sidecar, or you can point it at a hosted endpoint. See Use VLM analysis in VIF for details.
Synthetic video detection
VIF can also flag AI-generated or manipulated video using NVIDIA's Maxine Synthetic Video Detector (SVD) model, returning a per-window real/fake verdict. This is opt-in, and bring-your-own-endpoint is a local detector sidecar VIF can bring up for you, or any hosted/self-hosted SVD endpoint. See Use the Synthetic Video Detector in VIF for details.
Output
Out of the box, scene classifications are:
- Rendered as overlays on the transcoded -vi stream.
- Injected as ID3 metadata into HLS output.
- Available via webhook delivery and local log file events (if enabled).
Scene descriptions to analyse can be configured in the VIF configuration page in WSE Manager or through the WSE REST API.
Limitations
- As it is currently experimental, output quality may change between releases.
- Scene analysis requires internet access on the first run to download OpenAI CLIP base weights. The scene detection checkpoint can also be pre-downloaded for air-gapped environments (see VIS deployment guide). Subsequent runs use the cached weights from the
vis/models/directory. - For best results, keep source videos under 720p resolution.
