How To Configure Custom AI Analysis In Wowza Video Intelligence Framework

Choose among object detection, scene analysis, vision-language models, and synthetic video detection for live streams, all configurable options in Wowza Video Intelligence Framework (VIF).

Wowza Video Intelligence Framework (VIF)® gives teams several ways to analyze live streams without building a second video pipeline. Object detection, scene analysis, vision-language model analysis, and synthetic video detection solve different problems. They should not be treated as interchangeable, and a more complex model is not automatically a better choice.

Adding AI to a live video workflow starts with a simple question: What decision do you need the video to support? The answer should determine the analysis method, the model, the output, and the amount of compute required. Starting with the model and working backward usually creates unnecessary complexity.

What Type Of Model Is Best?

There is no one “best” model for every use case, rather the most important distinction is what the output needs are. Do you need a bounding box around an object? A score against a defined scene description? A natural-language interpretation of what is happening? Or an assessment of whether the video may be synthetic or manipulated? Answering those questions first determines what type of model is the best fit for your use case.

Each path has a different purpose, cost profile, and failure mode. In many production systems, the right answer may combine multiple model types. Object detection may provide precise localization and tracking, while a VLM is invoked less frequently to interpret a more complex event. Synthetic video detection may be the first step in the ingest pipeline, and anything returning a score above the threshold could get sent to a VLM to generate a justification or rationale for why it was scored as such.

What Does VIF Support?

Object Detection and Tracking in VIF

Object detection is the right starting point when the operational requirement depends on where an object is, how many objects are present, or how an object moves through the scene. Examples include counting vehicles, identifying a person in a restricted zone, or following a piece of equipment across successive frames.

VIF uses RF-DETR for object detection because, based on our evaluation, it provides the best combination of detection accuracy, real-time performance, model-size options, trainability, active maintenance, and licensing for VIF. The detector returns the class, confidence score, and bounding box for each detected object. ByteTrack can be enabled when the workflow needs a stable track ID across analyzed frames.

A custom RF-DETR model can extend detection beyond the standard COCO classes. That is useful for domain-specific objects such as specialized equipment, product defects, or industry-specific safety gear. The quality of the result depends heavily on the quality and representativeness of the training data.

Scene Analysis

Scene analysis compares sampled frames with natural-language descriptions configured by the operator. It is useful when the question is broader than a single object but still well-defined, such as whether a roadway appears congested or whether a loading area is occupied. VIF uses ViFi-CLIP for scene analysis, currently experimental, so the outputs may change between released versions.

This path is lighter and more constrained than a VLM. It does not generate an open-ended, AI-generated explanation of the scene. It evaluates the descriptions you provide, which can make it a better fit for repeated, narrowly defined checks where predictable output and lower compute cost matter more than detailed descriptions.

Vision-Language Model (VLM) Analysis

A vision-language model (VLM) combines visual input with natural-language instructions. It can interpret a sampled frame or analysis window, relate multiple elements in the scene, describe activity, read visible text or instruments when the image quality supports it, and return either natural language or structured data.

A VLM is not simply an object detector with a larger list of labels. It is most useful when the task depends on context, relationships, sequence, or instructions that cannot be reduced cleanly to a set of trained object classes. That could include tasks like:

• Summarizing Activity: Describing what changed during the analysis window and identifying the events that appear operationally significant.
• Interpreting A Procedure: Assessing whether the visible activity appears consistent with a written safety procedure and identifying the visual evidence used for the assessment.
• Extracting Structured Information: Reading a visible gauge, sign, or status panel and returning the observed values in a defined JSON schema.
• Explaining Scene Context: Describing the relationship among people, equipment, and the surrounding environment, rather than only listing the objects present.

VLMs can miss details, misread text, or generate a plausible explanation that is not supported by the video. Production use requires representative testing, clear thresholds for automated action, and human review when the consequence of an error is material.

Synthetic Video Detection

VIF is integrated with the NVIDIA Synthetic Video Detector (SVD) to assess live video for signs indicating whether it may be AI-generated or manipulated. Each analysis window returns a verdict of synthetic, real, or unknown, along with a score between 0.0 and 1.0. This score is measured against a configurable threshold set by the user. It should be evaluated as one control within a broader verification or compliance workflow, with thresholds and escalation paths appropriate to the risk of the use case. The verdict is not absolute proof of authenticity. Rather, it should be used as a signal to further evaluate if content was AI-generated or manipulated. Customers can authenticate with their own NVIDIA key. Detection is opt-in per stream and bring-your-own-endpoint, which could be a local sidecar or an endpoint you already run. The source stream must be H.264.

What Does VIF Do With The Detection Results?

VIF can route results through enriched HLS with ID3 timed metadata, burned-in overlays, JSONL logs, webhooks, and Java listeners. Multiple outputs can be enabled at the same time. It’s worth noting that the overlay requires transcoding, and the verdict overlay is off by default for synthetic video detection unless you enable it on the stream.

The delivery mechanisms are consistent across VIF, but the result schema depends on the detector. Object detection includes spatial coordinates. Scene analysis returns results for configured descriptions. VLM results may contain a description, a detected concept, an AI-generated explanation, or fields defined by a custom schema. Synthetic video detection returns its own score or verdict. Downstream systems should be designed for the detector-specific payload rather than assuming every result has the same fields.

Inference runs independently from the live delivery path. If analysis slows or a model endpoint becomes unavailable, the live stream should remain available. VIF can skip analysis samples, report degraded AI status, and resume analysis when the service recovers without making inference a dependency for stream uptime.

How Do You Choose An RF-DETR Model Size?

There is no universally correct model size. Published per-frame timing numbers are not a capacity plan. Real performance depends on the GPU, software stack, source resolution, preprocessing, sampling rate, number of concurrent streams, overlays, and other workloads sharing the system.

VIF currently supports four RF-DETR variants. Larger variants analyze a higher-resolution model input and generally improve accuracy, especially for smaller or more difficult objects, but they also require more GPU resources and take longer to process.

The Medium model variant often is a strong default starting point for most use cases, and Wowza recommends Medium or Large for the best performance. But, it’s best to test representative production footage, measure the errors that matter, and then select the smallest model that meets the required accuracy. VIF’s adaptive frame skipping protects real-time delivery when inference cannot keep up, but skipped analysis frames can still reduce event coverage. Stream uptime and analysis completeness are different service objectives and should be measured separately.

How Does VLM Analysis Work In VIF?

VLM analysis in VIF is configured in three modes: Detect, Describe, and Custom. Within the packaged Detect workflow, a Reasoning level can be set to Low, Medium, or High (High is the default). These levels control how much the model deliberates for each analysis window.

Wowza documents and validates model and serving configurations for VIF’s VLM integration for each release. This allows us to qualify multimodal input, API behavior, structured output, performance, hardware requirements, licensing, and operational reliability before a configuration is made available to customers.

VIF ships a ready-to-run VLM sidecar running Qwen3-VL-4B-Instruct-FP8 on an OpenAI-compatible vLLM server, and added NVIDIA Nemotron Nano 12B VL, Google Gemma 3 4B, NVIDIA Cosmos3 Edge, and NVIDIA Cosmos3 Nano as selectable models. Selecting a model doesn’t deploy it automatically. A matching VLM sidecar must be running on the server to deploy it.

Detect

Detect mode accepts operator-defined concepts expressed as short phrases. This is often described as open-vocabulary detection because the concepts are not limited to the fixed label set of an object detector. VIF returns the concepts that the model judges to be present. The output also includes the model’s reasoning for the judgment.

The term “detection” can be misleading here. VLM Detect mode does not provide the bounding boxes, object counts, or stable tracks expected from an object detector. Instead, it operates more at a conceptual level and simply says whether the specific concept is present or not. Use it for concept-level judgment calls, and use object detection when spatial precision and repeatable per-object tracking results are required.

Describe

Describe mode uses the built-in descriptive prompt and returns a natural-language description of the analysis window. It is useful for investigation, indexing, content discovery, and operator context. It is usually a weaker fit for deterministic alerting because the language and level of detail can vary across results.

Custom

Custom mode gives the developer control over the system prompt, user prompt, and optional response schema. Without a schema, the result remains free-form. With a schema, the model is instructed to return structured fields that downstream systems can consume.

Reasoning

Detect returns reasoning by default, with the level set to High unless configured otherwise. A custom result contains only what the prompt and schema request, and Describe mode returns a description rather than a separate reasoning field.

Within the packaged Detect workflow, the reasoning-level setting controls the amount of model deliberation used for the result. Higher settings trade speed for additional verification effort. This is a model behavior and performance control. It does not turn the AI-generated explanation into proof that the answer is correct.

What Happens When The VLM Endpoint Is Unavailable?

If the VLM endpoint becomes unavailable, or the AI inferencing fails, the live stream remains live and available. VIF reports the VLM path as degraded, emits empty analysis results while the endpoint is unreachable, and resumes analysis when the endpoint recovers. This can distinguish an AI outage from a scene that simply produced no detections. AI analysis can add operational value without becoming a potential point of failure for the primary video workflow.

When Is Scene Analysis Better Than A VLM?

Use scene analysis when the concepts are known in advance, the output can be expressed as a set of configured descriptions, and the workload benefits from a lighter, more repeatable analysis path. VIF uses ViFi-CLIP for scene analysis (still experimental). Use a VLM when the task requires broader language understanding, relationships among elements, interpretation across an analysis window, free-form description, or a custom structured response. A VLM should not be the default simply because it is more flexible. Flexibility carries compute cost, latency, and greater output variability. The model with the least complexity, that reliably supports the decision, is a more efficient production choice.

How Do You Train And Import A Custom RF-DETR Model?

The process for training and importing a custom RF-DETR model in VIF includes building a representative dataset, fine-tuning and evaluating it against a test set, and then deploying the supported checkpoint to production. Train a custom RF-DETR model when the required objects are not covered by the standard AI model, or when the standard model performs poorly in the customer’s environment. The full training workflow benefits from labeling data consistently, evaluating errors thoroughly, and monitoring performance after release.

1. Build a representative dataset

Annotate images with bounding boxes, then export them in COCO JSON format, with an _annotations.coco.json file in each split directory. Letterbox the images to your target model resolution, and separate training, validation, and test data before training begins. The dataset should represent the real camera angles, lighting, weather, motion blur, occlusion, compression, object sizes, and negative examples the model will see in production.

There is no universal minimum number of images per class, but Wowza’s documentation recommends a minimum of 50 labeled images per class. 500 or more is even better. A simple object in a controlled scene may need far fewer annotated images than a more variable object across hundreds of cameras. Dataset quality, coverage, label consistency, and the cost of false positives/negatives matter more than a single image-count rule.

2. Fine-tune and evaluate the model

Start from a supported pre-trained RF-DETR checkpoint. Select batch size and other training parameters based on the chosen model and available GPU memory. Monitor validation performance and stop when additional training no longer improves generalization.

Evaluate the model on a held-out test set and on complete representative video, not only still images. Review per-class precision and recall, missed events, false alarms, small-object performance, and behavior under difficult conditions. The acceptance criteria should reflect the operational consequence of each error type.

3. Deploy the supported checkpoint

VIF’s object-detection path expects a compatible PyTorch .pth checkpoint produced by the supported RF-DETR training pipeline. In the current Docker layout, custom checkpoints are placed in the models directory and referenced through checkpoint_path in the matching stream configuration. Restart the service so the new checkpoint loads, then verify the load and model metadata in the logs.

{
         "type": "config",
         "detector_type": "object",
         "checkpoint_path": "models/checkpoint_best_total.pth",
         "class_names": ["custom_class_a", "custom_class_b"],
         "confidence_threshold": 0.5
 }

Field names, file locations, and supported versions should be checked against the configuration reference for the VIF release being deployed.

The Practical Recommendation

Choose the analysis path from the decision the customer needs to make, then validate it with the video they actually operate. Object detection is good for localization, counting, and tracking. Use scene analysis for constrained comparisons against known descriptions. A VLM is good at language-driven interpretation, context, and custom outputs. Synthetic video detection is non-negotiable when authenticity risk is part of the threat model.

VIF’s value is not that every workload uses the same AI model, but rather that different analysis methods can operate within the live video infrastructure, use a common delivery and integration layer, and remain independent from the availability of the primary stream. For more information and a demo, reach out to a Wowza expert.

Frequently Asked Questions

Does a VLM explain its reasoning with every result?

Not necessarily, the reasoning output depends on the selected mode, prompt, and response schema. Detect returns reasoning alongside every class it reports. Describe returns a natural-language description rather than a reasoning field. Custom returns the fields or text requested by the developer. Explanation is optional and should not be treated as proof of correctness.

Can VLM analysis replace object detection?

Object detection and VLM analysis serve two different purposes. VLM analysis is not suitable when the workflow requires bounding boxes, accurate per-object counts, or stable tracking. A VLM can interpret concepts and context, but VIF’s VLM results do not provide the spatial localization of the object-detection path.

Which RF-DETR model size should run on a live stream?

Wowza VIF supports four RF-DETR model sizes: Nano, Small, Medium, and Large, at 384, 512, 576, and 704 pixel input resolution. That resolution is separate from the input stream’s resolution, which VIF resizes transparently, though the recommendation is to keep source video at or below 720p for the best results. Use representative production video to select the smallest model that meets the accuracy requirement. Larger models can improve difficult or small-object detection, but capacity depends on the full deployment, not only the model’s published per-frame latency.

What are the three VLM analysis modes in Wowza VIF?

VLM analysis in Wowza VIF operates in Detect, Describe, and Custom modes. Detect returns the classes that are present in each window along with the model’s reasoning, against an open-vocabulary list. Describe returns free-text descriptions of each analysis window, and Custom applies a user-supplied prompt and optional output schema.

Do VLM results in VIF include bounding boxes?

No, VLM results in VIF do not include bounding boxes because a vision-language model does not localize objects spatially. Overlays render the detected class names or description text instead of boxes, while ID3 timed metadata, webhooks, JSONL logs, and Java Listeners carry the VLM results the same way they do for object detection.

What happens to a live stream if the AI analysis fails?

If AI analysis in VIF fails, the live stream stays live. VIF emits empty results, resumes analysis once the endpoint is back online, displays an “AI Offline” badge on the overlay, and reports a degraded flag for operators. The live stream remains available. VIF isolates inference from delivery, reports degraded analysis status, and resumes analysis after recovery.

How much training data does a custom model need?

There is no universal minimum, the amount of training data depends on visual variability, class balance, camera diversity, object size, environmental conditions, and the error tolerance of the use case. A custom RF-DETR model typically requires 50 labeled images per class at minimum, with 500 or more per class yielding the best performance. Detection accuracy depends on the quality and quantity of the labeled dataset, not the framework. Build coverage around production conditions and measure performance on held-out data and complete video sequences.

How long does it take to train a custom model for VIF?

Training a custom RF-DETR model for VIF, running on an NVIDIA T4 or better, typically takes between one to three hours for 20 epochs on a dataset of 500 to 1,000 images. Wowza provides a guided training notebook in the VIF Github repository to support the training process.

What format does Wowza VIF use for custom models?

Wowza VIF expects PyTorch .pth checkpoint files for custom-trained models, using the RF-DETR architecture. The checkpoint is placed in the models directory and referenced through checkpoint_path in the stream’s configuration file.