Use object detection models with Video Intelligence Framework

Modern object detection models are deep neural networks trained on large datasets to recognize patterns associated with specific object classes. During inference, these models process video frames and output structured predictions containing: 

•   Class ID – Defines the category of the detected object (e.g., person, vehicle, animal)
•   Confidence score – Reflects the probability value (0.0 to 1.0) indicating the model's certainty
•   Bounding box – Specifies the object's location in the frame in rectangular coordinates

Within the Video Intelligence Framework (VIF), object detection runs in real time as part of the streaming pipeline in Wowza Streaming Engine. As video frames pass through the pipeline, the system analyzes them and converts detections into structured events. These events can be logged, displayed during playback through overlays or ID3 tags, or delivered to downstream systems using webhooks or custom Java listeners.

The framework supports compatible RF-DETR object detection models, allowing customers to deploy pre-trained models or integrate custom models trained on domain-specific datasets. This approach enables organizations to add tailored intelligence to video workflows for use cases such as security monitoring, traffic analysis, and industrial automation.

About RF-DETR

RF-DETR is a real-time transformer architecture for object detection developed by Roboflow and released under the Apache 2.0 license. Built on a DINOv2 vision transformer backbone, RF-DETR delivers state-of-the-art accuracy and latency trade-offs on benchmark datasets like Microsoft COCO and RF100-VL.

Key characteristics:

• Transformer-based architecture – Uses attention mechanisms instead of traditional convolutional networks, improving spatial reasoning and feature representation.
• Real-time performance – Optimized for low-latency inference, making it well-suited for live video processing and streaming pipelines.
• Multiple model variants – Available in four sizes, allowing deployments to balance compute requirements with detection accuracy.
• Native PyTorch implementation – Implemented in PyTorch, enabling direct integration with the Video Intelligence Framework model execution layer.

Model variants:

Variant	Input resolution	Use case	Inference time per frame on Nvidia T4 (ms)
Nano	384x384 px	Highest throughout, resource-constrained environments	2
Small	512x512 px	Balanced performance for general use	3
Medium	576x576 px	Higher accuracy for moderate-resolution streams	4.5
Large	704x704 px	Maximum accuracy for high-stakes detection scenarios	6

Larger models process higher-resolution inputs and typically achieve better detection performance, particularly for small objects. However, they require more GPU memory and take longer to process each frame. VIF's adaptive frame skip mechanism automatically adjusts processing rates to maintain real-time performance regardless of model size. For most use cases, we recommend using either the Medium or Large variant for best performance.

RF-DETR is our chosen object detection model for VIF due to its strong performance characteristics, active maintenance, ease of fine-tuning for specific needs, and permissive licensing. The framework's model management layer handles default checkpoint downloads (based on the Microsoft COCO dataset) automatically, with variants cached locally in ./models/ after first use.

Note: Custom model weights (.pth) must be placed in the VIS container's models directory (./vis/models/ when using Docker volume mounts). VIS loads the model on startup; VIC references it via the checkpoint_path setting in the stream configuration.

How to custom-train models

Custom training allows you to adapt RF-DETR to detect domain-specific objects not present in the standard COCO dataset. The process involves preparing a labeled dataset, configuring the training pipeline, and executing training on GPU hardware.

Note: For more information, Wowza provides a comprehensive training notebook that walks through the complete process.

High-level process overview

1. Dataset preparation
   • Labeling: Annotate images with bounding boxes using a labeling tool such as Roboflow, makesense.ai, or a similar platform.
   • Format: Export the dataset in COCO JSON format with separate train, validation, and test splits.
   • Structure: Place an _annotations.coco.json file inside each split directory alongside the images.
   • Preprocessing: Resize images to match the target model resolution (384px, 512px, 576px, or 704px) using letterboxing.
   • Recommended dataset size: Minimum 50 images per class, ideally 500+ for best performance.
2. Environment setup
   • Install RF-DETR training dependencies: pip install rfdetr==1.4.1 torch supervision roboflow.
   • Configure dataset access (for example, a Roboflow API key if using Roboflow Universe).
3. Model selection
   • Nano (384×384): Fastest inference with lower accuracy. Ideal for hardware-constrained environments.
   • Small (512×512): Higher resolution with improved detection performance.
   • Medium (576×576): Higher accuracy for moderate-resolution streams and a strong default for most use cases.
   • Large (704×704): Highest accuracy for critical detection tasks with slightly higher inference time.
4. Training configuration
   • Batch size: Adjust based on GPU memory (for example, T4: 4, A100: 16).
   • Gradient accumulation: Scale to maintain an effective batch size of approximately 16.
   • Epochs: Start with 20 and monitor convergence. If accuracy is insufficient, increase to as many as 50 epochs, especially for Medium and Large models.
   • Learning rate: Default configuration typically works well for fine-tuning.
   • Example training code:

     from rfdetr import RFDETRNano
     
     model = RFDETRNano()
     
     model.train(
         dataset_dir="/path/to/dataset",
         epochs=20,
         batch_size=4,
         grad_accum_steps=4
     )

5. Evaluation and validation
   • After training completes, evaluate performance on the test split.

   • model = RFDETRMedium(pretrain_weights="/path/to/checkpoint_best_total.pth")
     model.optimize_for_inference()
     
     # Run inference on test set
     detections = model.predict(image, threshold=0.5)

   • Review metrics_plot.png for training curves and results.json for final mAP scores.
6. Integration into VIF
   • Once satisfied with model performance, deploy the checkpoint_best_total.pth file using the model import process described below.

Training infrastructure recommendations

• GPU: NVIDIA T4 or better (16GB+ VRAM recommended for larger models)
• Environment: Google Colab (free tier with T4), AWS EC2 (g4dn instances), or on-premise GPU servers
• Training time: Typically 1–3 hours for 20 epochs on datasets with 500–1000 images
• Disk space: Reserve at least 10GB for checkpoints and training outputs

Best practices

• Start with a pre-trained checkpoint to accelerate fine-tuning.
• Monitor validation metrics to detect overfitting when validation mAP plateaus but training mAP continues to increase.
• Ensure the test split contains representative and edge-case data similar to production scenarios.
• Improve dataset quality if performance is poor by refining labels and adding more examples.
• Use preprocessing and augmentation pipelines to increase dataset diversity.

Note: For more information, Wowza provides a comprehensive training notebook that walks through the complete process.

For additional support with custom model training or integration, contact support.

Importing models to Wowza VIF

VIF supports custom RF-DETR models trained on domain-specific datasets. The process involves training a model using the RF-DETR training pipeline and integrating the resulting checkpoint into the VIF model management system.

Prerequisites

• A trained RF-DETR checkpoint file (.pth format)
• Access to the machine(s) where the VIF service is deployed

Integration steps

1. Export the optimized checkpoint
   • After training completes, locate the best-performing checkpoint in the training output directory.
   • The training process typically produces several checkpoint files:
   • • checkpoint_best_total.pth – Best overall performance (recommended)
     • checkpoint_best_ema.pth – Best exponential moving average weights
     • checkpoint_best_regular.pth – Best regular (non-EMA) weights
   • Select checkpoint_best_total.pth for production use and rename it if needed for clarity. For simplicity, the examples below continue to reference checkpoint_best_total.pth.
2. Upload to accessible storage
   • Place the checkpoint file in a location accessible to your VIF deployment.
     • Local filesystem: For on-premise deployments, copy the file to the VIF server (for example,  models/checkpoint_best_total.pth).
3. Configure the model in VIF
   • Reference the custom model in your video intelligence configuration within Engine by specifying the checkpoint path.

   • {
       "type": "config",
       "detector_type": "object",
       "checkpoint_path": "models/checkpoint_best_total.pth",
       "class_names": ["person", "vehicle", "animal"],
       "confidence_threshold": 0.5
     }

   • The model_checkpoint parameter overrides the default pre-trained weights so VIF loads your custom model instead of the standard checkpoint.
4. Verify the integration
   • Start a stream using the custom model configuration.
   • Monitor the logs to confirm that the model loads successfully.

   • INFO: Loading custom checkpoint from /path/to/checkpoint_best_total.pth
     INFO: Initialized ObjectAnalyzer with 3 classes on cuda:0

Important considerations

• GPU memory requirements: Ensure sufficient VRAM for your model variant (no less than 8GB RAM, but we recommend 16GB+ for optimal performance)
• Checkpoint format: VIF expects PyTorch .pth files in the format produced by RF-DETR's training pipeline