Release Notes

Wowza Video Intelligence Framework (VIF) brings real-time AI video analysis directly into live streaming workflows in Wowza Streaming Engine, with no separate video pipeline required. This release includes:

Detection and analysis

• Object detection: Real-time detection and classification of objects (people, vehicles, animals, and more) using RF-DETR models, with four size and speed variants (nano, small, medium, large) to match your latency and accuracy needs.
• Scene understanding: Whole-frame scene classification for detecting broader events and conditions, such as fire, smoke, and crowding, without needing per-object bounding boxes.
• Vision-language model (VLM) analysis: Open-vocabulary analysis using a vision-language model for cases where what you're looking for doesn't map to a fixed, pre-trained class list. Three modes cover different needs:
  • Detect: Checks for any class you describe in plain language, with per-class reasoning
  • Describe: Gets a free-text description of what's happening in each analysis window
  • Custom: Write your own prompt and output schema for fully custom structured results
• Object tracking: Persistent tracking IDs across frames using ByteTrack, so the same object keeps a stable identity as it moves through a stream instead of being re-detected from scratch every frame.
• Synthetic video detection: Flag AI-generated or manipulated video using NVIDIA's Maxine Synthetic Video Detector, returning a real/fake verdict per analysis window.

Deployment flexibility

• Docker Compose deployment: A single docker-compose.yaml with named profiles supports everything from an all-in-one single-server setup to fully split multi-server topologies (Streaming Engine and Manager on one machine, the AI inference service on a dedicated GPU server).
• Non-Docker installation: For environments that can't use Docker, both VIC and VIS can be installed natively; the Streaming Engine plugin onto an existing WSE instance, and the AI inference service as a self-contained Windows or Linux installer with GPU dependencies (CUDA, TensorRT) bundled in.
• NVIDIA Jetson support: A dedicated build for NVIDIA Jetson (JetPack, TensorRT-accelerated) for edge deployments, separate from standard amd64/arm64 GPU servers.
• Air-gapped deployment support: Model weights and container images can be pre-downloaded and transferred into environments without outbound internet access.

Integration and extensibility

• Multiple built-in event listeners: Deliver detection results as ID3 metadata (for player-side overlays and triggers), webhooks (for external system integration), structured JSON log files, or directly as video overlays.
• Custom event listeners: Build your own listener for use cases the built-in options don't cover, such as triggering recordings, alerting external systems, or custom data pipelines, using a documented Java interface and a reference implementation to build from.
• REST API: Manage and query stream configurations, detection results, and service status programmatically.

Management

• Wowza Streaming Engine Manager dashboard: Configure detection types, sensitivity, event listeners, and VLM settings for each stream via a UI, without hand-editing configuration files.

Reliability

• Resilience to AI endpoint outages: If a VLM endpoint becomes temporarily unreachable, the stream keeps running. VIF automatically resumes analysis once the endpoint recovers, with clear status indicators (an on-overlay badge and a stream status flag) distinguishing a real outage from a quiet scene.
• Adaptive frame throttling and fair GPU scheduling: The inference service automatically balances processing load across concurrent streams sharing a GPU and can throttle frame analysis to stay caught up with live instead of drifting behind in real time.
• Bounded synthetic-detection latency: If the synthetic video detector endpoint is unreachable or a request hangs, VIF reports that window as "unknown" rather than blocking the stream. A per-window timeout keeps latency bounded.