Use model chaining in VIF

A chain is a detector built out of other detectors. Instead of running one model on a stream, a chain runs named stages, each an ordinary Object, Scene, Motion, VLM, or Synthetic detector over the same captured media and decides for each window which stages actually run. The most common reason is cost: run a cheap stage on everything and wake an expensive one only when the cheap one saw something. Ask a VLM about a frame only when motion was detected, review a clip only when it scored as synthetic, or read a license plate only when a car is in view.

Use a chain anywhere the v2 API accepts a config.detector: inline VOD jobs, saved stream-group configs, per-stream overrides, or a running stream’s detector. A frame-only chain works on VOD and live streams; a chain containing a Synthetic stage is VOD-only. Chains cannot be nested inside a stage.

How a chain runs

Everything a detector does happens inside an analysis window. The engine captures and sends the unit of media for analysis. Depending on the detector, a window is a single frame, a batch of frames covering a few seconds, or an encoded clip, and within each window, several stages take their turn on that same captured media.

Every window starts at the chain’s entry stage (stages[0]). Each time a stage runs is one execution (called a ring in stored results). After an execution, the chain picks what happens next. The next stage in order (linear) is a routing rule’s target (conditional), or whatever your Java code returns (dynamic), until a stage ends the window or nothing is left to run. The sequence actually taken is the window’s path; it can differ window to window, skip stages, and revisit a stage. max_rings bounds the executions per window.

A chain captures the largest unit any of its stages consumes:

A chain whose Object or Motion entry is followed by a Scene or VLM stage runs at window cadence and needs a service that advertises chain.window_entry.

When a window ends, its results flow where any detector’s results flow. Each VOD window is stored as one result row, and event listeners (webhooks, ID3 tags, log files, overlays) receive the final stage’s result or every execution when result_mode is combined.

Build a chain in the Manager

1. In Wowza Streaming Engine Manager, open VIF ▸ Configs ▸ Stream Configs and select or create a config.
2. Set Detector Type to Chain.
3. Choose an execution mode (Linear, Conditional, or Dynamic).
4. Add stages with Add stage, name each one, and pick its detector type. Reorder with Up and Down.
5. In Conditional or Dynamic mode, add routes that choose the next stage from what the current stage found; a stage with no route falls through to the next one.
6. Watch the validity indicator, use View JSON to see the equivalent configuration, and Save all changes.

Capture settings (window length, inference fps) live in the config’s Video Processing block, not inside a stage. The rest of this article covers the v2 API, which the Manager editor writes for you.

Before you start: detector baselines and endpoints

Use matching Engine plugin and Video Intelligence Service versions that support the chains API. Configure the ordinary detector baselines first; each stage inherits its own type’s baseline. No separate chain baseline is needed.

The API uses Engine REST credentials. Set VIF_USER and VIF_PASSWORD:

VIF_URL=http://localhost:8087/v2/vif
curl -u "$VIF_USER:$VIF_PASSWORD" -i "$VIF_URL/persist/configs/default"

To change default endpoints, PATCH that resource with the ETag from the read, using Content-Type: application/merge-patch+json and If-Match: <ETag>:

{
  "detectors": {
    "vlm":       { "type": "vlm", "mode": "describe", "endpoint": { "url": "http://your-vlm:8000/v1" } },
    "synthetic": { "type": "synthetic", "endpoint": { "address": "your-svd:8001" } }
  }
}

Addresses must be reachable from the analysis service. VLM uses an OpenAI-compatible HTTP base URL; SVD uses a gRPC host:port with no HTTP scheme. Add an endpoint’s api_key only when it needs one. A stage may override any ordinary detector setting, including endpoints, thresholds, object tracking, and VLM generation settings. The API never echoes stage credentials.

Run a first chain

List the files under the Engine’s content directory, then submit a job:

curl -u "$VIF_USER:$VIF_PASSWORD" "$VIF_URL/vod/files"

{
  "file": "uploads/clip.mp4",
  "config": {
    "detector": {
      "type": "chain",
      "stages": [
        { "name": "objects",  "detector": { "type": "object", "classes": ["person"] } },
        { "name": "describe", "detector": { "type": "vlm", "mode": "describe" } }
      ]
    }
  }
}

curl -u "$VIF_USER:$VIF_PASSWORD" -X POST \
  -H 'Content-Type: application/json' \
  --data-binary @chain-job.json \
  "$VIF_URL/vod/jobs"

The response includes job_id, state, and detector_type: chain. Without a mode, stages run in listed order. The second stage sees the same captured media; it does not automatically receive crops of the detected people.

Put capture settings such as window_seconds and inference_fps in the parent’s config.processing, never inside a stage. Sharing a window is not sharing a rate: the chain captures once, and each stage chooses how much of that capture it analyzes. A stage without frame_selection reads every captured frame; a stage that needs fewer adds "frame_selection": {"mode": "middle"} to its stage wrapper. Modes are all, first, middle, last, and every_nth (which requires stride >= 2 and always includes the first and last captured frames). A frame-cadence chain has only one frame per request, so it refuses any frame_selection other than all.

Choose an execution mode

Start with conditional when its rules can express the decision, and move to dynamic when they cannot. Dynamic mode supports frame detectors only (no Synthetic stages).

Conditional mode: route on what a stage found

This chain scores each window and asks the VLM to review only windows classified as synthetic:

{
  "processing": { "window_seconds": 5, "inference_fps": 1 },
  "detector": {
    "type": "chain",
    "mode": "conditional",
    "stages": [
      {
        "name": "score",
        "detector": { "type": "synthetic", "classification_threshold": 0.7 },
        "routing": {
          "rules": [{ "classes": ["synthetic"] }],
          "on_match":    { "stage": "describe" },
          "on_no_match": { "end": true }
        }
      },
      {
        "name": "describe",
        "detector": { "type": "vlm", "mode": "describe" },
        "frame_selection": { "mode": "middle" }
      }
    ]
  }
}

A routing rule counts detections whose class matches (case-insensitive) and whose confidence is at least min_confidence (default 0). It matches when min_count (default 1) qualify. match: any is the default; match: all requires every rule. Missing or empty classes never match, and an empty rules list never matches.

A target is exactly one of {"stage": "name"} or {"end": true}. A conditional stage without routing falls through to the next stage. Omitting on_no_match ends the chain on a miss. Revisits are allowed; max_rings limits executions per window to 8 by default (range 1–64) — it is a loop bound, not the number of configured stages.

A window that ends on a miss was filtered out. Under the default result_mode: final its event listeners receive the gate stage’s response type with no detections; the window, timing, and end_reason: "no_match" still arrive, so a listener with suppress_empty_detections: true skips filtered windows entirely, while one without it can tell “filtered by the gate” from “analyzed, nothing found.” Ending through on_match (end_reason: "match_end") is an affirmative answer and delivers the stage’s detections. Use result_mode: combined when listeners should see every executed stage.

Wake expensive stages on motion

A Motion entry stage is a cheap gate: it compares consecutive frames on the CPU and reports each moving region as a detection of class motion. A quiet request yields no detections, no rule matches, and the chain ends before any downstream stage runs:

{ "type": "chain", "mode": "conditional", "stages": [
  { "name": "watch", "detector": { "type": "motion" },
    "routing": { "rules": [{ "classes": ["motion"] }],
      "on_match": { "stage": "describe" }, "on_no_match": { "end": true } } },
  { "name": "describe", "detector": { "type": "vlm", "mode": "describe" },
    "frame_selection": { "mode": "middle" } } ] }

motion → object and motion → scene compose the same way. The gate and the expensive stage need not analyze at the same rate: the capture rate is set once in config.processing, and each stage’s frame_selection narrows that stage alone. For example, on a 30 fps stream with window_seconds: 2 and inference_fps: -1, every window carries 60 frames — the motion entry differences all 60 (nothing slips past it), while giving the VLM stage {"mode": "every_nth", "stride": 15} sends it five of them.

Key rules:

• Motion is supported only as the entry stage (stages[0]) and cannot be cropped onto.
• An Object stage behind a Motion entry must set "tracking": {"method": "none"} — the gate filters which frames reach later stages, so track IDs would be unreliable. (Object tracking requires a chain of only Object stages.)
• No route may target the Motion entry, including its own routing.
• A Motion-entry chain cannot contain a Synthetic stage (a Synthetic stage switches the chain to clip-window transport, which the Motion entry does not support yet).
• Tune the gate with the motion detector’s sensitivity (1–10, default 5), min_displacement (default 0.02), and merge_distance (default 0.08) — not a routing min_confidence, because motion confidence is net displacement, not a quality score. Use min_count to require several moving regions. See Use motion detection in VIF.
• Ask the VLM what the scene shows, not whether it moved: sampled frames reach a VLM as separate stills, and a modest position change reads as near-identical images.

Crop detected regions

To run a stage on the preceding execution’s boxes, add a crop to its stage wrapper:

{ "crop": { "classes": ["person"], "min_confidence": 0.5, "padding": 0.1, "max_crops": 16 } }

Crop defaults are all classes, confidence 0, padding 0, and at most 16 crops per source frame (highest-confidence boxes kept). No matching boxes produces an empty stage result. The first stage, Motion stages, and Synthetic stages cannot crop. In linear mode, the preceding stage must be an Object or Motion detector (the two types whose detections carry boxes). A stage after a Motion entry crops to the moving regions. A cropped Object stage must disable tracking and tiling. Crop child coordinates refer to the original full frame; source_ring_index and source_detection_index identify the parent.

Dynamic mode: let an installed Java listener choose the next stage

In dynamic mode, Engine calls your Java class after every stage execution, and the class returns the next stage to run (optionally on crops) or ends the window. Because it is ordinary Java running inside Engine, it can base that choice on anything it can compute or reach: the full result of the stage that just ran (including VLM text), earlier executions in the window, state kept across windows, or an external system. Dynamic chains run on VOD and live streams and support frame detectors only.

A minimal listener

This chain looks for people on every window but asks the VLM to describe them at most once per cooldown period — a limit routing rules cannot express:

{
  "type": "chain",
  "mode": "dynamic",
  "decision_timeout_seconds": 5,
  "decision_listener": {
    "class_name": "com.example.PersonReviewDecision",
    "properties": { "cooldownSeconds": 60 }
  },
  "stages": [
    { "name": "objects",  "detector": { "type": "object", "classes": ["person"] } },
    { "name": "describe", "detector": { "type": "vlm", "mode": "describe" } }
  ]
}

Compile the class against the installed Engine and plugin APIs, put its JAR in the Engine’s library directory, and restart Engine:

package com.example;

import java.util.HashMap;
import com.wowza.wms.application.IApplicationInstance;
import com.wowza.wms.stream.IMediaStream;
import com.wowza.wms.plugin.videointelligence.api.ChainDecision;
import com.wowza.wms.plugin.videointelligence.api.ChainDecisionContext;
import com.wowza.wms.plugin.videointelligence.api.IVifChainDecisionListener;

public class PersonReviewDecision implements IVifChainDecisionListener {
  private long cooldownMs;
  private long lastReviewMs;

  public static String getVersion() { return "1.0.0"; }

  public void onInit(IApplicationInstance app, IMediaStream stream,
                     HashMap<String, Object> properties) {
    cooldownMs = 1000L * Long.parseLong(
        String.valueOf(properties.getOrDefault("cooldownSeconds", 60)));
  }

  public void onShutdown() {}

  public ChainDecision decide(ChainDecisionContext context) {
    if (!"objects".equals(context.stageName()) || !context.sawClass("person"))
      return ChainDecision.done();
    long now = System.currentTimeMillis();
    if (now - lastReviewMs < cooldownMs) return ChainDecision.done();
    lastReviewMs = now;
    return ChainDecision.advance("describe");
  }
}

ChainDecision.advance(name) runs that stage next; ChainDecision.done() completes the window. Returning null or an unknown stage name also completes it, with a warning in the Engine log. A stage may be revisited, up to max_rings executions per window. The class must be public and concrete, implement IVifChainDecisionListener, and expose a public no-argument constructor; a static getVersion() is recommended.

What a decision can read

ChainDecisionContext describes the execution that just finished:

A StageResult carries stageName(), detections(), the class queries, and window() with the covered media range. Every Detection carries kind() (object, scene, vlm, synthetic, or motion) next to className() and confidence(); type-specific data lives on object() (box, trackId) and vlm() (reasoning, data). A VLM in describe mode yields one detection whose class is description and whose vlm().reasoning() holds the text. Treat an unrecognized kind() as a detection with only the common members. Because the context is built from small interfaces, the listener is unit-testable without Engine.

Choosing crops in code

return ChainDecision.advanceCropped("plates",
    CropSpec.ofClasses("car").withMinConfidence(0.5).withPadding(0.1).withMaxCrops(8));

CropSpec.all() crops every class; unset fields use the same defaults as a configured crop. Crops come from detections that carry boxes (Object stages and a Motion entry’s regions). A cropped Object stage must have tracking and tiling disabled.

Lifecycle, threading, and timeouts

• Engine creates one instance per live stream and per VOD job and calls onInit once before the first window. Instance fields hold per-stream state; share state across streams with your own static or external store. On a VOD job the application instance is null and the stream answers only getName().
• A decision runs on an Engine worker thread, one at a time. Keep decisions fast and never wait on the network inside one; fetch outside data ahead of time (for example, a background poller publishing to a volatile field). When a decision exceeds decision_timeout_seconds (0.001–2147483.647), Engine interrupts it, completes the window with the results obtained, and runs later decisions on a fresh thread.
• Five consecutive callback exceptions disable the listener and call onShutdown.
• dispatch_intermediate_rings: true also sends intermediate events to ordinary event listeners (they do not count as completed VOD windows or create resume checkpoints).
• When a custom event listener on the same stream names the decision listener’s class, both roles share one instance, so fields can carry state between deciding and reacting. onInit then runs twice (event role first, then decision role) and onShutdown once. A config whose decision class matches more than one event listener is refused at submit.

A full worked example — gating a VLM PPE check against a loading-dock access-control system, with a background poller and a per-stream cooldown — is in the dev docs/MODEL_CHAINING.md. Reproduce or link it here if this KB should carry the complete listener code.

Service compatibility

Chains created or changed through v2 require the service to support their selected options: crop (even empty) requires chain.crop; a frame_selection other than all requires chain.frame_selection; an Object or Motion entry followed by a Scene or VLM stage requires chain.window_entry. Missing support fails the handshake before config or media is sent, with UNSUPPORTED_CHAIN_CAPABILITY naming the setting (for example, config.detector.stages[1].crop) and the missing capability. The chain does not silently fall back to full frames.

A native chain holding settings the v2 contract cannot express reads through v2 without them, and a v2 write to that config is refused with 409 Conflict (edit it natively).

Save and reuse the definition

Create a named config with POST /persist/stream-group-configs (a name, a match pattern, and the chain config), then submit jobs with {"file": "uploads/clip.mp4", "stream_group_config": "PeopleReview"}. Add config.processing to vary capture per job. Manager’s New Analysis → Stored config picker shows the chain’s stages and any VOD-only restriction, and its Edit link opens the chain in the Stream Configs editor.

When PATCHing, read for the ETag. The stages array is replaced as a whole, so send every stage you want to keep, each one complete. Stage names must be unique, nonblank, and free of surrounding whitespace or control characters (__terminal__ and __done__ are reserved). Reordering preserves credentials by stage name and detector type; renaming a stage or changing its type creates a new identity.

Read results and resume

curl -u "$VIF_USER:$VIF_PASSWORD" "$VIF_URL/vod/jobs/<job_id>"
curl -u "$VIF_USER:$VIF_PASSWORD" "$VIF_URL/vod/jobs/<job_id>/results?offset=0&limit=50"
curl -u "$VIF_USER:$VIF_PASSWORD" -o chain-results.jsonl "$VIF_URL/vod/jobs/<job_id>/results/file"

Each stored row is one completed media window. A two-execution result carries detections_type: "chain", detection_window, path_taken, is_terminal, end_reason, and a rings array (execution order, not configured order) where each ring holds its stage’s typed payload — Object/Scene/Motion stages carry detections, VLM stages carry vlm, and Synthetic stages carry synthetic.

result_mode: final (the default) sends only the last stage’s result to listeners (a missed routing window sends the gate type empty); combined sends all stages. Stored VOD results keep the complete terminal history either way.

Terminal results carry end_reason:

Cancel and resume use POST …/jobs/<job_id>/cancel and …/resume. Resume keeps committed windows and reruns an interrupted window from its entry stage; it does not resume inside a stage or preserve arbitrary Java listener state, and external listener side effects from an interrupted window can repeat. A changed source or nested detector config causes a drift refusal.

Latency

Event listeners receive a window’s results only after every executed stage finishes. Conditional chains run serially (a window costs the sum of its stages); crop-free linear chains run concurrently (a window costs its slowest stage); dynamic chains emit each ring as it completes while the window stays open. Overlay boxes are composited onto the live frame when results arrive, so they trail the live video by roughly the chain’s processing time plus the window’s age in the buffer. Put the Object stage first when box timeliness matters. Auto Frame Throttle lowers the inference fps when the stages cannot keep up.

Live streams

Use a frame-only chain in a saved matching group, a per-stream override, or the running stream’s /runtime/apps/{app}/streams/{stream}/detector resource (runtime edits require that resource’s current ETag). A Synthetic stage causes a clear live-stream refusal; use a VOD job for that chain.

Related articles

• Use motion detection in VIF
• Use object detection models with VIF
• Configure VIF in Wowza Streaming Engine Manager