Analyze VOD files with VIF

The Wowza Video Intelligence Framework (VIF) can analyze stored video files as well as live streams. An On-Demand job (also called a VOD job) points one of the framework's detectors - object detection, scene analysis, VLM analysis, or synthetic video detection at a video file that already sits on the Wowza Streaming Engine host, runs the detector over the whole file as fast as the analysis backend allows, and leaves behind a complete, queryable record. Every detection includes its position on the video's timeline, a job manifest describing exactly what ran, and a thumbnail.

On-Demand analysis is driven over the Wowza Streaming Engine REST API — the v2 Video Intelligence API, under /v2/vif, and it can also be run from the On-Demand tab in Wowza Streaming Engine Manager. Jobs run in the background, survive Engine restarts, recover automatically from transient failures, and can push status updates to your own service through webhooks.

Prerequisites: A running VIF stack (see Deploy WSE with VIF) and the Engine REST admin credentials. This article assumes familiarity with VIF detectors and configuration — see Configure Video Intelligence Framework in Wowza Streaming Engine.


How it works

An On-Demand job reuses the same configuration model as live streams. The same detector types, models, thresholds, class names, and listeners are in the same configuration document a stream group config carries. The differences that matter for a file job are:

  • The file is the clock — A live stream is analyzed at the rate it arrives; an On-Demand job runs as fast as the analysis service answers. A 10-minute file can finish in well under a minute, or take longer than 10 minutes with a heavyweight model. The wall-clock time depends on inference speed, not video duration.
  • Progress is media time — A job reports how far through the video's timeline it has gotten (media_time_ms), not how long it has been running for.
  • Results are always kept raw — Every response from the analysis service is stored as received, independent of any listener filtering. You can page through the results over REST, filter them by time range, or download the whole file.
  • There is no output stream — Listeners that write into a stream (overlay rendering, ID3 injection) do not apply to a file job and are skipped. Log and webhook listeners work exactly as they do for live streams.

The following detector types are supported:

Detector type Analysis Windowing Thumbnail
object Object detection (RF-DETR family) Decoded frames at inference_fps Yes
scene Scene analysis Decoded frames at inference_fps Yes
vlm Vision-language model Decoded frames at inference_fps Yes
synthetic Synthetic / AI-generated video detection Keyframe-aligned clips of ~duration seconds No (clips are relayed encoded, never decoded)

Run an On-Demand job in Wowza Streaming Engine Manager

The On-Demand tab of the VIF interface provides a UI for the same workflow described below. For the full field-by-field reference of the On-Demand jobs list, the New Analysis page, the results viewer, and On-Demand Configs, see Configure VIF in Wowza Streaming Engine Manager.

At a high level:

  1. Open the VIF tab, then select On-Demand.
  2. Select New Analysis.
  3. Choose a source file (or upload one), choose a Config, and set output options.
  4. Select Start analysis.
  5. Track the job in the jobs list, and select View to open its results viewer once it completes.

Run an On-Demand job using the VIF REST API

All On-Demand endpoints reside under a single base URL and authenticate using the Engine's REST admin credentials.

Authentication note: The framework configures the Engine REST API for HTTP Basic authentication, which curl -u uses by default. If your Engine is set to digest authentication instead (<AuthenticationMethod> in Server.xml, the default on standalone Engine installs), add --digest to every curl call.

Set your credentials and the base URL:

export WSE_ADMIN_USER=admin WSE_ADMIN_PASSWORD=your-password
VIF="http://localhost:8087/v2/vif"

1. Put a video where the Engine can see it

Jobs analyze files under the Engine's content directory. Copy a file into that directory, or upload it over the API (the request body is the file; the target name rides the query string):

curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" -X POST \
  -H "Content-Type: video/mp4" \
  --data-binary @my-clip.mp4 \
  "$VIF/vod/files?file=my-clip.mp4"

List what is analyzable at any time. The listing returns exactly the file strings a job submission takes:

curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" "$VIF/vod/files"

2. Submit a job

This example runs object detection with an inline config, sampling 5 frames per second. Anything you don't specify is inherited from the default config:

curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" \
  -H "Content-Type: application/json" \
  -d '{
    "file": "my-clip.mp4",
    "config": {
      "detector": { "type": "object", "classes": ["person", "car", "truck"] },
      "processing": { "inference_fps": 5 }
    },
    "tag": "quickstart"
  }' \
  "$VIF/vod/jobs"

The response is the job record, including its job_id and initial state.

3. Watch it run

Poll the job until its state is terminal (completed, failed, or cancelled):

curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" "$VIF/vod/jobs/$JOB_ID"

4. Read the results

Detections are served as pages of JSON rows, one row per analysis window, each stamped with its position on the video's timeline:

curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" "$VIF/vod/jobs/$JOB_ID/results?limit=2"

Each row carries the analysis window (detection_window with frame IDs and timecodes) and the detections inside it — for object detection, each detection has a class_name, confidence, frame_id, and a bbox. Or download everything as one newline-delimited JSON file:

curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" \
  -o results.jsonl "$VIF/vod/jobs/$JOB_ID/results/file"

5. Grab the thumbnail

The thumbnail is the frame currently under analysis while the job runs, and a representative frame once it is done:

curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" \
  -o thumbnail.jpg "$VIF/vod/jobs/$JOB_ID/thumbnail"

6. Clean up

Remove the job record when you no longer need it, and the video once nothing will analyze it again. Each answers 204 with no body:

curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" -X DELETE "$VIF/vod/jobs/$JOB_ID"
curl -s -u "$WSE_ADMIN_USER:$WSE_ADMIN_PASSWORD" -X DELETE "$VIF/vod/files?file=my-clip.mp4"

Choosing the analysis: stream group configs and inline configs

A job names its configuration the way the rest of the v2 API does, using either or both of two members. At least one is required.

  • Stream group config ("stream_group_config": "live_objectDotStar") names a saved group by its name — the same documents GET /v2/vif/persist/stream-group-configs lists and the Manager's Stream Configs page edits. A group's match rule plays no part in a file job; only its configuration is used. The framework ships ready-made groups — live_objectDotStar, live_sceneDotStar, live_vlmDotStar, and live_syntheticDotStar — and configuration files written before v2 answer to their file name without the extension.
  • Inline config ("config": { … }) carries a configuration document in the request itself (detector, listeners, processing, service, diagnostics, active). It is sparse — anything you leave out is inherited.

The two layer, in this order:

default config  <  stream_group_config  <  config
  • detector is atomic. An inline detector replaces the group's whole detector; whatever it leaves unset comes from the default config's per-type baseline, not from the detector it replaced.
  • listeners layer per entry, by name. A name the inline config declares owns that whole entry; entries it doesn't name pass through from the group.
  • Everything else merges field by fieldprocessing, service, diagnostics, active.

So a group plus a small inline config is the usual shape — reuse a tuned analysis and override the one thing this job needs:

{ "file": "clips/one.mp4",
  "stream_group_config": "live_objectDotStar",
  "config": { "processing": { "inference_fps": 2 } } }

Which to use: Stream group configs suit recurring work managed centrally through the Manager UI, and jobs that should be resumable unattended (the group's credentials live in the group, so the Engine can always reconstruct the job). Inline configs suit programmatic, per-request variation. Inline credentials (a VLM endpoint.api_key, for example) are never written to disk. The job's record stores a redacted copy so a failed inline job that held credentials can only be resumed by re-supplying them.

The single-job read carries config (the inline document as submitted, redacted) and effective_config (what actually ran, redacted). Editing a group later never changes a job that has already been submitted.


Inputs

  • Containers: .mp4, .m4v, .mov, .f4v. Anything else is neither listed nor accepted.
  • Codec: the file must carry an H.264 video track.
  • Location: files are named relative to the content directory (the VOD setting content_dir, by default the Engine's content/ folder). Subdirectories work (archive/cam3/monday.mp4); paths that resolve outside the content root are refused.
  • Audio is ignored — only the video track is analyzed.

Note: Frame-decoded detector types (object, scene, vlm) decode the file with FFmpeg, which must be available on the Engine's PATH. Clip-based jobs (synthetic) relay encoded H.264 and don't need it.

Manage files with GET /vod/files (list), POST /vod/files?file=<relative path> (upload, raw bytes as the body), and DELETE /vod/files?file=<relative path> (remove — refused with 409 while a queued or running job is using the file).


Outputs

A job produces up to four things.

1. The job view

Live progress and final status over REST (GET /vod/jobs/{jobId}), which is also the payload of every lifecycle webhook.

Field Meaning
job_id The job's identifier, assigned at submit.
file The source path as submitted.
tag The label the job was submitted with, if any.
detector_type The analysis the job runs: object, scene, vlm, or synthetic.
stream_group_config The stream group config the job was submitted with, by name; absent for an inline-only job.
listener_warning Configured listeners a file job cannot serve (overlay, ID3), which were skipped; absent when there is nothing to warn about.
store_results Whether the job keeps its detections for …/results.
results_truncated true when a write failure cut the stored results short of the file's end; a resume fills the gap.
state pending, connecting, running, completed, failed, or cancelled.
error Human-readable failure description; only on a failed job.
error_cause Machine-readable failure kind (see Failure causes); only on a failed job.
requests_sent Analysis requests answered so far — the exact meter.
requests_total Up-front estimate of the total (see note below).
media_time_ms How far through the video's timeline the analysis has gotten.
source_duration_ms The length of that timeline — the denominator for media_time_ms.
queued_at, started_at, ended_at RFC 3339 timestamps; started_at/ended_at are absent until they happen.
resumes How many times the job has been resumed; 0 on a first run.
config The inline config as submitted, credentials redacted; null for a group-only job. Single-job read only.
effective_config The configuration the job actually ran with, credentials redacted. Single-job read only.

Note: requests_total is estimated once at start from the container's duration and is a ceiling — on some files the realized count lands one short. Treat state as the completion signal and media_time_ms / source_duration_ms as the progress meter; don't wait for requests_sent == requests_total, and don't treat the ratio as a percentage that reaches 100.

2. Stored results

Every response the analysis service returned, one JSON row per analysis window, exactly as received (no listener filtering). Served as pages (GET …/results), filterable by a source time range (?from_ms=…&to_ms=…), or downloadable as one .jsonl file (GET …/results/file). Results are readable while the job is still running — a partial read is simply a shorter page. Each row echoes the job's identity as stream_name: "vod-<job id>".

Submitting with "store_results": false runs a status-only job: nothing is written, …/results answers 404, and the job cannot be resumed. A job with no runnable listener and no stored results is refused at submit, since its detections would go nowhere.

3. The manifest

A provenance record written from the moment the job is accepted and kept current as it runs: the submitted and effective configuration (credentials redacted), a hash of the submitted configuration, the resolved file with its size/duration/frame rate at submit time, progress counters, final state, and failure cause. It makes a week-old job's results auditable and is what resume and restart recovery are built on.

4. The thumbnail

A JPEG of the frame currently being analyzed (running) or a representative frame (finished), at the size the detector actually saw. Clip-based (synthetic) jobs never decode a frame and have none — expect 404.

Detection events also flow through the listeners configured on the job's config, exactly as for a live stream. Events from a file job carry the job's identity (vod-<job id>) as the stream name, plus job_id and source_file properties. For Webhooks.json routing, a file job's events carry the context vHost: _defaultVHost_, app: vod, appInstance: _definst_, stream: vod-<job id>.


Play the source file

The Manager's On-Demand results viewer can play the analyzed file and seek to a detection by selecting a cell in the score strip. Playback uses the Engine's built-in HLS vod application, which serves the same content directory jobs analyze. The file must use H.264 video with AAC audio, and the Engine's streaming port (1935) must be reachable from the browser.

If the Manager is served over HTTPS, browsers block plain-HTTP media; point the player at an HTTPS streaming port with the playback_host / playback_port Video Intelligence plugin properties.


Job lifecycle, failures, and resume

pending ──▶ connecting ──▶ running ──▶ completed | failed | cancelled
                                  ▲                    │
                                  └──────── resume ◀───┘
                            (continues from the last answered window)
  • pending — accepted and queued.
  • connecting / running — the job holds its own connection to the analysis service and works through the file.
  • completed — the whole file was analyzed. This is the only state that means the results are complete.
  • failed — the job stopped early; error says why in words, error_cause says what kind of failure it was.
  • cancelled — a cancel request stopped it. Partial results and the manifest remain readable until the record is removed, and the job can be resumed later.

By default only one job runs at a time (max_concurrent_jobs); the rest wait in submit order. If you raise concurrency above 1, jobs can finish out of order, so rely on state — not the order you submitted them — to know when a particular job is done.

Failure causes

A failed job reports why in two fields: error (a human-readable message) and error_cause (a stable code for automation). Most failures are transient — a dropped connection, a timeout, or an endpoint that is still loading — and are retried automatically (see Resume below). A few are not retried because someone needs to act first: the source file is missing or damaged, results can't be written to disk, the configuration no longer matches the run, or the Engine was restarted mid-job. For automation, route on error_cause rather than parsing error; the full set of error_cause values is listed in the OpenAPI specification.

Resume

POST /vod/jobs/{jobId}/resume continues a stopped job from the last window so nothing is analyzed twice, and nothing is skipped. Before continuing, the Engine verifies that the source file is unchanged (size/duration/frame rate as recorded in the manifest) and that the configuration still resolves to the same analysis; a group edited in the meantime is caught here, and the resume is refused with a 409 naming the drift.

Automatic resume is on by default (the VOD setting auto_resume, or per job with "auto_resume": false at submit). When a job fails for a transient reason, the Engine re-runs it on a backoff; seconds apart for blips, a more patient schedule stretching to minutes for an endpoint that is down or still loading which is up to 3 attempts. The attempt counter resets whenever a retry gets further than the run before it, so a long file with occasional blips always makes progress while a dead endpoint stops costing anything after three tries.

Engine restarts

Job records survive Engine restarts. A job interrupted by a restart is marked failed (error_cause: "engine_restart") and is not resumed automatically; resume it manually to continue from where it stopped.


Lifecycle webhooks

If polling doesn't fit your integration, have the job call you: one POST per persisted transition (running, then whichever terminal state the job reaches) carries the job view to a URL, with event: "status_changed" and a pointer to the results endpoint. There is no event for pending — the submit's own response already carries it.

The payload is the collection view of the job (so it carries no config or effective_config) plus a server-relative results pointer, for example:

{
  "event": "status_changed",
  "job_id": "…",
  "file": "my-clip.mp4",
  "tag": "quickstart",
  "detector_type": "object",
  "state": "completed",
  "store_results": true,
  "results_truncated": false,
  "requests_sent": 370,
  "requests_total": 371,
  "media_time_ms": 73800,
  "source_duration_ms": 74066,
  "queued_at": "…", "started_at": "…", "ended_at": "…",
  "resumes": 0,
  "results": "/v2/vif/vod/jobs/…/results"
}

Configure the destination globally (the VOD setting lifecycle_webhook, every job) or per job ("lifecycle_webhook": "https://…" at submit, which overrides the global; an empty string "" opts a job out). Delivery is ordered per job, retried (3 attempts), never blocks or fails the job, and terminal events are re-delivered after an Engine restart if delivery was never confirmed — design your handler to tolerate an occasional duplicate.

Caution: A per-job webhook URL is recorded in the job's manifest on disk, so never embed a token in one. Authorization is always a named webhook secret instead.

Named webhook secrets

Every webhook credential lives once, under a name, in the secrets document — GET/PATCH /v2/vif/persist/secrets. A read answers the names only; a value never travels back out. The patch is a JSON merge patch over values: a string sets or rotates a name, null removes it, and a name you don't mention is kept.

A submit then refers to a name:

{ "file": "clip.mp4", "stream_group_config": "live_objectDotStar",
  "lifecycle_webhook": "https://partner-a.example/hook",
  "lifecycle_webhook_secret": "partner-a" }

Every delivery of that job's notifications carries the named secret's current value as the Authorization header. The job record stores the name only, so the value is never at rest in a manifest; rotating it applies immediately to jobs already submitted. Values may be ${ENV_VAR} placeholders, resolved when the Engine reads the file.

Trust boundary: Any client that can submit jobs can direct any named secret's value at a URL of its choosing, so file only webhook-consumption credentials here — nothing else.


VOD settings and secrets

How the Engine runs On-Demand analysis is a persist document of its own — GET/PATCH /v2/vif/persist/vod-settings, stored beside the stream configuration and never part of it. Every member is optional; absence means the built-in default. Values may be ${ENV_VAR} placeholders. (These are the same settings exposed on the Manager's On-Demand Configs page.)

Setting Default Meaning
max_concurrent_jobs 1 How many jobs run at once; the rest queue. One job already saturates one analysis-model slot, so raise this only if your VIS deployment has capacity to spare.
max_jobs 25 Job records kept; the oldest finished jobs (and their stored files) are evicted past the cap. Queued/running jobs are never evicted.
job_ttl_seconds 0 (off) Additionally, forget finished jobs this many seconds after they end. 0 disables the TTL.
content_dir Engine content/ Where job file paths are resolved and uploads land. Applies at the next Engine start.
jobs_dir Engine vif-vod-jobs/ Where manifests, results, and thumbnails are stored. Applies at the next Engine start.
lifecycle_webhook none Default status-webhook URL for every job; an absolute http(s) URL, or a ${ENV_VAR} placeholder resolving to one.
lifecycle_webhook_secret none Name of a secrets entry whose value is the Authorization header on every lifecycle delivery to lifecycle_webhook; never sent to a job-supplied destination.
auto_resume true Whether jobs that fail for transient reasons are resumed automatically.
max_upload_bytes 10737418240 (10 GiB) The largest upload POST /vod/files accepts; a bigger one is refused with 413 before a byte is written.

Edit settings with the standard persist get→edit→save cycle: read the document, then send a merge patch that quotes its revision with an If-Match header (omitting it is a 428; quoting a stale one is a 412). Caps take effect on the running Engine at once; the two directories are read at Engine start and take effect at the next one.

Retention composes: max_jobs bounds how many jobs are kept, and job_ttl_seconds bounds how long — either one on its own can forget a job, taking its stored results and thumbnail with it. A queued or running job is never touched, however old it is.


Best practices

  • Track progress by state and media_time_ms / source_duration_ms, not by requests_sent / requests_total (the total is an estimate).
  • Prefer webhooks to polling for automation; poll for dashboards and ad-hoc checks. Handle the occasional duplicate terminal event.
  • Tag your jobs — tag is free-form, and the job list filters on it exactly (GET /vod/jobs?tag=…).
  • Use stream group configs for recurring, unattended work. They keep credentials out of your requests, make jobs resumable without re-supplying secrets, and centralize tuning in the Manager UI.
  • Mind the analysis cost before submitting. A file job sends roughly duration_seconds × inference_fps ÷ frames_per_request requests. Lowering inference_fps is the single biggest lever on how long a job takes and what it costs; for many use cases (finding whether/where something appears) 1–5 fps is plenty.
  • Size retention to your workflow. The defaults keep the last 25 jobs forever; a busy pipeline should set job_ttl_seconds and copy results into its own storage as jobs complete.
  • Don't parse error; route on error_cause.
  • Cancel first, then delete — DELETE only accepts a terminal job.
  • Delete source files over the API once their jobs are terminal.

API reference

All endpoints are under the Engine REST API (default http://localhost:8087), path prefix /v2/vif, authenticated with the Engine's REST admin credentials, JSON in and out unless noted.

Method & path Purpose
GET /vod/files List analyzable files under the content root
POST /vod/files?file=… Upload one source file into the content root
DELETE /vod/files?file=… Remove one source file from the content root
POST /vod/jobs Submit a job
GET /vod/jobs List jobs (paged, newest first, filterable by tag and state)
GET /vod/jobs/{jobId} One job, with the configuration it was given and the one it ran with
DELETE /vod/jobs/{jobId} Remove a finished job — its record, results, and thumbnail
POST /vod/jobs/{jobId}/cancel Stop a queued or running job
POST /vod/jobs/{jobId}/resume Continue a stopped job from where it got to
GET /vod/jobs/{jobId}/results The job's detections, one page at a time
GET /vod/jobs/{jobId}/results/file The whole results file as newline-delimited JSON
GET /vod/jobs/{jobId}/thumbnail The job's thumbnail (image/jpeg)
GET/PATCH /persist/vod-settings How this Engine runs On-Demand analysis
GET/PATCH /persist/secrets The named webhook credentials

Job submission body (POST /vod/jobs):

Field Required Meaning
file Yes Path relative to the content root.
stream_group_config At least one of the two A stream group config by name; its config is the middle layer.
config An inline config, layered over the group (or over the default config alone).
store_results No (default true) false runs the job status-only: no stored rows, no resume.
tag No Free-form label; the job list filters on it.
lifecycle_webhook No Status-webhook URL for this job. Overrides the global; "" disables notifications.
lifecycle_webhook_secret No Name of an entry in the secrets document; its value is the Authorization header on this job's notifications.
auto_resume No Overrides the VOD setting auto_resume for this job.

Responses. A successful call returns the operation's payload — a submit answers 201 with the job, a read returns the resource, an action answers 202 with the job as it stood, and a DELETE answers 204 with no body. Every refusal raised inside the API is application/problem+json (RFC 7807) with title, status, and detail.

Status When
400 Anything wrong with the request body or a query parameter (unknown group, neither config member, a config selecting no detector, a missing/unsupported file, a bad webhook URL, an unknown secret name, an invalid state or page number).
404 The job id names no job (and, on results endpoints, a job that stored no rows); a file delete naming a file that isn't there.
409 A conflict with the job's state (cancelling a terminal job, deleting a non-terminal one, a resume refusal, an upload whose name is taken, a file delete while a job is using it).
413 An upload larger than max_upload_bytes.
503 On-Demand is unavailable on this Engine — its content or jobs directory could not be resolved or created.

Troubleshooting

  • Submit refused with "neither stream_group_config nor config." A job must name its analysis — a group by name, an inline config, or both.
  • Submit refused with "no stream group config named …." The name must be a group's name as GET /v2/vif/persist/stream-group-configs lists it (a pre-v2 file answers to its file name without .json). The refusal's detail lists the names the Engine has.
  • Submit refused with "selects no detector." The layers resolved to a configuration with no detector. Add one at whichever layer should own it.
  • GET /vod/files answers 503. The content directory could not be resolved or created — check content_dir in the VOD settings and the Video Intelligence log from startup.
  • An upload answers 413. The file is bigger than max_upload_bytes (10 GiB by default). Raise it, or place the file in the content directory by other means.
  • A job stays pending. It is waiting for the jobs ahead of it (max_concurrent_jobs).
  • A job fails with endpoint_degraded almost immediately. Working as designed: the upstream endpoint (VLM, SVD…) is unreachable or still loading, and the job fails fast rather than burning the file into empty results. Auto-resume retries on a patient backoff; or fix the endpoint and POST …/resume.
  • A resume answers 409 "the source file has changed." The file under that path differs from the one the job analyzed. Restore the original file, or submit a new job.
  • A resume answers 409 naming the stream group config. The group has been edited or deleted. Submit a new job against the current group, or restore what the group said.
  • A DELETE answers 409 "cancel it first." DELETE only removes terminal jobs. POST …/cancel, poll until cancelled, then DELETE.
  • A cancel answers 409 "only a queued or running job can be cancelled." The job already reached a terminal state. (A failed job still waiting on an automatic retry is the one exception — cancelling it drops the retry.)
  • A settings or secrets PATCH answers 428 or 412. Persist writes are guarded by If-Match: 428 means the header was missing, 412 that the revision it quoted is stale. Read the document again and quote the ETag it answers with.
  • The thumbnail is 404 for a synthetic job. Expected — clip-based jobs never decode a frame.

Related articles