How to inject metadata into HLS at exactly the correct time/frame?

[Dan_Farrow]

I have an RTMP stream coming from my app (via ffmpeg). Into this stream, I am injecting timed metadata using an HTTP Provider/Module like so.
https://www.wowza.com/docs/how-to-inject-cue-points-or-metadata

I read Wowza’s information on Metadata Workflows Here:
https://www.wowza.com/docs/how-to-use-timed-metadata-in-wowza-live-streaming-workflows

That document states:

To prevent directly-ingested metadata from getting ahead of the video, either inject the metadata at the source or delay the direct injection into Wowza Streaming Engine: watch RTMP or WOWZ playback of the stream from Wowza Streaming Engine, and initiate the API call to inject the metadata as soon as you see the frame you want to sync to. (There’s less latency with RTMP and WOWZ playback, so you can see the desired frame before it appears for HLS playback.)

I want my metadata very closely aligned with the video time/frame. I am wondering how to accomplish this "Initiate the API call to inject the metadata as soon as you see the frame you want to sync to? What event can I use to inspect the video as it is processed to know what the absoluteTimeCode is on each frame?

I was thinking of storing an array of metadata being sent from my app with a timestamp, and I would store this data array in a Wowza Module. As each frame was processed I would get the encoded video time and inject the metadata when the current frame time matches the data timestamp (within some tolerance). What event runs on each frame of an HLS encoding session? Is there an example of this to look at?

[Rose_Power-Wowza_Com]

Hey there @Dan_Farrow, I wasn’t sure, but checked with Professional Services and this is what I was told. I’ll copy and paste so I don’t translate the info incorrectly. Hope it helps!

---

There is an onLivePacket method that he could use.

https://www.wowza.com/resources/serverapi/com/wowza/wms/stream/IMediaStreamLivePacketNotify.html

registered through stream.addLivePacketListener()

He would then need to call stream.sendDirect() with his AMFData, which will be injected right after the video packet (will probably have then same timestamp)

Then in his HLS data handler class, he should receive a notification with the new data packet.

[Dan_Farrow]

Interesting. I used that listenter to print out the timecode and I am not sure what to make of it:

Each frame I write out the timecode/framerate using the IMediaStreamLivePacketNotify class. I have set the absTimecode() start time in ffmpeg as -output_ts_offset 19.40252) so I know the server matches the client time (I have checked). I log the packet.absTimecode on each frame():

2022-02-09 17:50:57 EST comment server INFO 200 - ModuleLiveFrameRate fps '30.303030303030305' packet.absTimecode = 123341 _defaultVHost_ live_test _definst_ 940.873 - - - -

I am continually sending metadata via an HTTP provider, I send in the metadata from my client app and timestamp it on the client with the current client time (which should match server time)… I see this in the Wowza logs as the metadata is injected (right after the line above)
:

2022-02-09 17:50:57 EST comment server INFO 200 - injectMetadata client metadata timecode 120662 - - - 940.898 - - - - - - - - - - - - -

As you can see, the Server absolute timecode is “123341” and the client timestamp of when the metadata was sent is “120662” (which is obviously earlier). So…This data from client time ‘120662’ gets injects at server Time ( packet absTimecode() ) 123341, which would seemingly be too late, HOWEVER, the metadata arrives in the player before the associated video frame it was sent with.

I want to figure out how to re-synchronize the metadata so it plays back aligned exactly to video frame at the time it was sent. Right now the metadata is off in client playback by varying amounts between -500 <–> +500 ms.

It could be a benefit that the data arrives early, as that gives me time to buffer, delay and then play the metadata back correctly, but I want it to play at exactly the correct offset so the video/data are aligned again…

I am trying to determine the exact offset between metadata time as it arrives on the server and it’s associated video time, but this behavior is confusing. How can the metadata from 120662 on the client get injected at 123341 on the server, yet arrive before the video frame it was sent with when received in the client player? What am I missing here?