Convert timed metadata from AMF to emsg using the Wowza Streaming Engine Java API

You can use the Wowza Streaming Engine™ media server software Java API to convert AMF metadata to Event Message (emsg) metadata for live streams. This allows metadata ingested with an RTMP source stream or injected by Wowza Streaming Engine to be delivered with streams played using the MPEG-DASH protocol.

Notes:
  • Wowza Streaming Engine 4.8.10 or later is required.
  • Wowza Streaming Engine currently only supports converting AMF metadata to emsg metadata with live streams—not with VOD or nDVR streams.
  • Sending streams with emsg timed metadata to CDN destinations using Stream Targets (push publishing) is currently not supported. To send a stream with emsg timed metadata to a CDN, the CDN must pull the stream from a Wowza Streaming Engine Live HTTP Origin application.

About converting timed metadata for MPEG-DASH


Producing interactive live streams often requires the use of timed metadata. Wowza Streaming Engine can ingest timed metadata over RTMP in Action Message Format (AMF)—a binary format developed by Adobe for exchanging messages between servers—from any encoder that supports AMF metadata. Wowza Streaming Engine can also inject AMF metadata directly into a stream as it enters the Wowza Streaming Engine server.

In order for Wowza Streaming Engine to ingest AMF metadata, the metadata must be wrapped in a top-level AMF data object (AMFDataObj). To inject AMF metadata into a stream sent to Wowza Streaming Engine, see Inject timed metadata using a Wowza Streaming Engine HTTP provider.

Whether Wowza Streaming Engine ingests the timed metadata from an encoder or injects it into a stream, once the AMF metadata is in the stream, it must be converted to emsg metadata in order for it to be used when the stream is played with an MPEG-DASH client. An emsg contains timing information, a string payload, and an event scheme identifier that differentiates between different types of metadata events (for example, captions, slide transitions, etc.). The MPEG-DASH manifest identifies any event schemes with the InbandEventStream element and schemeIdUri and value attributes. The player then knows which metadata to retrieve from the stream. For more information about emsg metadata, see About Event Message (emsg).

Convert AMF metadata to emsg metadata in a live stream


For this example, Wowza Streaming Engine is broadcasting a live stream of a marathon. When a runner crosses the finish line, their name and the time that they cross should appear as an overlay in the player. This example is a follow-on to the example for injecting AMF timed metadata into a stream with Wowza Streaming Engine in Inject timed metadata using a Wowza Streaming Engine HTTP provider. A similar example for converting AMF timed metadata to ID3 for HLS playback is also available: Convert timed metadata from AMF to ID3 using the Wowza Streaming Engine Java API.

In the following code, Wowza Streaming Engine listens for an AMF data event, parses it, and maps the event to an emsg, which is inserted into an audio or video segment. Then it appends the emsg to the appropriate event scheme in the MPEG-DASH stream's manifest. This example uses a single event scheme but includes some code for registering multiple event schemes commented out. Also, this example uses emsg version 0 but includes code for emsg version 1 commented out.

Note: Refer to the code comments in the example and the following packages in the Wowza Streaming Engine Java API reference documentation for information about the classes used in this example and how to adapt the code to suit your needs:
 
  1. Create a custom Java module with the following example code by compiling it and packaging it into a .jar file in the [install-dir]/lib folder. This module is called during MPEG-DASH packetization and segment creation. See Use Wowza Streaming Engine Java modules for more information about extending Wowza Streaming Engine functionality with Java modules.
     
    package com.example;
    
    import com.wowza.wms.amf.*;
    import com.wowza.wms.application.IApplicationInstance;
    import com.wowza.wms.httpstreamer.model.LiveStreamPacketizerPacketHolder;
    import com.wowza.wms.httpstreamer.mpegdashstreaming.file.*;
    import com.wowza.wms.httpstreamer.mpegdashstreaming.livestreampacketizer.*;
    import com.wowza.wms.media.metadata.emsg.*;
    import com.wowza.wms.module.ModuleBase;
    import com.wowza.wms.stream.livepacketizer.*;
    
    public class ModuleMPEGDashLiveMarathonAMFToEmsg extends ModuleBase
    {
            // The scheme URI identifies a unique event scheme that differentiates between different types of metadata events (for example, captions, slide transitions, etc.). 
    	private static final String EVENT_SCHEME1_URI = "urn:com:example:raceevent";
    	// The value is something meaningful to whomever defines the schema of the SCHEME_URI. It can also be an empty string.
            private static final String EVENT_SCHEME1_VALUE = "1";
    
    	class LiveStreamPacketizerDataHandler implements IHTTPStreamerMPEGDashLivePacketizerDataHandler
    	{
    		private static final String AMF_PAYLOAD_TYPE = "onRaceFinisher"; // Identifies the AMF metadata to convert to emsg.
    
    		private LiveStreamPacketizerMPEGDash liveStreamPacketizer = null;
    		
    		private boolean hasRegisteredEventStreams = false;
    		
    		private int uniqueIDEventScheme1 = 0; 
    		
    		private long segmentStart;
    
    		
    		public LiveStreamPacketizerDataHandler(LiveStreamPacketizerMPEGDash liveStreamPacketizer)
    		{
    			this.liveStreamPacketizer = liveStreamPacketizer;
    		}
    
    		@Override
    		public void onFillSegmentStart(long startTimecode, long endTimecode, InbandEventStreams inbandEventStreams) {
    			// Only do this once with the first segment's start.
    			if (! this.hasRegisteredEventStreams) {
    				this.hasRegisteredEventStreams = true;
    
    				// If you want to put emsgs in video segments rather than audio segments, uncomment the following line or use the mpegdashDataEventsTrackType property in your application's configuration.
    				// inbandEventStreams.inAudio = false;   
    
    				// Register one or more event schemes. Uncomment the second call to register two event schemes.
    				inbandEventStreams.registerEventStream(new InbandEventStream(EVENT_SCHEME1_URI, EVENT_SCHEME1_VALUE));
                                    // inbandEventStreams.registerEventStream(new InbandEventStream(EVENT_SCHEME2_URI, EVENT_SCHEME2_VALUE));
    			}
    			
    			this.segmentStart = startTimecode;			
    		}
    
    
    		@Override
    		public void onFillSegmentDataPacket(LiveStreamPacketizerPacketHolder holder, AMFPacket packet,
    				InbandEventStreams inbandEventStreams) {
    			
    			String payload = extractPayload(packet);
    			if (payload == null)
    				return;
    			
    			//
    			// Extract AMF metadata strings and create emsgs using emsg version 0.
    			//
    
    			int id = uniqueIDEventScheme1++;  // A unique ID for each emsg. Each emsg per event scheme needs a unique ID.
    
    			long time = packet.getAbsTimecode();
    			int delta = Math.max(0, (int) (time - this.segmentStart));
    			int duration = 0; // The emsg has no duration. The value of duration is in the specified timescale. 				
    
    			int timescale = 1000; // The timescale is in milliseconds (1000 ticks per second). 	
    			IEmsgFrame emsg = this.createEmsgV0(EVENT_SCHEME1_URI, EVENT_SCHEME1_VALUE, id, timescale, delta, duration, payload);
    
    			// To create emsgs using emsg version 1, use the following call instead.
    			// IEmsgFrame emsg = this.createEmsgV1(EVENT_SCHEME1_URI, EVENT_SCHEME1_VALUE, id, timescale, time, duration, payload);
    			
    			//
    			// Add emsgs to segments.
    			//
    
    			EmsgFrames emsgFrames = inbandEventStreams.getEmsgFrames();
    			emsgFrames.addFrame(emsg);
    		}
    
    		@Override
    		public void onFillSegmentEnd(long endTimecodeVideo, long endTimecodeAudio, InbandEventStreams inbandEventStreams) { }
    		@Override
    		public void onFillSegmentMediaPacket(LiveStreamPacketizerPacketHolder holder, AMFPacket packet) { }
    
    		protected String extractPayload(AMFPacket packet) {
    			byte[] buffer = packet.getData();
    			
    			if (buffer == null) return null;
    
    			if (packet.getSize() <= 2) return null;
    
    			int offset = 0;
    			if (buffer[0] == 0)
    				offset++;
    
    			AMFDataList amfList = new AMFDataList(buffer, offset, buffer.length-offset);
    
    			if (amfList.size() <= 1) return null;
    
    			if (amfList.get(0).getType() != AMFData.DATA_TYPE_STRING || amfList.get(1).getType() != AMFData.DATA_TYPE_OBJECT)
    				return null;
    
    			String metaDataStr = amfList.getString(0);
    			AMFDataObj dataObj = amfList.getObject(1);
    
    			if (! metaDataStr.equalsIgnoreCase(AMF_PAYLOAD_TYPE)) return null;
    
    			AMFData amfData = dataObj.get("payload");
    			
    			if (!(amfData instanceof AMFDataItem)) return null;
    
    			return amfData.toString();
    		}
    
    		private IEmsgFrame createEmsgV0(String schemeUri, String schemeValue, int id, int timescale, long delta, int duration, String data) {
    			// Convert from milliseconds to timescale.
    			delta = ((int)convertMillisToTimescale(timescale, delta));	
    			duration = ((int)convertMillisToTimescale(timescale, duration));	
    
    			// Wowza Streaming Engine uses the EmsgBuilder to create an emsg using emsg version 0.
    			IEmsgFrame emsg = new EmsgBuilder(0, schemeUri, schemeValue)
    			          .setId(id)
    			          .setTimescale(timescale)
    			          .setTime(delta)
    			          .setEventDuration(duration)
    			          .setMessage(data)
    			          .build();
    			
    			return emsg;
    		}
    		
                    // To create emsgs using emsg version 1, use the following call instead.
    		/* private IEmsgFrame createEmsgV1(String schemeUri, String schemeValue, int id, int timescale, long time, int duration, String data) {
    			// Convert from milliseconds to timescale.
    			time = ((int)convertMillisToTimescale(timescale, time));	
    			duration = ((int)convertMillisToTimescale(timescale, duration));	
    
    			// Wowza Streaming Engine uses the EmsgBuilder to create an emsg using emsg version 1.
    			IEmsgFrame emsg = new EmsgBuilder(1, schemeUri, schemeValue)
    			          .setId(id)
    			          .setTimescale(timescale)
    			          .setTime(time)
    			          .setEventDuration(duration)
    			          .setMessage(data)
    			          .build();
    			
    			return emsg;
    		} 
                    */
    
    		private long convertMillisToTimescale(long timescale, long millis) {
    			return timescale*millis/1000;
    		}
    	}
    
    
    
    	class LiveStreamPacketizerListener extends LiveStreamPacketizerActionNotifyBase
    	{
    		// When the packetizer starts, register data handler for MPEG-DASH packetizer.
    		public void onLiveStreamPacketizerCreate(ILiveStreamPacketizer liveStreamPacketizer, String streamName)
    		{
    			if (liveStreamPacketizer instanceof LiveStreamPacketizerMPEGDash)
    			{
    				getLogger().info("ModuleMPEGDashLiveMarathonAMFToEmsg.onLiveStreamPacketizerCreate["+((LiveStreamPacketizerMPEGDash)liveStreamPacketizer).getContextStr()+"]");
    				((LiveStreamPacketizerMPEGDash)liveStreamPacketizer).setDataHandler(new LiveStreamPacketizerDataHandler((LiveStreamPacketizerMPEGDash)liveStreamPacketizer));
    			}
    		}
    	}
    
    	// When application starts, register to listen to packetizer events.
    	public void onAppStart(IApplicationInstance appInstance)
    	{
    		appInstance.addLiveStreamPacketizerListener(new LiveStreamPacketizerListener());
    
    		getLogger().info("ModuleMPEGDashLiveMarathonAMFToEmsg.onAppStart["+appInstance.getContextStr()+"]");
    	}
    }

     
  2. Add the module to your live application. The module should be added in the <Application>/<Modules> container element in your application's Application.xml file.
     
    <Module>
    	<Name>ModuleMarathonLiveAMFToEmsg</Name>
    	<Description>Module to convert AMF marathon data to emsg in live streams</Description>
    	<Class>com.wowza.wms.plugin.ModuleMPEGDashLiveMarathonAMFToEmsg</Class>
    </Module>

    See Use custom modules for information about how to add custom Java modules using Wowza Streaming Engine Manager.
  3. Enable MPEG-DASH or CMAF MPEG-DASH packetization for your application, if you haven't already. 
  4. (Optional) Add the mpegdashDataEventsTrackType custom property to your application to control whether emsgs are added to audio segments or video segments for streams sent to the application. Valid values are audio or video. The property should be added in the <Application>/<LiveStreamPacketizer>/<Properties> container element in the Application.xml file.
     
    Note: The example module adds emsgs to audio segments unless you uncomment this line: inbandEventStreams.inAudio = false;. If this line is uncommented, it overrides the value of the mpegdashDataEventsTrackType property.
    <Property>
    	<Name>mpegdashDataEventsTrackType</Name>
    	<Value>video</Value>
    	<Type>String</Type>
    </Property>

    See Add a custom property for information about how to add custom properties using Wowza Streaming Engine Manager or XML configuration files.

After sending a stream to your live application with AMF metadata added by an encoder or injected by Wowza Streaming Engine, you can confirm the module is converting the AMF metadata to emsgs by requesting the MPEG-DASH manifest with a curl command:

curl http://[address]:1935/[application-name]/[stream-name]/manifest.mpd

The manifest returned should include the InbandEventStream element with schemeIdUri and value attributes in either the audio or video adaptation set depending on if the emsgs were inserted into the audio or video segments. If you registered multiple event schemes, you should see an InbandEventStream element with schemeIdUri and value attributes for each event scheme. The following example includes the event scheme registered in the example code identified by InbandEventStream in the audio adaptation set:

<AdaptationSet id="1" group="2" mimeType="audio/mp4" lang="eng" segmentAlignment="true" startWithSAP="1" subsegmentAlignment="true" subsegmentStartsWithSAP="1">
    <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
    <InbandEventStream schemeIdUri="urn:com:example:raceevent" value="1"/>

To test playback, use an MPEG-DASH player that supports emsgs such as Bitmovin or Dash.js. When configuring the player for emsgs, you need to specify any event schemes' URIs.

More resources