• How to add graphic overlays to live streams with Wowza Transcoder

    This article provides instructions for developers about how to use Wowza Transcoder to place overlay images onto live broadcast streams. This article describes the steps of creating a custom Transcoder module, adding a static image to the video, adding text to the video, fading the image and text, and animating the image and text. The examples and classes in this article provide a starting point for development. More elaborate overlays are possible through custom development by expanding the examples or by creating custom classes.

    Note: Wowza Media Server™ 3.5 or later is required.

    Contents


    Tutorial


    More resources


    Tutorial



    Overview


    Wowza Transcoder is called for each frame that is created, allowing a developer to modify that frame with custom images and text. Because Transcoder is called repeatedly, the developer can also add animation to the newly encoded streams.

    Note: You can download the sample overlay images and helper classes that are referenced in this article. Download TranscoderOverlayExampleFiles.zip.

    Prerequisites


    This feature requires advanced technical skills. This article is intended for developers who are familiar with Wowza Media Server and are experienced with Java programming and XML.

    We recommend that you get a live stream playing through the Wowza Media Server to a client by doing the following:

    1. It's important that your server is properly tuned. For more information, see Performance Tuning.

    2. Follow the tutorial, How to set up and run Wowza Transcoder for live streaming for step-by-step configuration instructions. Before creating the Transcoder module for overlays, be sure to test basic live setup and playback of a single transcoded 360p stream. See Test #1 and Test #2 in the Troubleshooting Tests section of the tutorial. The overlay example in this article uses the application name live and the stream name myStream.

    3. Copy the example overlay content to your Wowza Media Server.

      Copy all the files in the content directory that is included in TranscoderOverlayExampleFiles.zip to the content directory of your Wowza Media Server installation ([install-dir]/content).

      Note: If you choose to create your own overlay images, be sure to review the Transcoder overlay image requirements.

    Create a new Transcoder module


    To overlay an image or text on a live stream, a Transcoder module must be created. This example uses the Wowza IDE.

    1. Create a new project.

      1. Use the Wowza IDE to create a new Wowza Media Server Project:
        • Project name: ModuleTranscoderOverlayExample
        • Wowza Media Server location: Select the install directory where WMS300 is installed.
        • Package: com.wowza.wms.plugin.transcoderoverlays
        • Name: ModuleTranscoderOverlayExample

      2. Under Run > Run Configurations, select ModuleTranscoderOverlayExample.

      3. On the Arguments tab, add the following to the VM arguments:
        -Dcom.wowza.wms.native.base="win"

    2. Set up the configuration files.

      Add the following code to [install-dir]/conf/live/Application.xml:
      <Module>
             <Name>ModuleTranscoderOverlayExample</Name>
             <Description>Example Overlay</Description>
             <Class>com.wowza.wms.plugin.transcoderoverlays.ModuleTranscoderOverlayExample</Class>
      </Module>
    3. Start Wowza Media Server and check for the message, "INFO server comment - onAppStart: live/_definst_" in the default message log or set a break point on onAppStart.


    Create a Transcoder module


    The overlayExample uses the following Wowza base classes and interface to overlay a graphic on the video:

    • IliveStreamTranscoderNotify
    • LiveStreamTranscoderActionNotifyBase
    • TranscoderVideoDecoderNotifyBase


    The example also includes the OverlayImage and AnimationEvents classes for quick drawing and animation. You must add these two classes to the project.

    1. Modify the ModuleTranscoderOverlayExample class.

      1. Import the following:
        import java.awt.Color;
        import java.awt.Font;
        import java.text.SimpleDateFormat;
        import java.util.ArrayList;
        import java.util.Date;
        import java.util.HashMap;
        import java.util.List;
        import java.util.Map;
        
        import com.wowza.util.SystemUtils;
        import com.wowza.wms.application.*;
        import com.wowza.wms.amf.*;
        import com.wowza.wms.client.*;
        import com.wowza.wms.module.*;
        import com.wowza.wms.request.*;
        import com.wowza.wms.stream.*;
        import com.wowza.wms.stream.livetranscoder.ILiveStreamTranscoder;
        import com.wowza.wms.stream.livetranscoder.ILiveStreamTranscoderNotify;
        import com.wowza.wms.transcoder.model.LiveStreamTranscoder;
        import com.wowza.wms.transcoder.model.LiveStreamTranscoderActionNotifyBase;
        import com.wowza.wms.transcoder.model.TranscoderSession;
        import com.wowza.wms.transcoder.model.TranscoderSessionVideo;
        import com.wowza.wms.transcoder.model.TranscoderSessionVideoEncode;
        import com.wowza.wms.transcoder.model.TranscoderStream;
        import com.wowza.wms.transcoder.model.TranscoderStreamDestination;
        import com.wowza.wms.transcoder.model.TranscoderStreamDestinationVideo;
        import com.wowza.wms.transcoder.model.TranscoderStreamSourceVideo;
        import com.wowza.wms.transcoder.model.TranscoderVideoDecoderNotifyBase;
        import com.wowza.wms.transcoder.model.TranscoderVideoOverlayFrame;
      2. Add these member variables to the class.
        String graphicName = "logo_${com.wowza.wms.plugin.transcoderoverlays.overlayimage.step}.png";
        int overlayIndex = 1;
            
        private IApplicationInstance appInstance = null;
        private String basePath = null;
        private Object lock = new Object();
        overlayIndex

        The overlayIndex defines how the overlay image is drawn relative to an overlay image defined in transcode/transcode.xml. The overlay image is drawn in the following order:

        1. Encode/Source with a lower number.
        2. Encode/Source with a higher number.
        3. Decode/Destination with a lower number.
        4. Decode/Destination with a higher number.

        -where-

        • Encode relates to the Root/Transcode/Encodes/Encode/Video/Overlays/Overlay/Index element in transcode/transcode.xml.
        • Source relates to encodeSource=true in the module source code.
        • Decode relates to the Root/Transcode/Decode/Video/Overlays/Overlay/Index element.
        • Destination relates to encodeSource=false in the module source code.


        A decode/destination with the highest number will be on top. An encode/source with a value of 0 will be on the bottom.

        The overlay image defined in transcode/transcode.xml will not be displayed if the overlayIndex value specified in transcode/transcode.xml is the same as the overlayIndex value in the module and both are for the encode/source or both are for the decode/destination.

        basePath

        The basePath variable is used to store the full path name of the content directory where the graphics are located.

      3. Modify the onAppStart method of the class.
        public void onAppStart(IApplicationInstance appInstance)
        {
               String fullname = appInstance.getApplication().getName() + "/" + appInstance.getName();
               getLogger().info("onAppStart: " + fullname);
               this.appInstance = appInstance;
               String artworkPath = "${com.wowza.wms.context.VHostConfigHome}/content/" + appInstance.getApplication().getName();
               Map<String, String> envMap = new HashMap<String, String>();
               if (appInstance.getVHost() != null)
               {
                    envMap.put("com.wowza.wms.context.VHost", appInstance.getVHost().getName());
                    envMap.put("com.wowza.wms.context.VHostConfigHome", appInstance.getVHost().getHomePath());
               }
               envMap.put("com.wowza.wms.context.Application", appInstance.getApplication().getName());
               if (this != null)
                     envMap.put("com.wowza.wms.context.ApplicationInstance", appInstance.getName());
               this.basePath =  SystemUtils.expandEnvironmentVariables(artworkPath, envMap);
               this.basePath = this.basePath.replace("\", "/");
               if (!this.basePath.endsWith("/"))
                    this.basePath = this.basePath+"/";
               this.appInstance.addLiveStreamTranscoderListener(new TranscoderCreateNotifierExample());
        }

    2. Create EncoderInfo as a nested class. This class is used to bundle the input and output video streams for each encoder.
      class EncoderInfo
      {
          public String encodeName;
          public TranscoderSessionVideoEncode sessionVideoEncode = null;
          public TranscoderStreamDestinationVideo destinationVideo = null;
          public int[] videoPadding = new int[4];
          public EncoderInfo(String name, TranscoderSessionVideoEncode sessionVideoEncode, TranscoderStreamDestinationVideo destinationVideo)
          {
              this.encodeName = name;
              this.sessionVideoEncode = sessionVideoEncode;
              this.destinationVideo = destinationVideo;
          }
      }
    3. Create the TranscoderCreateNotifierExample class as a nested class.

      1. Create the class.
        class TranscoderCreateNotifierExample implements ILiveStreamTranscoderNotify
        {
        }
      2. Add required functions.
        @Override
        public void onLiveStreamTranscoderCreate(ILiveStreamTranscoder liveStreamTranscoder, IMediaStream stream) 
        {
        }
         
        @Override
        public void onLiveStreamTranscoderDestroy(ILiveStreamTranscoder arg0, IMediaStream arg1) 
        {
        }
         
        @Override
        public void onLiveStreamTranscoderInit(ILiveStreamTranscoder arg0, IMediaStream arg1) 
        {
        }
      3. Modify the onLiveStreamTranscoderCreate method.
        public void onLiveStreamTranscoderCreate (ILiveStreamTranscoder liveStreamTranscoder, IMediaStream stream)
        {
              getLogger().info("ModuleTranscoderOverlayExample#TranscoderCreateNotifierExample.onLiveStreamTranscoderCreate["+appInstance.getContextStr()+"]: "+stream.getName());
              ((LiveStreamTranscoder)liveStreamTranscoder).addActionListener(new TranscoderActionNotifierExample());
        }

    4. Create the TranscoderActionNotifierExample class as a nested class.

      1. Create the class.
        class TranscoderActionNotifierExample extends LiveStreamTranscoderActionNotifyBase
        {
              TranscoderVideoDecoderNotifyExample transcoder=null;
        }
      2. Create the onSessionVideoEncodeSetup method. This will create the TranscoderVideoDecoderNotifyExample class, which extends the TranscoderVideoDecoderNotifier class. This class will invoke its onBeforeScaleFrame method for each frame that passes through the system.
        public void onSessionVideoEncodeSetup(LiveStreamTranscoder liveStreamTranscoder, TranscoderSessionVideoEncode sessionVideoEncode)
        {
        	getLogger().info("ModuleTranscoderOverlayExample#TranscoderActionNotifierExample.onSessionVideoEncodeSetup["+appInstance.getContextStr()+"]");
        	TranscoderStream transcoderStream = liveStreamTranscoder.getTranscodingStream();
        	if (transcoderStream != null && transcoder==null)
        	{
        		TranscoderSession transcoderSession = liveStreamTranscoder.getTranscodingSession();
        		TranscoderSessionVideo transcoderVideoSession = transcoderSession.getSessionVideo();
        		List<TranscoderStreamDestination> alltrans = transcoderStream.getDestinations();
        		
        		int w = transcoderVideoSession.getDecoderWidth();
        		int h = transcoderVideoSession.getDecoderHeight();
        		transcoder = new TranscoderVideoDecoderNotifyExample(w,h);
        		transcoderVideoSession.addFrameListener(transcoder);
        		
        		//apply an overlay to all outputs
        		for(TranscoderStreamDestination destination:alltrans)
        		{
        			//TranscoderSessionVideoEncode sessionVideoEncode = transcoderVideoSession.getEncode(destination.getName());
        			TranscoderStreamDestinationVideo videoDestination = destination.getVideo();
        			System.out.println("sessionVideoEncode:"+sessionVideoEncode);
        			System.out.println("videoDestination:"+videoDestination);
        			if (sessionVideoEncode != null && videoDestination !=null)
        			{
        				transcoder.addEncoder(destination.getName(),sessionVideoEncode,videoDestination);
        			} 
        		}
        	}
        	return;
        }

    5. Create the TranscoderVideoDecoderNotifyExample class as a nested class.

      1. Create the class.
        class TranscoderVideoDecoderNotifyExample extends TranscoderVideoDecoderNotifyBase
        {
        }
      2. Add member variables.
        private OverlayImage mainImage=null;private OverlayImage wowzaImage=null;
        private OverlayImage wowzaText = null;
        private OverlayImage wowzaTextShadow = null;
        List<EncoderInfo> encoderInfoList = new ArrayList<EncoderInfo>();
        AnimationEvents videoBottomPadding = new AnimationEvents();
      3. Add the TranscoderVideoDecoderNotifyExample constructor.
        public TranscoderVideoDecoderNotifyExample (int srcWidth, int srcHeight)
        {
        int lowerThirdHeight = 70;
        }
      4. Add the addEncoder method.
        public void addEncoder(String name, TranscoderSessionVideoEncode sessionVideoEncode, TranscoderStreamDestinationVideo destinationVideo)
        {
            encoderInfoList.add(new EncoderInfo(name, sessionVideoEncode,destinationVideo));
        }
      5. Add the onBeforeScaleFrame method. This method is called for each frame that is ingested by the transcoder (source). Here you can add a graphic either to the source or to each destination.
        		public void onBeforeScaleFrame(TranscoderSessionVideo sessionVideo, TranscoderStreamSourceVideo sourceVideo, long frameCount)
        		{
        			boolean encodeSource=false;
        			boolean showTime=false;
        			double scalingFactor=1.0;
        			synchronized(lock)
        			{
        				if (mainImage != null)
        				{
        					//does not need to be done for a static graphic, but left here to build on (transparency/animation)
        					videoBottomPadding.step();
        					mainImage.step();
        					int sourceHeight = sessionVideo.getDecoderHeight();
        					int sourceWidth = sessionVideo.getDecoderWidth();
        					if(showTime)
        					{
        						Date dNow = new Date( );
        						SimpleDateFormat ft = new SimpleDateFormat("hh:mm:ss");
        						wowzaText.SetText(ft.format(dNow));
        						wowzaTextShadow.SetText(ft.format(dNow));
        					}
        					if(encodeSource)
        					{
        						//put the image onto the source
        						scalingFactor = 1.0;
        						TranscoderVideoOverlayFrame overlay = new TranscoderVideoOverlayFrame(mainImage.GetWidth(scalingFactor),
        								mainImage.GetHeight(scalingFactor), mainImage.GetBuffer(scalingFactor));
        						overlay.setDstX(mainImage.GetxPos(scalingFactor));
        						overlay.setDstY(mainImage.GetyPos(scalingFactor));
        						sourceVideo.addOverlay(overlayIndex, overlay);
        					} 
        					else	
        					{
        						///put the image onto each destination but scaled to fit
        						for(EncoderInfo encoderInfo: encoderInfoList)
        						{
        							if (!encoderInfo.destinationVideo.isPassThrough())
        							{
        								int destinationHeight = encoderInfo.destinationVideo.getFrameSizeHeight();
        								scalingFactor = (double)destinationHeight/(double)sourceHeight;
        								TranscoderVideoOverlayFrame overlay = new TranscoderVideoOverlayFrame(mainImage.GetWidth(scalingFactor),
        										mainImage.GetHeight(scalingFactor), mainImage.GetBuffer(scalingFactor));
        								overlay.setDstX(mainImage.GetxPos(scalingFactor));
        								overlay.setDstY(mainImage.GetyPos(scalingFactor));
        								encoderInfo.destinationVideo.addOverlay(overlayIndex,	overlay);
        								//Add padding to the destination video i.e. pinch
        								encoderInfo.videoPadding[0] = 0; // left
        								encoderInfo.videoPadding[1] = 0; // top
        								encoderInfo.videoPadding[2] = 0; // right
        								encoderInfo.videoPadding[3] = (int)(((double)videoBottomPadding.getStepValue())*scalingFactor); // bottom
        								encoderInfo.destinationVideo.setPadding(encoderInfo.videoPadding);
        							}
        						}
        					}
        				} 
        			}
        			return; 
        		}
        encodeSource

        With the module, you can set the encodeSource variable to place an overlay image either on the source video before it's encoded to different destination videos (encodeSource=true) or on each destination video (encodeSource=false). Both methods have benefits and drawbacks.

        Source (encodeSource=true):

        • Benefits: Only one graphic operation is used.
        • Drawbacks: The graphics are rescaled and may not appear as the user intends. Also, fonts aren't redrawn and are rescaled with image reduction. You can put graphics outside the source video if the video is scaled back ("pinched"). For example, if a video was originally 640 x 480 and is pinched to 640 x 400 to provide space at the bottom of the TV for graphics.


        Destination (encodeSource=false):

        • Benefits: Individual graphics can be used for each encoder if needed and fonts are drawn at a size to match the destination. Also, the source can be pinched and graphics drawn outside the video.
        • Drawbacks: Multiple graphic operations are used for each encoder (720p, 360p, etc.)

        showTime

        Setting the showTime variable in onBeforeScaleFrame will cause the code to send the current time as the text for wowzaText. This variable is used in this example to demonstrate dynamic text by using the current time in an overlay image.

        scalingFactor

        This double value defines the relationship between the destination video and source video. A value of .5 means that the destination video is half the size of the source video while a value for 2.0 means that the destination video is twice the size of the source video.


    After you complete these steps, the project should compile, build, and run. However, you won't see an image on the video yet.

    Create an image on the video


    • Add the following command to the end of the TranscoderVideoDecoderNotifyExample constructor to add the logo image (logo_1.png) to the video.
      //create a transparent container for the bottom third of the screen.
      mainImage = new OverlayImage(0,srcHeight-lowerThirdHeight,srcWidth,lowerThirdHeight,100);
       
      //Create the Wowza logo image
      wowzaImage = new OverlayImage(basePath+graphicName,100);
      mainImage.addOverlayImage(wowzaImage,srcWidth-wowzaImage.GetWidth(1.0),0);
      mainImage is the base container for all animation. It represents a transparent lower 3rd for all other images and text.

      wowzaImage is the logo that is drawn on mainImage relative to its x,y position.


    When viewing myStream_360p and all other streams in [install-dir]/examples/LiveVideoStreaming/FlashHTTPPlayer/player.html, the Wowza logo should be displayed in the lower-right corner.

    Create text on the video


    • Add text to the bottom of the image.
      //Add Text with a drop shadow
      wowzaText = new OverlayImage("Wowza", 12, "SansSerif", Font.BOLD, Color.white, 66,15,100);
      wowzaTextShadow = new OverlayImage("Wowza", 12, "SansSerif", Font.BOLD, Color.darkGray, 66,15,100);
      mainImage.addOverlayImage(wowzaText, wowzaImage.GetxPos(1.0)+12, 54);
      wowzaText.addOverlayImage(wowzaTextShadow, 1, 1);
      This will add the the text "Wowza" to the bottom of the OverlayImage with a drop shadow.


    When viewing myStream_360p and all other streams in [install-dir/examples/LiveVideoStreaming/FlashHTTPPlayer/player.html, the Wowza logo and text should be displayed in the lower-right corner.

    Fade the image and text in and out


    To fade the image and text in and out, the OverlayImage class has the addFadingStep method for adding an event to fade the overlay.
    //Fade the Logo and text independently
    wowzaImage.addFadingStep(100,0,25);
    wowzaImage.addFadingStep(0,100,25);
    wowzaText.addFadingStep(0,100,25);
    wowzaText.addFadingStep(100,0,25);
    The addFadingStep method takes the parameters: (<start value>,<end value>,<number of steps>).

    The example code above takes the image opacity value from 0 to 100 in increments of 4 ((end-start)/steps) or ((100-0)/25) and then the reverse (taking the image opacity from 100 to 0). It will keep repeating these events. If the values are addFadingStep(0,100,200), the image would fade in increments of 0.5 ((100-0)/200)) and take longer.

    With the above example code, the Wowza logo and text should be displayed in the lower-right corner and fade in-and-out independently of each other when viewing myStream_360p and all other streams in [install-dir]/examples/LiveVideoStreaming/FlashHTTPPlayer/player.html.

    Note: Perform the following procedure to enable creation of animation sequences described later in this article.
    To fade the image and text in and out

    • Remove or comment-out the example code above and replace it with the following code.
      //do nothing for a bit
      mainImage.addFadingStep(50);
      wowzaImage.addImageStep(50);
      wowzaText.addMovementStep(50);
       
      //Fade the logo and text
      mainImage.addFadingStep(0,100,100);
      This bit of code overloads the addFadingStep method to include only the step value and perform a 'no operation'.


    Rotate the image


    To animate the graphic, the OverlayImage class has an addImageStep method for updating the graphic to be used. The following code will rotate the image four times.
    //Rotate the image while fading
    wowzaImage.addImageStep(1,37,25);
    wowzaImage.addImageStep(1,37,25);
    wowzaImage.addImageStep(1,37,25);
    wowzaImage.addImageStep(1,37,25);
    The addImageStep method takes the same start/end/step parameters as the addFadingStep method. The class sets the variable ${com.wowza.wms.plugin.transcoderoverlays.overlayimage.step} to the current step that the image animation is on in order to rotate the image. We use this variable when defining the image (variable graphicName). Each image is rotated by 5 degrees to create a spinning effect.

    The above code example spins the image 4 times (25 steps) during the time that it takes to fade the image (100 steps).

    Animate the text


    To animate the text, the OverlayImage class has the addMovementStep method for updating the x coordinates of the text that is used.
    //Animate the text off screen to original location
    wowzaText.addMovementStep(-75, 0, wowzaText.GetxPos(1.0), 54, 100);
    The addMovementStep takes a starting x1,y1 coordinate and moves to the x2,y2 coordinate. Moving the text takes the step value and sets the x and y coordinates of the text.

    To finish the animation sequence, add the following code.
    //hold everything for a bit
    mainImage.addFadingStep(50);
    wowzaImage.addImageStep(50);
    wowzaText.addMovementStep(50);
    
    //Fade out
    mainImage.addFadingStep(100,0,50);
    wowzaImage.addImageStep(50);
    wowzaText.addMovementStep(50);
    After the animation is completed, it will hold for 50 steps and then fade out.

    Pinch the video


    When encoding the destination, we can scale back (pinch) the source to leave room for the graphic sequence.
    //Pinch back video
    videoBottomPadding.addAnimationStep(0, 60, 50);
    videoBottomPadding.addAnimationStep(100);
    
    //unpinch the video
    videoBottomPadding.addAnimationStep(60, 0, 50);
    mainImage.addFadingStep(50);
    wowzaImage.addImageStep(50);
    wowzaText.addMovementStep(50);
    videoBottomPadding.addAnimationStep(100);

    Troubleshooting


    • Using overlays can put additional load on your server because the complicated image manipulation that occurs when creating image overlays can consume a lot of processing power. This can cause Transcoder to skip frames when transcoding and delay the video. It's important that your server be properly tuned for maximum resources. For more information, see Performance Tuning.

    • It's recommended that you keep the dimensions of your overlay image small. If your throughput is 30 frames-per-second (fps) and it takes more than 1/30 second to render the image, then the Transcoder may start to skip frames.

    • This feature supports manipulation of images and text to generate animation sequences. It does not support stream manipulation such as Picture-in-picture (PiP) or multi-stream compositing.

    • Using Transcoder to place overlay images onto video-on-demand (VOD) streams isn't supported.

    • Static overlay images that you create by setting Overlays properties in transcoder template files for decoded and encoded streams will be overridden by the dynamic overlays that you create using this feature.

    • Closed captions that you create in Wowza Media Server will be displayed on top of dynamic overlays that you create using this feature.


    Originally Published: For Wowza Media Server 3.5 on 11-08-2012.

    If you're having problems or want to discuss this article, post in our forum.