How to transport data to Amazon S3 services

Wowza Streaming Engine™ media server software (4.6.0.03 and later) can be used to transport data to external services. This article describes how to transport data to Amazon S3.

Contents


Requirements
Create an Amazon S3 transport provider Configure the MediaTransport listeners Configure an Amazon S3 bucket Configure the Amazon S3 transport object

Requirements


To use the TransportMedia packages to transport data to Amazon S3, you must have the following:

  • Wowza Streaming Engine 4.6.0.03 or later
  • Knowledge of programming in Java
  • Wowza IDE
  • An Amazon S3 account with the following:
    • Amazon access key
    • Amazon secret key
    For more information, see About access keys.
  • Amazon S3 bucket name
  • Amazon S3 region name

Create an Amazon S3 transport provider


The following example demonstrates how to create  and initialize a transport provider:

String buckeName = “”;
String regionName = “”:
String accessKey = “”;
String secretKey = “”;

MediaTransport mediaTransport = new MediaTransport();
mediaTransport.setDebug(this.debug);
mediaTransport.init();
mediaTransport.addTransportListener(new MediaTransportListener());
MediaTransportConfigurationAmazonS3 amazonConfig = new MediaTransportConfigurationAmazonS3();
MediaTransportProviderAmazonS3 amazonProvider = new MediaTransportProviderAmazonS3();

// Now we configure the provider configuration with our base configuration items
amazonConfig.setAttributeString(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_BUCKET, bucketName);
amazonConfig.setAttributeString(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_REGION, regionName);
amazonConfig.setAttributeString(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_ACCESSID, accessKey);
amazonConfig.setAttributeString(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_SECRETACCESSKEY, secretKey);
amazonConfig.init();
// Now we add the configuration to the transport provider
amazonProvider.setTransportProviderConfiguration(amazonConfig);
// Now we initialise the transport provider
// The transport provider can be done SYNC or ASYNC, ASYNC is default.
amazonProvider.init();
amazonProvider.setTransportProcessType(IMediaTransportProvider.TRANSPORT_PROVIDER_PROCESS_SYNC);
if ( amazonProvider.isReady() )
{
	mediaTransport.addTransportProvider(amazonProvider);
}

Where:

  • bucketName is the name of the bucket in Amazon S3. You'll need to use the same bucketName when you Create a bucket.
  • regionName is the bucket's region. If you don't specify a region, Amazon uses US East as the default region.
  • accessKey and secretKey are the access key and secret access key associated with your AWS account.

Amazon S3 transport provider parameters

The following static parameters are designed so that they can be used to easily added to a provider or, in some cases, on a per-transport object basis.

AMZ_CONFIG_BUCKET

The AMZ_CONFIG_BUCKET parameter sets the name of the bucket.

Use the following command to apply this parameter to the transport configuration:

amazonConfig.setAttributeString(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_BUCKET ,”mybucket”);AMZ_CONFIG_REGION

AMZ_CONFIG_REGION

The AMZ_CONFIG_REGION parameter sets the region for the bucket.

Use the following command to apply this parameter to the transport configuration:

amazonConfig.setAttributeString(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_REGION, “myregion”);

AMZ_CONFIG_REGION_ENABLED

The AMZ_CONFIG_REGION_ENABLED parameter enables or disables the region.

Use the following command to apply this parameter to the transport configuration:

amazonConfig.setAttributeBoolean(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_REGION_ENABLED, true);

AMZ_CONFIG_SECRETACCESSKEY

The AMZ_CONFIG_SECRETACCESSKEY parameter sets the secret access key associated with your AWS account.

Use the following command to apply this parameter to the transport configuration:

amazonConfig.setAttributeString(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_SECRETACCESSKEY, “mySecretKey”);

AMZ_CONFIG_ACCESSID

The AMZ_CONFIG_ACCESSID parameter sets the access key associated with your AWS account.

Use the following command to apply this parameter to the transport configuration:

amazonConfig.setAttributeString(MediaTransportConfigurationAmazonS3.AMZ_CONFIG_ACCESSID, “myAccessKey”);

Configure the MediaTransport listeners


The transport model is asynchronous by default, so two listeners are available to determine when object transports are requested and whether they were successful: IMediaTransportListener and IMediaTransportProviderListener.

IMediaTransportListener

The IMediaTransportListener provides two callback functions:

  • public void onTransportProviderAdd (IMediaTransportProvider thisProvider)
  • public void onTranportProviderRemove (IMediaTransportProvider thisProvider)

The following example logs the addition or removal of providers:

public class MediaTransportListener implements IMediaTransportListener
{
    public void onTransportProviderAdd(IMediaTransportProvider thisProvider)
    {
      getLogger().info("Adding Provider: "+thisProvider.getTransportProviderName());
    }

    public void onTransportProviderRemove(IMediaTransportProvider thisProvider)
    {
      getLogger().info("Removing Provider: "+thisProvider.getTransportProviderName());
    }
}

Use the following command to add the IMediaTransportListener to the transport media system:

mediaTransport.addTransportListener(new MediaTransportListener());

IMediaTransportProviderListener

The IMediaTransportProviderListener providers one callback function:

public void onTransportObjectStatus(int count, IMediaTransportResponse responseObject, String transportProviderName)

Each time a transport provider attempts to send an object, the call back function is called to access the status of the transport. In the callback function:

  • count is the current attempt count for the object being transported. The transport provider will attempt to transport the requested object up to the maximum number of retries set for the object. By default, each object's maximum retry value is 5.
  • responseObject is additional information about the transport attempt.
  • transportProviderName is the name of the transport provider used in the attempt to transport the object.

The following example checks the success of the responseObject. If successful, the response data and headers are displayed. If unsuccessful, the response data is displayed.

public class MediaTransportProviderListener implements IMediaTransportProviderListener
{
  public void onTransportObjectStatus(int count, IMediaTransportResponse responseObject, String transportProviderName)
  {
  getLogger().info("onTransportObjectStatus: count: "+count);
  getLogger().info("onTransportObjectStatus: transportProviderName: "+transportProviderName);
  getLogger().info("onTransportObjectStatus: success: "+responseObject.getSuccess());
  getLogger().info("onTransportObjectStatus: basic message: "+responseObject.getBasicMessage());

  byte[] responseCode = responseObject.getSuccessData(IMediaTransportResponse.RESPONSE_INDICATOR);
  if ( responseCode != null )
  {
    String thisResponseCode = new String(responseCode);
    getLogger().info("onTransportObjectStatus: Response code: "+thisResponseCode);
  }

  if ( responseObject.getSuccess() == true )
  {
    getLogger().info("onTransportObjectStatus: Path of object: "+responseObject.getPath());
    byte[] responseData = responseObject.getSuccessData(IMediaTransportResponse.RESPONSE_DATA);
    if ( responseData != null )
    {
      getLogger().info("onTransportObjectStatus: responseData Length: "+responseData.length);
      byte[] responseHeaders = responseObject.getSuccessData(IMediaTransportResponse.RESPONSE_HEADERS);
      if ( responseHeaders != null )
      {
      getLogger().info("onTransportObjectStatus: responseHeaders Length: "+responseHeaders.length);
      getLogger().info("onTransportObjectStatus: responseHeaders :"+new String(responseHeaders));
      }
    }
  }
  else
  {
    byte[] responseData = responseObject.getSuccessData(IMediaTransportResponse.RESPONSE_DATA);
    if ( responseData != null )
    {
      getLogger().info("onTransportObjectStatus: responseData Length: "+responseData.length);
      getLogger().info("onTransportObjectStatus: responseData :"+new String(responseData));
    }
  }
}

Use the following command to add the IMediaTransportProviderListener to the transport provider:

amazonProvider.addProviderListener(new MediaTransportProviderListener());

Configure an Amazon S3 bucket


Create a bucket

The following example creates a bucket named asimpletestbucket in your account. Note that the XML is placed as the body of the object and the setPath is set to a forward slash (/).

Note: This example assumes that the transport provider has been initialized.

String bucketCreateRequest = "";

bucketCreateRequest +="<CreateBucketConfiguration xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">\n";
bucketCreateRequest +="  <LocationConstraint>eu-west-1</LocationConstraint>\n";
bucketCreateRequest +="  </CreateBucketConfiguration>\n";
MediaTransportDataObject transportDataObject = new MediaTransportDataObject();
// Override the transport provider bucket name configured
transportDataObject.setOption(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_OPTION_BUCKET, "asimpletestbucket");
// When making requests direct to a bucket, so delete, create you need to specify a region, even if
// it is multi region
transportDataObject.setOption(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_OPTION_REGION_ENABLED, "true");
// Override the region for the bucket
transportDataObject.setOption(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_OPTON_REGION, "eu-west-1");
transportDataObject.setPath("/");
transportDataObject.setData(bucketCreateRequest.getBytes());
transportDataObject.setCommand("PUT");
mediaTransport.addTransportDataObject(transportDataObject);

Upload content to a bucket

The following example transports an object to the destination path set. Note that Amazon S3 also has options to set storage class and access type for objects.

// We get the bytes for the file and display the size of the byte array.
// We assume the file contents are provided as a byte array.
byte[] fileBytes = getBytesFromFile(uploadFile);
getLogger().info("Filename: "+filename+" File: FileByte Count: "+fileBytes.length);
if ( fileBytes != null )
{
getLogger().info("Filename: "+filename+" UploadFile:  "+uploadFile);
  // We do not override anything so this goes into the transport provider configured settings.
  MediaTransportDataObject transportDataObject = new MediaTransportDataObject();
  // Set the path as to the destination of the object
  transportDataObject.setPath("/"+filename);
  // Set the content type
  transportDataObject.setContentType("video/mp4");
  // Set the data to be transported
  transportDataObject.setData(fileBytes);
  // Set the command to use
  transportDataObject.setCommand("PUT");

  /**
   * Storage Class options are
   *
   * STANDARD
   * STANDARD_IS
   * GLACIER (you should not specify this during upload)
   * REDUCED_REDUNDANCY
   *
   * If you do not specify a storage class STANDARD is the default
   */
        transportDataObject.setHeader(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_HEADER_STORAGE_CLASS, "REDUCED_REDUNDANCY");

  /**
   * Access list options are
   *
   * private
   * public-read
   * public-read-write
   * aws-exec-read
   * authenticated-read
   * bucket-owner-read
   * bucket-owner-full-control
   *
   * If you do not specify an access list the default is private
 */
  transportDataObject.setHeader(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_HEADER_ACCESS_LIST, "public-read");
  getLogger().info("Filename: "+filename+" Object: "+transportDataObject);
  getLogger().info("Filename: "+filename+" Object.command: "+transportDataObject.getCommand());
  getLogger().info("Filename: "+filename+" Object.length: "+fileBytes.length);
  mediaTransport.addTransportDataObject(transportDataObject);
}

Delete a bucket

The following example deletes the bucket named asimpletestbucket from your account. Note that the setPath is set to  a forward slash (/).

Note: This example assumes that the transport provider has been initialized.

MediaTransportDataObject transportDataObject = new MediaTransportDataObject();
// This parameter overrides the default bucket set for the transport method. You can do this for each object if you wish.
transportDataObject.setOption(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_OPTION_BUCKET, "asimpletestbucket");
// You do need to enable region when deleting  bucket to be 100% sure it will work.
// ALL buckets do have a region.
transportDataObject.setOption(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_OPTION_REGION_ENABLED, "true");
// You can also override the project id with
//transportDataObject.setOption(MediaTransportConfigurationGoogleCloud.GC_CONFIG_PROJECT_ID,"a new project id");
transportDataObject.setPath("/");
transportDataObject.setCommand("DELETE");
mediaTransport.addTransportDataObject(transportDataObject);

Configure the Amazon S3 transport object


Transport objects have two parts: options and headers. The options override default values set by the transport provider configuration, and headers provide additional information in the HTTP request.

Transport object options

The following static options are available for transport objects:

  • AMZ_OBJECT_OPTION_BUCKET - Use this option to set the bucket's name to something other than the name configured by the transport provider. This is useful when you want objects to go to different buckets.
  • AMZ_OBJECT_OPTION_REGION - Use this option to set the bucket's region to something other than the region configured by the transport provider. This is useful when you want objects to go to different buckets.

    Note: This is primarily needed to create and delete buckets.

  • AMZ_OBJECT_OPTION_REGION_ENABLED - Use this option if you want the HTTP request to use the full Amazon S3 hostname rather than just the bucket name. Note that this is required for some older buckets.

The following example demonstrates sets the AMZ_OBJECT_OPTION_BUCKET option:

transportDataObject.setOption(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_OPTION_BUCKET, "anothersimplebucket");

The same syntax can be used to configure the other transport object options, too.

Transport object headers

Transport object headers enable you to add HTTP headers for specific objects. The following is a list of pre-defined headers:

  • AMZ_OBJECT_HEADER_STORAGE_CLASS
  • AMZ_OBJECT_HEADER_ACCESS_LIST
  • AMZ_OBJECT_HEADER_SECURITY_TOKEN
  • AMZ_OBJECT_HEADER_COPY_SOURCE
  • AMZ_OBJECT_HEADER_META_DIRECTIVE
  • AMZ_OBJECT_HEADER_COPY_SOURCE_IF_MATCH
  • AMZ_OBJECT_HEADER_COPY_SOURCE_IF_NONE_MATCH
  • AMZ_OBJECT_HEADER_COPY_SOURCE_IF_UNMODIFIED_SINCE
  • AMZ_OBJECT_HEADER_COPY_SOURCE_IF_MODIFIED_SINCE
  • AMZ_OBJECT_HEADER_GRANT_READ
  • AMZ_OBJECT_HEADER_GRANT_WRITE
  • AMZ_OBJECT_HEADER_GRANT_READ_ACP
  • AMZ_OBJECT_HEADER_GRANT_WRITE_ACP
  • AMZ_OBJECT_HEADER_GRANT_FULL_CONTROL
  • AMZ_OBJECT_HEADER_SERVER_ENCRYPT
  • AMZ_OBJECT_HEADER_SERVER_ENCRYPT_KEY_ID
  • AMZ_OBJECT_HEADER_SERVER_ENCRYPT_CONTEXT

Note: You can directly add any HTTP header that you want to, and aren't restricted to the headers listed above. Headers that begin with x-amz- will be added correctly to the Amazon S3 signature.

When setting these headers, you must provide the correct value. For example, the following command demonstrates how to set the access list header:

transportDataObject.setHeader(MediaTransportConfigurationAmazonS3.AMZ_OBJECT_HEADER_ACCESS_LIST, "public-read");

If you're having problems or want to discuss this article, post in our forum.