Wowza Community

/etc/pki/tls/cert.pem symlink missing in EC2 AMIs? (breaks wget)

In at least instances launched from ami-4aff667a in us-west-2, /etc/pki/tls/cert.pem is missing:

[ec2-user@ip-10-x-y-z tls]$ rpm -V ca-certificates-2010.63-3.7.amzn1.noarch

missing /etc/pki/tls/cert.pem

This breaks “wget” connecting to HTTPS sites, which breaks our initial automated provisioning. Could this get fixed?

Thanks,

Andy

You may be encountering a recent issue that has developed in EC2.

(From the Pre Built AMI’s launch page…)

An update to the EC2 Management Console prevents the Wowza Media Server AMIs from launching properly. The workaround is to specify the default startup package using the User data field as part of the AMI launch process. To do this:

In the Configure Instance Details step, open the Advanced Details section.

In the User data field, enter the following user data and then select the As text option.

WZA_startupPackageURL=http://s3.amazonaws.com/wowzamediasystems/com/wowza/startup/default_3.6.0.zip

You may replace your startup package (if applicable) using the same approach from your download source.

-Tim

I checked our current and recently replaced AMI’s. The folder /etc/pki/tls is certainly present, however, cert.pem is evidently not a default file (I didn’t see it). I don’t expect this file missing on your end is the result of a Wowza issue.

-Tim

Hi Andy,

We have worked out what is happening.

It is actually the EC2 tools supplied by Amazon that are removing the symlink when the ami is created. It looks like they added a script to the tool that removes all *.pem files when the new image is built.

We have found a workaround and will most likely be raising it as a bug with Amazon but we feel does not warrant a complete rebuild of all AMIs at this stage.

For the current AMIs, the symlink can be reinstated with the following commands.

cd /etc/pki/tls/
sudo ln -s certs/ca-bundle.crt cert.pem

Regards,

Roger.

I’m running on a “3.6.2.16 build7566 (October 14, 2013)” AMI (and launching via the EC2 API not the console), so I don’t think that’s related here.

(When I wrote it breaks provisioning, I was talking about the in-house tools we use to manage all of our EC2 instances. Right now, I’m manually fixing this, but I figured it best to report the issue upstream.)

My guess would be that /etc/pki/tls/cert.pem was inadvertently removed prior to bundling when the AMI was built?

Thanks,

Andy

Hi Tim,

Thanks for continuing to look into this.

If it’s missing in current and recently replaced AMIs, it most definitely is a Wowza issue: the AMIs as packaged are broken, because SSL client connections cannot validate against the installed CA certificates. That’s been my point all along.

rpm -V reports that it was removed from the RPM as distributed by Amazon:

[ec2-user@ip-10-x-y-z tls]$ rpm -V ca-certificates-2010.63-3.7.amzn1.noarch
missing /etc/pki/tls/cert.pem

It’s also present in the (presumably) upstream Amazon Linux 2013.09 AMI. You can test the difference by doing the following on a Wowza AMI and an Amazon Linux AMI:

Wowza, before /etc/pki/tls/cert.pem is fixed:

$ wget [url]https://www.google.com/[/url]
--2013-10-17 18:55:23--  [url]https://www.google.com/[/url]
Resolving [url]www.google.com[/url] ([url]www.google.com[/url])... 74.125.20.103, 74.125.20.104, 74.125.20.105, ...
Connecting to [url]www.google.com[/url] ([url]www.google.com)|74.125.20.103|:443[/url]... connected.
ERROR: cannot verify [url]www.google.com's[/url] certificate, issued by ‘/C=US/O=Google Inc/CN=Google Internet Authority G2’:
  Unable to locally verify the issuer's authority.
To connect to [url]www.google.com[/url] insecurely, use `--no-check-certificate'.

Amazon Linux, or Wowza with proper /etc/pki/tls/cert.pem:

$ wget [url]https://www.google.com/[/url]
--2013-10-17 18:57:31--  [url]https://www.google.com/[/url]
Resolving [url]www.google.com[/url] ([url]www.google.com[/url])... 173.194.33.179, 173.194.33.180, 173.194.33.176, ...
Connecting to [url]www.google.com[/url] ([url]www.google.com)|173.194.33.179|:443[/url]... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’
    [ <=>                                   ] 18,626      --.-K/s   in 0.006s
2013-10-17 18:57:31 (2.86 MB/s) - ‘index.html’ saved [18626]

Thanks,

Andy

Thanks for the follow through, much appreciated. Looking forward to when you will be able to spin your workaround into a new AMI.

-Andy