Hi,
Video encoding is a very cpu intensive process unless a hardware encoder is used. Even running something like FMLE on a normal desktop environment will use between 50% & 80% of a single cpu core. The Wowza transcoder is no exception. Even with a real server with 2 cpu cores and without quicksync would struggle to do more than 1 or 2 transcoding sessions.
Because of the way that virtual environments like vmware allocate resources, you will find very difficult to do any successful transcoding using a virtual server. Even if the host system has QuickSync or other accelerations available, these will not be available to the vm so you will be restricted to doing software encoding only.
You might be able to get satisfactory results if you can allocate more cpu cores to the vm but it will never be as good as a real server which has full access to the resources.
See here, Wowza-Transcoder-AddOn-Performance-Benchmark. Server 3 & Server 4 are EC2 instances with 4 & 8 virtual cores respectively.
The rtmpt with jw player is an automatic failover if it cannot make an rtmp connection. It first tries rtmp and then fails over to rtmpt. If it cannot connect on either then you will get that message.
Regards,
Roger.