When logged in to Wowza Streaming Engine Manager, selecting Server and then Performance Tuning > Java Settings, there is an option for Development Level and Production Level under Java Heap Size. By chance did you select Production Level? If so, try switching back to Development Level, restart (you will be promoted) and evaluate updated memory utilization.
So you compiled JDK directly from Oracle. That rules out any incompatibility with OpenJDK. :/ I have sent an email to Wowza support asking for their help. Once I hear back I will let you know what happens.