"NetStream.Publish.Denied" is a notification sent to the RTMP encoder, it is not logged in Wowza. That module does write to the log just before sending that notificaiton:
What do you mean by "memory leak"? Transcoding uses a lot of cpu and memory. Take a look at these Transcoder benchmark tests for comparison, to see what is realistic for your setup
getLogger().info("ModuleBlockDuplicateStreamNames.publish["+appInstance.getContextStr()+"]: Stream name is already in use: "+streamName);
Remember that you have to leave room for playback. It is common to use Live Repeater (origin/edge) configuration with the Transcoder, where the transcoding is done on the origin and the edge(s) handle playback clients. This leaves more room on the origin for transcoding.