Is this a virtual private server? How much memory is in the machine? You may need to set the heap size to a very low value to start the server. To do this edit the file [install-dir]/bin/setenv.sh and change the Java heap size:
You may need to try lower values. The problem with this setup is that you are not going to be able to handle many concurrent connections with this much memory. I am also concerned about the initial Java installation that failed. I am not confident that Java is installed properly.