First, find some general information on the environment:

Show the JVM system properties

jinfo [pid]

Show heap space usage

jmap -heap [pid]

Show object memory usage

jmap -histo [pid]

Next, find out some more specifics about what the app is actually doing:

Take a thread dump

pkill -3 java; sleep 5; pkill -3 java

We do the kill -3 twice to show if threads are stuck in a particular state – add more as required.  The thread dump is written to stdout for the java process.  This will be catalina.out for a Tomcat app, elsewhere for others, check the init script for clues.

Take a stack trace

jstack [pid]

For apps using JNI you can take a mixed-mode trace by doing

jstack -m [pid]

and piping the output through c++filt. On Solaris, c++filt can be obtained as part of Sun Studio Express

Hopefully by this point you should have enough data to at least give you some clues as to where the problems are occuring.