7.1 KiB
layout | title | tags | |||
---|---|---|---|---|---|
post | Profiling of remote JVMs with VisualVM and JConsole |
|
Note: I originally posted this here, on the Lookout hackers blog. I encourage you to check the blog out and follow @LookoutEng.
Recently I found myself hosting a bit of a "bake-off competition" between servlet containers for JRuby applications here at Lookout.
The goal of the bake-off was to determine whether we should host some warbled JRuby applications in Tomcat or Jetty. Not having a huge amount of experience, or bias, towards one or the other I elected to run a simple Hello Warld application in both and see how well the containers performed.
The Bake-Off Environment
For the bake-off of servlet containers I used to identical EC2 instances. EC2 instances were chosen instead of running both containers on my local machine to keep the machines consistent, isolated and more representative of the environment we would be running webapps in. The test bed specs were:
- Ubuntu 12.04 LTS
m1.small
instance size (hey, I'm not made of money!)- us-west-2 region
- A security group with everything open to the other machines in that security group. This is important for later.
- OpenJDK 7 (u55)
The machines were then provisioned with the Tomcat 7 and Jetty 6 respectively, only because those were available directly from the native packages on Ubuntu 12.04.
Both containers were also set up to perform hot-deploys; a feature which relies on live-reloading of an application without restarting the JVM.
Problems, ahoy!
After performing a number of successive hot-deploys in Tomcat, I found my logs clobbered with the following errors:
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
The actual problem here will need to be covered in another blog post, but something fishy was clearly going on with hot-deployments in Tomcat.
Profiling in the cloud
My favorite tool for understanding a running JVM is definitely VisualVM, with JConsole running in a close second place. Fortunately, both tools are quite easy to set up for connecting to remote JVMs.
Setting up jstatd
Previously I mentioned the importance of the EC2 security group. It's important
that port 1099 is open within the security group. This is the port
jstatd
runs on by default. jstatd
is what will provide VisualVM with live
instrumentation data from the JVMs running on the machines.
It's also important to provide a liberal security policy file to jstatd
. If
this were anything more than a simple test implementation, I would recommend a
more restrictive policy, but we're optimizing for easiness here, so a wide open
policy is fine:
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
Save the policy above into a file named jstatd.policy
, which we can run
on each machine:
ec2-tomcat% sudo jstatd -J-Djava.security.policy=./jstatd.policy &
ec2-jetty% sudo jstatd -J-Djava.security.policy=./jstatd.policy &
JMX for interactivity
jstatd
only gives us half the picture we want. We also want
JMX
to be configured for both containers to allow us to extract more information
and interact with the running JVMs.
Again, we'll set up very liberal security policies since this is for testing
only! The native packages for both containers put a file in /etc/defaults
which contains a JAVA_OPTS
variable, to which the following should be added:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.port=1098
After restarting the container processes, they will now have the right JMX settings and we should be able to finally be able to connect VisualVM or JConsole to the JVMs.
Proxying for connectivity
In order to give tools running locally on my machine access to these processes
running inside of a security group within EC2, I'll rely on ssh
's ability to
provide a SOCKS5 proxy;
kiwi% ssh -D 9696 ubuntu@ec2-66-166-66-66.us-west-2.compute.amazonaws.com
This will provide a path for both VisualVM and JConsole to use when accessing
the JMX information (port 1098) and the jstatd
information (port 1099) on the
machines running inside of a security group within EC2. While not wholly
necessary, exposing these ports to the wide-open internet seems like a Bad
Idea™.
Running JConsole with a proxy
JConsole doesn't have any GUI configuration for a proxy, so it's necessary to set some command line parameters:
kiwi% jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=9696
Once JConsole is up and running, you only need to enter the EC2 hostname and appropriate JMX port (1098) to connect to the running JVM:
After clicking "Connect", JConsole will use the SSH-based proxy to connect to the host, and you should be able to poke around with a real live JVM:
Running VisualVM with a proxy
Unlike JConsole, VisualVM allows for a GUI-based configuration of a SOCKS proxy:
With the proxy configuration saved, we can then add a remote host by right-clicking on "Remote" and selecting "Add remote host".
Provided jstatd
is running on the remote host, your SSH-based proxy is
running and the remote JVM is running, you should be able to connect to the
remote JVM and start profiling it like you would a local JVM!
Both JConsole and VisualVM give you access to a lot of the instrumentation data available from a running Java Virtual Machine, but neither will magically identify or solve performance problems. There's still more work to be done to triage and ultimately resolve those kinds of issues, but at least these tools give you the information you need to know what's going on, and knowing is half the battle.