203 lines
7.1 KiB
Markdown
203 lines
7.1 KiB
Markdown
|
---
|
||
|
layout: post
|
||
|
title: "Profiling of remote JVMs with VisualVM and JConsole"
|
||
|
tags:
|
||
|
- jvm
|
||
|
- jruby
|
||
|
- visualvm
|
||
|
---
|
||
|
|
||
|
**Note:** I originally posted this
|
||
|
[here](http://hackers.lookout.com/2014/06/profiling-remote-jvms/), on the
|
||
|
[Lookout hackers blog](http://hackers.lookout.com). I encourage you to check
|
||
|
the blog out and follow [@LookoutEng](https://twitter.com/lookouteng).
|
||
|
|
||
|
---
|
||
|
|
||
|
Recently I found myself hosting a bit of a "bake-off competition" between
|
||
|
servlet containers for JRuby applications here at
|
||
|
[Lookout](https://www.lookout.com/about/careers).
|
||
|
|
||
|
|
||
|
The goal of the bake-off was to determine whether we should host some
|
||
|
[warbled](https://github.com/jruby/warbler) JRuby applications in Tomcat or
|
||
|
Jetty. Not having a huge amount of experience, or bias, towards one or the
|
||
|
other I elected to run a simple [Hello
|
||
|
Warld](https://github.com/rtyler/hellowarld) application in both and see how
|
||
|
well the containers performed.
|
||
|
|
||
|
|
||
|
### The Bake-Off Environment
|
||
|
|
||
|
For the bake-off of servlet containers I used to identical EC2 instances. EC2
|
||
|
instances were chosen instead of running both containers on my local machine to
|
||
|
keep the machines consistent, isolated and more representative of the
|
||
|
environment we would be running webapps in. The test bed specs were:
|
||
|
|
||
|
* Ubuntu 12.04 LTS
|
||
|
* `m1.small` instance size (hey, I'm not made of money!)
|
||
|
* us-west-2 region
|
||
|
* A security group with everything open to the other machines in that security
|
||
|
group. This is important for later.
|
||
|
* OpenJDK 7 (u55)
|
||
|
|
||
|
The machines were then provisioned with the Tomcat 7 and Jetty 6 respectively,
|
||
|
only because those were available directly from the native packages on Ubuntu 12.04.
|
||
|
|
||
|
Both containers were also set up to perform hot-deploys; a feature which relies
|
||
|
on live-reloading of an application without restarting the JVM.
|
||
|
|
||
|
|
||
|
### Problems, ahoy!
|
||
|
|
||
|
After performing a number of successive hot-deploys in Tomcat, I found my logs
|
||
|
clobbered with the following errors:
|
||
|
|
||
|
~~~
|
||
|
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
|
||
|
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
|
||
|
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
|
||
|
~~~
|
||
|
|
||
|
The actual problem here will need to be covered in another blog post, but
|
||
|
something fishy was clearly going on with hot-deployments in Tomcat.
|
||
|
|
||
|
|
||
|
## Profiling in the cloud
|
||
|
|
||
|
My favorite tool for understanding a running JVM is
|
||
|
definitely [VisualVM](http://visualvm.java.net/), with
|
||
|
[JConsole](http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html)
|
||
|
running in a close second place. Fortunately, both tools are quite easy to set
|
||
|
up for connecting to remote JVMs.
|
||
|
|
||
|
|
||
|
### Setting up jstatd
|
||
|
|
||
|
Previously I mentioned the importance of the EC2 security group. It's important
|
||
|
that port **1099** is open within the security group. This is the port
|
||
|
[jstatd](http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstatd.html)
|
||
|
runs on by default. `jstatd` is what will provide VisualVM with live
|
||
|
instrumentation data from the JVMs running on the machines.
|
||
|
|
||
|
|
||
|
It's also important to provide a liberal security policy file to `jstatd`. If
|
||
|
this were anything more than a simple test implementation, I would recommend a
|
||
|
more restrictive policy, but we're optimizing for easiness here, so a wide open
|
||
|
policy is fine:
|
||
|
|
||
|
~~~
|
||
|
grant codebase "file:${java.home}/../lib/tools.jar" {
|
||
|
permission java.security.AllPermission;
|
||
|
};
|
||
|
~~~
|
||
|
|
||
|
Save the policy above into a file named `jstatd.policy`, which we can run
|
||
|
on each machine:
|
||
|
|
||
|
~~~
|
||
|
ec2-tomcat% sudo jstatd -J-Djava.security.policy=./jstatd.policy &
|
||
|
~~~
|
||
|
|
||
|
~~~
|
||
|
ec2-jetty% sudo jstatd -J-Djava.security.policy=./jstatd.policy &
|
||
|
~~~
|
||
|
|
||
|
### JMX for interactivity
|
||
|
|
||
|
`jstatd` only gives us half the picture we want. We also want
|
||
|
[JMX](http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html)
|
||
|
to be configured for both containers to allow us to extract more information
|
||
|
and interact with the running JVMs.
|
||
|
|
||
|
|
||
|
Again, we'll set up very liberal security policies since this is for testing
|
||
|
only! The native packages for both containers put a file in `/etc/defaults`
|
||
|
which contains a `JAVA_OPTS` variable, to which the following should be added:
|
||
|
|
||
|
~~~
|
||
|
-Dcom.sun.management.jmxremote \
|
||
|
-Dcom.sun.management.jmxremote.ssl=false \
|
||
|
-Dcom.sun.management.jmxremote.authenticate=false \
|
||
|
-Dcom.sun.management.jmxremote.port=1098
|
||
|
~~~
|
||
|
|
||
|
After restarting the container processes, they will now have the right JMX
|
||
|
settings and we should be able to finally be able to connect VisualVM or
|
||
|
JConsole to the JVMs.
|
||
|
|
||
|
|
||
|
|
||
|
### Proxying for connectivity
|
||
|
|
||
|
In order to give tools running locally on my machine access to these processes
|
||
|
running inside of a security group within EC2, I'll rely on `ssh`'s ability to
|
||
|
provide a SOCKS5 proxy;
|
||
|
|
||
|
~~~
|
||
|
kiwi% ssh -D 9696 ubuntu@ec2-66-166-66-66.us-west-2.compute.amazonaws.com
|
||
|
~~~
|
||
|
|
||
|
This will provide a path for both VisualVM and JConsole to use when accessing
|
||
|
the JMX information (port 1098) and the `jstatd` information (port 1099) on the
|
||
|
machines running inside of a security group within EC2. While not wholly
|
||
|
necessary, exposing these ports to the wide-open internet seems like a Bad
|
||
|
Idea™.
|
||
|
|
||
|
|
||
|
#### Running JConsole with a proxy
|
||
|
|
||
|
JConsole doesn't have any GUI configuration for a proxy, so it's necessary to
|
||
|
set some command line parameters:
|
||
|
|
||
|
~~~
|
||
|
kiwi% jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=9696
|
||
|
~~~
|
||
|
|
||
|
Once JConsole is up and running, you only need to enter the EC2 hostname and
|
||
|
appropriate JMX port (1098) to connect to the running JVM:
|
||
|
|
||
|
![JConsole with EC2
|
||
|
hostnames](http://hackers.lookout.com/images/post-images/profiling-remote-jvms/jconsole-with-proxy.png)
|
||
|
|
||
|
After clicking "Connect", JConsole will use the SSH-based proxy to connect to
|
||
|
the host, and you should be able to poke around with a real live JVM:
|
||
|
|
||
|
![JConsole connected to
|
||
|
EC2](http://hackers.lookout.com/images/post-images/profiling-remote-jvms/jconsole-connected-remotely.png)
|
||
|
|
||
|
|
||
|
### Running VisualVM with a proxy
|
||
|
|
||
|
Unlike JConsole, VisualVM allows for a GUI-based configuration of a SOCKS
|
||
|
proxy:
|
||
|
|
||
|
![VisualVM proxy
|
||
|
configuration](http://hackers.lookout.com/images/post-images/profiling-remote-jvms/visualvm-proxy-conf.png)
|
||
|
|
||
|
With the proxy configuration saved, we can then add a remote host by
|
||
|
right-clicking on "Remote" and selecting "Add remote host".
|
||
|
|
||
|
![Adding remote VisualVM
|
||
|
host](http://hackers.lookout.com/images/post-images/profiling-remote-jvms/adding-remote-host-visualvm.png)
|
||
|
|
||
|
|
||
|
Provided `jstatd` is running on the remote host, your SSH-based proxy is
|
||
|
running and the remote JVM is running, you should be able to connect to the
|
||
|
remote JVM and start profiling it like you would a local JVM!
|
||
|
|
||
|
![Profiling Tomcat
|
||
|
remotely](http://hackers.lookout.com/images/post-images/profiling-remote-jvms/profiling-tomcat-remotely.png)
|
||
|
|
||
|
---
|
||
|
|
||
|
Both JConsole and VisualVM give you access to a lot of the instrumentation data
|
||
|
available from a running Java Virtual Machine, but neither will magically
|
||
|
identify or solve performance problems. There's still more work to be done to
|
||
|
triage and ultimately resolve those kinds of issues, but at least these tools
|
||
|
give you the information you need to know what's going on, and [knowing is half
|
||
|
the
|
||
|
battle](http://cdn.churchm.ag/wp-content/uploads/2014/03/knowing-is-half-the-battle.jpg).
|
||
|
|
||
|
|