Add JVM cross-post
This commit is contained in:
R. Tyler Croy
parent
b9c0e9359d
commit
a985949e73
|
|
@ -0,0 +1,202 @@
|
|||
---
|
||||
layout: post
|
||||
title: "Profiling of remote JVMs with VisualVM and JConsole"
|
||||
tags:
|
||||
- jvm
|
||||
- jruby
|
||||
- visualvm
|
||||
---
|
||||
|
||||
**Note:** I originally posted this
|
||||
[here](http://hackers.lookout.com/2014/06/profiling-remote-jvms/), on the
|
||||
[Lookout hackers blog](http://hackers.lookout.com). I encourage you to check
|
||||
the blog out and follow [@LookoutEng](https://twitter.com/lookouteng).
|
||||
|
||||
---
|
||||
|
||||
Recently I found myself hosting a bit of a "bake-off competition" between
|
||||
servlet containers for JRuby applications here at
|
||||
[Lookout](https://www.lookout.com/about/careers).
|
||||
|
||||
|
||||
The goal of the bake-off was to determine whether we should host some
|
||||
[warbled](https://github.com/jruby/warbler) JRuby applications in Tomcat or
|
||||
Jetty. Not having a huge amount of experience, or bias, towards one or the
|
||||
other I elected to run a simple [Hello
|
||||
Warld](https://github.com/rtyler/hellowarld) application in both and see how
|
||||
well the containers performed.
|
||||
|
||||
|
||||
### The Bake-Off Environment
|
||||
|
||||
For the bake-off of servlet containers I used to identical EC2 instances. EC2
|
||||
instances were chosen instead of running both containers on my local machine to
|
||||
keep the machines consistent, isolated and more representative of the
|
||||
environment we would be running webapps in. The test bed specs were:
|
||||
|
||||
* Ubuntu 12.04 LTS
|
||||
* `m1.small` instance size (hey, I'm not made of money!)
|
||||
* us-west-2 region
|
||||
* A security group with everything open to the other machines in that security
|
||||
group. This is important for later.
|
||||
* OpenJDK 7 (u55)
|
||||
|
||||
The machines were then provisioned with the Tomcat 7 and Jetty 6 respectively,
|
||||
only because those were available directly from the native packages on Ubuntu 12.04.
|
||||
|
||||
Both containers were also set up to perform hot-deploys; a feature which relies
|
||||
on live-reloading of an application without restarting the JVM.
|
||||
|
||||
|
||||
### Problems, ahoy!
|
||||
|
||||
After performing a number of successive hot-deploys in Tomcat, I found my logs
|
||||
clobbered with the following errors:
|
||||
|
||||
~~~
|
||||
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
|
||||
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
|
||||
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
|
||||
~~~
|
||||
|
||||
The actual problem here will need to be covered in another blog post, but
|
||||
something fishy was clearly going on with hot-deployments in Tomcat.
|
||||
|
||||
|
||||
## Profiling in the cloud
|
||||
|
||||
My favorite tool for understanding a running JVM is
|
||||
definitely [VisualVM](http://visualvm.java.net/), with
|
||||
[JConsole](http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html)
|
||||
running in a close second place. Fortunately, both tools are quite easy to set
|
||||
up for connecting to remote JVMs.
|
||||
|
||||
|
||||
### Setting up jstatd
|
||||
|
||||
Previously I mentioned the importance of the EC2 security group. It's important
|
||||
that port **1099** is open within the security group. This is the port
|
||||
[jstatd](http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstatd.html)
|
||||
runs on by default. `jstatd` is what will provide VisualVM with live
|
||||
instrumentation data from the JVMs running on the machines.
|
||||
|
||||
|
||||
It's also important to provide a liberal security policy file to `jstatd`. If
|
||||
this were anything more than a simple test implementation, I would recommend a
|
||||
more restrictive policy, but we're optimizing for easiness here, so a wide open
|
||||
policy is fine:
|
||||
|
||||
~~~
|
||||
grant codebase "file:${java.home}/../lib/tools.jar" {
|
||||
permission java.security.AllPermission;
|
||||
};
|
||||
~~~
|
||||
|
||||
Save the policy above into a file named `jstatd.policy`, which we can run
|
||||
on each machine:
|
||||
|
||||
~~~
|
||||
ec2-tomcat% sudo jstatd -J-Djava.security.policy=./jstatd.policy &
|
||||
~~~
|
||||
|
||||
~~~
|
||||
ec2-jetty% sudo jstatd -J-Djava.security.policy=./jstatd.policy &
|
||||
~~~
|
||||
|
||||
### JMX for interactivity
|
||||
|
||||
`jstatd` only gives us half the picture we want. We also want
|
||||
[JMX](http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html)
|
||||
to be configured for both containers to allow us to extract more information
|
||||
and interact with the running JVMs.
|
||||
|
||||
|
||||
Again, we'll set up very liberal security policies since this is for testing
|
||||
only! The native packages for both containers put a file in `/etc/defaults`
|
||||
which contains a `JAVA_OPTS` variable, to which the following should be added:
|
||||
|
||||
~~~
|
||||
-Dcom.sun.management.jmxremote \
|
||||
-Dcom.sun.management.jmxremote.ssl=false \
|
||||
-Dcom.sun.management.jmxremote.authenticate=false \
|
||||
-Dcom.sun.management.jmxremote.port=1098
|
||||
~~~
|
||||
|
||||
After restarting the container processes, they will now have the right JMX
|
||||
settings and we should be able to finally be able to connect VisualVM or
|
||||
JConsole to the JVMs.
|
||||
|
||||
|
||||
|
||||
### Proxying for connectivity
|
||||
|
||||
In order to give tools running locally on my machine access to these processes
|
||||
running inside of a security group within EC2, I'll rely on `ssh`'s ability to
|
||||
provide a SOCKS5 proxy;
|
||||
|
||||
~~~
|
||||
kiwi% ssh -D 9696 ubuntu@ec2-66-166-66-66.us-west-2.compute.amazonaws.com
|
||||
~~~
|
||||
|
||||
This will provide a path for both VisualVM and JConsole to use when accessing
|
||||
the JMX information (port 1098) and the `jstatd` information (port 1099) on the
|
||||
machines running inside of a security group within EC2. While not wholly
|
||||
necessary, exposing these ports to the wide-open internet seems like a Bad
|
||||
Idea™.
|
||||
|
||||
|
||||
#### Running JConsole with a proxy
|
||||
|
||||
JConsole doesn't have any GUI configuration for a proxy, so it's necessary to
|
||||
set some command line parameters:
|
||||
|
||||
~~~
|
||||
kiwi% jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=9696
|
||||
~~~
|
||||
|
||||
Once JConsole is up and running, you only need to enter the EC2 hostname and
|
||||
appropriate JMX port (1098) to connect to the running JVM:
|
||||
|
||||
|
||||
|
||||
After clicking "Connect", JConsole will use the SSH-based proxy to connect to
|
||||
the host, and you should be able to poke around with a real live JVM:
|
||||
|
||||
|
||||
|
||||
|
||||
### Running VisualVM with a proxy
|
||||
|
||||
Unlike JConsole, VisualVM allows for a GUI-based configuration of a SOCKS
|
||||
proxy:
|
||||
|
||||
|
||||
|
||||
With the proxy configuration saved, we can then add a remote host by
|
||||
right-clicking on "Remote" and selecting "Add remote host".
|
||||
|
||||
|
||||
|
||||
|
||||
Provided `jstatd` is running on the remote host, your SSH-based proxy is
|
||||
running and the remote JVM is running, you should be able to connect to the
|
||||
remote JVM and start profiling it like you would a local JVM!
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
Both JConsole and VisualVM give you access to a lot of the instrumentation data
|
||||
available from a running Java Virtual Machine, but neither will magically
|
||||
identify or solve performance problems. There's still more work to be done to
|
||||
triage and ultimately resolve those kinds of issues, but at least these tools
|
||||
give you the information you need to know what's going on, and [knowing is half
|
||||
the
|
||||
battle](http://cdn.churchm.ag/wp-content/uploads/2014/03/knowing-is-half-the-battle.jpg).
|
||||
|
||||
|
||||
Loading…
Reference in New Issue