My production servers run
Jetty (v6) and are instrumented with JMX for runtime monitoring. They're also, of course, behind a number of firewalls. Most are only readily available via ssh. I need to monitor any of these servers with
VisualVM easily through a ssh tunnel. This was considerably harder to get working than you'd hope!
Here are the high-level steps I eventually settled on:
- Enable Jetty's JMX instrumentation
- Have Jetty listen for management connections over JMXMP not RMI
- Start VisualVM in such a way that it can speak JMXMP.
- Setup a tunnel and you're off!
The rest of this article explains how to make this work, and also why I selected this approach.
(Don't Use) RMI
The default way to use JMX is over RMI. This works just fine if you're on the same network as the target server and there is no firewall. It's a mess if there is a firewall. There is a level of indirection in the RMI approach that makes management through a tunnel hard or impossible. Here's what normally happens:
- Client connects to a RMI Registry on the server
- Client looks up in the registry where to connect to for JMX using magic name jmxrmi
- RMI Registry replies: ok, connect to jmxhost:jmxport
- Client connects to jmxhost:jmxport ... if possible
The problems with this when tunneling are:
- jmxhost has to be resolvable on both sides of the tunnel. If the servers are NAT'ed (and they will be), jmxhost will be a an unroutable private IP like 192.168.1.x
- by default, jmxport is randomly chosen by the runtime
So with the default config it completely doesn't work through a tunnel. The RMI registry will tell you to connect to some random endpoint that isn't tunneled! You
can make this work -- through a single port -- with some effort, however.
You can use
-Djava.rmi.server.hostname=127.0.0.1 on the server. This makes
jmxhost routable on both sides of the tunnel, but it restricts the server to accepting connections on 127.0.0.1 only (probably fine since you're tunneling anyway).
You can make
jmxport deterministic using a
JMXServiceURL like this:
service:jmx:rmi://127.0.0.1:1099/jndi/rmi://127.0.0.1:1099/jmxrmi. (That crazy URL is not a typo!) This forces
jmxhost:jmxport to be 127.0.0.1:1099, so as long as you've tunneled to that location you're good. You can use a port other than 1099, but you have to make sure there's an RMI registry listening on whatever port you specify. I've read that this single-port approach is likely to cause grief if you want to use
TLS, but I haven't tried it.
I find this to be way over-complicated. A simpler approach is to use the JMXMP protocol instead of RMI.
JMXMP
JMXMP is a simple protocol: serialized Java objects over a TCP connection. No indirection like RMI. It's what you wish the default was. The catch is it's not part of the core JDK. You have to download Sun^H^H^HOracle's freely available
JMX Remote Reference Implementation and put
jmxremote_optional.jar in the classpath of both the client and server. This is a pain, but way less of a pain than having to understand that RMI stuff above.
To use JMX over the JMXMP protocol:
- Ensure jmxremote_optional.jar in the classpath of both client and server
- Use service:jmx:jmxmp://127.0.0.1:5555 (selecting whatver IP and port you want) as the JMXServiceUrl on the server.
- Have the client (i.e. VisualVM) connect to service:jmx:jmxmp://127.0.0.1:5555 (assuming 127.0.0.1:5555 is a tunnel to the same location on the server)
Pretty easy.
Jetty 6 config for JMX over JMXMP
Put
jmxremote_optional.jar in
$JETTY_HOME/lib, and make sure the following is in your
jetty.xml:
<Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer">
<Arg>
<New class="javax.management.remote.JMXServiceURL">
<Arg>service:jmx:jmxmp://127.0.0.1:5555</Arg>
</New>
</Arg>
<Arg>
<Map>
<Entry>
<Item>jmx.remote.server.address.wildcard</Item>
<Item>false</Item>
</Entry>
</Map>
</Arg>
<Arg><Ref id="MBeanServer"/></Arg>
<Call name="start"/>
</Call>
This will cause Jetty to listen on 127.0.0.1:5555 for JMX connections using the JMXMP protocol.
Starting VisualVM with JMXMP Support
Simply need to ensure jmxremote_optional.jar is on the classpath:
visualvm -cp:a /path/to/jmxremote_optional.jar
I use a little script to launch it (adjust paths as necessary):
#!/bin/bash
/usr/local/visualvm_131/bin/visualvm -cp:a ~/jmx/jmxremote_optional.jar "$@"
Jetty 6 config for JMX over RMI, single port
I'm not actually using this method, but I did get it working. For completeness, here is the snippet of config from
jetty.xml:
<!-- Setup the RMIRegistry on a specific port -->
<Call id="rmiRegistry" class="java.rmi.registry.LocateRegistry" name="createRegistry">
<Arg type="int">5555</Arg>
</Call>
<!-- setup the JMXConnectorServer on a specific rmi server port -->
<Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer">
<Arg>
<New class="javax.management.remote.JMXServiceURL">
<Arg>service:jmx:rmi://127.0.0.1:5555/jndi/rmi://127.0.0.1:5555/jmxrmi</Arg>
</New>
</Arg>
<Arg>
<Map>
<Entry>
<Item>jmx.remote.server.address.wildcard</Item>
<Item>false</Item>
</Entry>
</Map>
</Arg>
<Arg><Ref id="MBeanServer"/></Arg>
<Call name="start"/>
</Call>
References: