Friday, October 15, 2010

JMX through a ssh tunnel

My production servers run Jetty (v6) and are instrumented with JMX for runtime monitoring. They're also, of course, behind a number of firewalls. Most are only readily available via ssh. I need to monitor any of these servers with VisualVM easily through a ssh tunnel. This was considerably harder to get working than you'd hope!

Here are the high-level steps I eventually settled on:
  1. Enable Jetty's JMX instrumentation
  2. Have Jetty listen for management connections over JMXMP not RMI
  3. Start VisualVM in such a way that it can speak JMXMP.
  4. Setup a tunnel and you're off!
The rest of this article explains how to make this work, and also why I selected this approach.

(Don't Use) RMI

The default way to use JMX is over RMI. This works just fine if you're on the same network as the target server and there is no firewall. It's a mess if there is a firewall. There is a level of indirection in the RMI approach that makes management through a tunnel hard or impossible. Here's what normally happens:
  1. Client connects to a RMI Registry on the server
  2. Client looks up in the registry where to connect to for JMX using magic name jmxrmi
  3. RMI Registry replies: ok, connect to jmxhost:jmxport
  4. Client connects to jmxhost:jmxport ... if possible
The problems with this when tunneling are:
  • jmxhost has to be resolvable on both sides of the tunnel. If the servers are NAT'ed (and they will be), jmxhost will be a an unroutable private IP like 192.168.1.x
  • by default, jmxport is randomly chosen by the runtime
So with the default config it completely doesn't work through a tunnel. The RMI registry will tell you to connect to some random endpoint that isn't tunneled! You can make this work -- through a single port -- with some effort, however.
You can use -Djava.rmi.server.hostname=127.0.0.1 on the server. This makes jmxhost routable on both sides of the tunnel, but it restricts the server to accepting connections on 127.0.0.1 only (probably fine since you're tunneling anyway).
You can make jmxport deterministic using a JMXServiceURL like this: service:jmx:rmi://127.0.0.1:1099/jndi/rmi://127.0.0.1:1099/jmxrmi. (That crazy URL is not a typo!) This forces jmxhost:jmxport to be 127.0.0.1:1099, so as long as you've tunneled to that location you're good. You can use a port other than 1099, but you have to make sure there's an RMI registry listening on whatever port you specify. I've read that this single-port approach is likely to cause grief if you want to use TLS, but I haven't tried it.
I find this to be way over-complicated. A simpler approach is to use the JMXMP protocol instead of RMI.

JMXMP

JMXMP is a simple protocol: serialized Java objects over a TCP connection. No indirection like RMI. It's what you wish the default was. The catch is it's not part of the core JDK. You have to download Sun^H^H^HOracle's freely available JMX Remote Reference Implementation and put jmxremote_optional.jar in the classpath of both the client and server. This is a pain, but way less of a pain than having to understand that RMI stuff above.
To use JMX over the JMXMP protocol:
  1. Ensure jmxremote_optional.jar in the classpath of both client and server
  2. Use service:jmx:jmxmp://127.0.0.1:5555 (selecting whatver IP and port you want) as the JMXServiceUrl on the server.
  3. Have the client (i.e. VisualVM) connect to service:jmx:jmxmp://127.0.0.1:5555 (assuming 127.0.0.1:5555 is a tunnel to the same location on the server)
Pretty easy.

Jetty 6 config for JMX over JMXMP

Put jmxremote_optional.jar in $JETTY_HOME/lib, and make sure the following is in your jetty.xml:
<Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer">
  <Arg>
    <New  class="javax.management.remote.JMXServiceURL">
      <Arg>service:jmx:jmxmp://127.0.0.1:5555</Arg>
    </New>
  </Arg>
  <Arg>
    <Map>
      <Entry>
        <Item>jmx.remote.server.address.wildcard</Item>
        <Item>false</Item>                                                                                                                                              
      </Entry>
    </Map>
  </Arg>
  <Arg><Ref id="MBeanServer"/></Arg>
  <Call name="start"/>
</Call>
This will cause Jetty to listen on 127.0.0.1:5555 for JMX connections using the JMXMP protocol.

Starting VisualVM with JMXMP Support

Simply need to ensure jmxremote_optional.jar is on the classpath:
visualvm -cp:a /path/to/jmxremote_optional.jar
I use a little script to launch it (adjust paths as necessary):
#!/bin/bash
/usr/local/visualvm_131/bin/visualvm -cp:a ~/jmx/jmxremote_optional.jar "$@"

Jetty 6 config for JMX over RMI, single port

I'm not actually using this method, but I did get it working. For completeness, here is the snippet of config from jetty.xml:
<!-- Setup the RMIRegistry on a specific port -->
<Call id="rmiRegistry" class="java.rmi.registry.LocateRegistry" name="createRegistry">
  <Arg type="int">5555</Arg>
</Call> 
<!-- setup the JMXConnectorServer on a specific rmi server port -->
<Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer">
  <Arg>
    <New class="javax.management.remote.JMXServiceURL">
      <Arg>service:jmx:rmi://127.0.0.1:5555/jndi/rmi://127.0.0.1:5555/jmxrmi</Arg>
    </New>
  </Arg>
  <Arg>
    <Map>
      <Entry>
        <Item>jmx.remote.server.address.wildcard</Item>
        <Item>false</Item>                                                                                                                                              
      </Entry>
    </Map>
  </Arg>
  <Arg><Ref id="MBeanServer"/></Arg>
  <Call name="start"/>
</Call>
References:

13 comments:

  1. Nice article, but needs a fix.

    JMXMP will always listen on the any address (0.0.0.0 on IPv4) regardless of the address in the server URL unless, that "feature" is turned off.

    The following snippet is good for Jetty 7.x and




    service:jmx:jmxmp://127.0.0.1:5555





    jmx.remote.server.address.wildcard
    false

    ReplyDelete
  2. Dammit, my snippet was snipped. So, in words

    replace the second blank arg to jmxConnector with a new HashMap that has the following string entry

    name "jmx.remote.server.address.wildcard"

    value "false"

    ReplyDelete
  3. @lord.buddha Good catch! I will update the article to reflect this. Thanks for the note.

    ReplyDelete
  4. Another AMAZING solution:

    http://bowerstudios.com/node/731

    ReplyDelete
  5. @Marcelo Cool, thanks for the link. Could definitely be easier in some cases.

    ReplyDelete
  6. "It's what you wish the default was." - Yes! I don't understand why they make remote profiling so difficult.
    Anyway, do you happen to know how to easily configure JMXMP in Tomcat?

    (http://stackoverflow.com/questions/11413178/how-to-enable-jmxmp-in-tomcat)

    ReplyDelete
  7. @Bart Sorry, but I don't know much about Tomcat. I've only done this with Jetty and a couple of custom apps. Best of luck!

    ReplyDelete
  8. I've written a custom javaagent with a premain hook as outlined here: https://blogs.oracle.com/jmxetc/entry/jmx_connecting_through_firewalls_using

    Instead of using RMI I am starting a JMXMP connector as outlined in this post.
    The JMXMP agent starts up correctly and if I telnet to 127.0.0.1:5555 it appears to be listening:

    $ telnet 127.0.0.1 5555
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    ??sr5javax.management.remote.message.HandshakeBeginMessage?,???6profilestLjava/lang/String;Lversionq~xppt1.0^]

    However, VisualVM cannot seem to retrieve any information from the JVM using a JMX connection URL of "service:jmx:jmxmp://127.0.0.1:5555".
    Every tab shows "Not supported for this JVM".

    The OSX client is running:
    java version "1.6.0_35"
    Java(TM) SE Runtime Environment (build 1.6.0_35-b10-428-11M3811)
    Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01-428, mixed mode)

    The Gentoo server is running:
    java version "1.6.0_35"
    Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
    Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)

    Any idea what might be wrong?

    Thanks

    ReplyDelete
  9. Silly me, I wasn't initializing with ManagementFactory.getPlatformMBeanServer()... it works fine.

    ReplyDelete
    Replies
    1. Seems I spoke too soon... it's all working except that the Sample Memory button is disabled.

      It says "Memory Sampling Not available. Cannot connect to target application. Make sure the application is running on a supported JDK 6 or JDK 7."

      Any help/hints would be much appreciated...

      Delete
  10. When you say

    put jmxremote_optional.jar in the classpath of both the client and server.

    Anyone know how to change the classpath of VisualVM on Windows?

    ReplyDelete
  11. $ jvisualvm --help
    Usage: /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/visualvm/platform/lib/nbexec {options} arguments

    General options:
    --help show this help
    --jdkhome path to Java(TM) 2 SDK, Standard Edition
    -J pass to JVM

    --cp:p prepend to classpath
    --cp:a append to classpath
    Module reload options:
    --reload /path/to/module.jar install or reinstall a module JAR file

    Additional module options:
    --openfile open file specified by , file can be Application snapshot, NetBeans Profiler snapshot or HPROF heap dump.
    --openid open application with id
    --openpid open application with process id
    --openjmx open application specified by JMX connection (host:port)
    --modules
    --refresh Refresh all catalogs
    --list Prints the list of all modules, their versions and enablement status
    --install ... Installs provided JAR files as modules
    --disable ... Disable modules for specified codebase names
    --enable ... Enable modules for specified codebase names
    --update ... Updates all or specified modules
    --update-all Updates all modules
    --extra-uc Add a extra Update Center (URL)

    Core options:
    --laf use given LookAndFeel class instead of the default
    --fontsize set the base font size of the user interface, in points
    --locale use specified locale
    --userdir use specified directory to store user settings
    --cachedir use specified directory to store user cache, must be different from userdir
    --nosplash do not show the splash screen

    ReplyDelete