Here are the high-level steps I eventually settled on:
- Enable Jetty's JMX instrumentation
- Have Jetty listen for management connections over JMXMP not RMI
- Start VisualVM in such a way that it can speak JMXMP.
- Setup a tunnel and you're off!
(Don't Use) RMI
The default way to use JMX is over RMI. This works just fine if you're on the same network as the target server and there is no firewall. It's a mess if there is a firewall. There is a level of indirection in the RMI approach that makes management through a tunnel hard or impossible. Here's what normally happens:- Client connects to a RMI Registry on the server
- Client looks up in the registry where to connect to for JMX using magic name jmxrmi
- RMI Registry replies: ok, connect to jmxhost:jmxport
- Client connects to jmxhost:jmxport ... if possible
- jmxhost has to be resolvable on both sides of the tunnel. If the servers are NAT'ed (and they will be), jmxhost will be a an unroutable private IP like 192.168.1.x
- by default, jmxport is randomly chosen by the runtime
You can use -Djava.rmi.server.hostname=127.0.0.1 on the server. This makes jmxhost routable on both sides of the tunnel, but it restricts the server to accepting connections on 127.0.0.1 only (probably fine since you're tunneling anyway).
You can make jmxport deterministic using a JMXServiceURL like this: service:jmx:rmi://127.0.0.1:1099/jndi/rmi://127.0.0.1:1099/jmxrmi. (That crazy URL is not a typo!) This forces jmxhost:jmxport to be 127.0.0.1:1099, so as long as you've tunneled to that location you're good. You can use a port other than 1099, but you have to make sure there's an RMI registry listening on whatever port you specify. I've read that this single-port approach is likely to cause grief if you want to use TLS, but I haven't tried it.
I find this to be way over-complicated. A simpler approach is to use the JMXMP protocol instead of RMI.
JMXMP
JMXMP is a simple protocol: serialized Java objects over a TCP connection. No indirection like RMI. It's what you wish the default was. The catch is it's not part of the core JDK. You have to download Sun^H^H^HOracle's freely available JMX Remote Reference Implementation and put jmxremote_optional.jar in the classpath of both the client and server. This is a pain, but way less of a pain than having to understand that RMI stuff above.To use JMX over the JMXMP protocol:
- Ensure jmxremote_optional.jar in the classpath of both client and server
- Use service:jmx:jmxmp://127.0.0.1:5555 (selecting whatver IP and port you want) as the JMXServiceUrl on the server.
- Have the client (i.e. VisualVM) connect to service:jmx:jmxmp://127.0.0.1:5555 (assuming 127.0.0.1:5555 is a tunnel to the same location on the server)
Jetty 6 config for JMX over JMXMP
Put jmxremote_optional.jar in $JETTY_HOME/lib, and make sure the following is in your jetty.xml:<Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer"> <Arg> <New class="javax.management.remote.JMXServiceURL"> <Arg>service:jmx:jmxmp://127.0.0.1:5555</Arg> </New> </Arg> <Arg> <Map> <Entry> <Item>jmx.remote.server.address.wildcard</Item> <Item>false</Item> </Entry> </Map> </Arg> <Arg><Ref id="MBeanServer"/></Arg> <Call name="start"/> </Call>This will cause Jetty to listen on 127.0.0.1:5555 for JMX connections using the JMXMP protocol.
Starting VisualVM with JMXMP Support
Simply need to ensure jmxremote_optional.jar is on the classpath:visualvm -cp:a /path/to/jmxremote_optional.jarI use a little script to launch it (adjust paths as necessary):
#!/bin/bash /usr/local/visualvm_131/bin/visualvm -cp:a ~/jmx/jmxremote_optional.jar "$@"
Jetty 6 config for JMX over RMI, single port
I'm not actually using this method, but I did get it working. For completeness, here is the snippet of config from jetty.xml:<!-- Setup the RMIRegistry on a specific port --> <Call id="rmiRegistry" class="java.rmi.registry.LocateRegistry" name="createRegistry"> <Arg type="int">5555</Arg> </Call> <!-- setup the JMXConnectorServer on a specific rmi server port --> <Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer"> <Arg> <New class="javax.management.remote.JMXServiceURL"> <Arg>service:jmx:rmi://127.0.0.1:5555/jndi/rmi://127.0.0.1:5555/jmxrmi</Arg> </New> </Arg> <Arg> <Map> <Entry> <Item>jmx.remote.server.address.wildcard</Item> <Item>false</Item> </Entry> </Map> </Arg> <Arg><Ref id="MBeanServer"/></Arg> <Call name="start"/> </Call>References:
- The best JMX tutorial I've ever read. Careful definitions, clearly stated, and calls out the bad parts of JMX so you know what to avoid.
- JMX Remote Reference Implementation (jmxremote_optional.jar)
- JMX over RMI through a firewall using a single port
- Jetty 6 JMX over RMI specifying RMI registry port
- Programmatic use of JMXServiceUrl
Nice article, but needs a fix.
ReplyDeleteJMXMP will always listen on the any address (0.0.0.0 on IPv4) regardless of the address in the server URL unless, that "feature" is turned off.
The following snippet is good for Jetty 7.x and
service:jmx:jmxmp://127.0.0.1:5555
jmx.remote.server.address.wildcard
false
Dammit, my snippet was snipped. So, in words
ReplyDeletereplace the second blank arg to jmxConnector with a new HashMap that has the following string entry
name "jmx.remote.server.address.wildcard"
value "false"
@lord.buddha Good catch! I will update the article to reflect this. Thanks for the note.
ReplyDeleteAnother AMAZING solution:
ReplyDeletehttp://bowerstudios.com/node/731
@Marcelo Cool, thanks for the link. Could definitely be easier in some cases.
ReplyDelete"It's what you wish the default was." - Yes! I don't understand why they make remote profiling so difficult.
ReplyDeleteAnyway, do you happen to know how to easily configure JMXMP in Tomcat?
(http://stackoverflow.com/questions/11413178/how-to-enable-jmxmp-in-tomcat)
@Bart Sorry, but I don't know much about Tomcat. I've only done this with Jetty and a couple of custom apps. Best of luck!
ReplyDeleteI've written a custom javaagent with a premain hook as outlined here: https://blogs.oracle.com/jmxetc/entry/jmx_connecting_through_firewalls_using
ReplyDeleteInstead of using RMI I am starting a JMXMP connector as outlined in this post.
The JMXMP agent starts up correctly and if I telnet to 127.0.0.1:5555 it appears to be listening:
$ telnet 127.0.0.1 5555
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
??sr5javax.management.remote.message.HandshakeBeginMessage?,???6profilestLjava/lang/String;Lversionq~xppt1.0^]
However, VisualVM cannot seem to retrieve any information from the JVM using a JMX connection URL of "service:jmx:jmxmp://127.0.0.1:5555".
Every tab shows "Not supported for this JVM".
The OSX client is running:
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10-428-11M3811)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01-428, mixed mode)
The Gentoo server is running:
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)
Any idea what might be wrong?
Thanks
Silly me, I wasn't initializing with ManagementFactory.getPlatformMBeanServer()... it works fine.
ReplyDeleteSeems I spoke too soon... it's all working except that the Sample Memory button is disabled.
DeleteIt says "Memory Sampling Not available. Cannot connect to target application. Make sure the application is running on a supported JDK 6 or JDK 7."
Any help/hints would be much appreciated...
Same problem
DeleteWhen you say
ReplyDeleteput jmxremote_optional.jar in the classpath of both the client and server.
Anyone know how to change the classpath of VisualVM on Windows?
$ jvisualvm --help
ReplyDeleteUsage: /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/visualvm/platform/lib/nbexec {options} arguments
General options:
--help show this help
--jdkhome path to Java(TM) 2 SDK, Standard Edition
-J pass to JVM
--cp:p prepend to classpath
--cp:a append to classpath
Module reload options:
--reload /path/to/module.jar install or reinstall a module JAR file
Additional module options:
--openfile open file specified by , file can be Application snapshot, NetBeans Profiler snapshot or HPROF heap dump.
--openid open application with id
--openpid open application with process id
--openjmx open application specified by JMX connection (host:port)
--modules
--refresh Refresh all catalogs
--list Prints the list of all modules, their versions and enablement status
--install ... Installs provided JAR files as modules
--disable ... Disable modules for specified codebase names
--enable ... Enable modules for specified codebase names
--update ... Updates all or specified modules
--update-all Updates all modules
--extra-uc Add a extra Update Center (URL)
Core options:
--laf use given LookAndFeel class instead of the default
--fontsize set the base font size of the user interface, in points
--locale use specified locale
--userdir use specified directory to store user settings
--cachedir use specified directory to store user cache, must be different from userdir
--nosplash do not show the splash screen