Tuesday, October 26, 2010

Scala 2.8, Maven scala:cc and FSC

I recently upgraded a large Maven-based Scala 2.7 project to Scala 2.8. After doing this I discovered that mvn scala:cc was no longer working. The error message was:

[INFO] Cannot start compilation daemon.
[INFO] tried command: List(scala, scala.tools.nsc.CompileServer)

Running mvn scala:cc -Dfsc=false worked fine, but I lost the benefits of FSC.

I managed to fix this, but I never tracked down exactly what has happening, so this is one of those "it works now, who cares" type of things.

  • I noticed java -version was not returning what I expected; an old manually installed version in /opt was apparently eclipsing the java-6-sun version installed through apt. Fixed this with: sudo update-java-alternatives -s java-6-sun
  • Noted the ... tried command: bit of the error above mentioned it was trying the scala command first. Made sure that when I typed scala I got the appropriate 2.8 version.

Started working after this. Wish I knew in more detail what was happening, but I don't have the time right now. Perhaps this will help someone else.


Monday, October 25, 2010

Wednesday, October 20, 2010

Friday, October 15, 2010

JMX through a ssh tunnel

My production servers run Jetty (v6) and are instrumented with JMX for runtime monitoring. They're also, of course, behind a number of firewalls. Most are only readily available via ssh. I need to monitor any of these servers with VisualVM easily through a ssh tunnel. This was considerably harder to get working than you'd hope!

Here are the high-level steps I eventually settled on:
  1. Enable Jetty's JMX instrumentation
  2. Have Jetty listen for management connections over JMXMP not RMI
  3. Start VisualVM in such a way that it can speak JMXMP.
  4. Setup a tunnel and you're off!
The rest of this article explains how to make this work, and also why I selected this approach.

(Don't Use) RMI

The default way to use JMX is over RMI. This works just fine if you're on the same network as the target server and there is no firewall. It's a mess if there is a firewall. There is a level of indirection in the RMI approach that makes management through a tunnel hard or impossible. Here's what normally happens:
  1. Client connects to a RMI Registry on the server
  2. Client looks up in the registry where to connect to for JMX using magic name jmxrmi
  3. RMI Registry replies: ok, connect to jmxhost:jmxport
  4. Client connects to jmxhost:jmxport ... if possible
The problems with this when tunneling are:
  • jmxhost has to be resolvable on both sides of the tunnel. If the servers are NAT'ed (and they will be), jmxhost will be a an unroutable private IP like 192.168.1.x
  • by default, jmxport is randomly chosen by the runtime
So with the default config it completely doesn't work through a tunnel. The RMI registry will tell you to connect to some random endpoint that isn't tunneled! You can make this work -- through a single port -- with some effort, however.
You can use -Djava.rmi.server.hostname= on the server. This makes jmxhost routable on both sides of the tunnel, but it restricts the server to accepting connections on only (probably fine since you're tunneling anyway).
You can make jmxport deterministic using a JMXServiceURL like this: service:jmx:rmi:// (That crazy URL is not a typo!) This forces jmxhost:jmxport to be, so as long as you've tunneled to that location you're good. You can use a port other than 1099, but you have to make sure there's an RMI registry listening on whatever port you specify. I've read that this single-port approach is likely to cause grief if you want to use TLS, but I haven't tried it.
I find this to be way over-complicated. A simpler approach is to use the JMXMP protocol instead of RMI.


JMXMP is a simple protocol: serialized Java objects over a TCP connection. No indirection like RMI. It's what you wish the default was. The catch is it's not part of the core JDK. You have to download Sun^H^H^HOracle's freely available JMX Remote Reference Implementation and put jmxremote_optional.jar in the classpath of both the client and server. This is a pain, but way less of a pain than having to understand that RMI stuff above.
To use JMX over the JMXMP protocol:
  1. Ensure jmxremote_optional.jar in the classpath of both client and server
  2. Use service:jmx:jmxmp:// (selecting whatver IP and port you want) as the JMXServiceUrl on the server.
  3. Have the client (i.e. VisualVM) connect to service:jmx:jmxmp:// (assuming is a tunnel to the same location on the server)
Pretty easy.

Jetty 6 config for JMX over JMXMP

Put jmxremote_optional.jar in $JETTY_HOME/lib, and make sure the following is in your jetty.xml:
<Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer">
    <New  class="javax.management.remote.JMXServiceURL">
  <Arg><Ref id="MBeanServer"/></Arg>
  <Call name="start"/>
This will cause Jetty to listen on for JMX connections using the JMXMP protocol.

Starting VisualVM with JMXMP Support

Simply need to ensure jmxremote_optional.jar is on the classpath:
visualvm -cp:a /path/to/jmxremote_optional.jar
I use a little script to launch it (adjust paths as necessary):
/usr/local/visualvm_131/bin/visualvm -cp:a ~/jmx/jmxremote_optional.jar "$@"

Jetty 6 config for JMX over RMI, single port

I'm not actually using this method, but I did get it working. For completeness, here is the snippet of config from jetty.xml:
<!-- Setup the RMIRegistry on a specific port -->
<Call id="rmiRegistry" class="java.rmi.registry.LocateRegistry" name="createRegistry">
  <Arg type="int">5555</Arg>
<!-- setup the JMXConnectorServer on a specific rmi server port -->
<Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer">
    <New class="javax.management.remote.JMXServiceURL">
  <Arg><Ref id="MBeanServer"/></Arg>
  <Call name="start"/>

JMX Statistics in Jetty 6 (6.1.22)

Jetty has a bunch of JMX instrumentation available, but it is normally not active by default. There is a little bit of documentation out there describing it, but not a simple explanation of how to really enable it. I eventually figured it out, so here goes.

This was all tested with Jetty 6.1.22 and Java 1.6u16.

At the top of your jetty.xml, setup the MBeanServer by adding this:

<Call id="MBeanServer" class="java.lang.management.ManagementFactory" name="getPlatformMBeanServer"/>

<Get id="Container" name="container">
  <Call name="addEventListener">
      <New class="org.mortbay.management.MBeanContainer">
        <Arg><Ref id="MBeanServer"/></Arg>
        <Call name="start" />

Then, at the bottom of your jetty.xml add this (sets a request stats handler as top-level handler, see here for more):

<Get id="oldhandler" name="handler"/>
<Set name="handler">
 <New id="StatsHandler" class="org.mortbay.jetty.handler.AtomicStatisticsHandler">
  <Set name="handler"><Ref id="oldhandler"/></Set>

Note that the order in which this stuff appears in the jetty.xml file matters. If you don't setup the MBeanServer at the beginning, subsequent components won't register themselves with JMX. This was the key that I missed out on at first, but finally realized after reading this post on the jetty-user mailing list.

You may also want to make sure you have statsOn enabled on your Connector:

<Call name="addConnector">
      <New class="org.mortbay.jetty.nio.SelectChannelConnector">
        <Set name="host"><SystemProperty name="jetty.host" /></Set>
        <Set name="port"><SystemProperty name="jetty.port" default="8080"/></Set>
        <Set name="maxIdleTime">30000</Set>
        <Set name="Acceptors">2</Set>
        <Set name="statsOn">true</Set>
        <Set name="confidentialPort">8443</Set>
        <Set name="lowResourcesConnections">5000</Set>
        <Set name="lowResourcesMaxIdleTime">5000</Set>

"Simple." Ugh.

For command line monitoring of these stats, JMXTerm seems reasonable.

Now, if you want to remotely monitor these stats with jconsole or VisualVM and you've got a firewall... that's another story, and is a complete clusterfuck (not Jetty's fault though). I will write more on that topic later. Read the comments in etc/jetty-jmx.xml that ships with Jetty to see more.

Thursday, October 14, 2010

CAP^H Theorem

Brewer's CAP Theorem is often stated as, "Consistency, Availability, and Partition tolerance: choose two." This is catchy, but in real systems you can't actually choose both C and A. If your network is down, how can a multiple node database remain consistent? I'll admit you can mess around with the definition of "available" to make this "work" (oh, it just returns '503 try again later' when the network is down), but you're kind of lying to yourself.

Coda Hale has written an excellent article going over why you can't sacrifice partition tolerance. Simply stated, logical, and not too long. Read it a few times.