tag:blogger.com,1999:blog-62001031715374336222024-03-05T22:25:10.777-05:00Mark FeeneyRemember: you chose to read thisUnknownnoreply@blogger.comBlogger42125tag:blogger.com,1999:blog-6200103171537433622.post-58954506396941800492013-12-05T16:01:00.001-05:002022-03-29T11:10:45.631-04:00New blog has RSS nowAmazingly, there are people reading this blog, and some have actually requested an RSS feed of the new blog, <a href="http://proofbyexample.com/">Proof By Example</a>. That's now done, via the I-can't-believe-it's-still-alive Feedburner service. <a href="http://feeds.feedburner.com/proofbyexample/">Direct link to the feed</a>.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-52028048438482117282013-12-04T11:38:00.001-05:002022-03-29T11:11:06.611-04:00New programming-related blogI've started a new blog for <a href="http://proofbyexample.com/">programming</a> and computer stuff. Just wanted to leave a pointer since this blog is pretty desolate these days.Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6200103171537433622.post-68878775819745019212013-04-30T09:43:00.001-04:002022-03-29T11:11:23.048-04:00DevHouse WaterlooAfter many years, the DevHouse Waterloo web page has moved from the old aiderss.com domain. The old page is apparently inaccessible for edits, so it'll linger. This quick post is mostly to help teach Google where the new <a href="http://artbarnlabs.com/devhouse">DevHouse Waterloo</a> page is. Hopefully the hugely powerful PageRank of my site will help.<div>
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-86270184715883423462012-03-26T10:02:00.001-04:002022-03-30T08:28:08.165-04:00Code Shapes<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDu1qaRtFdi4dZIdL9DLAn2Q0kT3BJlBmpgSI5Nh5PC-zoBr5KVYrNQHnLohZl8JEUVHXLPLsswqeUo1f9kEOtEtYiyqp0xiuS13uGMv59udfIv0P8Swe3y6nhB33AtoD2-9DY4fzcsg/s1600/Selection_021.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="320" width="286" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDu1qaRtFdi4dZIdL9DLAn2Q0kT3BJlBmpgSI5Nh5PC-zoBr5KVYrNQHnLohZl8JEUVHXLPLsswqeUo1f9kEOtEtYiyqp0xiuS13uGMv59udfIv0P8Swe3y6nhB33AtoD2-9DY4fzcsg/s320/Selection_021.png" /></a></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-91629298914310724412012-02-10T17:15:00.000-05:002012-02-10T17:24:51.681-05:00Ubuntu Lucid 10.04 at EC2: cloudimg-rootfs does not exist<p>I've been having a seemingly intermittent problem where I can't boot a custom AMI built from one of Ubuntu's default AMIs.</t>
<p>My process:</p>
<ul>
<li>Start with stock 10.04 LTS, instance-store, 64-bit, us-east-1 image: <a href="http://uec-images.ubuntu.com/releases/10.04/release/">ami-35de095c</a></li>
<li>Fire it up: works</li>
<li>Install a bunch of stuff I need: no prob</li>
<li>Create a new AMI for next time with <tt>ec2-bundle-vol</tt>: "works"</li>
<li>Upload it, register it</li>
<li>Launch a new instance with this new custom AMI: box fails to come up!</li>
</ul>
<p>The error message, available from <tt>get server log</tt> in the AWS Console, is this:</p>
<pre>
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/disk/by-label/cloudimg-rootfs does not exist. Dropping to a shell!
</pre>
<p>Some Googling turned up <a href="http://ubuntu-smoser.blogspot.com/2010/11/create-image-with-xfs-root-filesystem.html">a similar error for EBS instances using the XFS filesystem</a>, but I was using ext3 and instance-store, so it didn't really apply.</p>
<p>The issue appeared to be that <tt>/dev/disk/by-label/cloudimg-rootfs</tt> is expected, but not present. What does this really mean? Took some help from a friend for me to realize that the filesystem name is stored on the physical (or virutal...) hard drive. When a new machine is imaged, it obviously won't have "cloudimg-rootfs" as it's filesystem name automatically. Apparently the AMI I was creating was not capturing the filesystem name. Found some <a href="https://forums.aws.amazon.com/thread.jspa?threadID=58232">evidence</a> supporting this theory. I think the upshot is my version of ec2-ami-tools is missing the patch that records the filesystem name. I have <tt>1.3.49953-0ubuntu1~lucid1</tt> (<strong>update:</strong> This is an old backport which is probably the source of the issues), which by version number suggests it should have the patch, but I guess it does not (<a href="https://launchpad.net/ubuntu/+source/ec2-ami-tools">different versions listed here</a>).</p>
<p>To deal with all this I just decided to not rely on <tt>LABEL=foo</tt> for referring to drives, and instead just use the good old direct notation <tt>/dev/sda1</tt> (or whatever).</p>
<p>So the fix is: edit <tt>/etc/fstab</tt>, <tt>/boot/grub/menu.lst</tt>, and <tt>/boot/grub/grub.cfg</tt> and replace uses of <tt>LABEL=cloudimg-rootfs</tt> with <tt>/dev/sda1</tt>. This is less robust, but has the side-effect of actually working.
<p>Work-around process:</p>
<ul>
<li>Start with stock 10.04 LTS, instance-store, 64-bit, us-east-1 image: <a href="http://uec-images.ubuntu.com/releases/10.04/release/">ami-35de095c</a></li>
<li>Fire it up: works</li>
<li><strong>edit <tt>/etc/fstab</tt>, <tt>/boot/grub/menu.lst</tt>, and <tt>/boot/grub/grub.cfg</tt> as above</strong></li>
<li>Install a bunch of stuff I need: no prob</li>
<li>Create a new AMI for next time with <tt>ec2-bundle-vol</tt>: "works"</li>
<li>Upload it, register it</li>
<li>Launch a new instance with this new custom AMI: boots!</li>
</ul>
<p>I'll update this if I get to the bottom of the <tt>ec2-ami-tools</tt> version numbers.</p>
<p><strong>Update:</strong> Looks like <tt><a href="https://launchpad.net/ubuntu/+source/ec2-ami-tools/1.3-45758-0ubuntu1.1">1.3-45758-0ubuntu1.1</a></tt> is the correct version of ec2-bundle-vol to avoid all this nonsense. Haven't tested yet.</p>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6200103171537433622.post-3254213355765817162011-08-07T15:18:00.006-04:002022-03-29T11:11:42.949-04:00Image Manipulation in Ubuntu with ImageMagick Tools<p>Can't believe I haven't come across <a href="http://www.imagemagick.org/script/command-line-tools.php"><tt>convert</tt> and <tt>mogrify</tt></a> before now. Much time could have been saved in the past.</p>
<p>I used Ubuntu 10.04 Lucid for this, but I doubt it matters much as long as you have a recent Linux distribution.</p>
<pre>sudo apt-get install imagemagick</pre>
<p>Example: resize an image to fit in a 100x100 pixel box, and place it in the centre of a 100x100 pixel box:</p>
<pre>convert foo.png -resize '100x100' \
-background transparent \
-gravity center \
-extent '100x100' \
bar.png
</pre>
<p>Lots more <a href="http://www.imagemagick.org/Usage/thumbnails/">good documentation</a> here.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-20928310518827002102011-08-02T10:52:00.014-04:002011-08-02T20:59:39.097-04:00Ubuntu Remote Desktop<h2>Overview</h2>
<p>I recently had the need to get reliable remote access to the desktop of an Ubuntu 10.04 install. There are many ways to enable remote access to your X11 desktop in Ubuntu, each with different trade-offs. Here are three (all <a href="http://en.wikipedia.org/wiki/Virtual_Network_Computing">VNC</a>-based) that I recently looked into:</p>
<ul>
<li><b>Built-in</b> - System > Preferences > Remote Desktop. Easy, but only works if you're already logged in locally. i.e. after a system restart you can't connect remotely until you login locally again. Common work-around for that: use auto-login (yuck).</li>
<li><b>Xvnc</b> - Create a new X display only viewable via VNC. i.e. the remote user has its own desktop unrelated to what is seen on the monitor if a local user logs in.</li>
<li><b>x11vnc</b> - Serve up what is being shown on the local monitor (i.e. a "real" X display) over VNC. Similar to the built-in solution, but without the "must be logged in" restriction.</li>
</ul>
<p>My requirements were:</p>
<ul>
<li>Must be able to login remotely after a system restart</li>
<li>Must be able to plug in a monitor and see what the remote user sees/last saw (i.e. be able to fix things in the GUI if I break them remotely)</li>
</ul>
<h2>What I Actually Did</h2>
<p>Used <b>x11vnc</b> and started right after gdm starts. Here are all the steps:</p>
<ol>
<li><tt>sudo apt-get install x11vnc</tt></li>
<li>Create a VNC password in a stable location:<pre>sudo mkdir -p /etc/x11vnc
x11vnc -storepasswd <PASSWORD> /etc/x11vnc/passwd</pre></li>
<li>Add the following to <tt>/etc/gdm/Init/Default</tt>, right before the <tt>exit 0</tt> at the end of the script:<pre>/usr/bin/x11vnc -safer -allow 127.0.0.1,192.168.1. -o /var/log/x11vnc.log -xkb -bg -forever -rfbauth /etc/x11vnc/passwd -display :0</pre>
<li>Reboot, or just restart gdm (<tt>sudo restart gdm</tt>).</li>
<li>Should now be able to connect to port 5901 with a VNC client (I used TightVNC).</li>
</ol>
<p>If things don't work, check <tt>/var/log/x11vnc.log</tt> -- there's usually a lot of detail in there explaining exactly what is going on.</p>
<h2>Clusterfuck Alert</h2>
<p>Ubuntu (at least in 10.04) has a crazy issue where X won't start if a monitor is not plugged in when the system restarts. I don't know -- and don't want to know -- why, but there are work-arounds. I'll link to a few below, but the gist of it is to not use the fancy NVidia or ATI drivers, but instead just use the VESA driver. If you're going to have a monitor attached all the time, this is not an issue.
<h2>References</h2>
<ul>
<li>Forum thread on <a href="http://ubuntuforums.org/showthread.php?t=1452600&page=3">how to "fix" the issue around rebooting without a monitor</a></li>
<li><a href="http://www.realvnc.com/products/free/4.1/man/Xvnc.html">Xvnc man page</a></li>
<li><a href="http://www.karlrunge.com/x11vnc/">x11vnc</a>, the most configurable command line app yet written by man</li>
<li><a href="https://help.ubuntu.com/community/VNC/Servers#x11vnc-before-login">x11vnc in Ubuntu before login</a></li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-89151700605747843422011-07-30T23:47:00.004-04:002011-07-30T23:54:10.410-04:00Mount NTFS drives using /etc/fstab in Ubuntu 10.04 Lucid<p>This is mostly a note to myself, but here's the line to add to <tt>/etc/fstab</tt> to mount an existing NTFS drive read-only in Linux (Ubuntu Lucid 10.04).</p>
<pre>
/dev/sdc2 /mnt/ntfs500 ntfs nls=utf8,uid=mark,gid=mark,dmask=007,fmask=117,ro 0 0
</pre>
<p>I used the <b>GUI System -> Administration -> Disk Utility</b> to figure out that the drive I wanted was <tt>/dev/sdc2</tt>.</p>
<p>
References:
<ul><li><a href="http://www.psychocats.net/ubuntu/mountwindowsfstab">Mounting windows partitions in Ubuntu</a></li></ul>
</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-13190056016118489562011-06-27T17:30:00.004-04:002011-08-25T14:36:51.388-04:00mkdir -p "trick"<p>Had no idea you could do this:</p>
<pre class="brush: bash">
mark@pc:~/tmp/x$ mkdir -p src/{main,test}/{scala,java,resources}
mark@pc:~/tmp/x$ find .
.
./src
./src/main
./src/main/java
./src/main/resources
./src/main/scala
./src/test
./src/test/java
./src/test/resources
./src/test/scala
</pre>
<p>Awesome. Discovered <a href="http://dcsobral.blogspot.com/2011/06/very-quick-guide-to-project-creation-on.html?showComment=1309205892704#c4343987043982045713">here</a>.</p>
<b>Update: 2011-08-25</b>
<p>I was wondering why this wasn't mentioned on the <tt>mkdir</tt> man page: the reason is it's a general bash thing for generating strings called <a href="http://www.gnu.org/s/bash/manual/bash.html#Brace-Expansion">brace expansion</a>.
<pre>
$ echo foo{bar,baz,boo}
foobar foobaz fooboo
</pre>
</p>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6200103171537433622.post-28395014046228110602011-04-13T10:44:00.005-04:002011-04-13T11:00:49.239-04:00Bash history, caret, search & replace<p>I've been a (blind) user of bash's "caret replacement" for ages, but have occasionally wondered how to replace all occurences of a word, rather than the first.</p>
<pre class="brush: bash">
$ echo "foo bar baz foo"
foo bar baz foo
$ ^foo^quux
echo "quux bar baz foo"
quux bar baz foo
</pre>
<p>The answer is to use the more robust event designator syntax: <tt>!!:gs/search/replace/</tt></p>
<pre class="brush: bash">
$ echo "foo bar baz foo"
foo bar baz foo
$ !!:gs/foo/quux/
echo "quux bar baz quux"
quux bar baz quux
</pre>
<p>While on the topic of esoteric bash commands, here's another good one. You may (?) know of <tt>!$</tt> which references the last argument of the last command in history. That's just an alias for <tt>!!:$</tt>, which shows the more general form of the command. You can access any argument of any command in history.</p>
<pre class="brush: bash">
$ echo "test" > /tmp/blah.txt
$ cat !!:3
cat /tmp/blah.txt
test
</pre>
<p>There's lots more. In particular, you can operate on any command in history, not just the last one (<tt>!!</tt>). See the bash manual link in the references below.</p>
<h4>References</h4>
<ul>
<li><a href="http://www.gnu.org/software/bash/manual/bashref.html#History-Interaction">History Interaction</a> in the bash manual</li>
<li>"<a href="http://stackoverflow.com/questions/2149482/caret-search-and-replace-in-bash-shell">Caret search and replace</a>" question at StackOverflow</li>
</ul>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6200103171537433622.post-28517233148823252932011-04-07T15:04:00.005-04:002017-01-20T09:12:06.336-05:00Scala method signature rulesI came across a great post on the IDEA Scala Plugin message board the other day. Apparently there are best-practices/conventions for writing method signatures in Scala. Adhering to these rules communicates to the user more than just argument and return types: it can tell you if a method has side-effects, or is meant to be a "property".<br />
Check it out: <a href="http://devnet.jetbrains.net/thread/297990">Rules for Scala method signatures</a>.<br />
Also check out IDEA 10: the <a href="http://www.jetbrains.com/idea/">best Scala IDE</a>.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-45284292080419791802011-03-25T11:52:00.011-04:002015-06-10T11:47:00.836-04:00Scala XML GotchasScala's built-in XML support is <a href="http://www.codecommit.com/blog/scala/working-with-scalas-xml-support">perhaps flawed</a>, but still offers very convenient syntax for simple XML manipulation. Even ignoring performance concerns and concurrency issues, there are still weird gotchas that the average user may need to deal with...<br />
<h4>
CDATA magically escaped</h4>
<pre>scala> val xml = <xml><test><![CDATA[a < b]]></test></xml>
xml: scala.xml.Elem = <xml><test>a &lt; b</test></xml> <-- WTF?
</pre>
Same when loading from a String:<br />
<pre>scala> val xml = XML.loadString("<xml><test><![CDATA[a < b]]></test></xml>")
xml: scala.xml.Elem = <xml><test>a &lt; b</test></xml>
</pre>
This is not what you want. The stuff in the CDATA is meant to be left alone. Instead, it seems that the CDATA is eaten and its contents magically escaped. This causes lots of grief if the contents of the CDATA are Javascript, for example.<br />
One workaround is to use the built-in <tt><a href="http://www.scala-lang.org/api/current/scala/xml/parsing/ConstructingParser.html">ConstructingParser</a> </tt> to load XML.<br />
<pre>scala> val xml2 = ConstructingParser.fromSource(Source.fromString("<xml><test><![CDATA[a < b]]></test></xml>"), preserveWS = true).document.docElem
xml2: scala.xml.Node = <xml><test><![CDATA[a < b]]></test></xml></pre>
Looks good.<br />
You can also use <tt><xml:unparsed></tt>. Check out this <a href="http://scala.sygneca.com/faqs/xml">Scala XML faq</a> for more.<br />
<h4>
XML Comments eaten</h4>
When loading XML from a string, XML comments disappear. Example:<br />
<pre>scala> val looksGood = <xml><test><!-- comment --></test></xml>
looksGood: scala.xml.Elem = <xml><test><!-- comment --></test></xml>
scala> val wtf = XML.loadString("<xml><test><!-- comment --></test></xml>")
wtf: scala.xml.Elem = <xml><test></test></xml>
</pre>
Again, <tt>ConstructingParser</tt> can fix this:<br />
<pre>scala> val correct = ConstructingParser.fromSource(Source.fromString("<xml><test><!-- comment --></test></xml>"), preserveWS = true).document.docElem
correct: scala.xml.Node = <xml><test><!-- comment --></test></xml>
</pre>
There are some alternatives if you run into these issues.<br />
<br />
<ul>
<li>As described above, use <tt>scala.xml.parsers.ConstructingParser</tt> to load XML</li>
<li>Use the Lift <a href="http://liftweb.net/">web framework</a>'s <a href="https://github.com/lift/framework/blob/master/core/util/src/main/scala/net/liftweb/util/PCDataMarkupParser.scala">PCDataMarkupParser</a> (extends Scala's built-in MarkupParser with various improvments)</li>
<li>Daniel Spiewak's <a href="https://github.com/djspiewak/anti-xml">Anti-XML</a> project looks promising</li>
<li>Use any of the million Java XML parsers that are out there (but give up the convenient <tt>scala.xml</tt> syntax</li>
</ul>
<br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-23621557941796041122011-03-25T11:20:00.006-04:002011-03-25T13:38:13.223-04:00ssh client config: hosts<p>Slightly embarrassed that in more than a decade of daily <tt><a href="http://www.openssh.com/">ssh</a></tt> use I've never made use of ssh client config to simplify connecting to commonly used hosts. The idea is you can just <tt>ssh foo</tt> rather than <tt>ssh -p 12345 fluffy@foo.blah-blah.on.ca ...</tt>. This is especially useful if you connect to a lot of <a href="http://aws.amazon.com/ec2/">EC2</a> hosts frequently and don't want to remember their ugly names (or setup DNS). Best understood by example:</p>
Contents of <tt>~/.ssh/config</tt>:
<pre>
host ec2-webserver
hostname ec2-123-456-78-90.compute-1.amazonaws.com
user root
identityfile ~/my-ec2-key.pem
compression yes
protocol 2
host home
hostname my.place.com
port 51000
user fluffy
identityfile ~/.ssh/id_dsa
ServerAliveInterval 15
ServerAliveCountMax 4
compression yes
protocol 2
</pre>
<p>After this is setup, you can simply type <tt>ssh ec2-webserver</tt> or <tt>ssh home</tt> rather than the full ssh command. There are a million other <a href="http://manpages.ubuntu.com/manpages/man5/ssh_config.5.html">ssh client config options</a> you can set, too. As expected, all the <tt>ssh</tt> tools like <tt>scp</tt> honour these settings.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-62172261985122171592010-12-27T00:17:00.005-05:002010-12-30T12:02:16.421-05:00Maven, SBT, and "Impossible to load parent"<p>I have a multi-module Scala project with Maven as the primary build tool. Lately I've been trying to run <a href="http://code.google.com/p/simple-build-tool/">SBT</a> beside it. The model seems good: SBT examines existing pom.xml files for dependency info, but gives you all the other good SBT stuff. It should be possible for the two systems to co-exist, even in large, multi-module systems (see <a href="https://github.com/lift/lift/blob/master/framework/project/build/LiftFrameworkProject.scala">Lift</a>, or <a href="https://github.com/scalate/scalate/blob/master/project/build/Scalate.scala">Scalate</a>, for example.)<p>
<p>In my project, I ran into "java.io.IOException: Impossible to load parent ..." when running sbt update for the first time. The issue is that the parent (aggregator) pom.xml for my project was not in any repository that SBT was checking. Maven pom.xml files support the <a href="http://maven.apache.org/ref/3.0/maven-model/maven.html#class_parent">relativePath</a> tag to deal with this, but SBT uses <a href="http://ant.apache.org/ivy/">Ivy</a> to get the dependencies out of the pom files, and Ivy has a bug with finding parents via <tt>relativePath</tt>. So my conclusion is there's no "good" way to handle this right now.</p>
<p>The ugly work-around:</p>
<ul>
<li>Add <tt>val mavenLocal = "Local Maven Repository" at "file://" + (Path.userHome / ".m2" / "repository").absolutePath</tt> to the SBT project definition. This causes SBT to check the local Maven repo for dependencies.</li>
<li>Run <tt>mvn install</tt> once before using SBT. This will ensure the parent pom.xml files are put in the local repository for SBT to see.</li>
</ul>
<h3>References</h3>
<ul>
<li>Lift mailing list post summing up the <a href="http://groups.google.com/group/liftweb/msg/d8bffa7732a2838c">workaround</a></li>
<li><a href="http://scalate.fusesource.org/">Scalate</a> has the same issue and <a href="http://scalate.fusesource.org/faq.html#Why_does_the_SBT_build_not_find_any_dependent_jars_">the same workaround</a></li>
<li><a href="https://issues.apache.org/jira/browse/IVY-1173">Ivy bug related to relativePath</a></li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-58938531078952701622010-11-14T16:36:00.004-05:002011-03-25T13:40:21.376-04:00Lift: including and embedding templates<p>For some reason I am incapable of remembering that it's <tt>lift:embed</tt> and not <tt>lift:include</tt>, so I'm writing it down. Along the same lines, I can never remember how to change the evaluation order of a snippet call (is it even called a 'snippet call'?). Examples follow.</p>
<h4>Embedding a fragment of HTML in a template</h4>
<pre class="brush: xml">
<lift:embed what="path/to/fragment"/>
<!--
DO NOT add .html to the what= part
what= is relative to src/main/webapp in source tree
-->
</pre>
<h4>Embedding a fragment of HTML containing lift bindings in a template</h4>
<pre class="brush: xml">
<lift:MySnippet.foo eager_eval="true">
<lift:embed what="path/to/fragment_with_bindings"/>
</lift:MySnippet.foo>
</pre>
<h4>References</h4>
<ul>
<li>Mailing list thread about <a href="http://comments.gmane.org/gmane.comp.web.lift/30854">embedding HTML fragments in Scala Lift</a></li>
<li><a href="http://books.google.com/books?id=B10oUsyUz2AC&pg=PA38&dq=scala+lift+embed+eager&hl=en&ei=PVfgTOb0L86onQeh6aWgDw&sa=X&oi=book_result&ct=result&resnum=1&ved=0CC8Q6AEwAA#v=onepage&q&f=false">Eager evaluation in the Lift Book</a></li>
</ul>Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-6200103171537433622.post-41839300357127748902010-11-02T12:53:00.005-04:002018-02-24T16:20:47.042-05:00Java: URI vs URL<p>I was just writing some HTTP-related code using java.net.URL when I noticed that <a href="http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/methods/HttpGet.html#HttpGet(java.net.URI)">Apache httpclient 4.0's API seems to want java.net.URI instances</a>. "Why's that, I wonder?" The answer, it seems, is that Java's java.net.URL class is broken: its equals() method is blocking! It goes out on the network and does a reverse lookup of the hostname. This is very unfortunate since in every other way that class is what I want.</p>
<p>From the Javadoc for <a href="http://download.oracle.com/javase/6/docs/api/java/net/URL.html#equals(java.lang.Object)">java.net.URL#equals</a>:</p>
<pre>
Two hosts are considered equivalent <b>if both host names can be resolved
into the same IP addresses</b>; else if either host name can't be
resolved, the host names must be equal without regard to case; or both
host names equal to null.
Since hosts comparison requires name resolution, <b>this operation is a
blocking operation</b>. (Emphasis mine)
</pre>
<p>Good times. So, to avoid abitrary thread "hanging" at some point down the road I guess I'll use java.net.URI. Too bad these are all valid URIs, but nonsense in an HTTP context: "mailto:me@foo.com", "abc:123", "quux"</p>
<p>This begs the question: what precisely is the difference between a URI and URL? There is tons written on this (Google it), but I'll add my semi-informed $0.02 as well:</p>
<p><b>URI</b>: an identifier (name) for a resource. Doesn't <em>necessarily</em> say anything about how to locate the idientified resource, but sometimes does. e.g. "/foo", "http://test.com/bar", "x:y:z/a/b/c"</p>
<p><b>URL</b>: a URI that MUST include how to locate the resource. i.e. it starts with "http", "https", "ftp", etc. e.g. "http://www.google.com", "https://bank.com", http://abc.com/foo/bar/baz.html"</p>
<p>So, URI is very general, and URLs are a specialization of URIs. There is another subset of URI called URN that adds even more complexity, so I'm going mostly ignore that here. I'll just paraphrase from the SO link below and say that URNs are supposed to be a unique name (over time and space) for a resource, and they say nothing about locating said resource.</p>
<h2>References</h2>
<ul>
<li>Stack Overflow: <a href="http://stackoverflow.com/questions/1984213/difference-between-uri-and-url">difference between URI and URL</a></li>
<li><a href="http://www.symphonious.net/2007/03/29/javaneturl-or-javaneturi/">Another guy asking same question with some good discussion</a></li>
</ul>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6200103171537433622.post-45238381660591659982010-10-26T11:06:00.004-04:002010-10-26T11:19:25.604-04:00Scala 2.8, Maven scala:cc and FSC<p>I recently upgraded a large Maven-based Scala 2.7 project to Scala 2.8. After doing this I discovered that <tt>mvn scala:cc</tt> was no longer working. The error message was:</p>
<pre>[INFO] Cannot start compilation daemon.
[INFO] tried command: List(scala, scala.tools.nsc.CompileServer)
</pre>
<p>Running <tt>mvn scala:cc -Dfsc=false</tt> worked fine, but I lost the benefits of <a href="http://www.scala-lang.org/docu/files/tools/fsc.html">FSC</a>.</p>
<p>I managed to fix this, but I never tracked down exactly what has happening, so this is one of those "it works now, who cares" type of things.</p>
<ul>
<li>I noticed <tt>java -version</tt> was not returning what I expected; an old manually installed version in <tt>/opt</tt> was apparently eclipsing the <tt>java-6-sun</tt> version installed through <tt>apt</tt>. Fixed this with: <tt>sudo update-java-alternatives -s java-6-sun</tt></li>
<li>Noted the <tt>... tried command:</tt> bit of the error above mentioned it was trying the <tt>scala</tt> command first. Made sure that when I typed <tt>scala</tt> I got the appropriate 2.8 version.</tt></li>
</ul>
<p>Started working after this. Wish I knew in more detail what was happening, but I don't have the time right now. Perhaps this will help someone else.</p>
<h2>References</h2>
<ul>
<li><a href="http://www.coldcode.net/2010/04/scala-28-maven-and-continuous.html">Maven, Scala 2.8, continuous compiliation</a></li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-30642472241590099622010-10-25T12:08:00.003-04:002010-10-25T12:20:42.167-04:00Deep insights from the ACM<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMCCxsBr6Sl1n3ulOxpu0bUbsGrgoWf1LqwRXMWv0YXCy2jshceDe0_z7hIhQq2735gJ7wynWhr7VqQDIcfAIPyY3dt2rgQdQKF2maMUotXm_WSF5WebcLfrAKXxefonZHFMAC6n4K4A/s1600/Selection_020.png"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;width: 320px; height: 130px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMCCxsBr6Sl1n3ulOxpu0bUbsGrgoWf1LqwRXMWv0YXCy2jshceDe0_z7hIhQq2735gJ7wynWhr7VqQDIcfAIPyY3dt2rgQdQKF2maMUotXm_WSF5WebcLfrAKXxefonZHFMAC6n4K4A/s320/Selection_020.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5532016686754497746" /></a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-49496021477434038252010-10-20T16:15:00.000-04:002010-10-20T16:16:05.213-04:00Fortunately, not implemented<pre>mvn jetty:ruin</pre>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-39644848967921641902010-10-15T16:35:00.031-04:002016-09-01T09:29:36.779-04:00JMX through a ssh tunnelMy production servers run <a href="http://jetty.codehaus.org/jetty/">Jetty</a> (v6) and are instrumented with JMX for runtime monitoring. They're also, of course, behind a number of firewalls. Most are only readily available via ssh. I need to monitor any of these servers with <a href="https://visualvm.dev.java.net/">VisualVM</a> easily through a ssh tunnel. This was considerably harder to get working than you'd hope!<br />
<br />
Here are the high-level steps I eventually settled on:<br />
<ol>
<li><a href="http://blog.markfeeney.com/2010/10/jmx-statistics-in-jetty-6-6122.html">Enable Jetty's JMX instrumentation</a></li>
<li>Have Jetty listen for management connections over <a href="http://download.oracle.com/javase/6/docs/api/javax/management/remote/package-summary.html">JMXMP</a> <span style="font-weight: bold;">not RMI</span></li>
<li>Start VisualVM in such a way that it can speak JMXMP.</li>
<li>Setup a tunnel and you're off!</li>
</ol>
The rest of this article explains how to make this work, and also why I selected this approach.<br />
<h2>
(Don't Use) RMI</h2>
The default way to use JMX is over RMI. This works just fine if you're on the same network as the target server and there is no firewall. It's a mess if there is a firewall. There is a level of indirection in the RMI approach that makes management through a tunnel hard or impossible. Here's what normally happens:<br />
<ol>
<li>Client connects to a RMI Registry on the server</li>
<li>Client looks up in the registry where to connect to for JMX using magic name <tt>jmxrmi</tt></li>
<li>RMI Registry replies: ok, connect to <tt>jmxhost:jmxport</tt></li>
<li>Client connects to <tt>jmxhost:jmxport</tt> ... if possible</li>
</ol>
The problems with this when tunneling are:<br />
<ul>
<li><tt>jmxhost</tt> has to be resolvable on both sides of the tunnel. If the servers are NAT'ed (and they will be), <tt>jmxhost</tt> will be a an unroutable private IP like 192.168.1.x</li>
<li>by default, <tt>jmxport</tt> is randomly chosen by the runtime</li>
</ul>
So with the default config it completely doesn't work through a tunnel. The RMI registry will tell you to connect to some random endpoint that isn't tunneled! You <em>can</em> make this work -- through a single port -- with some effort, however.<br />
You can use <tt>-Djava.rmi.server.hostname=127.0.0.1</tt> on the server. This makes <tt>jmxhost</tt> routable on both sides of the tunnel, but it restricts the server to accepting connections on 127.0.0.1 only (probably fine since you're tunneling anyway).<br />
You can make <tt>jmxport</tt> deterministic using a <a href="http://download.oracle.com/javase/6/docs/api/javax/management/remote/JMXServiceURL.html">JMXServiceURL</a> like this: <tt>service:jmx:rmi://127.0.0.1:1099/jndi/rmi://127.0.0.1:1099/jmxrmi</tt>. (That crazy URL is not a typo!) This forces <tt>jmxhost:jmxport</tt> to be 127.0.0.1:1099, so as long as you've tunneled to that location you're good. You can use a port other than 1099, but you have to make sure there's an RMI registry listening on whatever port you specify. I've read that this single-port approach is likely to cause grief if you want to use <a href="http://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a>, but I haven't tried it.<br />
I find this to be way over-complicated. A simpler approach is to use the JMXMP protocol instead of RMI.<br />
<h2>
JMXMP</h2>
JMXMP is a simple protocol: serialized Java objects over a TCP connection. No indirection like RMI. It's what you wish the default was. The catch is it's not part of the core JDK. You have to download Sun^H^H^HOracle's freely available <a href="http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/download.jsp">JMX Remote Reference Implementation</a> and put <tt>jmxremote_optional.jar</tt> in the classpath of both the client and server. This is a pain, but way less of a pain than having to understand that RMI stuff above.<br />
To use JMX over the JMXMP protocol:<br />
<ol>
<li>Ensure <tt>jmxremote_optional.jar</tt> in the classpath of both client and server</li>
<li>Use <tt>service:jmx:jmxmp://127.0.0.1:5555</tt> (selecting whatver IP and port you want) as the JMXServiceUrl on the server.</li>
<li>Have the client (i.e. VisualVM) connect to <tt>service:jmx:jmxmp://127.0.0.1:5555</tt> (assuming 127.0.0.1:5555 is a tunnel to the same location on the server)</li>
</ol>
Pretty easy.<br />
<h2>
Jetty 6 config for JMX over JMXMP</h2>
Put <tt>jmxremote_optional.jar</tt> in <tt>$JETTY_HOME/lib</tt>, and make sure the following is in your <tt>jetty.xml</tt>:<br />
<pre class="brush: xml"><Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer">
<Arg>
<New class="javax.management.remote.JMXServiceURL">
<Arg>service:jmx:jmxmp://127.0.0.1:5555</Arg>
</New>
</Arg>
<Arg>
<Map>
<Entry>
<Item>jmx.remote.server.address.wildcard</Item>
<Item>false</Item>
</Entry>
</Map>
</Arg>
<Arg><Ref id="MBeanServer"/></Arg>
<Call name="start"/>
</Call>
</pre>
This will cause Jetty to listen on 127.0.0.1:5555 for JMX connections using the JMXMP protocol.<br />
<h2>
Starting VisualVM with JMXMP Support</h2>
Simply need to ensure jmxremote_optional.jar is on the classpath:<br />
<pre>visualvm -cp:a /path/to/jmxremote_optional.jar</pre>
I use a little script to launch it (adjust paths as necessary):<br />
<pre class="brush: bash">#!/bin/bash
/usr/local/visualvm_131/bin/visualvm -cp:a ~/jmx/jmxremote_optional.jar "$@"
</pre>
<h2>
Jetty 6 config for JMX over RMI, single port</h2>
I'm not actually using this method, but I did get it working. For completeness, here is the snippet of config from <tt>jetty.xml</tt>:<br />
<pre class="brush: xml"><!-- Setup the RMIRegistry on a specific port -->
<Call id="rmiRegistry" class="java.rmi.registry.LocateRegistry" name="createRegistry">
<Arg type="int">5555</Arg>
</Call>
<!-- setup the JMXConnectorServer on a specific rmi server port -->
<Call id="jmxConnector" class="javax.management.remote.JMXConnectorServerFactory" name="newJMXConnectorServer">
<Arg>
<New class="javax.management.remote.JMXServiceURL">
<Arg>service:jmx:rmi://127.0.0.1:5555/jndi/rmi://127.0.0.1:5555/jmxrmi</Arg>
</New>
</Arg>
<Arg>
<Map>
<Entry>
<Item>jmx.remote.server.address.wildcard</Item>
<Item>false</Item>
</Entry>
</Map>
</Arg>
<Arg><Ref id="MBeanServer"/></Arg>
<Call name="start"/>
</Call>
</pre>
References:<br />
<ul>
<li>The <a href="http://pub.admc.com/howtos/jmx/">best JMX tutorial</a> I've ever read. Careful definitions, clearly stated, and calls out the bad parts of JMX so you know what to avoid.</li>
<li><a href="http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/download.jsp">JMX Remote Reference Implementation</a> (<tt>jmxremote_optional.jar</tt>)</li>
<li><a href="http://blogs.sun.com/jmxetc/entry/connecting_through_firewall_using_jmx">JMX over RMI through a firewall</a> using a single port</li>
<li><a href="http://groups.google.com/group/cometd-users/msg/ac184730fce8212a?pli=1">Jetty 6 JMX over RMI specifying RMI registry port</a></li>
<li><a href="http://download.oracle.com/javase/6/docs/technotes/guides/management/agent.html#gdfvq">Programmatic use of JMXServiceUrl</a></li>
</ul>
<br />
Unknownnoreply@blogger.com13tag:blogger.com,1999:blog-6200103171537433622.post-52353865977376731142010-10-15T15:53:00.012-04:002010-11-04T16:28:33.252-04:00JMX Statistics in Jetty 6 (6.1.22)<p><a href="http://jetty.codehaus.org/jetty/">Jetty</a> has a bunch of JMX instrumentation available, but it is normally not active by default. There is a little bit of <a href="http://docs.codehaus.org/display/JETTY/JMX">documentation</a> out there describing it, but not a simple explanation of how to really enable it. I eventually figured it out, so here goes.</p>
<p>This was all tested with Jetty 6.1.22 and Java 1.6u16.</p>
<p>At the top of your <tt>jetty.xml</tt>, setup the <tt>MBeanServer</tt> by adding this:</p>
<pre class="brush: xml">
<Call id="MBeanServer" class="java.lang.management.ManagementFactory" name="getPlatformMBeanServer"/>
<Get id="Container" name="container">
<Call name="addEventListener">
<Arg>
<New class="org.mortbay.management.MBeanContainer">
<Arg><Ref id="MBeanServer"/></Arg>
<Call name="start" />
</New>
</Arg>
</Call>
</Get>
</pre>
<p>Then, at the bottom of your jetty.xml add this (sets a request stats handler as top-level handler, see <a href="http://docs.codehaus.org/display/JETTY/Statistics">here</a> for more):</p>
<pre class="brush: xml">
<Get id="oldhandler" name="handler"/>
<Set name="handler">
<New id="StatsHandler" class="org.mortbay.jetty.handler.AtomicStatisticsHandler">
<Set name="handler"><Ref id="oldhandler"/></Set>
</New>
</Set>
</pre>
<p>Note that <em>the order in which this stuff appears in the <tt>jetty.xml</tt> file matters</em>. If you don't setup the <tt>MBeanServer</tt> at the beginning, subsequent components won't register themselves with JMX. This was the key that I missed out on at first, but finally realized after reading <a href="http://markmail.org/message/5c2zvw6acqtougwj">this post</a> on the jetty-user mailing list.</p>
<p>You may also want to make sure you have <tt>statsOn</tt> enabled on your <tt>Connector</tt>:</p>
<pre class="brush: xml">
<Call name="addConnector">
<Arg>
<New class="org.mortbay.jetty.nio.SelectChannelConnector">
<Set name="host"><SystemProperty name="jetty.host" /></Set>
<Set name="port"><SystemProperty name="jetty.port" default="8080"/></Set>
<Set name="maxIdleTime">30000</Set>
<Set name="Acceptors">2</Set>
<Set name="statsOn">true</Set>
<Set name="confidentialPort">8443</Set>
<Set name="lowResourcesConnections">5000</Set>
<Set name="lowResourcesMaxIdleTime">5000</Set>
</New>
</Arg>
</Call>
</pre>
<p>"Simple." Ugh.</p>
<p>For command line monitoring of these stats, <a href="http://www.cyclopsgroup.org/projects/jmxterm/">JMXTerm</a> seems reasonable.</p>
<p>Now, if you want to remotely monitor these stats with jconsole or VisualVM and you've got a firewall... that's another story, and is a complete clusterfuck (not Jetty's fault though). I will write more on that topic later. Read the comments in <tt>etc/jetty-jmx.xml</tt> that ships with Jetty to see more.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-20364932533445583652010-10-14T22:57:00.004-04:002010-10-14T23:29:10.584-04:00CAP^H Theorem<p><a href="http://en.wikipedia.org/wiki/CAP_theorem">Brewer's CAP Theorem</a> is often stated as, "Consistency, Availability, and Partition tolerance: choose two." This is catchy, but in real systems you can't actually choose both C and A. If your network is down, how can a multiple node database remain consistent? I'll admit you can mess around with the definition of "available" to make this "work" (oh, it just returns '503 try again later' when the network is down), but you're kind of lying to yourself.</p>
<p>Coda Hale has written an excellent article going over why <a href="http://codahale.com/you-cant-sacrifice-partition-tolerance/">you can't sacrifice partition tolerance</a>. Simply stated, logical, and not too long. Read it a few times.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-82221509448066190412010-09-24T09:51:00.004-04:002010-09-24T10:06:20.668-04:00Disabling JSESSIONID in Jetty context XML<p>The following snippet shows how to deploy a web application/.war via a Jetty context XML file with JSESSIONID disabled.</p>
<pre class="brush: xml"><Configure id="cs" class="org.mortbay.jetty.webapp.WebAppContext">
<Set name="contextPath">/</Set>
<Set name="war">/foo/bar/myapp.war</Set>
<!-- Turn off JSESSIONID -->
<Get id="sh" name="sessionHandler"/>
<Ref id="sh">
<Get name="sessionManager">
<Set name="sessionURL">none</Set>
</Get>
</Ref>
</Configure>
</pre>
<p>To be clear: you'd save this as foo.xml and place it in <tt>$JETTY_HOME/contexts</tt> and your app would be deployed.</p>
<p>If you want JSESSIONID disabled when working with the <tt>maven-jetty-plugin</tt> or when embedding Jetty, check out Alex's post on <a href="http://code.alexblack.ca/disabling-jsessionid-url-sessions-in-jetty">removing JSESSIONID</a>.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-62528774542613725642010-09-18T15:42:00.002-04:002017-01-21T11:28:47.081-05:00Used Car Research: TrueDelta<p>A friend recently pointed me to a great site for <a href="http://www.truedelta.com">used car reliability information</a> called TrueDelta. These guys collect repair and fuel economy data from over 70,000 cars around North America. From this data they publish a realistic picture of how cars truly perform out in the wild. The data is updated quarterly, so you always have an up to date picture on how a car performs. Really cool idea.</p>
<p>Anyone with a car can join and contribute data to the research. If you're participating, you get free access to all the reports, so it's well worth it. It takes a minute or two to join, and a minute or two each quarter to submit some basic data (number of repairs, odometer, etc.). It's not a big time investment, and the reward could be very valuable next time you're buying a car. If you don't want to participate you can buy your way in for cheap, too.</p>
<p><a href="http://www.truedelta.com/tish.php">Sign up today</a> and make more informed decisions next time you have to buy a car.</p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6200103171537433622.post-84134386454537519882010-09-15T11:42:00.004-04:002018-02-20T10:14:46.565-05:00Using a custom WebAppClassLoader in Jetty<p>I recently ran into a situation where I wanted to log details about what Jetty's class loader was doing for one of our web apps. There is a hook in <a href="http://jetty.codehaus.org/jetty/jetty-6/apidocs/org/mortbay/jetty/webapp/WebAppContext.html#setClassLoader(java.lang.ClassLoader)"><tt>WebAppContext</tt></a> to provide your own <a href="http://jetty.codehaus.org/jetty/jetty-6/apidocs/org/mortbay/jetty/webapp/WebAppClassLoader.html"><tt>WebAppClassLoader</tt></a> implementation, which was just what I needed, so I proceeded to write <tt>LoggingWebAppClassLoader</tt> (source below). The trouble was: where do I actually get a chance to insert my custom implementation?</p>
<p>It's easy enough to do this if you're embedding Jetty in your app:</p>
<pre>// Scala code
val context = new WebAppContext()
val lwacl = new LoggingWebAppClassLoader(context)
context.setClassLoader(lwacl)
...
</pre>
<p>Unfortunately we're not embedding, but just deploying a <tt>.war</tt> to an existing Jetty server. I messed around with this at length, and eventually <a href="http://stackoverflow.com/questions/3712057/how-does-one-set-a-custom-webappclassloader-in-jetty-through-config">asked on StackOverflow</a>. The answer there got me on the right path, though I had a few other issues along the way.</p>
<p>Here's how I got it working:</p>
<ul>
<li>Changed my deployment technique to Jetty's context deployer (<tt>$JETTY_HOME/contexts</tt>) instead of just copying <tt>.war</tt> files into <tt>$JETTY_HOME/webapps</tt>.</li>
<li>Wrote <tt>myapp.xml</tt> (see below) to define the context. It's in here that you can configure Jetty to use the custom <tt>WebAppClassLoader</tt>.</li>
<li>Copied the jar file containing my <tt>LoggingWebAppClassLoader</tt> class into <tt>$JETTY_HOME/lib</tt> so the class is available to the context deployer.</li>
</ul>
<p>Of course it seems pretty straight-forward now that I've figured it out :) The biggest issue was that I didn't know much about deploying via Jetty contexts. They seem to have a lot of advantages over the vanilla war deployer in that you can easily tweak any Jetty internals at deploy time. Downside is "programming" in XML...</p>
<tt>myapp.xml</tt> (Jetty context):
<pre class="brush: xml">
<Configure id="mycontext" class="org.mortbay.jetty.webapp.WebAppContext">
<Set name="contextPath">/</Set>
<Set name="war">/foo/myapp.war</Set>
<Set name="classLoader">
<New class="fully.qualified.name.LoggingWebAppClassLoader">
<Arg><Ref id="mycontext"/></Arg>
</New>
</Set>
</Configure>
</pre>
<tt>LoggingWebAppClassLoader.java</tt>:
<pre class="brush: java">import java.io.IOException;
import org.mortbay.jetty.webapp.WebAppContext;
import org.mortbay.jetty.webapp.WebAppClassLoader;
public class LoggingWebAppClassLoader extends WebAppClassLoader {
public LoggingWebAppClassLoader(ClassLoader parent, WebAppContext context) throws IOException {
super(parent, context);
}
public LoggingWebAppClassLoader(WebAppContext context) throws IOException {
super(context);
}
private void log(String s) {
System.out.println(s);
}
@Override
public void addClassPath(String classPath) throws IOException {
log(String.format("addClassPath: %s", classPath));
super.addClassPath(classPath);
}
@Override
public Class loadClass(String name) throws ClassNotFoundException {
log(String.format("loadClass: %s", name));
return super.loadClass(name);
}
}
</pre>Unknownnoreply@blogger.com0