Archive for the ‘technical’ Category

… and I nailed the javadoc

Thursday, January 7th, 2010

Yesterday was a red letter day. I found multiple bugs in the javac and javadoc. (1.6.0_17)

This javadoc issue occurred when there is a class that has no package line ( it is in the default package). Annoyingly enough it occurred when I was trying to create a test case for this javac bug. I was running maven 2.2.1 at the time.

java.lang.NullPointerException
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generatePackageUse(PackageUseWriter.java:180)
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generatePackageList(PackageUseWriter.java:124)
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generatePackageUse(PackageUseWriter.java:110)
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generatePackageUseFile(PackageUseWriter.java:99)
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generate(PackageUseWriter.java:78)
       at com.sun.tools.doclets.formats.html.ClassUseWriter.generate(ClassUseWriter.java:116)
       at com.sun.tools.doclets.formats.html.HtmlDoclet.generateOtherFiles(HtmlDoclet.java:92)
       at com.sun.tools.doclets.internal.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:122)
       at com.sun.tools.doclets.internal.toolkit.AbstractDoclet.start(AbstractDoclet.java:64)
       at com.sun.tools.doclets.formats.html.HtmlDoclet.start(HtmlDoclet.java:42)
       at com.sun.tools.doclets.standard.Standard.start(Standard.java:23)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at com.sun.tools.javadoc.DocletInvoker.invoke(DocletInvoker.java:269)
       at com.sun.tools.javadoc.DocletInvoker.start(DocletInvoker.java:143)
       at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:340)
       at com.sun.tools.javadoc.Start.begin(Start.java:128)
       at com.sun.tools.javadoc.Main.execute(Main.java:41)
       at com.sun.tools.javadoc.Main.main(Main.java:31)

Solution was to put the java code into a package.

Woot! I crashed the javac

Thursday, January 7th, 2010

My javac crash:

An exception has occurred in the compiler (1.6.0_17). Please file a bug at the Java Developer Connection (http://java.sun.com/webapps/bugreport) after checking the Bug Parade for duplicates. Include your program and the following diagnostic in your report. Thank you.

java.lang.AssertionError: isSubtype 15 ( 15 = TypeTags.WILDCARD )
	at com.sun.tools.javac.code.Types$5.visitType(Types.java:347)
	at com.sun.tools.javac.code.Types$5.visitType(Types.java:328)
	at com.sun.tools.javac.code.Types$DefaultTypeVisitor.visitWildcardType(Types.java:3163)
	at com.sun.tools.javac.code.Type$WildcardType.accept(Type.java:416)
	at com.sun.tools.javac.code.Types$DefaultTypeVisitor.visit(Types.java:3161)
	at com.sun.tools.javac.code.Types.isSubtype(Types.java:324)
	at com.sun.tools.javac.code.Types.isSubtype(Types.java:308)
	at com.sun.tools.javac.code.Types.isSubtypeUnchecked(Types.java:288)
	at com.sun.tools.javac.code.Types.isSubtypeUnchecked(Types.java:460)
	at com.sun.tools.javac.comp.Infer.checkWithinBounds(Infer.java:388)
	at com.sun.tools.javac.comp.Infer.instantiateExpr(Infer.java:241)
	at com.sun.tools.javac.comp.Check.instantiatePoly(Check.java:356)
	at com.sun.tools.javac.comp.Check.checkType(Check.java:324)
	at com.sun.tools.javac.comp.Attr.check(Attr.java:160)
	at com.sun.tools.javac.comp.Attr.visitApply(Attr.java:1276)
	at com.sun.tools.javac.tree.JCTree$JCMethodInvocation.accept(JCTree.java:1210)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribExpr(Attr.java:377)
	at com.sun.tools.javac.comp.Attr.visitAssign(Attr.java:1550)
	at com.sun.tools.javac.tree.JCTree$JCAssign.accept(JCTree.java:1342)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribExpr(Attr.java:384)
	at com.sun.tools.javac.comp.Attr.visitExec(Attr.java:1017)
	at com.sun.tools.javac.tree.JCTree$JCExpressionStatement.accept(JCTree.java:1074)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitIf(Attr.java:1009)
	at com.sun.tools.javac.tree.JCTree$JCIf.accept(JCTree.java:1050)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitForLoop(Attr.java:740)
	at com.sun.tools.javac.tree.JCTree$JCForLoop.accept(JCTree.java:818)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitIf(Attr.java:1009)
	at com.sun.tools.javac.tree.JCTree$JCIf.accept(JCTree.java:1050)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitIf(Attr.java:1009)
	at com.sun.tools.javac.tree.JCTree$JCIf.accept(JCTree.java:1050)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitMethodDef(Attr.java:634)
	at com.sun.tools.javac.tree.JCTree$JCMethodDecl.accept(JCTree.java:639)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribClassBody(Attr.java:2697)
	at com.sun.tools.javac.comp.Attr.attribClass(Attr.java:2628)
	at com.sun.tools.javac.comp.Attr.attribClass(Attr.java:2564)
	at com.sun.tools.javac.main.JavaCompiler.attribute(JavaCompiler.java:1036)
	at com.sun.tools.javac.main.JavaCompiler.compile2(JavaCompiler.java:765)
	at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:730)
	at com.sun.tools.javac.main.Main.compile(Main.java:353)
	at com.sun.tools.javac.main.Main.compile(Main.java:279)
	at com.sun.tools.javac.main.Main.compile(Main.java:270)
	at com.sun.tools.javac.Main.compile(Main.java:69)
	at com.sun.tools.javac.Main.main(Main.java:54)

I was not successful at reducing this crash to a simple test case.

However, the line that triggered the crash was:

ProxyMapperImplementor<?,?> childProxy = this;

Some points:

  • Once again (and again) the eclipse compiler is more robust than the sun javac
  • ProxyMapperImplementor is defined as:

    interface ProxyMapperImplementor<I, O extends I>

  • Removing the wildcards so the line looks like solved the problem:

    ProxyMapperImplementor childProxy = this;

    Update (email from sun):

    This issue is related to Bug ID: 6738538. You can review this bug at:

    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6738538

    This problem has been fixed in Java SE 7.0. Consider downloading a free copy at http://download.java.net/jdk7/binaries/ and checking if the problem persists. If the problem persists do let us know and we shall investigate this furthur.

    We greatly appreciate your efforts in identifying areas in the Java Standard Edition where we can improve upon and I would request you to continue doing so.

Generic annoyances in java

Wednesday, January 6th, 2010

For some reason, the type variable <O> cannot flow through from an external caller through a callee to another called method.

This compiles just fine in eclipse, but once again sun’s javac is temperamental.

The three solutions seem to be:

  1. pass an object of type <O> to the subcallee method (m0(), p0() )
  2. use wildcard instead of <O> – which is o.k. if O is not needed in the body of the method
  3. assign to a variable with no generic information (universal backup solution to java generic wierdness)

I filed a bug with Sun.

(Technical details: javac 1.6.0_17 )

public class Foo {
    public > I m(I i) {
/*line3*/        R r = (R) p(i);
        return i;
    }
    public > R p(I i) {
        return null;
    }

    public > I m0(O o) {
        K r= (K) p0(o);
        return o;
    }
    public > K p0(O o) {
        return null;
    }

    public > I m1(I i) {
        K r= (K) p1(i);
        return i;
    }
    public > K p1(I i) {
        return null;
    }

    public > I m2(I i) {
/*line 27*/        K r= (K) p2(i);
        return i;
    }
    public > K p2(I i) {
        return null;
    }
    interface FooInterface {

    }
}

Foo.java:3: incompatible types; inferred type argument(s) I,java.lang.Object do not conform to bounds of type variable(s) O,R
found : R
required: java.lang.Object
/*line3*/ R r = (R) p(i);
^
Foo.java:27: incompatible types; inferred type argument(s) I,java.lang.Object do not conform to bounds of type variable(s) O,K
found : K
required: java.lang.Object
K r= (K) p2(i);
^

Gmail and the Electrical Grid: Looks the same

Wednesday, September 2nd, 2009

GMail had a large-scale cascading failure yesterday:

At about 12:30 pm Pacific a few of the request routers became overloaded and in effect told the rest of the system “stop sending us traffic, we’re too slow!”. This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

This cascading failure sounds exactly like that other universal network we all share: The electrical grid.

Large blackouts are cascading failures compounded by the failure of the “fuses” meant to isolate still functioning parts of the grid from the failed part.

Sounds a lot like what happened to gmail.

Interestingly the smart people at Google have recognized exactly that:

we have concluded that request routers don’t have sufficient failure isolation (i.e. if there’s a problem in one datacenter, it shouldn’t affect servers in another datacenter) and do not degrade gracefully (e.g. if many request routers are overloaded simultaneously, they all should just get slower instead of refusing to accept traffic and shifting their load).

Sounds like a good start. But what happens if a datacenter catestrofically fails. The routers stop accepting requests because they are gone ( or can’t respond ). Prepare for the next GMail failure.

As wikipedia notes:

Modern power systems are designed to be resistant to this sort of cascading failure, but it may be unavoidable (see below). Moreover, since there is no short-term economic benefit to preventing rare large-scale failures, some observers have expressed concern that there is a tendency to erode the resilience of the network over time, which is only corrected after a major failure occurs. It has been claimed that reducing the likelihood of small outages only increases the likelihood of larger ones. In that case, the short-term economic benefit of keeping the individual customer happy increases the likelihood of large-scale blackouts.

Mac OSX — the most insecure OS around

Friday, August 7th, 2009

A year ago, I gave up on Windows. I was tired of the nervous eggshell feeling with anti-virus software, security patches, and a machine that would mysteriously be slower and slower no matter what I did.

Because of MacOSX Unix roots, I made the switch to MacOSX confident that my personal computer would be safe.

This faith has been severely shaken. I now regard MacOSX as one of the most INSECURE operation systems.

MacOSX has a dangerous default DNS/DHCP configuration. Even worse, this dangerous configuration can not be fixed from the UI. Even the command line fix is difficult. And worst of all Apple is aware of this and does nothing.

There are 5 bits of background you should know:

  1. DNS is a fundamental part of the internet. DNS is the ‘name resolution’ service that converts ‘mail.google.com’ into the ipaddress: ‘74.125.19.19′ which is what your computer really uses to contact GMail servers so you can read your email. This conversion from the human readable ‘mail.google.com’ to ‘74.125.19.19′ is analogous to the post office converting the postal address on your snail mail envelope to a Zip+4 encoding that is printed at the bottom of the envelope. This encoding is what is actually read by the postal service mail sorting machines to determine where your snail mail goes. Now imagine that the postal service’s encoding machine was compromised. This compromised postal encoding machine was changed so that no matter what the Zip+4 code was supposed to be the machine always encoded the location of Dick Cheney’s house. As a result, all your mail that was processed by that compromised postal service would go first to Dick Cheney. Dick Cheney would get a chance to open all your snail mail, read it, copy it, etc. He could then reseal the original envelope, reencode the envelope with the correct barcode and put the envelope back in the postal service system to be delivered to the correct address. so that no matter what address you had actually printed on your envelope. All this would happen without you being aware of the problem nor able to stop it from happening.

    Everything internet related depends on correct translation of ‘apple.com’ or ‘bankofamerica.com’ to the correct IP address, not some third party server. How does the your machine know that it is communicating with apple.com and not some evil server? Your computer relies on the DNS lookup being correct. If the DNS lookup is compromised then when your software update runs to check for the latest security patches it is really installing a virus from evilserver.com not apple.com.

    There has been recent concern about DNS spoofing. ( links ). Being the cautious person that I am, I decided to explicitly listed opendns.org’s DNS servers (208.67.222.222 and 208.67.220.220) as the DNS Servers to trust in my Network configuration. I felt pretty cocky and safe.

  2. The second bit of the puzzle is DHCP. In order to talk to the world, computers need to have their own personal unique ipaddress (it’s very own ZIP+4 code). Every time you go into an internet cafe and pop open your laptop, your laptop uses the DHCP service to figure out what unique ipaddress (192.168.1.101) it should use while you are in that cafe. DHCP is nice because otherwise you would have to manually figure out and set an ipaddress for your computer that is different than everyone else’s laptop. And if someone else picks the same address as you did, all of a sudden your internet connection starts behaving odd. In addition to supplying a ipaddress, the DHCP server also supplies a DNS server that should be used. This is useful when you have your laptop at work and you need to go to an internal website such as http://go/wave Notice there is no ‘.org’, or ‘.com’ after ‘http://go/’ this means that ‘go’ is only visible when you are at work and can access the internal DNS server using the information that the corporate DHCP server supplied to your laptop.

    So to summarize DHCP supplies your laptop with the information needed for the world to talk to your laptop ( by assigning an ipaddress to your computer) , and helps you find out about the world (by telling your computer about the corporate DNS server). DNS servers enables your laptop talk to the world by giving your laptop a ‘go-to-machine’ for all its addressing questions.

  3. Third, DNS servers are usually big expensive computers secured by “smart people”. However, the DHCP server is really just a bit of software running on a Linksys router at your internet cafe. Your internet cafe’s Linksys router probably has the default password and no one ever checks on it. There are millions of this routers, with minimal security, no one checking on them and your laptop is trusting these unsecured routers with the keys to your kingdom. Your laptop is asking this router — “tell me which DNS server to trust?”
  4. Fourth, Lets say that you are paranoid enough to say ” ohh this is bad. I am not going to trust such a router in a greasy, dark corner with telling my precious laptop which DNS server to trust.” So if you are like me you configure your laptop with an explicit list of DNS servers thinking that your laptop, especially your oh-so-secure Mac would never disobey you about something so critical as DNS.
  5. Fifth, you would be wrong. The insecure MacOSX does disobey and it does trust that greasy spoon router over you. MacOsX doesn’t let the user (YOU!) say that only certain machines are allowed to be your laptop’s DNS servers! Furthermore even if you have supplied your own custom DNS servers that you trust, the insecure MacOSX trusts the greasy spoon DHCP server’s DNS servers over your trusted DNS servers. And there is NO way to convince MacOSX otherwise.

This means that if the DHCP server at your internet cafe has been compromised you are as well.

How I found out

So here I am feeling all cocky and safe. I type in my company’s web address, ‘amplafi.net’ and amplafi.net resolved to 113.29.236.168 which offered that the website was for sale!… I freaked out!

hacked-dns

I discovered my MacOSX laptop was insisting on trusting these EVIL DNS servers 206.13.28.12 206.13.31.12 . Was the 10.5.8 OSX patch that was installed 6 hours ago really what it seemed? Who knows? When I installed updates to Firefox plugins was I really installing the correct versions or a compromised version that would report back to some site in Russia all my bank account information? I have no way of knowing.

This is the really scary part about everything. John Simpson reports:

Under 10.4 and earlier, when I specified a custom nameserver, the system would use only the nameserver(s) I specified. However, under 10.5 Apple has apparently changed that behavior, and uses my specified nameservers in addition to the DNS servers specified by the DHCP server. It shows the DHCP-provided server IP on the list, greyed out, so you can’t delete it.

For a while, I adopted a “grin and bear it” attitude — after all, the DHCP server at home is handing out the IP of my internal Linux server (also running djbdns) as the DNS server, so I was only unsafe when I used the laptop outside the house. However, with the recently announced vulnerability in the DNS protocol, the massive world-wide patch effort by major DNS vendors, and the fact that many networks haven’t applied the patches yet, I don’t really feel safe relying on anybody else’s nameservers.

I tried calling Apple about this, but it turns out that my AppleCare contract doesn’t cover technical support such as this.

My next approach was to just brute-force search the system for anything relating to DHCP. It took a while, but I was able to find the file which needed to be changed, and figure out the necessary changes. Basically, I found a file which controls which options are used by the DHCP client when handling a response from a DHCP server. I removed the DNS-related options from this list, and after rebooting the system, the laptop now ignores the DNS server options being sent by the DHCP server.

The file I found is named IPConfiguration.xml, and it’s buried in this folder: /System/Library/SystemConfiguration/IPConfiguration.bundle/Contents/Resources. You need to create a copy of that file, edit the copy, and remove a few entries in the DHCPRequestedParameterList key. (The entries to remove are those for 6, 15, and 119.) I have added full details on this process to my djbdns setup page, in the section titled Disabling DNS servers from DHCP.

Thankfully I found this web page from 2008(!) that showed how to fix this problem (thanks John M. Simpson):

It is possible to make the DHCP client ignore the “DNS server” options in the DHCP response. It’s not for the faint of heart, but if you’ve been able to handle the rest of the instructions on this page, you can handle this bit as well.

I have done this on my own laptop (a MacBook Pro) and it does work.

Be aware that this is a GLOBAL change. If you do this, your machine will not use the DNS servers specified by any DHCP server. This may affect your machine’s ability to easily work with corporate networks (especially those using Windows Active Directory) or other networks which use private DNS namespaces.

$ sudo -s
Password: You will not see your password as you enter it.
# cd /System/Library/SystemConfiguration/IPConfiguration.bundle/Contents/Resources
# vi IPConfiguration.xml

Find this block...

        <key>DHCPRequestedParameterList</key>
        <array>
                <integer>1</integer>
                <integer>3</integer>
                <integer>6</integer>
                <integer>15</integer>
                <integer>119</integer>
                <integer>95</integer>
                <integer>252</integer>
                <integer>44</integer>
                <integer>46</integer>
                <integer>47</integer>
        </array>

Comment out the 6, 15, and 119 entries. The result should look like this:

        <key>DHCPRequestedParameterList</key>
        <array>
                <integer>1</integer>
                <integer>3</integer>
                <!-- commented out so that Bad DNS servers coming from DHCP servers
                are not used.
                <integer>6</integer>
                <integer>15</integer>
                <integer>119</integer>
                -->
                <integer>95</integer>
                <integer>252</integer>
                <integer>44</integer>
                <integer>46</integer>
                <integer>47</integer>
        </array>

Save your changes.

Be sure to flush the DNS cache.

On MacOsX:

dscacheutil -flushcache

I have tried John’s suggestion and those scary DNS servers are no longer present. But has my machine been compromised already? I will be visiting the apple store in a few hours asking for answers.

The story continues. I was sitting next to a customer. On her windows box she was picking up the same bad DNS servers. It wasn’t until later when I got home that I discovered all this information. I suspect (but am not completely certain) that windows will not override an explicitly specified DNS server.

Update: So after talking with some people, its pretty clear that MacOsX shares this issue with Windows XP because offering out internal DNS servers is part of what DHCP does. However with Windows XP, it is easy to explicitly lock down the DNS servers.

How to lock down a Windows XP box with safe DNS servers:

On Windows:

network-connections-step1

before-changing-dns-step2

use-opendns-dns-servers-step3

To clear Windows DNS cache:

ipconfig /flushdns

Code Review #8: When to comment

Monday, March 16th, 2009

The last ? in a series of posts about commenting. See “the why”. See “not commenting is career threatening”. And the comment that started this off!

This post should have really been the second one I wrote. The first post was about who a developer needs to keep happy – the client or the manager.

But commenting everything is both hard, excessive, and wasteful of time. In “the why” post, I cover what should be in the comments – implicitly I was limiting comments to just the places where knowing “the why” is important.

However, still the question is “when” should a developer comment?

Here is my rule of thumb:

  • Any developer had to spend time figuring out what the code did. Refactor it and if still not clear comment the code with questions and assumptions. (Just in case the refactoring introduced a bug or caused an existing bug to appear.)
  • Another developer asks the implementing developer (or the current “owner”/responsible developer) about the code, interfaces, package. The developer asking the question copy the explanation into the code, cleaning up the chat log/email response and adding further edits and explanation as needed.
  • The code is not going to have further development for a while. Comment the code enough so that when development is resumed, the restart is faster.
  • The code is being handed off from one developer to another.
  • Key interfaces that are used by multiple groups.

And managers give developers time to wrap up and add these comments.

Conversely, what is not worth commenting:

  • Code that is actively, daily, being changed rearchitected. Every developer already knows what the comments would say and the structure is changing fast enough that the comments would become outdated.
  • Code that does not impact data integrity and end-user experience in a horrid way.
  • Utility classes/ methods.

Why and when to wrap external library classes

Monday, March 16th, 2009

At some point, every developer starts using an external library. They then have to decide if that external library should be wrapped in their own custom interfaces and classes.

If the external library would be pervasively imported in throughout the code then the answer is empathically YES — the external library should be wrapped.

At Amplafi, hibernate is just such a library. Without wrapping every time the hibernate library changed its structure or behavior, the change would have a large impact. However, by wrapping the libraries hibernate became loosely coupled.

This also enabled us to solve other more major issues with Hibernate Query and Session.

Specifically:

  • control when a transaction could be committed. We wanted to count how many times a tx was “started” and only commit when the tx was “ended” the same number of times it was started. Useful for code that doesn’t know if it needs to start a transaction. Now any code that needs a tx just “starts” one and ends it when done.
  • Performance metrics gathering.
  • Delaying starting the transaction until it is known that something will actually be done.
    More gentle behavior for query.uniqueResult() ( default Hibernate behavior is to throw an exception if there is more than one row returned. Amplafi’s wrapper logs the non-uniqueness + ids of the extra rows. )

How we did this:

  1. Create an interface (AmplafiQuery) that extends Query
  2. Create a class (AmplafiQueryImpl) that extends AmplafiQuery and wraps a org.hibernate.Query
  3. Create a Txmanager that returns a Tx.
  4. Tx has the various createQuery methods and returns AmplafiQuery

We also took the opportunity to add to AmplafiQuery:

  • a “asList()” that is a generic enabled version of Query.list()
  • a “unique()” that is a generic enabled version of Query.uniqueResult()

Interfaces vs. abstract classes

Saturday, March 14th, 2009

Sigh … some people just don’t get it…. :-P

Interfaces rock! Below is my comment from stackoverflow.com, a question about how to handle the “interfaces v. abstract classes” interview question in an interview.

  1. First, the “only one super class” answer is lame. Anyone who gave me that answer in an interview would be quickly countered with “C++ existed before Java and C++ had multiple super classes. Why do you think James Gosling only allowed one superclass for Java?”

    Understand the philosophy behind your answer otherwise you are toast (at least if I interview you.)

  2. Second, interfaces have multiple advantages over abstract classes, especially when designing interfaces. The biggest one is not having a particular class structure imposed on the caller of a method. There is nothing worse than trying to use a method call that demands a particular class structure. It is painful and awkward. Using an interface *anything* can be passed to the method with a minimum of expectations.

    Example:

    public void foo(Hashtable bar);

    vs.

    public void foo(Map bar);

    For the former, the caller will always be taking their existing data structure and slamming it into a new Hashtable.

  3. Third, interfaces allow public methods in the concrete class implementers to be “private”. If the method is not declared in the interface then the method cannot be used (or misused) by classes that have no business using the method. Which brings me to point 4….
  4. Fourth, Interfaces represent a minimal contract between the implementing class and the caller. This minimal contract specifies exactly *how* the concrete implementer expects to be used and no more. The calling class is not allowed to use any other method not specified by the “contract” of the interface. The interface name in use also flavors the developer’s expectation of how they should be using the object. If a developer is passed a

    public interface FragmentVisitor {
    public void visit(Node node);
    }

    The developer knows that the only method they can call is the visit method. They don’t get distracted by the bright shiny methods in the concrete class that they shouldn’t mess with.

  5. Lastly, abstract classes have many methods that are really only present for the subclasses to be using. So abstract classes tend to look a little like a mess to the outside developer, there is no guidance on which methods are intended to be used by outside code.

    Yes of course some such methods can be made protected. However, sadly protected methods are also visible to other classes in the same package. And if an abstract class’ method implements an interface the method must be public.

    However using interfaces all this innards that are hanging out when looking at the abstract super class or the concrete class are safely tucked away.

Yes I know that of course the developer may use some “special” knowledge to cast an object to another broader interface or the concrete class itself. But such a cast violates the expected contract, and the developer should be slapped with a salmon.

Code Review #7 – Comment the “why” not the “what”

Wednesday, March 4th, 2009

[This post continues the response to Mike.]

Clean “good” code is good but not enough. Code needs comments — but the right kind of comments.

“What” comments are useless and the most quickly out-dated. An example of a what comment is: “add the suffix path to the end”.

However, the post “Thoughts on how to evaluate code” was very specific referring to “why” comments.

Code no matter how “clean” can not self-document. Excellent comments are “why” comments. “Why” comments talk about :

  1. the assumptions / program state the code is expecting. (For example, the user is logged in, premium membership, or admin privileges )
  2. other code affected if this code is changed (For example, setting up a language default for the session, a default that another piece of code is expecting to be set.)
  3. if something is being done in a non-standard way – why it is being done that way (for example, iterating through a loop from high index to low-index rather than the standard low to high)
  4. should the code in question be used as an example pattern of how to do similar operations elsewhere in the code – or should a developer instead use a different pattern elsewhere.
  5. list the assumptions or conditions that require this implementation (with timestamp) so that it is easy to spot code that is present to handle conditions that no longer exist. For example, a comment like this: “12/20/2007 – yahoo’s open id implementation is using the openid 0.8-proposedspec.” Its pretty likely that the the spec has changed as has yahoo’s implementation. This wasn’t a HACK but knowing that reason behind the code is invaluable.
  6. Future work planned.
  7. Alternative solutions that were rejected (and why).
  8. Performance considerations – this code might be special because it executes millions of times a second. So there is good reasons for some “ugliness”.

The developer, at the moment they are writing the code, must be able to explain why they are writing the code a certain way. It is a red flag to any developer if they cannot explain the why. They should double-check to make sure they understand the requirements. Chances are they don’t and they are doing the wrong thing.

I started requiring developers to clearly document the “why” in their code. I discovered developers that were doing things “wrong” because they were unaware of the correct solution. I discovered that I was not as clear in conveying the requirements as I really thought. Without the developer’s “why” comment, it is easy to get frustrated with developers for the wrong reasons.

Parting thought

Why should the developers that come after the original developer have to spend any time ( and money! ) figuring out the previous work?

You may be the next developer.

Not commenting code is dangerous to your career

Wednesday, March 4th, 2009

There is this myth that code can be self-documenting and that comments are not necessary in good code. Michael recently comment on an earlier blog post advocating the idea of self-documenting code.

“Self-documenting” code is a career-damaging concept, because:

  • Your “manager” (person who decides you stay employed) is an “idiot”,
  • Your co-workers are “idiots”,
  • You are an “idiot”
  • The product manager is an “idiot”
  • Your client is an “idiot”

Your “manager” (person who decides you stay employed) is an “idiot”

That’s me. The guy who has 6+ people to manage has to figure out business plans and a much of non-technical stuff that has no doubt resulted in permanent brain damage.

Your manager is code reviewing all the changes by his direct reports. A task that takes a few minutes every day… unless there are no comments. I know that your manager should be able to immediately see what the code does. But remember your manager is an “idiot”. Unfortunately for you, he signs your paycheck.

Also unfortunately for you, if your manager doesn’t immediately understand your change — you are an “idiot” in his eyes.

You have also just turned a 15-minute task into a 2-hour task. He didn’t really want to be home with his family.

Your co-workers are idiots

Lets agree, your code was stellar but then your co-workers came in and mucked it up. They did a stupid refactoring and broke everything.

The code was “obvious” but not to them.

You are an idiot

Your manager returns from a two-week vacation and is code reviewing 2 weeks of changes. Your manager is asking about a “odd” change that you did 14 days ago.

  • Can you explain it clearly?
  • Do you remember your thinking at the time?
  • Alternative solutions that you tried and failed?

The product manager is an “idiot”

  1. Your stellar (comment-free) code is deployed into production.
  2. The product manager changes the product definition and a new feature is born.
  3. One of your (“idiot”) co-workers makes a change (but not to your code).
  4. The tests pass and 2am this Friday, the code is to be deployed
  5. You go out drinking.
  6. Your (“idiot”) manager deploys the code to production and it breaks.
  7. Unfortunately, it breaks in your code.
  8. Your manager looks at your code and can’t figure out why you wrote the code a certain way.
  9. Its 3am now. He calls you up. Are you going to remember why and are you going to be able to help him?

Lets pretend that the test coverage is better and the problem is discovered before deployment. But your change was made 3 months ago, are you going to be able to explain the thought process for the change?

Your client is an “idiot”

You are independent and you take on clients. Your new client is a non-technical person who needs your awesome coding skills. Now you are in trouble. A non-technical product manager and a non-technical manager (they are paying you!).

But your code is self-documenting right?

Not to them. The product is late. The doubts grow in their head. They ask a friend to look at your code. Their friend (“another idiot!”) says the code looks like garbage and he cannot tell why you wrote the code a certain way. The client starts to push back on paying your invoices. Things get ugly.