Archive for the ‘java’ Category

SQL Server + hbm2ddl + unicode columns

Tuesday, May 1st, 2007

Hibernate offers org.hibernate.dialect.SQLServerDialect as the dialect for SQL Server. When generating the database schema, using hbm2ddl, the string type columns do not support native characters. So the following mapping:

<property name="title" length="128" />

will produce the following SQL:

...
title varchar(128) null,
...

By extending the org.hibernate.dialect.SQLServerDialect we can achieve the generation of NCHAR, NVARCHAR, and NTEXT columns instead of CHAR, VARCHAR and TEXT.

package com.foo.hibernate;

import java.sql.Types;
import org.hibernate.dialect.SQLServerDialect;

public class SQLServerNativeDialect extends SQLServerDialect{

  public SQLServerNativeDialect() {
    super();
    registerColumnType(Types.CHAR, "nchar(1)");
    registerColumnType(Types.VARCHAR, "nvarchar($l)");
    registerColumnType(Types.LONGVARCHAR, "nvarchar($l)");
    registerColumnType(Types.CLOB, "ntext");
  }

}

All we need to do now is plug this dialect in our hibernate configuration:

<property name="hibernate.dialect">
  com.foo.hibernate.SQLServerNativeDialect
</property>

Related hibernate forums thread: http://forum.hibernate.org/viewtopic.php?t=972518
Related API method: http://www.hibernate.org/hib_docs/v3/api/org/hibernate/dialect/Dialect.html

Learn to use the debugger

Monday, April 30th, 2007

Your favourite IDE has a powerful debugger which you can use to debug your programs. If you are new to programming, chances are that you are aware of it’s existence, but never use it.

The problem

Here is an example method, which has a problem. (It actually does nothing of value, but demonstrates the problem case). The method doSomething is called with a parameter, but the expected result is not returned.

public Box doSomething(Box foo, Box bar) {
  Box temp = foo.cloneMe();
  if (bar!=null) {
    Box newBox = new Box(bar);
    if (newBox!=null) {
      temp = doSomethingElse(foo);
      if (foo==null) {
        temp = bar;
      }
    }
  }
  return temp;
}

Here you can see the most frequent (ab)use of System.out.println statements.

public Box doSomething(Box foo, Box bar) {
  System.out.println("1");
  Box temp = foo.cloneMe();
  if (bar!=null) {
    System.out.println("2");
    Box newBox = new Box(bar);
    if (newBox!=null) {
      System.out.println("3");
      temp = doSomethingElse(foo);
      if (foo==null) {
        System.out.println("4");
        temp = bar;
      }
    }
  }
  return temp;
}

The developer’s intent is to trace which if-statements execute, so as to find the bug. If 1 2 3 is displayed in the console, the developer knows that foo==null evaluated to false.

An “enhancement” of this method is to add variables of interest in those System.out.println statements.

public Box doSomething(Box foo, Box bar) {
  System.out.println("1: " + foo + " " + bar);
  Box temp = foo.cloneMe();
  if (bar!=null) {
    System.out.println("2: " + temp);
    Box newBox = new Box(bar);
    if (newBox!=null) {
      System.out.println("3: " + newBox);
      temp = doSomethingElse(foo);
      if (foo==null) {
        System.out.println("4: " + temp);
        temp = bar;
      }
    }
  }
  return temp;
}

This is one of the most crude ways to debug a program. Unfortunately it’s quite common between junior developers. Note that if logging needs to be performed (for monitoring or historical purposes) a proper logging framework has to be used.

Things get interesting when the developer forgets to delete those System.out.println statements. The application is deployed, in a servlet container which hosts more applications, featuring code “debugged” in this way.
It’s not rare to see catalina.out logs which look like this:

INFO: Find registry server-registry.xml at classpath resource
20 Απρ 2007 10:23:24 μμ org.apache.catalina.startup.Catalina start
INFO: Server startup in 14063 ms
20 Απρ 2007 10:23:24 μμ org.apache.catalina.core.StandardContext reload
INFO: Reloading this Context has started
1
2
is null
3
copying file pic_01.jpg->temp/pic_01.jpg
copying file pic_02.jpg->temp/pic_02.jpg
copying file pic_03.jpg->temp/pic_03.jpg
copying file pic_06.jpg->temp/pic_06.jpg
1
2
is null
3
true
4
5
6

** BEGIN NESTED EXCEPTION ** 

java.net.ConnectException
MESSAGE: Connection refused: connect

STACKTRACE:

java.net.ConnectException: Connection refused: connect
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
...
is null
false
java.lang.NullPointerException
BoxExample$Box@126b249
1
2
is null
3
resultset was null
1
resultset was null
...

The solution

Learn to use your debugger. All you have to do is go to the line you want debugging to start, set a breakpoint (CTRL+F8 in Netbeans) and start the debug process. You will either debug the whole application (F5) or that single file/unit test (CTRL+SHIFT+F5).
You can set watches, see the stacktrace and examine the contents of all the local variables at any time in the program execution. You get orders of magnitude more power, in less time; for free!

Try it out. When you get used to it, you’ll never look back.

Test First (#1)

Saturday, April 28th, 2007

Unit testing not only ensures that the code you write is correct, but also helps you develop your code. This can be achieved by testing unimplemented methods and functionalities first, and then “filling in” the code to satisfy the tests. This is the “test first” rule that your XP coworker will always remind you, at any occasion. (You do have an XP coworker, don’t you? :)

The Requirement

Suppose you have a webapp which displays some data from a database. Some of those texts are long, and you are asked to be able to truncate them at predefined lengths on some particular views. So instead of printing “Lorem ipsum dolor sit” you should be able to fit that in 12 chars and print “Lorem ips…”.

Wait

The first worst thing you could do there would be to stick that logic straight into the view (JSP, velocity or whatever templating engine you use). You will not be able to test that piece of logic, nor easily reuse it in some other project. You will need to do this in a Java class and then find a way to call it from the template.

The second worst thing you could do would be to implement this functionality yourself, as it already exists in commons lang. You should know what APIs exist out there and try to reuse as often as possible. But anyhow, we’ll assume that you want to do it yourself.

Think

You start by thinking of where to place this method and whether it will be a helper (static?) method or part of a full fledged class with state etc. Then you choose a good method name and what parameters it will accept. Think of how you would like to use this method.

Code

You write the method signature:

public static String truncate(String text, int length) {
}

Then fill it’s body to the absolute minimum to make it “compilable”:

public static String truncate(String text, int length) {
  throw new RuntimeException("not implemented");
}

…or…

public static String truncate(String text, int length) {
  return "";
}

And then stop, because that’s all you need to implement for now.

Test

You create a test (hit CTRL+SHIFT+U if you use http://www.netbeans.org/) and create a test for the method truncate

public void testTruncate() {
}

This test passes, because it doesn’t test anything. You make things more interesting by adding some basic assertions of what you expect from this method.

public void testTruncate() {
  assertEquals("Lorem ip...", Demo.truncate("Lorem ipsum dolor sit", 11));
  assertEquals("Lorem ips...", Demo.truncate("Lorem ipsum dolor sit", 12));
  assertEquals("Lorem ipsu...", Demo.truncate("Lorem ipsum dolor sit", 13));
}

You run the test and it fails.
You go back to the source code and implement some logic to satisfy this test.

Code

public static final String truncate(String text, int length) {
  return text.substring(0, length-3) + "...";
}

You run the test and it now passes.

Test

It’s time to test for some corner cases.

public void testTruncate() {
  assertEquals("", Demo.truncate("Lorem", 0));
  assertEquals(".", Demo.truncate("Lorem", 1));
  assertEquals("..", Demo.truncate("Lorem", 2));
  assertEquals("...", Demo.truncate("Lorem", 3));
  assertEquals("Lorem ip...", Demo.truncate("Lorem ipsum dolor sit", 11));
  assertEquals("Lorem ips...", Demo.truncate("Lorem ipsum dolor sit", 12));
  assertEquals("Lorem ipsu...", Demo.truncate("Lorem ipsum dolor sit", 13));
  assertEquals("Lorem ipsum dolor sit", Demo.truncate("Lorem ipsum dolor sit", 21));
}

All new assertions fail with a StringIndexOutOfBoundsException. You write code to satisfy these corner cases.

Code

public static final String truncate(String text, int length) {
  switch(length) {
    case 0: return "";
    case 1: return ".";
    case 2: return "..";
    case 3: return "...";
    default:
      return length<text.length() ?
        text.substring(0, length-3) + "..." :
        text;
  }
}

You run the test and it now passes.

Test

Then you test even more.

public void testTruncate() {
  assertEquals("", Demo.truncate("Lorem", 0));
  assertEquals(".", Demo.truncate("Lorem", 1));
  assertEquals("..", Demo.truncate("Lorem", 2));
  assertEquals("...", Demo.truncate("Lorem", 3));
  assertEquals("Lorem ip...", Demo.truncate("Lorem ipsum dolor sit", 11));
  assertEquals("Lorem ips...", Demo.truncate("Lorem ipsum dolor sit", 12));
  assertEquals("Lorem ipsu...", Demo.truncate("Lorem ipsum dolor sit", 13));
  assertEquals("Lorem ipsum dolor sit", Demo.truncate("Lorem ipsum dolor sit", 21));
  try {
    Demo.truncate("Lorem ipsum dolor sit", -1);
    fail("Should have thrown illegal argument exception");
  } catch (IllegalArgumentException expected) { }
}

This is a standard idiom for testing that an exception should be thrown. If this method is called with a negative length parameter, we’d like an IllegalArgumentException to be thrown. It is not mandatory to test for things like these, but some people like to seal their methods from really bad usage.

Code

public static final String truncate(String text, int length) {
  if (length<0) {
    throw new IllegalArgumentException("Length cannot be negative");
  }
  switch(length) {
    case 0: return "";
    case 1: return ".";
    case 2: return "..";
    case 3: return "...";
    default:
      return length<text.length() ?
        text.substring(0, length-3) + "..." :
        text;
  }
}

You run the test and it now passes. You are done.

Conclusion

This implementation might not be the best in the world, but right now this doesn’t matter. The code runs, and it’s robust. Whenever you feel like, you can refactor it and make it perform better. The test will be there to guide you.

p.s What we’ve omitted when we wrote the signature of this method, was to write Javadoc. It is very important to document your API, and we’ll discuss that in another post.

Unit testing needs time?

Tuesday, March 20th, 2007

New developers will sometimes complain about how Unit Testing requires a lot of time. How much it slows them down, and how they cannot see any good in writing tests for their code.
There are many scenarios which prove that unit testing is necessary. These include speed of development, ability to refactor easily, test-driven development, testing without the need of web container, testing with mock objects etc.

My favorite though is the “client calls to report something weird” scenario. I’ve seen it many times and it goes something like this:

Scenario:

  1. Your webapp is deployed, weeks ago, and you’ve moved on with a new, exciting project. Everything is feels good, as you’ve completely forgotten about sins of the past (not writing tests)
  2. Client calls you to report “something weird”.
  3. You stop whatever you are doing at that moment to switch to that project.
  4. You connect (VPN or whatever) to the remote server to see possibly logged exceptions.
  5. You collect your data.
  6. You try to reproduce the error locally, on the development server.
  7. You find the bug.
  8. You issue the bug in the bug tracking system.
  9. You fix the bug.
  10. You build for deployment.
  11. You deploy (while solving any possible application version issues).
  12. You contact the client.
  13. Wait for his confirmation that the bug has been corrected.
  14. You deliver the bug in the issue tracking system.
  15. Finally you commit the code back to SVN.

…or in whatever order feels more natural to you.

If the above scenario feels OK, and you need some hints on why you should try to minimize such cases, have a look at the costs involved:

Costs:

  • All of these actions need time. Your time.
  • Most of them require a context switch. Not only you lose X minutes from your previous work, but also need Y minutes to get back into the flow (mind state) you had previously.
  • Some of these steps might not be what you really want to be doing (talking directly to the client).
  • You become a slow worker producing bad code.
  • People will never trust you with that mission critical application, because your code has a tendency to develop “random features” on runtime (usually involving exciting names such as NullPointerException).

Facts:

This scenario can definitely happen for tested code. Bugs will always creep into your code no matter what. The point is try to at least minimize the stupid ones. Cases which can easily be covered by unit tests.
Unit testing is important (if not mandatory). If you feel that it needs time, you have to press yourself and do it. It’s a matter of weeks until you become test infected and experience how your software becomes better in less time.

Hints for NetBeans users:

  • Got a class that you want to create a JUnit test for? CTRL+SHIFT+U
  • Got a class and want to jump to the unit test (and vice versa)? ALT+SHIFT+E
  • Want to test the project? ALT+F6

Happy testing.

Tomcat vs JBoss Web

Wednesday, March 14th, 2007

JBoss Web is a web server and servlet container at the same time. It’s promise is that it can serve static and dynamic content, very fast, without the need of an Apache HTTPD fronting it. If that’s true, its party time, and I personally live for the day where it will be easy to get Java 5 enabled hosting for ~5USD/month (as it is the case today with LAMP stacks).

JBoss Web uses APR and native extensions in order to achieve better utilization of the resources of the O/S. Note that APR is also available for Tomcat now.

I’ve decided to give JBoss Web a try, locally, and stress test it against a regular Tomcat. Note that what I did was done for pure fun (and out of curiosity). I do not own a lab, I am definitely not a stress test expert and I do not understand many things at the low level (I/O, threads etc).

Test info

  1. JMeter was used and it was running on the same machine with the servers tested.
  2. During tests JMeter would use ~30% of cpu, and the server would consume the rest ~70%.
  3. O/S was Windows XP SP2 on an AMD64 3000+ with 1.5GB ram.
  4. Java 1.5.0_06 on -server mode for both servers.
  5. Default installations of JBoss Web 1.0.1 GA and Tomcat 5.5.23 were used.
  6. -Xms and -Xmx settings were not altered. Don’t think it mattered.
  7. I stress tested 10 URLs of a very small webapp with a front controller delegating to cached freemarker views. No logging, no persistence or database calls. JBoss’ CONSOLE appender’s threshold was changed to FATAL, to avoid any logging output which would slow down things. The most interesting operations in the webapp would be the GZIP filter, and multipart request using commons fileupload.
  8. Warm up of the servers was performed. I found out that even for small amount of concurrent threads hitting the server, if these all start immediately, it’s most likely you’ll get some 500s at the beginning. The warm up would be anything between 2500-5000 requests until the server throughput was stabilized.
  9. When the server was warmed up, I would get my sample from the next 5000-10000 requests.
  10. The “threads” column in the results table, is the amount of concurrent threads which where hitting the server.
  11. An http cookie manager was used on JMeter, so 10000 sessions were not being created.

Results

threads Tomcat 5.5.23 JBoss Web 1.0.1
50 95 requests/sec 88 requests/sec
75 105 requests/sec 95 requests/sec
100 123 requests/sec 100 requests/sec
125 75 requests/sec 104 requests/sec
150 110 requests/sec
at this point I had to increase the maxThreads
110 requests/sec
200 62 requests/sec 97 requests/sec
300 115 requests/sec 108 requests/sec
400 n/a
at this point JMeter would block.
[25 seconds per page]
80 requests/sec
500 n/a 75 requests/sec
600 n/a 84 requests/sec
700 n/a 55 requests/sec
[10 seconds per page]
800 n/a 48 requests/sec
[13 seconds per page]
1000 n/a n/a
at this point JMeter would block

Findings

Even this test can be considered rudimentary, JBoss Web looks very good. The biggest problem with the whole procedure is that JMeter was on the same machine as the servers. JMeter supports Remote Testing and Distributed Testing which would have produced more accurate results.

In any case, it was fun.

Default Servlet and Resin

Thursday, March 8th, 2007

Suppose you use a servlet as a front controller to catch and process all urls in a web app. If you want clean URLs you may have mapped it using:

<servlet-mapping>
  <servlet-name>FrontController</servlet-name>
  <url-pattern>/</url-pattern>
</servlet-mapping>

Your front controller will now attempt to serve all URLs, and this is something you don’t want. Static content (png, html, ico, css…) are being served by a default servlet. In tomcat that is org.apache.catalina.servlets.DefaultServlet, and has been configured for you in conf/web.xml with the name “default”.

So, in order to exclude all static content from the catch-all of your front controller, you have to map static content to the default servlet, before the mapping of the front controller:

<servlet-mapping>
  <servlet-name>default</servlet-name><url-pattern>*.css</url-pattern>
</servlet-mapping>
<servlet-mapping>
  <servlet-name>default</servlet-name><url-pattern>*.js</url-pattern>
</servlet-mapping>
<servlet-mapping>
  <servlet-name>default</servlet-name><url-pattern>*.png</url-pattern>
</servlet-mapping>
<servlet-mapping>
  <servlet-name>default</servlet-name><url-pattern>*.jpg</url-pattern>
</servlet-mapping>
...

That works nicely, when deploying in Tomcat, Jetty and JBoss Application Server.
On Resin, deployment fails with the following message:
WEB-INF/web.xml:89: `default’ is an unknown servlet-name. servlet-mapping requires that the named servlet be defined in a <servlet> configuration before the <servlet-mapping>.</servlet-mapping></servlet>
Resin’s static content servlet is com.caucho.servlets.FileServlet and until 3.0 was mapped using the name “file”. Then, on 3.1, and after some people complained that they couldn’t have a servlet called “file”, the name was changed to “resin-file”.
So, there are 2 solutions to make your application function properly. You can either change all references from “default” to “resin-file” in your web.xml, or change the FileServlet’s name from “resin-file” to “default” in Resin’s conf\app-default.xml.

Happy deployments.

Time to get Groovy

Friday, March 2nd, 2007

Groovy is a dynamic language for Java. It allows you to do all funky stuff that you can do in dynamic languages, but still be able to write Java and execute inside the JVM.

Let’s get busy and write a simple Person Java class, in Java:
Person.java

package com.cherouvim.blog;
public abstract class Person {
  private int age;

  public Person() {
  }

  public void setAge(int age) {
    this.age = age;
  }

  public int getAge() {
    return this.age;
  }

  /**
   * Determines whether this person should
   * start smoking.
   * Always returns false!
   */
  public boolean shouldStartSmoking() {
    return false;
  }

  /**
   * Determines whether this person is allowed
   * to drive a car.
   * To be implemented on subclass.
   */
  public abstract boolean isAllowedToDriveCar();

}

This is an abstract class; it could have also been an interface. We want to add more behaviour now with Groovy code, so we are going to extend it and define the isAllowedToDriveCar method:
Person.groovy

package com.cherouvim.blog
public class PersonGroovy extends Person {
  public boolean isAllowedToDriveCar() {
    return this.getAge()>18
  }
}

In order to make this groovy script available to our Java program, we need to load it via the GroovyClassLoader. I define a helper method so I can easily get instances of Groovy classes:
GroovyLoader

public static final Object loadGroovyObject(String groovyScriptLocation) {
  GroovyClassLoader gc = new GroovyClassLoader();
  Class groovyClass = gc.parseClass(
      ClassLoader.getSystemResourceAsStream(groovyScriptLocation));
  Object groovyObject = null;
  try {
    groovyObject = groovyClass.newInstance();
  } catch (Exception ex) {
    // log exception
    ex.printStackTrace();
  }
  return groovyObject;
}

Now let’s test our concrete PersonGroovy class:
PersonTest.java

  Person p;
  public void setUp() {
    p = (Person)GroovyLoader.loadGroovyObject(
        "com/cherouvim/blog/Person.groovy");
  }

  /**
   * Test java method
   */
  public void testShouldStartSmoking() {
    assertFalse(p.shouldStartSmoking());
  }

  /**
   * Test groovy method
   */
  public void testIsAllowedToDriveCar() {
    p.setAge(14);
    assertFalse(p.isAllowedToDriveCar());
    p.setAge(27);
    assertTrue(p.isAllowedToDriveCar());
  }

This test gets an instance of the PersonGroovy class, casts it to the known type of Person and we are ready to go. The isAllowedToDriveCar method is available because we defined it in the abstract superclass. The tests pass.

Note that the way I’ve presented parses the groovy file every time we call loadGroovyObject which is slow. You can cache the class.

An http session impersonation protection filter

Thursday, March 1st, 2007

Session Impersonation is an attack which works for webapps and dynamic websites. Someone steals your session cookie (possibly by using XSS – Cross Site Scripting), injects it into his browser visits the site and suddenly appears to be you. If you happened to be logged in as the single superadmin of the system, then he is a superadmin as well.

One of the ways to avoid this problem is by storing a hash (or token) the first time the http session is created. That hash will contain the user’s IP address and his user agent (the browser he uses). On each following request, the hash is being recalculated, and must match the hash previously stored in the http session. If it does not match, any of the 3 things might have happened:

  1. Client has changed his IP.
  2. Client has changed his user agent String.
  3. Client is using another clients session (session impersonation attack).

Changing you IP is hard (unless your ISP is AOL or you use an anonymity service such as TOR). Changing browsers will initiate a new http session anyway, and changing your user-agent String is rare. It can be done in Firefox using the about:config page but that’s not a thing that users do everyday.

Note that session impersonation protection is hard (impossible?) to do when people use the same IP. That can be the case in universities, companies and netcafes.

Here is the doFilter method of an http filter which you can use to protect your application from session impersonation attacks. It will invalidate the session when this happens.

if (request instanceof HttpServletRequest) {
  HttpServletRequest httpRequest = (HttpServletRequest)request;
  // get the session, without creating one if there isn't any
  HttpSession session = httpRequest.getSession(false);
  // if there is a session
  if (session!=null) {
    //calculate a hash using ip and user agent
    String hash = getHash(httpRequest.getRemoteAddr(),
        httpRequest.getHeader(USER_AGENT_KEY));
    if (session.getAttribute(HASH_KEY)==null) {
      // put hash in session
      session.setAttribute(HASH_KEY, hash);
    } else {
      // session does not contain hash
      if(!hash.equals(session.getAttribute(HASH_KEY))) {
        // TODO log session impersonation attempt?
        session.invalidate();
      }
    }
  }
}
chain.doFilter(request, response);

The getHash method could just return the two Strings concatenated, but ideally you should hash them.

public static final String getHash(String ip, String agent) {
  return Integer.toString(ip.hashCode()) + agent.hashCode();
}

MD5 would be good but usually it’s costly. Here I just used String#hashCode.
You’ll also need two constants for the filter:

public static final String HASH_KEY = "HASH";
public static final String USER_AGENT_KEY = "user-agent";

Thats it. Set the filter on the top of your filters chain and you are ready.

Java Code Conventions

Saturday, February 24th, 2007

When writing Java code you must follow the Code Conventions for the Java Programming Language. It’s a document written by Sun back in 1997. Why should you read such an ancient (in computer science terms) document?

Code conventions are important to programmers for a number of reasons:

* 80% of the lifetime cost of a piece of software goes to maintenance.
* Hardly any software is maintained for its whole life by the original author.
* Code conventions improve the readability of the software, allowing engineers to understand new code more quickly and thoroughly.

Some people tell me that they cannot change their style, because that’s how they are used to coding. Fair enough. Do whatever you want when coding alone, in your home. But not in a professional environment. If you can’t be bothered, do us a favour and leave. Resign. Start selling popcorn. Whatever.

I have a serious problem working with people who commit Java code which looks like this:

public class persons {
public void PersonSave(persons p) {
package foo.bar.personUtils;
public static final String foo = "whatever";
whatever()
{
  // do stuff
}

Seriously, please try to read the following piece of code found in a real life project. Does this look like Java?

if(e<0.0)d= -d;
if(d!=0.0)for(int i=0;i<dim;i++)this.n[i] = this.n[i]/d;
else this.tW.writeString("normalise: non simplex");
Object leftList = null, rightList = null;
try{ leftList = c.newInstance();}
catch(Exception e){tW.writeString("sort:error1 ");return null;};

OK, this style might have been good in an Obfuscated Code Contest, but it definitely does not get you going in the workplace.

Good luck

Database Connection Pooling

Friday, February 23rd, 2007

You are building a webapp. You want database connectivity. You want pooling (because its an expensive resource). You start building your own database connection pool. STOP!!!

Who told you that you can do it well? Why did you hack your own connection pool implementation, which is seriously broken, spawns thousands of threads and turns the server into a miserable piece of shit that needs restart every 24h?

Wrong choice my friend. Next time do us all a favour and use one of the following:

And don’t forget: The standard idiom for releasing a connection is to close (return) the connection in a finally block.

Connection con = getPooledConnectionFromSomewhere();
try {
  // do stuff with connection
} catch (SQLException e) {
  // handle problems
} finally {
  con.close();
}

Of course, closing the connection can throw an SQLException, but it’s up to you on how you will handle it.

Good luck