Archive for the ‘webapps’ Category

The * stupidest things I’ve done in my programming job

Saturday, February 7th, 2009

I’m not ashamed of those sins any more, so here you go :)

1. ORM

Stupidity
Building my own Object Relational Mapping framework.
Consequence
Project is a mess after 2 years of maintenance with hardcore hacks to bypass my own ORM and call custom SQL queries.
What should I have done
Use hibernate, iBATIS, Cayenne or something similar.

2. EAV

Stupidity
Using an Entity-Attribute-Value model database schema design.
Consequence
Non scalable solution and total impossibility to run any useful queries on the database level.
What should I have done
Use an ordinary normalized database schema design.

3. Database Access

Stupidity
Synchronize (serialize) database access using one shared connection.
Consequence
Zero scalability. Very slow response times when more than 10 users where using the application.
What should I have done
Don’t do that and use a connection pool such as c3p0 and use a “new” (reused) connection returned from the pool for every request/response cycle.

4. IDE

Stupidity
Avoided learning and using an Integrated development environment.
Consequence
Inability to build test and deploy the application quickly and generally do anything useful.
What should I have done
Get familiar with an IDE. NetBeans, eclipse etc.

5. Transactions

Stupidity
Not using them.
Consequence
Corrupt data in an application involving e-shop like functionality.
What should I have done
Use database transactions. When in MySQL use InnoDB.

6. Prepared Statements

Stupidity
Using Statements, string concatenation and naive character escaping to assemble my own “safe” queries.
Consequence
SQL Injections possible in my application. I managed to login using ‘or 1=1;delete from users;– and alter the database state in a very nasty way.
What should I have done
Use Prepared Statements which correctly assemble and escape the query properly depending on the JDBC driver used.

7. Business Logic

Stupidity
Doing it in the template (JSP).
Consequence
Messy non maintainable application.
What should I have done
Do it in an MVC style with servlets or with a Front Controller. Even better by using an existing open source MVC framework such as Struts, Spring MVC etc.

Of course, all the bad choices above have probably made me a better programmer.

FreeMarker exception handling

Wednesday, June 20th, 2007

FreeMarker is a very flexible templating engine for Java. Exception handling (while rendering the template) is a very important issue for a templating engine. As with JSP the default behaviour of FreeMarker is to completely cancel rendering and display an error page. When developing a webapp this might not be very helpful. Sometimes errors might need to be tolerated; at least for the development phase of the application development.

FreeMarker provides the TemplateExceptionHandler interface with some implementations but we’ll define our own in order to provide a more failsafe and usable behaviour.

public class MyTemplateExceptionHandler 
          implements TemplateExceptionHandler {

  public void handleTemplateException(TemplateException te, 
          Environment env, Writer out) {
    freemarkerlog.error("template error", te);
    try {
      out.write("<span style=\"cursor:help; color: red\" " +
                "title=\"" + ExceptionUtils.getMessage(te) + "\">" +
                "[e]" +
                "</span>\n");
    } catch (IOException ignored) { }
  }
  
}

Then, in the code where you configure FreeMarker you need:

config.setTemplateExceptionHandler(new MyTemplateExceptionHandler());

This is what you’ll see whenever there is an exception thrown while rendering the template:
FreeMarker exception handling
A nice little [e] with a tooltip containing the exception message.

Runtime dispatching freemarker macros for pojo views

Sunday, June 10th, 2007

One of the (many) reasons I switched from JSP to FreeMarker is that I couldn’t achieve what I describe in this post. Tutorials or blog posts regarding this situation were never to be found, and in addition it was really hard to find anyone considering this issue a real problem.

The problem

Suppose we are building an issue tracking system. We have a rich Domain Model which includes entities such as User, Project, Account, Role etc. We’ve also got an abstract Issue object which is the root of the issue’s hierarchy. Concrete classes extending Issue include Bug, Feature, Request and Change. These 4 POJOs inherit common fields from Issue but add fields, methods and logic of their own.

Each of the issue’s subclass will need to have a slightly different HTML view. I tend to use the composite design pattern for my views, so I can break the HTML down to small reusable components. So it is obvious that we’re going to need 4 different views, one for each of them. Here are those issue rendering methods (presented in an imaginary pseudolanguage which combines EL, HTML and functions):

renderBug(bug) {
  <fieldset>
    <legend>Bug #${bug.id}</legend>
    <p>Author: ${bug.author}</p>
    <p>Date: ${bug.date}</p>
    <p>Description: ${bug.description}</p>
    <p>Steps to recreate bug: ${bug.stepsToRecreate}</p>
  </fieldset>
}

renderFeature(feature) {
  <fieldset>
    <legend>feature #${feature.id}</legend>
    <p>Author: ${feature.author}</p>
    <p>Date: ${feature.date}</p>
    <p>Description: ${feature.description}</p>
    <p>Related URL: ${feature.url}</p>
    <p>Screenshot upload: <img src="${feature.screenshot}" /></p>
  </fieldset>
}

...

Our DAO (probably called IssueDao) is going to fetch a Collection<Issue> (a bunch of issues) from the database for a particular use case. The runtime type of each of those entities cannot be Issue but it will be Bug, Feature, Request or Change. The problem is that we are presenting them altogether in the same screen, so in order to render them we have to write code such as this:

foreach(issues as issue) {
  if (issue instanceof Bug) renderBug(issue); continue;
  if (issue instanceof Feature) renderFeature(issue); continue;
  if (issue instanceof Request) renderRequest(issue); continue;
  if (issue instanceof Change) renderChange(issue);
}

If this doesn’t seem very bad to you, here is an actual view implementation of a slightly bigger hierarchy using JSP 2.0 Tag Files:

if (t instanceof ActivityInternal) {%><p:activityInternalView pojo="${t}" /><%;}
if (t instanceof ActivityExternal) {%><p:activityExternalView pojo="${t}" /><%;}
if (t instanceof ActivityMilestone) {%><p:activityMilestoneView pojo="${t}" /><%;}
if (t instanceof Review) {%><p:reviewView pojo="${t}" /><%;}
if (t instanceof PublicationReport) {%><p:publicationReportView pojo="${t}" /><%;}
if (t instanceof PublicationWebsite) {%><p:publicationWebsiteView pojo="${t}" /><%;}
if (t instanceof InfoConference) {%><p:infoConferenceView pojo="${t}" /><%;}
if (t instanceof InfoBase) {%><p:infoBaseView pojo="${t}" /><%;} 
if (t instanceof InfoChannel) {%><p:infoChannelView pojo="${t}" /><%;}
if (t instanceof Meeting) {%><p:meetingView pojo="${t}" /><%;}
if (t instanceof Interpretation) {%><p:interpretationView pojo="${t}" /><%;}
if (t instanceof BudgetItem) {%><p:budgetItemView pojo="${t}" /><%;}
if (t instanceof FocusGeneral) {%><p:focusGeneralView pojo="${t}" /><%;}
if (t instanceof FocusResearch) {%><p:focusResearchView pojo="${t}" /><%;}
if (t instanceof Risk) {%><p:riskView pojo="${t}" /><%;}
if (t instanceof QAChecklist) {%><p:QAChecklistView pojo="${t}" /><%;}
if (t instanceof TargetAudience) {%><p:targetAudienceView pojo="${t}" /><%;}
if (t instanceof LessonsLearned) {%><p:lessonsLearnedView pojo="${t}" /><%;}

If you still don’t think this is bad, you can stop reading ;)

What not to do

In a project I did in my early JSP days, what I did was to put all the view logic in the Java class! So it looked like this (this is actual Java):

public class Bug extends Issue {

  ...

  public String renderMe() {
    return "<fieldset><legend>" + this.getName() + "</legend>" + 
           "<p>Author: " + this.getAuthor() + "</p>" +
           "<p>Date: " + this.getDate() + "</p>" +
           "<p>Description: " + this.getDescription() + "</p>" +
           "</fieldset>";
  }
}

Although this type of code is a perfect candidate for The Daily WTF, the (only) advantage was that I could now render my pojos using (pseudocode):

foreach(issues as issue) {
  issue.renderMe();
}

The solution

It seems that all we want is the ability to construct and dynamically (reflectively in Java terms) call the appropriate render tag each time. In freemarker we define macros which look like this:

<#macro renderBug bug>
  <fieldset>
    <legend>Bug #${bug.id}</legend>
    <p>Author: ${bug.author}</p>
    <p>Date: ${bug.date}</p>
    <p>Description: ${bug.description}</p>
    <p>Steps to recreate bug: ${bug.stepsToRecreate}</p>
  </fieldset>
</#macro>

We need a way to call renderXXX where XXX is the short class name of the issue in question. And here is how you can do this in freemarker:

<#local macroname='render' + issue.class.name?split(".")?last />
<@.vars[macroname] issue />

For an issue of runtime type com.example.Foo, it concatenates the word “render” with “Foo” and calls the macro with that name. The magic happens with the help of the .vars special variable. It allows us to access variables by name. The full code now becomes:

<#macro renderIssue issue>
  <#local macroname='render' + issue.class.name?split(".")?last />
  <@.vars[macroname] issue />
</#macro>

<#list issues as issue>
  <@renderIssue issue />
</#list>

By the way, this capability is usually present in dynamic scripting languages. So for example there are many ways to do that in PHP.

using dynamic evaluation
$functionName = "renderBug";
$functionName($issue);
using eval
eval("renderBug($issue);");
using call_user_func (probably safest of all)
call_user_func("renderBug", $issue);

Caching pages using ehcache

Monday, June 4th, 2007

When an http request to your /rss page needs 400 milliseconds to complete, it seems obvious that your website could benefit from some caching. Ehcache is a well known cache provider, which most of us know from hibernate. Since we are already “bound” to ehcache, lets see how we can benefit from caching some dynamically generated pages:

web.xml

<filter>
  <filter-name>SimplePageCachingFilter</filter-name>
  <filter-class>net.sf.ehcache.constructs.web.filter.SimplePageCachingFilter</filter-class>
</filter>
<filter-mapping>
  <filter-name>SimplePageCachingFilter</filter-name>
  <url-pattern>/rss</url-pattern>
</filter-mapping>

We set up the SimplePageCachingFilter in the web.xml of the web application and map it to one or more url patterns or servlets. All requests to /rss will be intercepted by the SimplePageCachingFilter.

ehcache.xml

<ehcache>
  <diskStore path="java.io.tmpdir" />
  <cache name="SimplePageCachingFilter"
         maxElementsInMemory="0"
         eternal="false"
         timeToIdleSeconds="600"
         timeToLiveSeconds="600"
         overflowToDisk="true"/>        
</ehcache>

We then configure the cache region for pages. We don’t want any elements kept in memory. Everything should be written to disk at the java.io.tmpdir location. The cache expires every 10 minutes.

Now hitting http://example.com/rss (our default rss page) results in a cache miss. The content is being generated from scratch but before returning to the client, the filter stores it locally. The second time we’ll get a cache hit. The content now is being fetched from the cache and its very fast. 10 minutes later this cache element will be invalidated. Note that the default implementation uses the URI together with the query string to calculate the cache key, so /rss?type=news and /rss?type=forum will result in two different cache elements.

Default Servlet and Resin

Thursday, March 8th, 2007

Suppose you use a servlet as a front controller to catch and process all urls in a web app. If you want clean URLs you may have mapped it using:

<servlet-mapping>
  <servlet-name>FrontController</servlet-name>
  <url-pattern>/</url-pattern>
</servlet-mapping>

Your front controller will now attempt to serve all URLs, and this is something you don’t want. Static content (png, html, ico, css…) are being served by a default servlet. In tomcat that is org.apache.catalina.servlets.DefaultServlet, and has been configured for you in conf/web.xml with the name “default”.

So, in order to exclude all static content from the catch-all of your front controller, you have to map static content to the default servlet, before the mapping of the front controller:

<servlet-mapping>
  <servlet-name>default</servlet-name><url-pattern>*.css</url-pattern>
</servlet-mapping>
<servlet-mapping>
  <servlet-name>default</servlet-name><url-pattern>*.js</url-pattern>
</servlet-mapping>
<servlet-mapping>
  <servlet-name>default</servlet-name><url-pattern>*.png</url-pattern>
</servlet-mapping>
<servlet-mapping>
  <servlet-name>default</servlet-name><url-pattern>*.jpg</url-pattern>
</servlet-mapping>
...

That works nicely, when deploying in Tomcat, Jetty and JBoss Application Server.
On Resin, deployment fails with the following message:
WEB-INF/web.xml:89: `default’ is an unknown servlet-name. servlet-mapping requires that the named servlet be defined in a <servlet> configuration before the <servlet-mapping>.</servlet-mapping></servlet>
Resin’s static content servlet is com.caucho.servlets.FileServlet and until 3.0 was mapped using the name “file”. Then, on 3.1, and after some people complained that they couldn’t have a servlet called “file”, the name was changed to “resin-file”.
So, there are 2 solutions to make your application function properly. You can either change all references from “default” to “resin-file” in your web.xml, or change the FileServlet’s name from “resin-file” to “default” in Resin’s conf\app-default.xml.

Happy deployments.

An http session impersonation protection filter

Thursday, March 1st, 2007

Session Impersonation is an attack which works for webapps and dynamic websites. Someone steals your session cookie (possibly by using XSS – Cross Site Scripting), injects it into his browser visits the site and suddenly appears to be you. If you happened to be logged in as the single superadmin of the system, then he is a superadmin as well.

One of the ways to avoid this problem is by storing a hash (or token) the first time the http session is created. That hash will contain the user’s IP address and his user agent (the browser he uses). On each following request, the hash is being recalculated, and must match the hash previously stored in the http session. If it does not match, any of the 3 things might have happened:

  1. Client has changed his IP.
  2. Client has changed his user agent String.
  3. Client is using another clients session (session impersonation attack).

Changing you IP is hard (unless your ISP is AOL or you use an anonymity service such as TOR). Changing browsers will initiate a new http session anyway, and changing your user-agent String is rare. It can be done in Firefox using the about:config page but that’s not a thing that users do everyday.

Note that session impersonation protection is hard (impossible?) to do when people use the same IP. That can be the case in universities, companies and netcafes.

Here is the doFilter method of an http filter which you can use to protect your application from session impersonation attacks. It will invalidate the session when this happens.

if (request instanceof HttpServletRequest) {
  HttpServletRequest httpRequest = (HttpServletRequest)request;
  // get the session, without creating one if there isn't any
  HttpSession session = httpRequest.getSession(false);
  // if there is a session
  if (session!=null) {
    //calculate a hash using ip and user agent
    String hash = getHash(httpRequest.getRemoteAddr(),
        httpRequest.getHeader(USER_AGENT_KEY));
    if (session.getAttribute(HASH_KEY)==null) {
      // put hash in session
      session.setAttribute(HASH_KEY, hash);
    } else {
      // session does not contain hash
      if(!hash.equals(session.getAttribute(HASH_KEY))) {
        // TODO log session impersonation attempt?
        session.invalidate();
      }
    }
  }
}
chain.doFilter(request, response);

The getHash method could just return the two Strings concatenated, but ideally you should hash them.

public static final String getHash(String ip, String agent) {
  return Integer.toString(ip.hashCode()) + agent.hashCode();
}

MD5 would be good but usually it’s costly. Here I just used String#hashCode.
You’ll also need two constants for the filter:

public static final String HASH_KEY = "HASH";
public static final String USER_AGENT_KEY = "user-agent";

Thats it. Set the filter on the top of your filters chain and you are ready.

Database Connection Pooling

Friday, February 23rd, 2007

You are building a webapp. You want database connectivity. You want pooling (because its an expensive resource). You start building your own database connection pool. STOP!!!

Who told you that you can do it well? Why did you hack your own connection pool implementation, which is seriously broken, spawns thousands of threads and turns the server into a miserable piece of shit that needs restart every 24h?

Wrong choice my friend. Next time do us all a favour and use one of the following:

And don’t forget: The standard idiom for releasing a connection is to close (return) the connection in a finally block.

Connection con = getPooledConnectionFromSomewhere();
try {
  // do stuff with connection
} catch (SQLException e) {
  // handle problems
} finally {
  con.close();
}

Of course, closing the connection can throw an SQLException, but it’s up to you on how you will handle it.

Good luck

DataSource exposed through JNDI

Friday, February 23rd, 2007

You are building a webapp and you want database connection pooling. Your container can help you manage this javax.sql.DataSource by configuring it and exposing it through the JNDI tree.

Containers usually come with Jakarta Commons DBCP out of the box. In order to use it edit the context.xml file of your webapp and set your datasource there.

<?xml version="1.0" encoding="UTF-8"?>
 <Context path="/foo-app">

 <!-- TOMCAT 5.5.xx DESCRIPTOR -->
 <Resource name="foo"
  auth="Container"
  type="javax.sql.DataSource"
  maxActive="1"
  maxIdle="1"
  maxWait="3000"
  username="user"
  password="pass"
  driverClassName="net.sourceforge.jtds.jdbc.Driver"
  url="jdbc:jtds:sqlserver://localhost:1433;DatabaseName=foobar;charset=utf8"
 />

 <!-- TOMCAT 5.0.28 DESCRIPTOR -->
 <!--
  <Resource name="foo" type="javax.sql.DataSource"/>
  <ResourceParams name="foo">
   <parameter><name>maxActive</name><value>1</value></parameter>
   <parameter><name>maxIdle</name><value>1</value></parameter>
   <parameter><name>maxWait</name><value>3000</value></parameter>
   <parameter><name>username</name><value>user</value></parameter>
   <parameter><name>password</name><value>pass</value></parameter>
   <parameter><name>driverClassName</name><value>net.sourceforge.jtds.jdbc.Driver</value></parameter>
   <parameter><name>url</name><value>jdbc:jtds:sqlserver://localhost:1433;DatabaseName=foobar;charset=utf8</value></parameter>
  </ResourceParams>
  -->

</Context>

Tomcat 5.0.xx and 5.5.xx uses different xml syntax for most of it’s configuration. Here I’m presenting both with 5.0.xx’s block commented out.

So, you’ve got your app called “foo-app”. Next time you’ll deploy it, tomcat will copy context.xml to ${catalina.home}/conf/localhost/foo-app.xml where from it will be reading the configuration on each context or container initialization. Your Datasource is called “foo”.
In your java code now, what you need to get a reference to “foo” is:

Context ctx = new InitialContext();
DataSource ds = (DataSource)ctx.lookup("java:comp/env/foo");

And then something along the lines:

Connection con = ds.getConnection();

Happy coding.