ioannis cherouvim

When running a longevity stress test with JMeter (a test which runs for many days) you may need to emulate a load which approximates the real traffic that the site is receiving in production. And that is definitelly not a steady and constant load during the duration of the full 24 hour cycle.

Most normal sites (not twitter or facebook) tend to receive different amounts of traffic during a day. Although it depends on the nature of the site, usually the traffic will look like a sine wave with a wave length of 1 day. Even if it doesn’t look as smooth as a sine wave, a sine modulating throughput will be much better than testing with constant one. Having a constant throuput can mess up with the data you receive from the test since the application, db and o/s level caches and other systems of the stack (e.g the GC) may tune to the specific constant throughput.

So, first of all we need to setup some variables in the JMeter test.

Setting oscillationsPerDay to 1 is what we want.

Next we setup a Constant Throughput Timer to reference the hitsPerMinute variable. Note that the initial value of this variable doesn’t play any role since we’ll be constantly changing this via a bean shell script.

Lastly we need a BeanShell PreProcessor with the following script:

// variables
double minHitsPerSec = Double.parseDouble(vars.get("minHitsPerSec"));
double maxHitsPerSec = Double.parseDouble(vars.get("maxHitsPerSec"));
double oscillationsPerDay  = Double.parseDouble(vars.get("oscillationsPerDay"));

// calculation
double oscillationFrequency = 1000L * 60 * 60 * 24 / oscillationsPerDay;
double range = maxHitsPerSec - minHitsPerSec;
double hitsPerSecond = Math.sin(System.currentTimeMillis()/oscillationFrequency*(Math.PI*2))*range/2+range/2+minHitsPerSec;

// set
vars.put("hitsPerMinute", String.valueOf(hitsPerSecond*60));

// log
log.info("throughput: " + hitsPerSecond + " hits per second, or " + vars.get("hitsPerMinute") + " hits per minute");

So this will generate a load which will modulate from minHitsPerSec to maxHitsPerSec for as many times per day you need. Of course, you can make the load and requests behavior more realistic by adding a Random Timer.

Last year Terracotta acquired ehcache and quartz and it was all good an exciting news. The problem is that since then they’ve included an automatic update checker on these two libraries which is turned on by default!

What this does is to connect to www.terracotta.org as soon as you bootstrap your application, send some info (!) and get a response back on whether you are currently using the latest version of the library.

You’ll get something this on your logs for ehcache:

2010-08-16 11:18:04,794 DEBUG (UpdateChecker.java:68) - Checking for update...
2010-08-16 11:18:05,934 INFO  (UpdateChecker.java:98) - New update(s) found: 2.2.0 Please check http://ehcache.org for the latest version.

and for quartz:

2010-08-16 11:15:58,218 DEBUG (UpdateChecker.java:56) - Checking for available updated version of Quartz...
2010-08-16 11:16:01,734 INFO  (UpdateChecker.java:86) - New Quartz update(s) found: 1.8.4 [http://www.terracotta.org/kit/reflector?kitID=default&pageID=QuartzChangeLog]

Terracotta gives an explanation on why they did this but no matter how you try it still makes your brain hurt and wonder what would happen if every vendor of Java libraries did this. Complete misery.

Disabling this check is highly recommended both on development and production.

For ehcache you need to add:

updateCheck="false"

in your ehcache.xml root element (<ehcache>) and:

org.quartz.scheduler.skipUpdateCheck=true

in your quartz.properties file.

More discussions:
ehcache UpdateChecker: http://forums.terracotta.org/forums/posts/list/2701.page
quartz UpdateChecker: http://forums.terracotta.org/forums/posts/list/3395.page

Moving from tomcat to weblogic may sound crazy. In case you need to do it though (e.g for business reasons) here are a couple of things which may go wrong.

First of all the classloader hierarchy in weblogic do not do what you usually expect from other servers such as tomcat, resin, jetty and jboss. If your application uses hibernate (and implicitly ANTLR) you may get the following exception:

Caused by: java.lang.Throwable: Substituted for missing class org.hibernate.QueryException - ClassNotFoundException: org.hibernate.hql.ast.HqlToken [from com.example.model.Person order by id]
        at org.hibernate.hql.ast.HqlLexer.panic(HqlLexer.java:80)
        at antlr.CharScanner.setTokenObjectClass(CharScanner.java:340)
        at org.hibernate.hql.ast.HqlLexer.setTokenObjectClass(HqlLexer.java:54)
        at antlr.CharScanner.<init>(CharScanner.java:51)
        at antlr.CharScanner.<init>(CharScanner.java:60)
        at org.hibernate.hql.antlr.HqlBaseLexer.<init>(HqlBaseLexer.java:56)
...

As explained in the Hibernate3 Migration Guide Weblogic doesn’t seem to support proper class loader isolation, will not see the Hibernate classes in the application’s context and will try to use it’s own version of ANTLR.

In the same fashion you may get the following exception for commons lang:

java.lang.NoSuchMethodError: org.apache.commons.lang.exception.ExceptionUtils.getMessage(Ljava/lang/Throwable;)Ljava/lang/String;

because weblogic internally uses commons lang 2.1 and the one you use may have more API methods.

For both these problems the solution is to instruct weblogic to prefer the jars from the WEB-INF of your application. You need to create a weblogic specific file called weblogic.xml and place it under WEB-INF:

<?xml version="1.0" encoding="UTF-8"?>
<weblogic-web-app>
    <container-descriptor>
        <prefer-web-inf-classes>true</prefer-web-inf-classes>
    </container-descriptor>
</weblogic-web-app>

Another problem is that, like in resin, the default servlet is not named “default” so if you depend on it in web.xml, your application may throw the following at the deployment phase:

Caused by: weblogic.management.DeploymentException: [HTTP:101170]The servlet default is referenced in servlet-mapping *.avi, but not defined in web.xml.

This is because the default servlet is called FileServlet in the web.xml, so you’ll need to change all references in your web.xml from “default” to “FileServlet”.

Last, but not least, tomcat will automatically issue a 302 redirect from http://localhost:8080/context to http://localhost:8080/context/ before allowing your application to do any processing. So all instances of request.getServletPath() will never return an empty string, but will always start with “/”. Weblogic doesn’t do this so http://localhost:8080/context resolves and if your code contains something like:

request.getServletPath().substring(1)

you’ll get:

java.lang.StringIndexOutOfBoundsException: String index out of range: -1

so a safer way to trim this leading slash is by doing:

request.getServletPath().replaceFirst("^/", "")

Good luck, and remember. Every time you use a full blown application server for something that a simple web container would be enough, god kills a kitten.

You’ve done everything right. You are using of the JavaMail API with the correct settings and still it doesn’t manage to connect to the SMTP host to dispatch the email. You are on a windows machine and the exception looks like:

javax.mail.MessagingException: Could not connect to SMTP host: your.smtp.host, port: 25;
  nested exception is:
	java.net.SocketException: Software caused connection abort: connect
	at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1545)
	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:453)
	at javax.mail.Service.connect(Service.java:291)
	at javax.mail.Service.connect(Service.java:172)
	at javax.mail.Service.connect(Service.java:121)
	at javax.mail.Transport.send0(Transport.java:190)
	at javax.mail.Transport.send(Transport.java:120)
        ...
Caused by: java.net.SocketException: Software caused connection abort: connect
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:519)
	at java.net.Socket.connect(Socket.java:469)
	at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:267)
	at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:227)
	at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1511)
	... 40 more

One possibility is that something is blocking JavaMail from connecting to the local or a remote SMTP host, and this something can be an anti-virus.

via: http://forums.sun.com/thread.jspa?threadID=590866

p.s Why would a sysadmin want a resident shield anti-virus on a production box serving web content via tomcat still remains a mystery to me

SMTPAppender for log4j is a type of appender which sends emails via an SMTP server. It’s very useful for applications released in production where you’d definitely need to know of all application errors logged. Of course every caring developer should look at the server logs every now and then, but if you’ve got hundreds of them (applications) then it becomes a full time job in itself.

Sometimes a fresh release of a high traffic website may produce hundreds or thousands of ERROR level log events. Many times this may be something minor which is being logged deep inside your code. Until the bug is fixed and a new release is deployed, your inbox and the mail server may suffer heavily.

What follows is an extension of SMTPAppender which limits the amount of emails sent in a specified period of time. It features sensible defaults which of course can be configured externally via the log4j configuration file.

package com.cherouvim;

import org.apache.log4j.Logger;
import org.apache.log4j.net.SMTPAppender;

public class LimitedSMTPAppender extends SMTPAppender {

    private int limit = 10;           // max at 10 mails ...
    private int cycleSeconds = 3600;  // ... per hour

    public void setLimit(int limit) {
        this.limit = limit;
    }

    public void setCycleSeconds(int cycleSeconds) {
        this.cycleSeconds = cycleSeconds;
    }

    private int lastVisited;
    private long lastCycle;

    protected boolean checkEntryConditions() {
        final long now = System.currentTimeMillis();
        final long thisCycle =  now - (now % (1000L*cycleSeconds));
        if (lastCycle!=thisCycle) {
            lastCycle = thisCycle;
            lastVisited = 0;
        }
        lastVisited++;
        return super.checkEntryConditions() && lastVisited<=limit;
    }

}

The configuration would look something like this:

log4j.appender.???=com.cherouvim.LimitedSMTPAppender
log4j.appender.???.limit=3
log4j.appender.???.cycleSeconds=60
log4j.appender.???.BufferSize=25
log4j.appender.???.SMTPHost=${mail.smtp.host}
log4j.appender.???.From=${mail-sender}
log4j.appender.???.To=${sysadmin.email}
log4j.appender.???.Subject=An error occured
log4j.appender.???.layout=org.apache.log4j.PatternLayout
log4j.appender.???.layout.ConversionPattern=%d{ISO8601} %-5p (%F:%L) - %m%n
log4j.appender.???.threshold=ERROR

The above configuration will limit the mail dispatch to only 3 emails per minute. Any further errors in that minute will not be emailed. The limit and cycleSeconds setting lines can be omitted and the defaults will be applied.

Happy logging!

If you are using an entity as a named parameter in a hibernate Query or Criteria which is cachable from ehcache then this entity needs to implement hashcode and equals using a business key. Otherwise the hibernate Query or Criteria may always “look different” to ehcache so it will be a constant cache miss.

DEBUG (MemoryStore.java:138) - query.FooBarCache: query.FooBarMemoryStore miss for sql: /* criteria query */ select this_.id as y0_ from foobars this_ where this_.state=?; parameters: LIVE; max rows: 1; transformer: org.hibernate.transform.PassThroughResultTransformer@294633f0
DEBUG (Cache.java:808) - query.FooBar cache - Miss

If you experience the following warning in your hibernate+ehcache application:

2008-11-20 13:02:42,937 WARN  (CacheManager.java:322) -
  Creating a new instance of CacheManager using the diskStorePath
  "D:\apache-tomcat-5.5.26\temp" which is already used by an
  existing CacheManager.
The source of the configuration was classpath.
The diskStore path for this CacheManager will be set to
  D:\apache-tomcat-5.5.26\temp\ehcache_auto_created_1227178962937.
To avoid this warning consider using the CacheManager factory
  methods to create a singleton CacheManager or specifying a
  separate ehcache configuration (ehcache.xml) for each
  CacheManager instance.

then you need to set the following in your hibernate.cfg.xml file:

<property name="hibernate.cache.provider_class">
  net.sf.ehcache.hibernate.SingletonEhCacheProvider
</property>

Ehcache Hibernate documentation

I’m not ashamed of those sins any more, so here you go :)

1. ORM

Stupidity

Building my own Object Relational Mapping framework.

Consequence

Project is a mess after 2 years of maintenance with hardcore hacks to bypass my own ORM and call custom SQL queries.

What should I have done

Use hibernate, iBATIS, Cayenne or something similar.

2. EAV

Stupidity

Using an Entity-Attribute-Value model database schema design.

Consequence

Non scalable solution and total impossibility to run any useful queries on the database level.

What should I have done

Use an ordinary normalized database schema design.

3. Database Access

Stupidity

Synchronize (serialize) database access using one shared connection.

Consequence

Zero scalability. Very slow response times when more than 10 users where using the application.

What should I have done

Don’t do that and use a connection pool such as c3p0 and use a “new” (reused) connection returned from the pool for every request/response cycle.

4. IDE

Stupidity

Avoided learning and using an Integrated development environment.

Consequence

Inability to build test and deploy the application quickly and generally do anything useful.

What should I have done

Get familiar with an IDE. NetBeans, eclipse etc.

5. Transactions

Stupidity

Not using them.

Consequence

Corrupt data in an application involving e-shop like functionality.

What should I have done

Use database transactions. When in MySQL use InnoDB.

6. Prepared Statements

Stupidity

Using Statements, string concatenation and naive character escaping to assemble my own “safe” queries.

Consequence

SQL Injections possible in my application. I managed to login using ‘or 1=1;delete from users;– and alter the database state in a very nasty way.

What should I have done

Use Prepared Statements which correctly assemble and escape the query properly depending on the JDBC driver used.

7. Business Logic

Stupidity

Doing it in the template (JSP).

Consequence

Messy non maintainable application.

What should I have done

Do it in an MVC style with servlets or with a Front Controller. Even better by using an existing open source MVC framework such as Struts, Spring MVC etc.

Of course, all the bad choices above have probably made me a better programmer.

What follows is usual modifications I apply to java code, when I code review. Most of the following changes have a stylistic nature, but some of them can impact performance. Bad code is in red and good code is in green. Feel free to comment.

Unnecessary if statement

if (foo>bar) {
  return true;
} else {
  return false;
}

return foo>bar;

Unnecessary nesting

if (foo) {
  if (bar) {
    // do something
  }
}

if (foo && bar) {
  // do something
}

A bit of redundancy here

if (foo!=null && foo.equals("bar")) {
  // do something
}

if ("bar".equals(foo)) {
  // do something
}

Looks ugly

String comment = request.getParameter("post");
if (comment==null) {
  comment = "n/a";
}

String comment = request.getParameter("post")!=null?
        request.getParameter("post"):
        "n/a";

This only holds if request.getParameter("post") is cheap.

Unnecessary variable allocation

Something something = new Something(foo);
return something;

return new Something(foo);

Too much variable scope

Something something = new Something();
if (foo) {
  something.prepare();
  request.setAttribute("something", something);
}

if (foo) {
  Something something = new Something();
  something.prepare();
  request.setAttribute("something", something);
}

Method exposes implementation

public ArrayList getSomething() {
  ...
}

public List getSomething() {
  ...
}

A bit slow

String foo = "something";
if (!foo.equals("")) {
  // do something
}

String foo = "something";
if (foo.length()>0) {
  // do something
}

Too many lines

List immutableList = new ArrayList();
immutableList.add(foo);
immutableList.add(bar);
immutableList.add(example);
Something something = new Something(immutableList);

Something something = new Something(
        Arrays.asList(new Object[] {foo, bar, example}));

When generating the database schema using the hbm2ddl tools, foreign keys are being created for every relation. You can enhance the schema by using the foreign-key attribute to specify your own name for a particular foreign key.

But, what happens when you don’t want a foreign key generated?

There is an undocumented behavior of the foreign-key attribute, and that is to specify foreign-key=”none”. hbm2ddl will not create an FK for a relation which has such an attribute. The hibernate documentation and the two bibles Hibernate in Action and Java Persistence with Hibernate state absolutely nothing about this feature. I found it after hardcore googling in the following hibernate changelog:

Changes in version 2.1.9 (xx.x.xxxx)
------------------------------------
* foreign-key="none" can be used to disable generation of a foreign key.

Later on, I found another blog mentioning this behavior: http://blog.xebia.com/2007/02/05/let-hibernate-connect-your-world/

The most logical question now is “why don’t you want an FK?”. That depends on the system you are building. Sometimes you might have a table in your application which will be CRUDed from another system, which you do not control. You may want to be able to keep references (ids) to entities which may be deleted. foreign-key=”none” together with not-found=”ignore” can solve these kind of problems.