Performance Tuning in JBoss AS

JBoss is an open source Java EE application server widely used for web development, testing and deployment. In this tutorial, I share some performance tips I recently applied on a number of JBoss servers for a telco company.

Performance has a significant impact on user experience, business and productivity, not to mention savings on hardware and software resources. For details on performance optimization and its benefits, I recommend this article.

Tuning is all about identifying bottlenecks! Performance bottlenecks are one of the nastiest to resolve: it could be due to poor AS configuration, bad coding, slow SQL statements, hardware limitations, etc. Surely though you'll aim to get the most out of your company's middleware investment :-) so upgrading the hardware is not an option.

This tutorial applies to JBoss version 4.x. Consider making a backup before changing any of your AS scripts.

General information before we start

To recap: processors, operating systems and applications can all be of 32 or 64 bit, as a result we get 32/64-bit CPU, OS and JVMs. For a more detailed explanation check out this article

If the processor (CPU) is 32-bit then the OS must also be 32-bit, however, if the CPU is 64-bit, then you can run both 32 and 64 bit OSes, just bear in mind that running a 32-bit OS on a 64-bit CPU means your memory can only go as high as 4GB (half of a 64-bit OS). In other words, The OS can only be installed if the hardware architecture supports it.

to find out whether the CPU is 32 or 64 bit, run the following on your Linux shell:

grep flags /proc/cpuinfo

lm (long mode): 64-bit
Protected mode: 32-bit
Real mode: 16-bit

Now, check whether your OS is 32-bit or 64 bit. Again from the Linux shell:

uname -m

Finally, try running your JVM in 64-bit mode:

java -d64 -version

1. Tuning JVM parameters

In JBoss, set your JAVA_HOME to point to your 64-bit JDK otherwise JBoss will probably run on its default 32-bit JVM. When you start JBoss, check the [Server Info] console log, It should say "64-bit Server VM", if not, you can activate the 64-bit JVM by adding the following inside run.conf file:

JAVA_OPTS="-D64..."

The run.conf file contains JVM parameters including memory configuration:

set JAVA_OPTS="$JAVA_OPTS -Xms3g -Xmx3g"

Note: Be careful not to set it too high, you need to give space to the native JVM.

Consider having fixed heap settings (same value for -Xms and -Xmx). If, for example, you only set "-Xmx" then the VM starts with a much smaller heap and re-sizes it. If your system, however, is under load for a relatively short period of time then you might want to set a smaller initial heap size so it can grow on demand. The actual heap size setting will depend on your application requirements and server RAM.

2. Garbage Collection

By default, JBoss RMI subsystem runs a full garbage collection every minute. Whether you're using RMI or not, all threads are paused and the entire heap is scanned. This can take a good time specially under load. You can change this behaviour to once an hour in run.conf.

This is set in the same way as the JVM parameters:

set JAVA_OPTS="... -Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000"

3. Stack Memory

Unlike the heap memory (whose size is controlled via -Xmx), the thread stack memory is not collected by the GC. JVM errors such as StackOverflowError and OutOfMemoryExceptions can also be related to the stack memory (where primitive vars and method call goes). Add the following to run.conf:

set JAVA_OPTS="... -XX:ThreadStackSize=256"

Note: 256K is the size of the thread, you can set 128 for smaller apps

The default thread stack memory space on 64-bit JVMs is 1M and 512K for 32-bit JVMs. Refer to Java Thread Stack Sizing doc for more details.

4. Turn off Debug Information in JSP Classes

By default, Apache's JSP compiler (Jasper) in JBoss, will compile JSP classes in debug mode. In production systems you should turn this off to increase the resources available to your application.

Edit file $JBOSS_HOME/server/default/jboss-web.deployer/conf/web.xml Add extra initialisation parameter (classdebuginfo) to the JspServlet:

<servlet>
  <servlet-name>jsp</servlet-name>
  <servlet-class>
     org.apache.jasper.servlet.JspServlet
  </servlet-class>
  <init-param>
     <param-name>classdebuginfo</param-name>
     <param-value>false</param-value>
  </init-param>
</servlet>

5. Hot Deployment

It's recommended to disable Hot Deployment (HDScanner) in production as it can compromise performance and potentially expose security risks. Add the following in file: $JBOSS_HOME/server/default/conf/jboss-service.xml

<attribute name="ScanPeriod">BIG_NUMBER_IN_MS</attribute>

Note: Default is 5000ms (5 seconds)

Or alternatively, turn it off:

<attribute name="ScanPeriod">false</attribute>

6. Binding JBoss to IP addresses

As of version 4.2, JBoss binds all its services to the 'localhost' interface (127.0.0.1). Which means it can only be accessed from the localhost. This change was done for security reasons, previously it was bound to any IP address (0.0.0.0) by default.

On production, the default localhost interface (127.0.0.1) configuration is not practical as the server will be accessed by different client IPs. Below are 3 options you can utilise to bind JBoss services to a desired IP or any network interface (0.0.0.0):

* Passing a parameter to the startup script

./run.sh -b0.0.0.0

* Editing $JBOSS_HOME/bin/run.conf

Add the following at the end of the file: JAVA_OPTS="$JAVA_OPTS -Djboss.bind.address=0.0.0.0"

* Editing $JBOSS_HOME/server/default/deploy/jboss-web.deployer/server.xml

For all <Connector> elements, add the following attribute <Connector address="0.0.0.0"

Summarising

The above are just some of JBoss parameters you can play with. In my opinion, the most effective ones but for a comprehensive list of parameters please refer to the "Further Reading" section below. There's plenty of more stuff out there.

Note that while optimizing your application, as soon as one area improves another can become a bottleneck, therefore it's important to keep testing the application under stress conditions for each and every change. Performance tuning is an iterative process, you must repeat the cycle until you're happy with the results. Such tests are accomplished using load test tools like Apache JMeter and HP LoadRunner. I'm more familiar with Grinder but in case you're wondering which one to go for, here is a list of open source performance test tools.

Hasta luego!

Further Reading
1. Best practices for performance tuning
2. JBoss Optimization 101
3. Solving common Java EE performance problems
4. User Guide - JBoss Installation and Tuning