Good transaction management isn’t simply a matter of proper configuration of your EJB deployment descriptors. There’re many more factors involved that may influence the proper behaviour of your transactions.
Author: Keesjan van Bunningen (photo), Finalist IT Group, e-mail:
, tel: +31 10 217 08 00
Understanding the default behaviour of the EJB container is an essential part in building robust transactions. Particularly when a transaction enters into an error condition, it is important to know how the container will handle this condition. For instance, in case of an uncaught runtime exception in the onMessage of a Message-Driven Bean, the EJB container will roll back the transaction which may or may not be what you desire.
Time-out behaviour is also a key factor to successfully master transactions. There’re many different levels in a total transaction where time-outs are configured and/or may occur; in any of the participating (back-end) systems, the EJB container, at the level of connection resources, and on the level of the EJB itself. Each of these settings and/or occurrences may have impact on any of the process steps inside a transaction. And this may not always be apparent to the casual observer (read: developer). When properly configured, the time span of the time-out value only becomes shorter for each consecutive process step. This ensures that the caller always waits longer than the maximum time that may elapse before finishing (or rolling back) the process step, before giving up. Not applying this basic principle can (or better said: will) cause erratic operation of your process.
Read about a real-life example here.
How your project can turn into a Whodunit when your transaction goes wrong.
The last couple of months I was forced to dig deep into the topic of EJB transaction management due to problems we encountered on our current project. There are plenty of text book cases on this topic, but nothing can quite prepare you for the real world issues that may arise. This article touches on some of these practical issues and shows that the day-to-day job of a software developer sometimes may look more like that of a detective.
Our project involves transaction requests on (amongst other back-end systems) a non-transactional back-end system. The recurring problem that kept cropping up over and over again was that of duplicate transaction requests sent to this back-end system.
The handling of an average transaction request in the application consists of the execution of a (limited) number of process steps. Basically, each process step is handled by a separate Enterprise Java Bean (EJB). In this particular case, the request was first put on a JMS queue and later picked up by a Message-Driven Bean (MDB) to be passed on to various EJBs for further processing.
Case 1: The EJB container did it The first time we were confronted with duplicate requests, an examination of the log file quickly made it obvious that our application ran into an uncaught runtime exception while executing the onMessage method of this MDB. The standard behaviour of the EJB container when encountering such an occurrence is to rollback the transaction. The developer of this MDB was unaware of this default behaviour and also hadn’t anticipated the (unchecked) runtime exception that brought this first bug to our attention.
The bug was easily repaired by surrounding the content of the onMessage method by a catch-all try-catch block with appropriate error handling. After testing the bugfix, the application was rolled-out to production and all-was-well, or so we thought. It wasn’t until a couple of days later that a business representative asked me whether we had actually put the bugfix in production, because it seemed as if the same bug somehow had reappeared.
Case 2: It was the connection pool manager This time around, it wasn’t as straightforward as the first time and it took considerably more effort to figure out what actually was going wrong. The initial impression was that the transaction took an extremely long time to execute in the back-end system. Could it therefore be poor performance of the back-end system that was to blame?! More elaborate logging was put into place and disproved this assumption. Only after incorporating even more debug logging, an extensive analysis of the application log file revealed that the problem was in fact a classic example of a scarce resource caused by, in this case, inappropriate (unnecessary) use of synchronized methods.
To connect to the back-end system, a connection pool was used. Unfortunately, it turned out that the connection pool manager was badly designed. It was not only used for retrieving a connection in a synchronized way, it was also implemented to close a connection by way of a synchronized method. In addition, the connection pool manager was designed to occasionally perform a clean-up of its pool. This process also locked the pool manager for the duration of the iteration over the pool while testing each free connection and throwing away the dead ones. This could cause an (otherwise successful) transaction to be blocked before it could close its connection and release it to the pool, eventually causing the transaction to roll back.
This situation only occurred when the application received a considerable amount of load, causing the connection pool manager to become a scarce resource. This problem was resolved by refactoring the code, and calling the close on the connection directly, instead of via the connection pool manager.
Case 3: It was the configuration after all Another cause of duplicate transactions turned out to be inappropriate configuration of time-out values. These settings may exist at many different levels; in any of the participating (back-end) systems, the EJB container, at the level of connection resources, and on the level of the EJB itself. Each of these settings may have impact on any of the process steps inside a transaction, which may not always be apparent to the casual observer (read: developer). However, it can have a big effect on the smooth or erratic operation of your process. When properly configured, the time span of the time-out value only becomes shorter for each consecutive process step. This ensures that the caller always waits longer than the maximum time that may elapse before finishing (or rolling back) the process step, before giving up.
Case 4: When duplicate requests are unavoidable After eliminating all the causes inside our application, we still encountered conditions out of our control (disruption of network communication) that caused the application to receive duplicate requests from an external system. This proofs the point of defensive programming and the only way to resolve this issue was to check the uniqueness of the request before start processing it.
As I hopefully have demonstrated with this story is that transaction management isn’t simply a matter of proper configuration of your EJB deployment descriptor (although it helps). There’re many more factors involved that can wreck your transaction. A solid understanding of the transactional behaviour of all the participating components is essential to design and build a robust enterprise application.
