in Coding, Java

Effective error handling for ActiveMQ topics

No matter what type of software you make, there will always be errors. There might be bugs in your code or an external system is just down at the moment. No matter what it is, it will cause you trouble.

Asynchronous messaging (with JMS) might help to deal with such situations. Especially when you depend on external systems. Instead of making a synchronous call (like a REST call), you just send a message to a queue. As soon as the message is out, the transaction management will assure that it is consumed successfully. Otherwise an error handling process will kick in.

In the following I want to give an example on how we deal with errors in our project with ActiveMQ.

Our system

Our system is composed of multiple microservices. Those services are event-driven and communicate with messages send over ActiveMQ. Since a single event is of interest for more then one system, we decided to use topics instead of queues. Since all parts of our systems belongs together, we use a single topics with typed messages. This means all events will be written to the same topic and all listeners will consume from that topic. However, listeners will filter for certain message types which they use (instead of getting every message which comes around).

Virtual Topics

The first action we took to improve our error handling was switching from normal topics to virtual topics. In contrast to normal topics, virtual topics will dispatch messages to physical queues – one for each consumer. This allows us to use topics but to have an individual error handling for every consumer. Another benefit is that consumer will become persistent. This means, even if a consumer is offline, the message will be delivered to its queue. When the consumer comes back online, it can consume all (old) messages. Nothing will be lost.

Read more in one of my previous posts:

Redelivery Policy

Our second step was to think about a redelivery policy. A redelivery policy comes into play as soon as the delivery of a message fails. ActiveMQ will automatically try to deliver the message again. The number of retries as well as the time period between every redelivery can be configured.

However, we came to the conclusion to stick with ActiveMQ’s default settings. The only change we made was to set the queue name (as shown below). This assures that if the message cannot be delivered finally, that the queue name is shown in AcitveMQ’s web interface.

Dead Letter Queues

If a message cannot be (re-) delivered finally, ActiveMQ will put it on a dead letter queue. This means that the message is moved to a special place where we can manually deal with the error. We can look into the message, analyse the bug and fix it. Afterwards, the message can be put back on the regular queue of the failed consumer. As we use virtual topics, this will not affect the other consumer as each of them has its own physical queue.

By default, ActiveMQ will put all failed messages to a single dead letter queue. This makes it hard to see which consumers have failed. Because of that, we decided to use individual dead letter queues for each consumer. Failing consumers will be easy to spot in ActiveMQ’s web console.

We used the following configuration in our activemq.xml:

Conclusion

By applying three simple techniques, we build an effective error handling for our event-driven application with ActiveMQ.

  • Virtual topics provide an individual error handling for consumer as well as persistent delivery.
  • Redelivery policies will automatically give consumers a second chance.
  • Individual dead letter queues make problems easy to spot and messages easy to recover.

Read more

My previous posts:

ActiveMQ documentation:

Best regards,
Thomas