Effective error handling for ActiveMQ topics

Thomas Uhrig · June 1, 2017

No matter what type of software you make, there will always be errors. There might be bugs in your code or an external system is just down at the moment. No matter what, it will cause you trouble.

Asynchronous messaging (for example with JMS) might help to deal with such situations. Especially when you depend on external systems. Instead of making a synchronous call (like a REST call), you send a message to a queue. As soon as the message is out, the transaction management will assure that it is consumed successfully. Otherwise, an error handling process will kick in.

In the following I want to give an example on how we deal with errors in our project with ActiveMQ.

Our system

Our system is composed of multiple microservices. Those services are event-driven and communicate via messages send over ActiveMQ. Since a single event is of interest for more than one system, we decided to use topics instead of queues. Since all parts of our system belong together, we use a single topics with typed messages. This means that all events will be written to the same topic and all listeners will consume from that topic. However, listeners will filter for certain message types which they use (instead of getting every message which comes around).

Virtual Topics

The first action we took to improve our error handling was switching from normal topics to virtual topics. In contrast to normal topics, virtual topics will dispatch messages to physical queues - one for each consumer. This allows us to use topics but also to have an individual error handling for every consumer. Another benefit is that consumers will become persistent. This means, even if a consumer is offline, the message will be delivered to its queue. When the consumer comes back, it can consume all (old) messages. Nothing will be lost.

Redelivery Policy

Our second step was to implement a redelivery policy. A redelivery policy comes into play as soon as the delivery of a message fails. ActiveMQ will automatically try to deliver the message again. The number of retries as well as the time period between every redelivery can be configured.

However, we came to the conclusion to stick with ActiveMQ’s default settings. The only change we made was to set the queue name (as shown below). This assures that if the message cannot be delivered finally, that the queue name is shown in AcitveMQ’s web interface.

@Bean
public RedeliveryPolicy redeliveryPolicy() {
    RedeliveryPolicy topicPolicy = new RedeliveryPolicy();
    topicPolicy.setQueue("Consumer.myConsumer.VirtualTopic.MY\_TOPIC");
    return topicPolicy;
}

Dead Letter Queues

If a message cannot be (re-)delivered finally, ActiveMQ will put it on a dead letter queue. This means that the message is moved to a special place where we can manually deal with the error. We can look into the message, analyse the bug and fix it. Afterwards, the message can be put back on the regular queue of the failed consumer. As we use virtual topics, this will not affect the other consumer as each of them has its own physical queue.

By default, ActiveMQ will put all failed messages to a single dead letter queue. This makes it hard to see which consumers have failed. Because of that, we decided to use individual dead letter queues for each consumer. Failing consumers will be easy to spot in ActiveMQ’s web console.

We used the following configuration in our activemq.xml:

<policyEntry queue=">">
  <deadLetterStrategy>
      <individualDeadLetterStrategy queuePrefix="DLQ." useQueueForQueueMessages="true"/>
  </deadLetterStrategy>
</policyEntry>

Conclusion

By applying three simple techniques, we build an effective error handling for our event-driven application with ActiveMQ.

Virtual topics provide an individual error handling for consumer as well as persistent delivery.
Redelivery policies will automatically give consumers a second chance.
Individual dead letter queues make problems easy to spot and messages easy to recover.

My previous posts:

ActiveMQ documentation:

Best regards, Thomas.

Thomas Uhrig's Blog