in Coding

Virtual Topics in ActiveMQ

A couple of days ago I published a post about the difference between queues, topics and virtual topics. Today I want to share some practical information on how to use virtual topics in ActiveMQ with Spring Boot.

Virtual Topics

Virtual topics are a combination of topics and queues. Producers will write messages to a topic while listeners will consume from their own queue. ActiveMQ will copy and duplicate each message from the topic to the actual consumer queues.

This combination of topics and queues has some advantages over conventional topics:

  • Even if a consumer is offline, no messages will be lost. ActiveMQ will copy all messages to the registered queues. As soon as a consumer comes back online, he can handle the messages stored on his queue.
  • If a consumer cannot handle a message, it will put on a dead letter queue. The consumer can be fixed and the message can be put back on his own dedicated queue without affecting the other consumer (as a topic would do).
  • We can register multiple instances of a consumer on a queue to implement a load balancing mechanism.

Using virtual topics with Spring Boot

Using virtual topics with Spring Boot is very straight forward. The configuration is based on conventions to which we will stick in the first part. However, you can easily change the conventions as we will see below.

Producer

From the point of view of the producer, virtual topics look exactly like normal topics. The only difference is the name of the topic which must have the syntax VirtualTopic.<name> by convention.

Consumer

On the side of the consumer, we find a similar naming convention. The destination must match the syntax Consumer.<name>.VirtualTopic.<name> by convention.

Conventions

By following these two simple naming conventions, ActiveMQ will automatically do all the magic in the background. Every message send to the virtual topic will be copied to each of the individual consumer queues.

Producer writes to: VirtualTopic.<name>
Consumer reads from: Consumer.<name>.VirtualTopic.<name>

Note that you need to explicitly state in you Java code that the destination of the producer is a topic (not a queue) by using new ActiveMQTopic(...) when sending the message.

Advanced configuration

The naming conventions above can be changed in ActiveMQ. To do this, we need to modify the activemq.xml. This means we cannot do this in our Java code – we must do it outside in our broker configuration! The configuration goes like this:

This small piece of configuration allows you to change the naming convention of your virtual queues. It also allows you to turn on the selector awareness which is de-activated by default.

Read more at http://activemq.apache.org/virtual-destinations.html.

Best regards,
Thomas

Write a Comment

Comment

18 Comments

  1. Hi,

    I did the annotation conguration as per the details mentioned.
    But when my producer send the message to Virtual Topic, the vitual topic enqueue this message but does not dispatch it the consumer.
    I have just one consumer and the consumer Queue has a subscriber as 1.
    Please help.Do I need to do any other configuration.??V

    • This looks like a configuration problem in ActiveMQ. If you are able to to send messages to the topic and to register a listener to the queue, the Spring part should be fine.

      – Did you follow the naming convention? VirtualTopic.?
      – Did you change anything in your activemq.xml?

  2. I’m concerned about lost messages when a virtual topic is marked as selector aware. What happens to messages when a consumer with selectors disconnects? Are those messages that arrive to the virtual topic lost (due to there being no consumers during that time frame)?

    • No, that’s not an issue. After a listener has registered once, its queue will receive all messages even if the listener is disconnected. So the messages will not be lost.

      • Thank you. I’ve observed somewhat opposite behavior to losing messages. In fact, it seems that when my consumers are disconnected, the virtual topic ends up getting some messages for which no consumers will have selectors. This results in the messages getting ‘stuck’ in that virtual topic until they are moved to the DLQ. This can be dangerous if too many appear because valid messages will be paged out if there are too many ‘orphaned’ messages.

  3. Hi Thomas,
    I tried out the following scenario,
    Selector Aware is enabled and registered subscriber is disabled.
    On the AMQ Web Console, observed that the messages are not getting persistent. Also, these messages are not received when the subscriber is started.

    Can you highlight any configurations that might lead to this scenario?
    Or if you can think of what possibly could have gone wrong, that would be even more helpful.
    FYI, using AMQ 5.15.2.

    Thanks a lot 🙂 Cheers!!

      • Hi Thomas,
        As per your suggestions in the above screens shots, I observed this is possible only when the selectorAware=”false” in activemq.xml file, if it’s toggled to true then messages are not persistent.

        Thanks

  4. Hi Thomas,
    We checked all the queues in ActiveMQ console and it’s looks exactly like you mentioned.
    On further scratching googles head, we found a solution.
    There’s a plugin called “virtualSelectorCacheBrokerPlugin” which needs to be configured in xml. Seems to work for my usecase.
    I’ll post the syntax for other users that need help with this.
    Here we go:

    Please review this, and shed some more light on it if you are familiar with the virtualSelectorCacheBrokerPlugin. Also let us know if you need more info regarding the same.

    We have one more query, is it possible to make certain subs non-durable in Virtual topic?

    Thanks for your input 🙂

  5. Hi Thomas,
    Do you have any solution for my problem because I’m stuck here:
    Problem: The messages (with no matching selector) in Virtual topic are not getting removed from queue.

    • I’ve two suggestions on this:

      #1

      Go to your activemq.xml and active the selector aware flag:

      By doing this, ActiveMQ will only dispatch messages to your queue for which a listener is registered. So you won’t have messages with no matching selector anymore.

      #2

      Write a single listener for your queue which receives all kinds of messages. Don’t use any selector in your listener anymore. Then do the dispatching of you messages by yourself and skip all messages for which you have no usage. See https://tuhrig.de/self-made-event-dispatcher for an example.

  6. Thanks for your response,
    I’m using virtual topic with multiple queues and each queue has a specific selector, when producer sends the messages to the virtual topic it will be delivered to the queue which satisfies selector condition (as per your first approach).

    To achieve message durability (when consumer goes offline) I configured virtualSelectorCacheBrokerPlugin in activemq.xml file.

    After applying above plugin, all the messages sent by producer are delivered to all the queues even though the selector condition does not match.

    Could you please suggest me any suitable approach for my use case?
    Thanks.

    • Hey Ranjit! Unfortunately, I’m not familiar with the virtualSelectorCacheBrokerPlugin plugin. However, message durability (when consumer goes offline) should be working without this plugin as I mentioned before (in my answer with the big screenshot). Good luck!

  7. Hi Thomas,

    I am trying to create virtual topics using go package stomp but not getting any success. Could you share any example to create virtual topics in golang ?
    I modified activemq.xml so that all my topics are virtual topics but when i create a topic, virtual topics are not created. Request your inputs.

  8. Hi Thomas,
    I got the fix in golang we need to user /topic/VirtualTopic. to create a virtual topic.

    Thanks,
    Manjit Singh

  9. Thomas Thank you very much for your help in resolving Virtual Topic.
    You should not have “spring.jms.pub-sub-domain: true” in your application.yml
    I wasted time because of this mistake.

    • Before you send data when using a virtual topic

      I wonder if I can make it by declaring the number of consumers beforehand.

      There are two consumers and ten messages.

      At first, one consumer connects to receive data up to 5 times, and after the consumer connects, I want to receive from 1 again

      If I have ever done this, can I get some advice?

Webmentions

  • Effective error handling for ActiveMQ topics – Thomas Uhrig August 7, 2018

    Before you send data when using a virtual topic

    I wonder if I can make it by declaring the number of consumers beforehand.

    There are two consumers and ten messages.

    At first, one consumer connects to receive data up to 5 times, and after the consumer connects, I want to receive from 1 again

    If I have ever done this, can I get some advice?