In this article, I am going to discuss on how to implement “Retry Pattern” with Microsoft Azure Service Bus. 

Intended Audience: Anyone who is excited to learn the significance of “Retry Pattern” implementation using Microsoft Azure Service Bus. We are not going to create any topics or queues in this article. We will just go through the pattern implementation.

Most of the modern applications in today’s world are distributed across the globe and they all need to talk to each other in processing the transactions. There are two ways that any two systems can talk. One is point to point communication where one system calls another system and they talk to each other. But this unlikely is going to happen in the modern world. Other one is asynchronous or disconnected talks through some other medium. That medium takes responsibility of providing the information to the intended recipient when the recipient is ready to process it. Also, information provided by one system may have many interested parties and one system cannot talk to all the interested parties to provide the same information and all interested parties may not be ready at the same time to receive the information. Owing to this topology of systems and their preferences, asynchronous communication is the only way for the successful communication between the systems.

In software world, the medium is called “Broker” and who ever provides the information or data is called “Publisher” and all the interested parties for the information or data is called “Consumers” or “Subscribers” and the information is sent in the form of “message”. 

Many vendors provide the message broker functionalities like RabbitMQ, ActiveMQ, IBM MQ and so on. Each messaging broker has its specific features. But almost all of them provide basic features like durability, reliability, high-availability etc. Some might be supporting modern languages and interfaces whereas some might be supporting single language. But all must support the basic pattern i.e. Retry Pattern.

Since applications talk over network like inserting data to a database, posting a message to a HTTP endpoint within the same domain or outside or over the cloud. Communication over network is always prone to have some glitches and we cannot guarantee successful connections every time due to many known or unknown reasons. So, most of the applications built are designed to retry N number of times before concluding the transaction failure. Some applications do not want to retry immediately but after some time and they need some intermediate storage. 

Message brokers discussed before supports the intermediate storage in the form of queues. For example, if database is down and the ETA is 1 hour the application should have to retry again inserting into the database table after 1 hour for all the failed transactions. So, the messages for those failed transactions will be stored in designated queue and the designated queues have time-to-live (TTL) configured. TTL for 1 hour enforces that any message that is published to this queue is going to live for 1 hour and after that it has to be deleted. If these queues are configured with dead letter exchanges, then the message will be published to the dead letter exchange when TTL expires.

I am going to show how we can implement the Retry mechanism in RabbitMQ which a message broker built on erlang.

Typical Structure:

 A Virtual Host is like a virtual container that houses the Exchanges, Queues. In a typical architecture, if same message has to be sent to multiple subscribers, we create one queue for one subscriber and the routing key will be same for all the queues. If a publisher publishes a message with routing key as ‘P1’ at the exchange level, the same message is sent to multiple queues that has been configured with the routing key as ‘p1’.

Figure 1 – Exchanges and Queues in RabbitMQ

Two consumers are interested in same message and so consumer 1 has been designated with Queue1 to consume message and Consumer 2 has been designated with Queue2 to consume the message. For each designated queue, corresponding retry queue have been configured.

RetryQueue1 has been configured to send the message to Exchange1 with routing key as ‘P1’ so that the message will be published back to Queue1 and so on for RetryQueue2.

If consumer 1 has failed in completing the transaction, then the consumer publishes the message to exchange with routing key as ‘R1’ so that the message lands in RetryQueue1 and so for consumer 2.

So, the message will be delayed for re-processing again by the consumer when published to retry queue.

Note: RabbitMQ supports Dead Letter Exchangewith Routing Key so that both main and retry queues can be configured in the same exchange and the routing to queues was so simple. This structure of exchange/queues with dead letter configurations are supported by almost all the Message brokers. But Azure Service Bus support this in a different way as you cannot configure the dead letter topic with routing key. So, we need to look for alternative design approach for this reason.

There are two alternative solutions for implementing the retry using Microsoft Azure Service Bus:

1. Create a new topic for retry subscriptions
2. Create a new Queue for retry mechanism

Create a new topic for retry subscriptions:

As shown in the Figure 2, we now have to create two topics: One for the Main Subscription where publisher publishes messages and Error Subscription where the errored-out messages will be published by the consumer. Each subscription has their own filters which does not collide with each other. Second topic is for retrying mechanism and the subscription created underneath the Retry Topic will have messages that are supposed to be published back to Main Subscription once the message’s TTL expires.

Retry subscription will have its own filter condition to route message i.e. RoutingKey=’Retry’used by consumer when it could not complete the transaction and also the property to Forward Dead letter Messagesfor the subscription is configured with Main Topicso that the message will be routed to the Main Topiconce TTL expires. In order for the message to be routed only to Main Subscription, but not to Error Subscription, the message should have the header property Destination=’MainSub’which is the filterfor Main Subscription. In this way, we can accomplish the retry mechanism with using only a Topic.

Figure 2 – Topics with Main subscription, Error Subscription and Retry Subscriptions

The flow is as follows:

• Publisher publishes a message with header property Destination=’MainSub’ to the“Main Topic” and the message reaches to the subscription “Main Subscription” as the header property matches with the subscription’s filter condition.
• Filters are added to subscriptions using Rules.
• Another Subscription is also created to route error messages in the “Main Topic”with the name “Error Subscription” which has the filter Destination=’ErrorSub’.
• Consumer consumes the message from “Main Subscription” and performs the transaction. If transaction is successful, Consumer logs the success message and process next message.
• If there is a transaction failure, Consumer will publish the message to “Retry Topic” with two header properties: 
  • Destination=’MainSub’
  • RoutingKey=’Retry’
• Since the RoutingKey value matches with the Retry Subscription’s filter condition, the message will be routed to the Retry Subscription.
• Retry Subscription is configured with TTL to some X seconds and the property “Forward Dead Letter Messages To” is configured with the “Main Topic” and also the setting for “Dead Lettering On Message Expiration” is set so that when the message’s TTL expires it will move the message to the Dead Letter configured for the subscription.
• Now when the message’s TTL expires, the message will be routed to Main Topicand to the subscription Main Subscription as the message header has property Destination=’MainSub’.
• Now the message is available for the Consumer to consume and re-process it again.

Note:It is the responsibility of the Consumer to make count of the re-processing the messages using Retry by adding either count property in the header and increment it every time it re-process or track the message using message Id and insert the count data in some storage and if it has re-processed for N number of times and still the transaction is failed, then Consumer needs to publish to Error Subscription. Otherwise, the message will be re-processed infinite times and will cause the system to behave abnormal.

Create a new Queue for retry mechanism:

Figure 3 – Retry using Queue

Instead of creating a new Topic for Retry as explained in Figure 2, we can add a Queue for retry mechanism and the advantage of having the Queue is we no more need to worry about adding additional header property to route message as the Consumer will be able to publish message directly to the Queue. 

The Queue is configured with “Forward Dead Letter Messages To” Main Topic and TTL is configured. The flow is same as discussed for previous mechanism except that the Consumer publishes the message to be retried to a Queue instead of a Topic and the message should only need a header property Destination so that message will be routed to “Main Subscription”.

Rest all functionality will remain same.

Note: Here we are talking about simple retry mechanism. But, if a project needs little complex like Consumer talks to 5 systems and Consumer has to retry if it has failed transactions with 3 systems by either adding header property to the message that tells which system has the transaction failed with or need to create retry topic/queue for each system and need to publish the message to that topic/queue assigned for the system. But overall, any new scenario will be an extension to the implementation shown above.

Hope you enjoyed the article. If Microsoft can add an option to specify the filter or subscription name along with the topic for forwarding dead letter messages it would have made easier for us to implement by keeping all subscriptions in one Topic instead of multiple topics.