How can I set up main and failover MQTT subscribers for a job queue with AWS IoT?

Question

I have an system where a client (let's call it ClientA) can publish requests to a particular MQTT topic. The broker, in case it matters, is Amazon Web Services. Then I have another client (let's call it MainSubscriber) which is always subscribed to the same topic so that it can pick up requests from ClientA and do some work that, in the end, turns into a database operation. The database, in case it matters, is DynamoDB.

Since the MainSubscriber may not be always accessible/online, there is a desire to have a failover subscriber to be the failover backup of the main subscriber. The idea is that if the main subscriber does not handle the request in a timely manner, then the failover subscriber would kick in and do the equivalent work/database operation. The challenge is that the "work" and the resulting "database operation" must not be duplicated by both main and failover subscribers.

Here's a logical system architecture drawing for this system.

                   -----> MainSubscriber ----
                  /                          \
ClientA --> Broker                            ---> Database
                  \                          /
                   ---> FailoverSubscriber --

Clearly, there are some challenges with such a system:

1. How does the main subscriber indicate to the failover subscriber that it is working on the request?
2. How does the failover subscriber detect that the main subscriber has not picked up the request and needs to start working on it?
3. How does the failover subscriber then hold off the main subscriber in case it all of a sudden comes back online and picks up the request?
4. How to deal with synchronicity issues between main and failover subscribers?

I would rather not have to reinvent the wheel if an existing solution already exists for such a scheme. So, my first question is whether there is something out there already?

If not, then I was thinking of using DynamoDB with Strongly Consistent reads to act as the mediator between the Main and Failover subscriber. So, my second question is whether there any well established schemes for doing this?

Answer 1

According to AWS SQS Documentation (as you said the broker is AWS) this should be native:

Immediately after the message is received, it remains in the queue. To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consuming components from receiving and processing the message.

The problem being finding the right visibility timeout according to your maximum processing time.

You still have a small chance both subscriber process the same message, in this case your subscriber code should try to create idempotent output for the database (same primary key at least) and should handle gracefully a fail when trying to insert the same record.

Answer 2

You might want to look at the concept of dead-letter queues of AWS SQS. From the AWS docs:

A dead letter queue is a queue that other (source) queues can target for messages that can't be processed (consumed) successfully. You can set aside and isolate these messages in the dead letter queue to determine why their processing did not succeed.

So, if you point the main subscriber to listen from the normal queue and the secondary subscriber to listen from the dead-letter queue, the failover problem should be solved.

Also, with this, 1, 2 and 3 of your problems are taken care of. The main and secondary subscribers don't need to talk to each other in this case.

Also, building upon Tensibai's answer, make sure your subscriber code is written so as to receive one message at a time if multiple subscribers are listening to the same queue due to the visibility timeout

---

Downside would be that it would introduce a delay in processing, messages enter the dead letter queue only after a while.

So, in case you wouldn't want that, then you can go ahead with Tensibai's answer. And if you can tolerate that, instead of having an extra Dynamo table for status checks, then you can use this.