I have an system where a client (let's call it ClientA) can publish requests to a particular MQTT topic. The broker, in case it matters, is Amazon Web Services. Then I have another client (let's call it MainSubscriber) which is always subscribed to the same topic so that it can pick up requests from ClientA and do some work that, in the end, turns into a database operation. The database, in case it matters, is DynamoDB.
Since the MainSubscriber may not be always accessible/online, there is a desire to have a failover subscriber to be the failover backup of the main subscriber. The idea is that if the main subscriber does not handle the request in a timely manner, then the failover subscriber would kick in and do the equivalent work/database operation. The challenge is that the "work" and the resulting "database operation" must not be duplicated by both main and failover subscribers.
Here's a logical system architecture drawing for this system.
-----> MainSubscriber ---- / \ ClientA --> Broker ---> Database \ / ---> FailoverSubscriber --
Clearly, there are some challenges with such a system:
- How does the main subscriber indicate to the failover subscriber that it is working on the request?
- How does the failover subscriber detect that the main subscriber has not picked up the request and needs to start working on it?
- How does the failover subscriber then hold off the main subscriber in case it all of a sudden comes back online and picks up the request?
- How to deal with synchronicity issues between main and failover subscribers?
I would rather not have to reinvent the wheel if an existing solution already exists for such a scheme. So, my first question is whether there is something out there already?
If not, then I was thinking of using DynamoDB with Strongly Consistent reads to act as the mediator between the Main and Failover subscriber. So, my second question is whether there any well established schemes for doing this?