Alternatives to MQTT for local / remote bridging

Question

Amongst the plethora of MQTT questions, I am wondering what are some alternatives to MQTT for when all messages sent to a topic need to be kept, and in a queue for a new subscriber.

At my company, we have remote deployments that we manage, and we are wanting to use MQTT for local data collection. The idea would be that data would be sent to the local broker onsite (running on a Raspberry Pi, for example), and the broker would have an MQTT bridge with our CloudMQTT deployment. If connectivity would be lost, the messages would collect locally, and synchronize again when connectivity was re-established.

The set up is typical, like this:

For my example, on the left side would be many (around ~100) MQTT local brokers running at each location, and on the right would be the CloudMQTT server we pay for.

When I read the article MQTT Essentials Part 8: Retained Messages, this part was disappointing:

A retained message is a normal MQTT message with the retained flag set to true. The broker stores the last retained message and the corresponding QoS for that topic. Each client that subscribes to a topic pattern that matches the topic of the retained message receives the retained message immediately after they subscribe. The broker stores only one retained message per topic.

Essentially what this means is that there would have to constantly be a subscriber on the CloudMQTT server listening for all incoming events from all of our locations; otherwise, data might be lost.

MQTT seems built to only keep the most recent message; are there any other software packages that can do this local <=> remote syncing, but keep all messages?

Answer 1

You are trying to use a system that is not designed for your situation.

MQTT is designed as an:

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport.

Ref.: https://mqtt.org/

RabbitMQ is what you are looking for, it can be deployed in distributed and federated configurations. It also has an MQTT 3.1.1 plugin that ships in the core distribution.

Ref.: https://www.rabbitmq.com

Answer 2

MQTT will handle this for you, retained messages is not the right thing, you need to make use of the higher QOS values (either 1 or 2 rather than the default 0). You need to read part 6 of the sequence of articles you mentioned.

Retained messages solve a different problem, which is to allow new clients to always grab the most up to date information. High QOS will ensure delivery of all messages.

You can set the mosquitto local brokers up to bridge messages at QOS 1/2 regardless to what they are published at by the sensors. When the network goes down the broker will then queue all the messages until the bridge comes back and will then deliver them to the cloud broker.

Answer 3

Depending upon support in the MQTT broker, Persistent Sessions may work. When a client connection, they can set the clean session flag to False and then subscribe to the topics of interest. In the event the client loses connection, any messages published to those topics under QoS 1 or 2 will be queued. When the client reconnects, the subscriptions will automatically be established again the queued messages delivered.

It does require more thought and potential changes on the clients to deal with the queued messages.

Answer 4

Another option is to create your own service to handle it. The service needs to be connected to your local MQTT broker on local RPi (acts as local gateway) as you have specified. The local gateway generally used to provide data filtering, but you can use it for temporary storage of data that your device sends. You can use python or any programming language that support MQTT client library. The client service is nothing but another MQTT client running on gateway and can perform logic and database operations. This client service, that you build, receives the data as soon as it arrives on MQTT broker at local gateway, try to forward it to CloudMQTT. If it couldn't connect to CloudMQTT due to no Internet connectivity available, it stores the data to temporary database with flag to keep track whether that data is published to CloudMQTT or not. The client service (always running in background) try to fetch this data from database and resend it. If that data is successfully sent to CloudMQTT, the service updates the flag as the data is sent. You can later on delete all data from database whose flag value indicates that it is sent to CloudMQTT. So in this case you local RPi machine which runs a local broker (of course with database like SQLite) can actually acts as an intelligent gateway. This solution should give you ability to send all collected data even if there is no Internet connectivity for hours and even for a day or more because you have all the backup.