Don’t subscribe to # - so how to dump all messages to database with Mosquitto?

Question

HiveMQ's blog lists under "best practices" not to subscribe to the multi level wildcard when attempting to dump all messages to a database. They claim that the subscribing client may not be able to keep up with a high load of messages and propose to use a broker plugin to directly hook into the stream of messages instead.

Sometimes it is necessary to subscribe to all messages, which are transferred over the broker, for example when persisting all of them into a database. This should not be done by using a MQTT client and subscribing to the multi level wildcard. The reason is that often the subscribing client is not able to process the load of messages that is coming its way. Especially if you have a massive throughput. Our recommended solution is to implement an extension in the MQTT broker, for example the plugin system of HiveMQ allows you to hook into the behavior of HiveMQ and add a asynchronous routine to process each incoming message and persist it to a database.

Is there either

• a similar system (extension/plugin) for the mosquitto broker,
• another recommended method that works with mosquitto, or
• reasonable evidence that this approach is unnecessary at all, i.e. that a client subscribing to # can do just fine?

---

^{https://stackoverflow.com/q/31584613/3984613 does not address this question exhaustively.}

Answer 1

a similar system (extension/plugin) for the mosquitto broker

As far as I know there is no plugin/extension for mosquitto broker ( at least no opensource one )

another recommended method that works with mosquitto

Well I can say per my experience with Mosquitto broker and AWS IoT, you can just directly subscribe to '#'

Reasonable evidence

After looking at this question, I was a bit curious to know the throughput limits and to find out whether there is a need for an extension system. So I set up the following:

• 100 AWS Lambda functions that act as virtual end devices to send some random data to the Gateway( EC2 instance t2.nano 500MB RAM )
• Every 60 seconds functions get triggered to publish data to the gateway to different topics ( lambdatoec2/{VariableTopicNumberFrom1-100}
• EC2 instance is running Mosquitto 1.4.10

As of now, I see that there is no problem subscribing to # without any extension system. But again I still have to test for few edge case scenarios(I will update answer once I will test them).

Answer 2

This discussion on the openHAB mailing list seems to suggest there's no issue with using # as a subscription to receive all messages:

Whilst troubleshooting MQTT devices, it occurred to me that sometimes I wish I could see all MQTT messages that the Mosquitto broker sees, instead of a on a specific topic. Is there a way to do this?

Somebody answered this question for you on the Mosquitto list; use a wildcard. (#)

This Stack Overflow question also suggests the same method:

Subscribing to # gives you a subscription to everything except for topics that start with a $ (these are normally control topics anyway).

It is better to know what you are subscribing to first though, of course, and note that some broker configurations may disallow subscribing to # explicitly.

As pointed out by Bence Kaulics, the specification does state that # is valid:

Non normative comment

• “#” is valid and will receive every Application Message

Honestly, I dispute whether the original claim really makes much sense at all:

The reason is that often the subscribing client is not able to process the load of messages that is coming its way.

If that's the case, how could the broker handle the messages in the first place? As long as your client has similar performance characteristics to the broker, I strongly doubt it would be possible to overwhelm the client, because that level of traffic would also overwhelm the broker and cause that to crash first.

In summary, the HiveMQ claim doesn't seem to be supported by much evidence from other sources and, when you consider what it would actually mean, it doesn't seem particularly logical.

Answer 3

I think it is important to consider that there are many different use cases for MQTT brokers, as with any piece of software.

Handling chat messages for a billion users (many users, relatively low message rate per user) is different to a system with few clients but a high message rate, and they are both different to a home automation system (few clients, low message rate).

HiveMQ are thinking about the very high client/message rate applications - in which case the capability of the broker almost certainly far exceeds that of a client.

If you want to subscribe to # in your home automation system then it's really unlikely to cause problems. You can check and see if the broker is using excessive CPU in any case.

As in the other answers, subscribing to # will give you all 'normal' topics, that is anything that doesn't start with a $. I interpret the spec as saying that each topic beginning with $ is a whole separate tree in itself, so you'd have to subscribe to $SYS/#, $whatever/# to get everything. You most likely don't want to do that anyway for a normal application.