How to make a MQTT Broker and a REST API communicate?

Question

I'm still very new to IoT and IoT protocols, so pardon my ignorance.

I'm trying to go about creating a home automation system to control lights, read room temperatures etc.

The user should be able to interact with the IoT devices via a mobile client that is being developed with React Native. Could I implement a MQTT client on that platform? Yes, but I think is way more convenient to have a REST API between the mobile client and the IoT devices. This way I can easily manage user-created routines, add authentication to control certain devices and implement a lot of other functionalities. Also, some devices may use different protocols so I'll let my API take care of this instead of implementing multiple clients on the front-end.

I've found ways to "bridge" MQTT and HTTP transforming HTTP posts into MQTT publishes, but this isn't exactly an elegant solution. I want to use my server as a subscriber to MQTT topics and serve the information over HTTP to the mobile client, as well as publish topics when requests arrive or scheduled routines are triggered. How can I do this?

I'll be preferably running the MQTT broker and the REST API on the same device (a RaspberryPi) connected to the local network.

Answer 1

Here's one way:
The actual business logic of your system would have a certain API. That API needs to have some doXYZ methods, some getXYZ methods and some events of onXYZ.
What I'm assuming you're asking is "How do I make this available via MQTT and via HTTP based transports?"
If you were doing something in the cloud, I would say take a look at some technologies such as AWS AppSync adds Real-Time enhancements with Pure WebSockets support for GraphQL Subscriptions.

However, you are wanting to make it all! I would suggest still look at doing this on the cloud. But if you want to develop something in a cave without internet for example, I would suggest using a pubsub system like Redis that can take care of state and HTTP aspects and then building the MQTT publish. Have message format as JSON to make it easier.
I would ask you to seriously consider a few possibilities before you do all this:

1. Check out cloud based systems such as the one I mentioned above.
2. See if you can just do everything with one interface (MQTT, say) and see if you can find some good libraries for your frontend.
3. Instead of MQTT, can you do things with a pubsub mechanism like that exposed by Redis
4. See if NodeRed does parts of what you need.

Answer 2

Edit

I've come across this stack overflow post. In case the link is dead: In short, the post describes a way to use JWT to authenticate a frontend user with your backend and mqtt broker.

While it doesn't really cover your question, I think it still is useful. Specifically, it solves your problem of authenticating a client and controlling access to topics. The mosquitto broker, compiled with mosquitto-go-auth, would be an example of how you could implement the architecture described in the post.

Further building on this architecture, to support multiple clients you could develop some sort of gateway from mqtt to other protocols. E.g. a backend service that connects to your mqtt broker and listens for messages on topic newprotocol. When it receives such message it translates the message content to your desired protocol. Your frontend clients would only need to speak http and mqtt then.

---

I'm trying also trying to build a DIY home automation system and I would very much appreciate it if you could keep this question up-to-date with your progress.

That being said, I found a few useful links that you'll maybe also find useful:

• Article from 2014 explaining a way to combine REST and MQTT
• MQTT-REST Bridge Presentation (Mentioned in the article)
• Redis' PUB/SUB messaging paradigm

The gist of the article is:

Essentially, you start with a normal REST API and add MQTT messages for REST endpoints that result in a state change (POST/PUT/PATCH/DELETE).

Then add to this approach, anytime a IOT device changes state, publish the new state to a REST endpoint.

What I've tried

I hope you'll let me share my thoughts on how to build a system like this and I also hope that this'll be useful to someone. One way I thought about structuring this system is to have 3 components: Clients, REST-API and MQTT-Broker.

In my case I have a react app running in the browser. It makes API calls to the REST backend (i.e. GET all IOT devices, POST new message to IOT device, etc.). The backend handles authentication, the PUB/SUB messaging with the broker (i.e. Handle IOT device status changes, find out what IOT devices exists, etc.) and any messages coming from the react app.

However, I ran into problem with this approach. How do you let the react app know the state of an IOT device changed while using a stateless protocol (HTTP in this case)? I thought about using websockets to transmit changes to the react app but I ended up not implementing it and went back to the drawing board. And that's were I am now. Another issue with this is that I would be restricting myself to use MQTT for every IOT device. But what if a device doesn't talk MQTT?

Redis

Kalyanswaroop's answer is very insightful. I don't know much about redis but it sounds very interesting. Instead of using MQTT as the sole protocol, we could use redis' messaging system. The IOT devices publish their messages into the message queue of redis instead of directly to the REST-API. All messages from various protocols are stored in the database. Then the API would only need to talk to the database and any clients, e.g. a react app.

To transmit changes in the database to a client in real-time, you could use websockets.