I've come across this stack overflow post. In case the link is dead: In short, the post describes a way to use JWT to authenticate a frontend user with your backend and mqtt broker.
While it doesn't really cover your question, I think it still is useful. Specifically, it solves your problem of authenticating a client and controlling access to topics. The mosquitto broker, compiled with mosquitto-go-auth, would be an example of how you could implement the architecture described in the post.
Further building on this architecture, to support multiple clients you could develop some sort of gateway from mqtt to other protocols. E.g. a backend service that connects to your mqtt broker and listens for messages on topic newprotocol. When it receives such message it translates the message content to your desired protocol. Your frontend clients would only need to speak http and mqtt then.
I'm trying also trying to build a DIY home automation system and I would very much appreciate it if you could keep this question up-to-date with your progress.
That being said, I found a few useful links that you'll maybe also find useful:
The gist of the article is:
Essentially, you start with a normal REST API and add MQTT messages for REST endpoints that result in a state change (POST/PUT/PATCH/DELETE).
Then add to this approach, anytime a IOT device changes state, publish the new state to a REST endpoint.
What I've tried
I hope you'll let me share my thoughts on how to build a system like this and I also hope that this'll be useful to someone. One way I thought about structuring this system is to have 3 components: Clients, REST-API and MQTT-Broker.
In my case I have a react app running in the browser. It makes API calls to the REST backend (i.e. GET all IOT devices, POST new message to IOT device, etc.). The backend handles authentication, the PUB/SUB messaging with the broker (i.e. Handle IOT device status changes, find out what IOT devices exists, etc.) and any messages coming from the react app.
However, I ran into problem with this approach. How do you let the react app know the state of an IOT device changed while using a stateless protocol (HTTP in this case)? I thought about using websockets to transmit changes to the react app but I ended up not implementing it and went back to the drawing board. And that's were I am now. Another issue with this is that I would be restricting myself to use MQTT for every IOT device. But what if a device doesn't talk MQTT?
Kalyanswaroop's answer is very insightful. I don't know much about redis but it sounds very interesting. Instead of using MQTT as the sole protocol, we could use redis' messaging system. The IOT devices publish their messages into the message queue of redis instead of directly to the REST-API. All messages from various protocols are stored in the database. Then the API would only need to talk to the database and any clients, e.g. a react app.
To transmit changes in the database to a client in real-time, you could use websockets.