Efficient Server- and Protocol structure for raspberry, smartphone and 3rd-party-device network

Question

I am new to this forum and this field. I am looking forward to good criticism and tutorials/links/instructing suggestions to build a nice system with as less effort as possible while achieving all things I want to have.

---

The Situation

I am right now working on a sensorik-system with the following design:

The main hub is the raspberry pi (newest v3).

• hosts MQTT-Server for save message transfer
• hosts python-based (socket-)server for communication with arduinos (hence: sensors)
• hosts data-storage
• hosts GUI using Kivy (nice and neat)

The Arduinos (nano) directly control the sensors/actuators/etc:

• connected via I²C
• handshake protocol exchanges possible functions of
• receive situation dependend different scripts from Python-server

The raspicam (v2):

• provides an image stream which can be previewed or stored.

Project-status: I started with a simpler approach and did not use any server. Hence right now, there is only 1 main program (Kivy-Gui) which has all the relevant code and interfaces. I am using the Pi-camera module and the Arduinos are connected to the pi via I²C interface. It works well. :)

---

So, why all this hassle if everything is basically running already on 1 machine?

Easily spoken: I want to rescale the whole setup. There will be:

• multiple Raspberrys each with their own set of sensors
• external (smartphone) for GUI/Human interfacing
• external (network) storage
• (optional) 3rd party devices for e.g. further sensing or IP-cam or...

Therefore I need to add:

• mQTT-server on Raspberry #1
• wifi-hub on Raspberry #2
• script hosting and executing Python-(socket)server on every Raspberry
• Network-streaming server to ip-stream Raspicam video etc to external GUI on every Raspberry
• net-drive to storage data from different systems

Finally, I want to still have the possibility to connect a screen to one of the raspberrys and directly have a GUI running on it. Therefore I will modify my existing GUI by outsourcing the executing code. Having the possibility to run the GUI directly on a raspi of choice inherently leads to various problems with the new setup. Just to mention 1: streaming the Raspicam-data to ip and reading it out with the streaming device might be slow or impossible. Here (e.g.) I am thinking to implement a workaround that uses different code decided by MQTT if topic-subscriber and content-poster is the same device.

---

My question:

What do you think regarding speed and efficiency of this setup with respect to the different tasks? Do you see any major mistakes in my reasoning and approaches? As the Arduinos have the init routine to be used correctly, it is good to have 1 instance running per rasperry with all the parameter etc as active objects. Do you see a more code efficient, stable and faster approuch instead of writing my own python-server? I e.g. read about using a node.js for this, but this might lead to problems with the arduino-handshaking...

Answer 1

Looking at your design diagram, I’m assuming that is what you are thinking of building (noted that some part are already there). I’m going to try and answer based on that.

What do you think regarding speed and efficiency of this setup with respect to the different tasks? It could be a lot better.

Do you see any major mistakes in my reasoning and approaches? Yes.

The first thing I notice is you are not making full use of MQTT. MQTT is a message broker and a very good and fast one. Consider moving it to be the centre of all your communications between different sensors, computers, servers, phones etc. Have everything talk MQTT.

Create a topic structure around the sensors for example:

house/location/device_type/

followed by either in or out with in being the device listens for messages and out is where the device sends messages. This will allow other things to listen to what interests them through the topic wild card methods eg to listen to all humidity sensors in the house you use house/+/humidity/out, so any humidity readings will be received.

With this model, you can have a database that listens and records data. A web GUI that sends on/off messages to the light, an air conditioner that listens to a temperature sensor. It opens up your options. Note the MQTT broker can be running on the same RPi as the data storage and gui, however they should be communicating via MQTT

As the Arduinos have the init routine to be used correctly, it is good to have 1 instance running per raspberry with all the parameter etc as active objects. Do you see a more code efficient, stable and faster approach instead of writing my own python-server? As suggested in the answer to the previous question, I would have the Arduino’s (assuming they are actually ESP’s) use MQTT directly which they are very capable of doing.