What do I need to create my own personal cloud for IoT devices?

Question

This is a subject I have been thinking of for a while, especially because the "IoT" concept has been floating around a lot lately.

I will start with what I mean when I say "IoT". I know that the term IoT could mean different things and that sometimes it's misused. It could be a term that is not clearly defined and can lead to big discussions around what does it exactly mean, I myself don't know the proper and widely accepted definition of the term. So for me IoT is a concept, a concept that defines the ability to connect to an embedded device remotely through the internet either from another embedded device or from a cell phone. As simple as that.

On this context, the purpose of the connection doesn't matter, if you can connect one device in your office with another one at home, or if you can connect to one device at home from your cell phone, all this through the internet, then we are talking about IoT devices (the embedded devices, not the phone).

So, having agreed on what I mean by IoT I will now describe what I'm trying to achieve.

What I'm trying to achieve is precisely that what I describe on my definition of IoT.

I want to have one or several embedded devices at home connected to my internet router, either by ethernet or wifi and be able to connect to them remotely with another embedded devices in a remote location (and by remote I mean not on the same network) and maybe also to be able to connect to them with a monitoring app on my phone

For example, I may have a simple embedded device acting as an on/off switch hooked op to my garage door opener and another embedded device acting as a big red button on my desk at work so that I can push the red button in my desk and the garage door opens.

Another example would be to have an embedded device with ADC capabilities that can monitor the temperature of my house and send me an alert when it reaches a threshold. The notification could be received either by a simple android app or by another embedded device with a little screen sitting on my desk at work.

These examples may be silly but are just to illustrate the possible scenarios and use cases for what I'm trying to achieve. At the end, the idea is the same, connect one embedded device with another through the internet.

Another thing to clarify is that the data exchange between these devices will be very lightweight, just a couple of bytes every time, it is not that hundreds of kilobytes are needed to be interchanged between devices.

Additionally, the kind of "embedded devices" I'm referring to are simple but capable devices based on 100MHz or 200MHz cortex-m4 microcontrollers. And that is important to clarify because there won't be any Linux or complex libraries running on those devices. In the end, is such a waste of resources and completely unnecessary to have a powerful processor running Linux just to turn on and off a light bulb. In any case, I'm planning to use a BeagleBoard, Raspberry Pi, or any other board like that as my embedded devices. Just Microcontrollers because no more complexity than that is needed.

I don't know much about IoT platforms and those kinds of complex solutions out there. When I started this journey of finding out a way of connecting one embedded device with another through the internet I stumbled upon a couple of sites with IoT services.

I know that there are some IoT cloud services like:

• Blynk
• Google Cloud Platform
• Exosite IoT Platform
• Ayla IoT Platform

Just to name a few. The main issues with those are cost and complexity. You have to pay to get those services and also you have to learn how to implement all the services they have, in case you need them all, and their APIs and maybe a bunch of other stuff that doesn't seem necessary to me to be able just to interchange some bytes between devices. I just want something simpler than that, something I can do myself.

You may say that implementing my own "cloud", if that is something I have to do, is not simple and sometimes is better to use those kinds of services for the sake of simplicity but there are two main reasons I want to know how to implement my own IoT services.

The main reason is that I want to do it myself. I don't want to rely on a 3rd party to connect my devices to each other and since I'll be developing the code and the hardware for my devices then it feels better to also create my own means to connect them as IoT devices.

The second reason is to learn how to do it. By knowing all the necessary things I need to achieve this, I will have a better understanding about the IoT world.

Also, I want to mention that I'm proficient in C and I use Linux as my everyday OS at work as well as in my home, so please avoid windows stuff because that is useless to me. I'm not afraid of anything I have to implement in C for my embedded devices or on Linux to implement whatever is needed to achieve my goal.

So my question is, what is it necessary to implement, and where, to be able to connect two or more embedded devices to each other with the purpose of data interchange between them?

This question What can I use to create an IoT on our own server? have something similar but is closed and doesn't have any answers, also assumes an already existing cloud infrastructure to be used. So it doesn't help me.

This other post What IoT services are available for storing/sending/publishing generic data in the cloud? has a similar question but the OP is asking explicitly for IoT services and I'm trying to avoid those.

Answer 1

Maybe I missed the point of the question, but I think this is a good start at an answer.

You need three things, at a minimum.

1. An always-on compute node to aggregate your data. This does not need to be powerful, It could be a process running on your NAS, or even (in theory) on your router. For simplicity, assume it is a Raspberry Pi. This could also provide any fancy radio protocols that you decide to support in the future. Although in theory you could run peer-to-peer with all nodes exposed to the internet, it is easier to nominate one as the master and have this handle the data collation and publishing (to the app/user). Of course, the hub can be a node too, particularly if you use WiFi nodes which are moderately powerful.
2. A suitable software stack to allow the endpoints to submit their data to your hub node. mqtt is the sort of thing you need here, plus an OS to run it on.
3. DNS and port-forwarding to facilitate access to your server from the wider internet.

Then don't forget security. By doing this, you will be closer to opening up everything on your home network to attack. Maybe only a little bit, but it's good to be aware of the risk. You can try and take care, but do assume you will make mistakes.

Answer 2

Lightweight devices, and couple of bytes date rates ask for using MQTT, as it has already been mentioned. Your sensor nodes could be based on standalone ESP8266 modules which are powerful enough to host an MQTT client implementation. Or you can simply use these modules as an AT command controlled Wi-Fi module along with your external microcontrollers.

You can implement your own MQTT solution on much less powerful microcontrollers like this guy who has used an Atmega48V with 4 kB of flash.

You can host a broker on your computer, though it would be more power efficient to run a Raspberry Pi instead. Again if you like coding you can implement your own MQTT broker. Personally I found Mosquitto great for this purpose.

Disadvantage that all of your sensor nodes would need TCP/IP connection.

---

Battery friendly solution could be to use LoraWAN or ZigBee enabled sensor/actuator nodes and implement a gateway on a Raspberry/BeagleBone which can host a simple web server too that can be accessed from your phone or other devices.

---

In every case it all comes down to make your hub, gateway or server accessible outside your private network. There are more ways of doing this and the main concern is always security. This is the most risky part in my opinion.

The basic requirements are summarised quite well by @Sean.

Answer 3

You've questioned both previous answers about the need for a controller/hub. Consider that to make things happen, you need rules to exist. If you want to push a big red button to open a garage door, some rule has to tie the sensor (button) to the desired action (opening the door). There are two ways to make that happen: you can put the rule directly in the button, or you can put the rule in a separate computer.

Let's think more about the direct solution. If you teach the button about the garage door, then your button holds the rules internally. The button needs the ID of the garage door, so if you replace the garage door, the button doesn't work. If the button is on your desk, and your house uses a proprietary network, the button has to know both the address of your home's gateway and the address of the door. The button needs to know the specific protocol to signal your door to open - do all manufacturers make compatible buttons that know about all door signals? The button can't do anything else unless you reprogram it - do you have a flash programmer for the button's chip laying around, and is that programmer compatible with any other devices? If you want the garage door to open, and 5 minutes later close, your button needs all the complexities of maintaining a real-time clock. Your button won't know the state of the door, making it hard to know if you're closing the door or opening it. And how do you back up the rules so that if your button breaks, your replacement button can do the job? On the plus side, it sounds cheap: you don't need a separate computer.

With a controller, things are different. All messages from all sensors are delivered to the controller. Each sensor is simple: send the signal to the controller. The controller can then apply whatever inputs are needed to very complex rules: it can check the sunshine sensor and not turn on the outdoor lights unless it's dark, or not run the sprinklers if the average rainfall for the month is above average and the current temperature is five degrees below average. The controller can keep track of state. This could be important if you want a "close garage door" button but not an "open garage door" button (when I'm away from home, I rarely want to open the door, but I definitely want to close it if it is accidentally left open.)

The controller can provide the place for device drivers that know how to listen to buttons and other drivers that know how to speak to doors. The controller may be more upgradeable to new devices and device types than a tiny chip tucked inside a button.

The controller can also have more complex logic for infrastructure tasks such as delivering messages by offering certain levels of service. For example, the MQTT protocol allows for three different levels: try to deliver the message once, deliver it repeatedly until it's been seen at least once, or deliver it once and only once.

The controller offers an architecturally logical place to consolidate all messaging to and from a communication gateway, allowing the use of an external interface. This means your button and your phone can both send signals, and the rules can figure out that either one is allowed to open the garage door. The gateway can also provide the security. You don't have to put your button and your garage door on the internet; you can put them both on private isolated networks and use the gateway to carry the signals. The controller also provides a single point to back up all the rules for your system.

The downsides to the controller are added latency and extra complexity. Surprisingly, a controller doesn't make the cost go up appreciably. You can implement a controller on a Raspberry Pi for less than the cost of one average remotely controllable light switch. Don't discount the idea on the basis of cost alone.