How to effectively control Arduino via a web application?

Question

I have a web application in PHP on remote hosting. The application is connected to a MySQL database, from which it reads states, draws graphs from the data that Arduino sends to the database.

The problem arises when I want to use the buttons to switch the status of the LEDs from the web application. The current solution is that the web application writes the status to the database at the touch of a button and the Arduino looks at the database every 3-5 seconds and changes the status of the LEDs. This method is not very dynamic and there is a large time delay, and if I go to lower time intervals then it will be a burden on the network ...

I can't handle how fast, dynamically Arduino can respond to database changes without having to open the network for incoming connections from the web to address security (port-forwarding).

How IoT systems solve this, I know that Google and Amazon IoT applications have this solved in some way even without port-forwarding and I don't see that they would ping the server every second

Answer 1

What most devices in this situation do is that they establish a permanent connection to the server, send enough information to identify and authenticate the device, and then sit there waiting for a message to be sent by the server.

That turns a pull (polling) situation into a push (notifications).

This however requires a “long lived” process on the server, which will accept those connections and keep them open, keeping track of which connection is for which device. Most HTTP-based servers, including those revolving around php, are not well suited for that: they work on a request basis: receive a request, process it, return data, close connection (with caveats for keep alive and the like).

So this usually involves running another server (software) which will handle that. Using node.js for this is very common, but there are probably plenty of other solutions.

Even though on the device side you could use raw TCP, it’s probably a good idea to encapsulate into HTTP as this will give you better compatibility with slightly complex cases (filtering, proxies, etc.), which means using the WebSocket protocol. This has the added advantage that you can use it for real time communication with web apps as well.

An alternative is to use MQTT, but the overall principle remains the same.

Note that this will work if your web host allows you to run long-lived processes. If you have a bare metal server, a VM, or some kind of docker-based hosting that should work. If you have a standard “web hosting” service that probably won’t be possible.

There are lots of ways to achieve what you want, I hope I have given you enough pointers to get you started.