What IoT software/web-services can be used to implement the basic "IoT device->Database ->Frontend" architecture?

Question

I'm looking for the simplest (i.e least components) way to create the data logging/visualization system with the following architecture:

IoT Data (Plaintext HTTP POST) -> Database+data processing backend -> Web-based frontend

Currently, my implementation is as such:

1. IoT data (Plaintext HTTP POST requests) is sent to the Database (Thingsboard community ed. running on the VM in the cloud).

2. Data is copied off Thingsboard manually into MSExcel spreadsheet containing the mathematical model (formulas) for processing.

3. Processed data from Excel is visualised using Excel’s graphs.

This approach worked as a proof-of-concept, but is obviously impractical for any serious applications.

Are there any web-based services (besides AWS) that offer complete, pre-made solutions/templates for Database+data processing and web-based frontend? I.e. web-hosting that provides VMs with database software (Thingsboard, Timescaledb, etc) already preinstalled and pre-configured (but still customizable).

Same with the frontends - are there web-services that can provide a pre-made (but still customizable) template-based solution similar to Thingsboard's dashboards or PowerBI?

Answer 1

I can't say enough good things about Node Red, a (NodeJS) web application framework. I feel that most technically-savvy people can get a node red server performing their basic IoT needs with virtually no coding, just some clever drag-and-drop modules connected by "wires" in the web front end.

It has a plug-in architecture that's backed by node packages (called "nodes" and managed by the palette manager) Simple databases are available. Simple web-front-end graphing can be accomplished with the "Node Red Dashboard" that is a well-documented node module.
The Dashboard is a collection of reasonably customizable UI widgets.
If you find the Dashboard widgets still don't meet your need, (you might be able to extend that code, OR more likely) Node Red allows other front end technologies (vue.js is a popular choice)

There are several hosting providers for Node Red (I'd probably start by looking at the Node Red architect's company "FlowForge") and it's pretty trivial to host it on any generic host service (Amazon EC2, DigitalOcean droplet) or Node-hosting service like Heroku. (I had node red instances running for free on Heroku until they cancelled the free developer tier).

One of the best things about Node Red is that you can drop into JavaScript code when you need to. I found I needed to in order to massage data and queries before injecting into a database (to match database vendor requirements), and I found I needed to do a bit of massaging of data from the database before I rendered it in graphs (matching the graphing software's API requirements) but the ability to isolate custom coding is wonderful. As it is a node application itself, you can embed it into a wrapping instance of Node if you find you need custom services that just don't fit within Node Red itself.

Basically it's a robust (open source) solution that can adapt to fulfill lots of requirements. Since your question question lacks exacting requirements (no specific database, just a list with an 'etc') I had to respond with a 'probably' knowing that there are many contributed node modules, and you can compare your exact requirements with those that are freely available.

Good luck!

(Since this is a question that asks for opinions and recommendations, I'm replying with my recommendation and opinions. I speak as an experienced user with all points above. Note, I have no monetary relationship with any company or product listed, excepting as a paying Digital Ocean customer.)