11

I have to provide the IoT service for my customer. MQTT, Kafka and Rest Services components will be used to ingest the data from the devices to the database. I need to do some analytics over the data in the backend. The data size would be 135 bytes/device and 6000 device/second. I have shared the architecture here to understand the requirement and components.

enter image description here

I have investigated about the data stores(MongoDB, Postgresql(TimescaleDB), Redis, Neo4j, Cassandra) and every vendors proved that their database is suitable for the IoT use-case. I have confused about using the proven/most reliable/scalable database for the IoT.

What could be the best suitable database to ingest this much of data and do the analytics?

Is there any proven benchmark for the suitable database for the IoT?

Please give your thoughts and suggestions.

  • I used ElasticSearch for a similar use case recently. But I can’t say why it’s better than others, that part is mostly opinion based. I literally used Kafka to connect sensors to DB. There are nice libraries that support stream processing of Kafka with Elasticsearch – atayenel Mar 15 '18 at 16:17
  • 2
    “IoT use-case” is far too broad to rank implementations. Each one has its strengths and weaknesses. – Gilles Mar 16 '18 at 9:05
  • 1
    Not my field, but I'd be surprised if any modern db would look like a bad fit here. Use what you are familiar with, or has the shiniest tooling. – Sean Houlihane Mar 16 '18 at 11:39
4

You are limited to either NoSQL databases, because any SQL database won't allow you 6K TPS directly on the server nor you may use any SaaS cloud service or platform already specialized in such kind of operations - e.g. receive telematics data via MQTT/Kafka, split it over and store for these 6000 devices and provide simple REST API to access the telemetry data. Like flespi or whatever similar.

  • got your point and thanks. Could you tell me what NoSQL database is the best fit for my use-case? – Mourish Khan Mar 20 '18 at 6:10
  • It really depends on your experience and runtime environment. For AWS/GoogleCloud it will be one choice, for local installation I would recommend to LevelDB or any of its competitors, just search levelDB on google and you will see full list of them. In any variant you will need to implement intermediate API between web application and database, so it also depends on what kind of backend you are using for this. Exactly your case described in this article, when you fill data with mqtt and access it and history from web. – shal Mar 20 '18 at 6:42
  • btw, I tried in last 15 years many of this NoSQL databases. Started from Berkeley DB on its early ages. At the end, when you need full power and performance in your applications and trying to squeeze from database maximum IOPs and throughput I find no other way, but to develop own database engine, specifically targeted to telematics (IoT) use case and requirements. But it was my experience +) – shal Mar 20 '18 at 6:48
1

IoT is pretty much time-series data. There are a few TSDB out there: InfluxDB, OpenTSDB, GridDB, etc. They all have the community/oss version so you can see if it suits your need. InfluxDB is a popular one but note that clustering is only available for paid version. OpenTSD is pure oss, and GridDB states it is IoT-oriented and faster than InfluxDB. Depending on your needs, maybe you want to look for one that has fast ingest.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.