23

I have to provide the IoT service for my customer. MQTT, Kafka and Rest Services components will be used to ingest the data from the devices to the database. I need to do some analytics over the data in the backend. The data size would be 135 bytes/device and 6000 device/second. I have shared the architecture here to understand the requirement and components.

enter image description here

I have investigated about the data stores(MongoDB, Postgresql(TimescaleDB), Redis, Neo4j, Cassandra) and every vendors proved that their database is suitable for the IoT use-case. I have confused about using the proven/most reliable/scalable database for the IoT.

What could be the best suitable database to ingest this much of data and do the analytics?

Is there any proven benchmark for the suitable database for the IoT?

Please give your thoughts and suggestions.

Improve this question
4
  • I used ElasticSearch for a similar use case recently. But I can’t say why it’s better than others, that part is mostly opinion based. I literally used Kafka to connect sensors to DB. There are nice libraries that support stream processing of Kafka with Elasticsearch
    – atakanyenel
    Mar 15, 2018 at 16:17
  • 2
    “IoT use-case” is far too broad to rank implementations. Each one has its strengths and weaknesses.
    – Gilles 'SO- stop being evil'
    Mar 16, 2018 at 9:05
  • 1
    Not my field, but I'd be surprised if any modern db would look like a bad fit here. Use what you are familiar with, or has the shiniest tooling.
    – Sean Houlihane
    Mar 16, 2018 at 11:39
  • @Mourish, Did you archive it? may I know which database you worked?
    – nrs jayaram
    May 30 at 11:17

5 Answers 5

Reset to default
7

Timescaledb, a postgres extension customized for timeseries datasets works really well. And you get the usual relational database features, use of SQL, reliability, indexes, scalability.

Improve this answer
6

IoT is pretty much time-series data. There are a few TSDB out there: InfluxDB, OpenTSDB, GridDB, etc. They all have the community/oss version so you can see if it suits your need. InfluxDB is a popular one but note that clustering is only available for paid version. OpenTSD is pure oss, and GridDB states it is IoT-oriented and faster than InfluxDB. Depending on your needs, maybe you want to look for one that has fast ingest.

Improve this answer
5

You are limited to either NoSQL databases, because any SQL database won't allow you 6K TPS directly on the server nor you may use any SaaS cloud service or platform already specialized in such kind of operations - e.g. receive telematics data via MQTT/Kafka, split it over and store for these 6000 devices and provide simple REST API to access the telemetry data. Like flespi or whatever similar.

Improve this answer
5
  • got your point and thanks. Could you tell me what NoSQL database is the best fit for my use-case?
    – Mourish Khan
    Mar 20, 2018 at 6:10
  • It really depends on your experience and runtime environment. For AWS/GoogleCloud it will be one choice, for local installation I would recommend to LevelDB or any of its competitors, just search levelDB on google and you will see full list of them. In any variant you will need to implement intermediate API between web application and database, so it also depends on what kind of backend you are using for this. Exactly your case described in this article, when you fill data with mqtt and access it and history from web.
    – shal
    Mar 20, 2018 at 6:42
  • 1
    btw, I tried in last 15 years many of this NoSQL databases. Started from Berkeley DB on its early ages. At the end, when you need full power and performance in your applications and trying to squeeze from database maximum IOPs and throughput I find no other way, but to develop own database engine, specifically targeted to telematics (IoT) use case and requirements. But it was my experience +)
    – shal
    Mar 20, 2018 at 6:48
  • " 6K TPS" ?? 6tB / second?
    – Mawg says reinstate Monica
    Apr 29, 2019 at 7:12
  • 1
    6.000 transactions/second
    – shal
    Apr 30, 2019 at 8:22
4

The question is broad and no accurate answer can be given, but these links can help:

http://outlyer.com/blog/top10-open-source-time-series-databases/ enter image description here

Followup with benchmarks: http://outlyer.com/blog/time-series-database-benchmarks/

Other comparison: https://gist.github.com/sacreman/00a85cf09251147175241d334aafa798

I set some rules to attempt to limit the scope otherwise this blog would never end.

Only free and open source time series databases and their features have been compared. Therefore is someone asks “have you tried Kdb+ and Informix?” the answer will be no. They are probably awesome though.

The list will only include databases that either classify themselves in their marketing material as time series, or have been written about in a blog by a cool company as something they are using for time series data.

What has been done is reading the official docs, reading StackOverflow, looking through Github issues and code and generally hacking the information together. With this in mind some facts may be incorrect.

If anyone spots anything factually wrong please let me know and I’ll update the blog.

Benchmarking has been based on marketing claims and estimation. Why? Because benchmarking is a sizeable chunk of work and prone to error. You always get “you should have tuned this special undocumented setting”. The numbers listed are highly favourable to most databases. They are either the numbers blogged about or claimed on Twitter at some time in the past. If you feel any numbers are wrong let me know and I’ll update them.

Improve this answer
0

Look into Hyprcubd. Full disclosure - I'm one of the co-founders. It has a number of features that sets us apart from others:

  • Serverless - No capacity to provision, just create your tables and go
  • High Availability out of the box
  • High Durability (11 9s)
  • High Ingestion - No need for Kafka in front of us
  • SQL Interface - Aggregations and functions
Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.