9

I am planning to start to implement the below IoT use case.

Use case

The IoT devices will send 100k messages/minute to the gateway via repeaters and the gateway will transfer the messages to the cloud. I want to track the employees in an organization. The sensors will be fixed on their ID card. The sensor sends the location related data (approx. 15KB/message) to the gateway via repeaters. It's for the analytical purpose. After the data passed to the cloud, I'll do some analytics and store into the DB and display on a web page. Based on this analytics data, I'll show the user's current location and also the user's moving locations of a certain passed time span (last 1hr or 2hr or 1day).

I'll do some processing over the data and send it to the front end/DB.

I have gone through the IoT basics and its architecture. Then I decided to use "SMACK" stack (Spark, Mesos, Akka, Cassandra, Kafka) architecture.

I decided to use "Kafka native client" in the gateway to publishing the messages to the cloud.

Should I use MQTT protocol to transfer the message to Kafka? Or MQTT is not needed for the above use case?

If yes, what would be the benefit of using MQTT with the "SMACK" architecture?

Improve this question
3
  • 2
    1.5 Megabytes per minute? Per employee, all day? Forget about protocols, you should probably take a step back and re consider the data itself, finding a way to distill it before transmission.
    – Chris Stratton
    Commented Jan 11, 2018 at 7:02
  • My thoughts exactly (+1). How many employees do you have? Ever on a large(ish) site with 2,000 employees, you are tracking their location almost once per second - who needs that accuracy? What can you do with it? And why do need 15kB for lat/long? What its the rest of it, and how often does it change?
    – Mawg
    Commented Jan 22, 2018 at 8:03
  • Just curious - how are you tracking the Ids? Long range passive RFID? Bluetooth? If anything other than passive RFID, I can foresee battery problems.
    – Mawg
    Commented Jan 22, 2018 at 8:05

2 Answers 2

Reset to default
5

You don't have to use MQTT. Kafka clients installed on employee ID cards can send data to the kafka broker in the cloud directly. So while you are using Kafka for the gateway, you can actually use kafka for the sensors itself.

Kafka and MQTT are not interchangeable, they have different strong sides(energy consumption, bandwidth consumption,throughtput...) but I think you know it , assuming from the complexity of your stack. Kafka can handle 100k messages/minute.

If you decide to use MQTT anyway , here is a blog post on how IBM uses MQTT and Kafka in the same project. Their project is also about mobility, so I think it might actually help.

Managing IoT devices with Kafka and MQTT

Improve this answer
6
  • atayenel, Thanks for explaining in detail. In my case, I should use the gateway instead directly push message from the sensor. Google(MQTT) and IBM IOT service providers using MQTT? After reading the below link, I have some confusions ... Do you know why, Is there any specific reason?cloud.google.com/iot/docs/concepts/protocols
    – SKK
    Commented Jan 11, 2018 at 14:09
  • Can Kafka handle that amount of data?
    – Mawg
    Commented Jan 22, 2018 at 8:05
  • Kafka is better than mqtt at data throughput. So if mqtt handle it, kafka can also handle it.
    – atakanyenel
    Commented Jan 22, 2018 at 8:43
  • @atayenel thanks for your comments. What do you suggest me the approximate minimum hardware requirements for my use-case?
    – SKK
    Commented Feb 15, 2018 at 5:46
  • @SKK I can't say it exactly , but you can look at this question.
    – atakanyenel
    Commented Feb 15, 2018 at 16:56
2

There is no problem with almost any kind MQTT broker to handle this load, especially for qos=0 (probably in your case) messages. We have constant load to our broker with incoming 100.000 messages (0.5KB) per second (+SSL). The problem may appear from traffic side, not from pps.

Regarding architecture of your system my personal advice - try to make it as simple as possible. And simple mean - just a few intermediate components/services. If you can connect directly two services - do it. You will always have the possibility to make it more complex when you will start adding features.

Improve this answer
3
  • Thank you. Do you want me to use simple architecture rather than SMACK?
    – SKK
    Commented Mar 13, 2018 at 5:01
  • Can you please look into this iot.stackexchange.com/q/2718/5382 ?
    – SKK
    Commented Mar 13, 2018 at 6:29
  • It's absolutely up to you, but in my opinion I would create it as simple as possible, just concentrating on the important tasks in the beginning. Usual case is that your understanding of the project before and after(or in the middle) is totally different and often you have a final conclusion "if I do it again I will do it different" ))
    – shal
    Commented Mar 14, 2018 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.