9

I am planning to start to implement the below IoT use case.

Use case

The IoT devices will send 100k messages/minute to the gateway via repeaters and the gateway will transfer the messages to the cloud. I want to track the employees in an organization. The sensors will be fixed on their ID card. The sensor sends the location related data (approx. 15KB/message) to the gateway via repeaters. It's for the analytical purpose. After the data passed to the cloud, I'll do some analytics and store into the DB and display on a web page. Based on this analytics data, I'll show the user's current location and also the user's moving locations of a certain passed time span (last 1hr or 2hr or 1day).

I'll do some processing over the data and send it to the front end/DB.

I have gone through the IoT basics and its architecture. Then I decided to use "SMACK" stack (Spark, Mesos, Akka, Cassandra, Kafka) architecture.

I decided to use "Kafka native client" in the gateway to publishing the messages to the cloud.

Should I use MQTT protocol to transfer the message to Kafka? Or MQTT is not needed for the above use case?

If yes, what would be the benefit of using MQTT with the "SMACK" architecture?

  • 2
    1.5 Megabytes per minute? Per employee, all day? Forget about protocols, you should probably take a step back and re consider the data itself, finding a way to distill it before transmission. – Chris Stratton Jan 11 '18 at 7:02
  • My thoughts exactly (+1). How many employees do you have? Ever on a large(ish) site with 2,000 employees, you are tracking their location almost once per second - who needs that accuracy? What can you do with it? And why do need 15kB for lat/long? What its the rest of it, and how often does it change? – Mawg Jan 22 '18 at 8:03
  • Just curious - how are you tracking the Ids? Long range passive RFID? Bluetooth? If anything other than passive RFID, I can foresee battery problems. – Mawg Jan 22 '18 at 8:05
4

You don't have to use MQTT. Kafka clients installed on employee ID cards can send data to the kafka broker in the cloud directly. So while you are using Kafka for the gateway, you can actually use kafka for the sensors itself.

Kafka and MQTT are not interchangeable, they have different strong sides(energy consumption, bandwidth consumption,throughtput...) but I think you know it , assuming from the complexity of your stack. Kafka can handle 100k messages/minute.

If you decide to use MQTT anyway , here is a blog post on how IBM uses MQTT and Kafka in the same project. Their project is also about mobility, so I think it might actually help.

Managing IoT devices with Kafka and MQTT

  • atayenel, Thanks for explaining in detail. In my case, I should use the gateway instead directly push message from the sensor. Google(MQTT) and IBM IOT service providers using MQTT? After reading the below link, I have some confusions ... Do you know why, Is there any specific reason?cloud.google.com/iot/docs/concepts/protocols – SKK Jan 11 '18 at 14:09
  • Can Kafka handle that amount of data? – Mawg Jan 22 '18 at 8:05
  • Kafka is better than mqtt at data throughput. So if mqtt handle it, kafka can also handle it. – atayenel Jan 22 '18 at 8:43
  • @atayenel thanks for your comments. What do you suggest me the approximate minimum hardware requirements for my use-case? – SKK Feb 15 '18 at 5:46
  • @SKK I can't say it exactly , but you can look at this question. – atayenel Feb 15 '18 at 16:56
2

There is no problem with almost any kind MQTT broker to handle this load, especially for qos=0 (probably in your case) messages. We have constant load to our broker with incoming 100.000 messages (0.5KB) per second (+SSL). The problem may appear from traffic side, not from pps.

Regarding architecture of your system my personal advice - try to make it as simple as possible. And simple mean - just a few intermediate components/services. If you can connect directly two services - do it. You will always have the possibility to make it more complex when you will start adding features.

  • Thank you. Do you want me to use simple architecture rather than SMACK? – SKK Mar 13 '18 at 5:01
  • Can you please look into this iot.stackexchange.com/q/2718/5382 ? – SKK Mar 13 '18 at 6:29
  • It's absolutely up to you, but in my opinion I would create it as simple as possible, just concentrating on the important tasks in the beginning. Usual case is that your understanding of the project before and after(or in the middle) is totally different and often you have a final conclusion "if I do it again I will do it different" )) – shal Mar 14 '18 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.