1

I'm working with industrial machines, not always connected to the internet in industrial environment.

I'm new in IoT and for my current setup, I have 2 usage: analytics and live data. For the analytics, I collect data with fluentbit and it ends in elasticsearch, where I fetch the data later on. I don't have anything setup yet for the live data.

The type of data I am sending is position of my vehicles (autonomous machines and robots), and part of the sensor data.

Now, I see many applications are using MQTT to send the data to a streaming process like kafka. My question is, can I keep and extend my setup to use live data, or it's not realistic without MQTT?

Edit: My IoT devices are robots or material handling equipment (e.g forklifts). They are working between 12 and 18h/day, then charging. They will likely have no internet access and will be on a local network.

My current approach is to send with fluentbit to a local PC that has access to internet which forwards it later on to ES. The PCs on each robot will have at least 1 core free and 1 Gb of ram available.

As I am testing, I have only 2 robots to test latency... so i'm not so sure what i should expect.

Last but not least, regarding the live part, I believe what I lack understanding of is the pipeline. I would need to collect the data for live display but also for future analysis. My current setup puts everything in elasticsearch, which doesn't sound right, but I don't know why. I'm spending time on reading about it, but most resources I find have emphasis on the tools instead of their usage.

Edit 2:So, I'm ditching fluent bit regarding the data acquisition. Since i need to stream data with different and unreliable network environment, I am going with MQTT, then transfers the data to Kafka and stream it from there. I did a small POC this week and it works fine. The challenge is the scaling which I'm taking care of right now.

Improve this question
3
  • 1
    Hello David, that depends on a bunch of further things. What is the requirement for the live data usage? How much data are we talking about? When not connected to the internet are you still working on a reliable local network were can set up stuff or are you constrained to the edge things? Generally no IoT solution needs MQTT, it's just one protocol of one communication pattern. Without knowing more about your use case it's hard to give you good answers.
    – Helmar ♦
    Commented May 28, 2021 at 15:51
  • 1
    It would be especially interesting to know how much processing power and battery constrains you have on your IoT devices.
    – Helmar ♦
    Commented May 28, 2021 at 15:55
  • Hi @Helmar, thanks! I added some more more information.
    – David Bensoussan
    Commented May 28, 2021 at 22:50

1 Answer 1

Reset to default
1

It isnt clear from your statement above, but I assume that you have fluentbit on each vehicle and also on a server where you're collecting from all the machines and putting it in elasticache. If you already have enough CPU and memory to have fluentbit running on your vehicles, you dont need MQTT.
MQTT is usually for more constrained devices. In fact, MQTT can be one of the inputs that feed into fluentbit on the server (cloud perhaps) side. Fluentbit is similar to kafka.
For the live part, depending on how many clients you'll have and security needs, etc, it maybe possible to just use Elasticache's pub/sub mechanism. You may find that the latency of the chain is sufficient for your live data needs. If it is not enough and you need lower latency, then MQTT will be better for you. At that point, whether you will use FluentBit or Kafka or Flink or Apache Nifi or ... is dependent on what you're comfortable with. Consider looking for something readymade if you wish to save time. AWS has their KDA and Azure and google have similar things. There's also Thingsboard.io, logz.io and... (I'm not affiliated with any)
Do let us know what you decided and how it went.

Improve this answer
1
  • Hi @kalyanswaroop, thanks! I edited the question and added details.
    – David Bensoussan
    Commented May 28, 2021 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.