2

I'm trying to aggregate logs from multiple embedded devices in centralized place to be able to analyze them.

This is how system looks like:

  • I have number of embedded devices.
  • Some deployments are on-site and system might not have access to the internet. So I need to request customer to dump the logs and send them to us.
  • Addition to on-site per-customer deployments, there is also a server running on cloud for multiple customers.
  • Devices use grpc to talk with server.
  • I also have RabbitMQ in place.

What do I do with logs:

  • Run generic analytics on them
  • When received a bug report, request logs from the customer to investigate.

How should I collect logs from devices? Create an endpoint in server? Let devices publish logs to rabbitmq? Deploy a local RabbitMQ instances to devices and collect logs from there? Use some other logging stack?

3
  • your question seeks opinions ... as such, it is off topic here ... please ask one specific, answerable question
    – jsotola
    Mar 20 at 18:40
  • how capable are your embedded devices ? How sensitive are your logs ? Mar 22 at 18:59
  • @kalyanswaroop They're connected to wifi, have quite good battery life and good processors. Not sensitive, mainly from internal components for debugging purposes.
    – bca
    Mar 22 at 21:47
2

There are solutions such as Apache Flume and logstash to deal with collecting logs. If you want to host your own. If you want a readymade solution, look at logz.io

In your case, maybe the easiest is for you to write them as JSON or CSV entries in files and put those files eventually into S3 either directly from the device or coming in via USB via some uploader. Then, you can run S3 select commands from the cli or if you want to do large processing, you could use AWS Athena.

There are other options such as loading the files into elastisearch and then using kibana. The S3 option will allow you to grow later, since you can do other things on S3 events later, including putting that data into some other tool.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.