Tracking exceptions and errors in production IoT devices?

Question

How are companies tracking errors on IoT devices, gateways and platforms at present? My company is using papertrail to aggregate logs from all devices, however this often leaves us scratching around between multiple systems when an error occurs in production.

I'm looking for a way to reduce our "time-to-root-cause" when resolving exceptions which may have been generated in one place (e.g. on an IoT platform), but have originated as a result of a problem elsewhere in the stack — e.g. data error from edge device.

In terms of what I've found in this space, Sentry and Rollbar are good for exception tracking on servers or apps, but don't provide a means to track cascading errors as discussed in the previous paragraph.

Are there systems for doing this better than text logging? I'm specifically looking to make use of the breadcrumb style events which you get from Sentry, but with tracking across a distributed system.

Answer 1

Distributed Tracing

The idea behind any worthwhile distributed tracing is most commonly known described in this Google whitepaper about their Dapper solution. Note that I'm not saying they invented it. In essence it works the same for IoT, just start the trace at the edge, either of your backend or even on the end devices.

While the Google whitepaper is more or less focused on server-side systems the concept can be easily adapted to include end devices. The magic of using trace and span ids to trace all your information throughout the systems can be seen by every visualization Netflix does via Vizceral which they recently open-sourced. What in the blog is visualized under Regional View is completely based on live log analyses where the calls are correlated via trace ids. Note that — like Google mentions in the Dapper paper — Netflix has a sampling of the calls pattering on their API. Google mentioned 1:1000 in the paper — which is a few years old. Apparently Netflix has reached 1:1 million on some of their request types already.

I don't know about your system but very likely you can start out with an actual 100% tracing.

Either way as long as you can either match the trace from the beginning to your IoT devices or even create the trace ids on your endpoints in the first place there's nothing preventing you from adapting these ideas in a way that include your edge devices.