14

This came to my attention recently when I found an amazing video on Youtube by:

Micheal E. Anderson: Comparing Messaging Techniques for the IoT, OpenIoTSummit, Linux Foundation.

The slides for his talk are Available Here

On Slide 26 and 41 minutes of the video he is discussing about how (let me paraphrase):

Cellular carriers prefer that their IoT consumers use HTML, XML or JSON type of messages since they consume more Data. More Data means they can charge the consumers more money for the service.

I understand that a lot of proprietary protocols viz. SigFox, Wireless HART or Z Wave have lower data rates and sending bulky data over such carriers can be an expensive affair.

Question

  • Are there some other light-weight messaging formats that are being used for usage in Proprietary Protocols which makes them cost-efficient solutions for current and future IoT consumers? (Shot in a dark: some format called lightweight XML or HTML or JSON is lying somewhere?)

  • Maybe something like CBOR is or maybe used ?

Improve this question
4
  • 1
    I suspect that data bandwidth is probably a 2nd-order cost, and not actually paid by the application developer. So, although it's worth worrying about, but there is probably more development coming in this area.
    – Sean Houlihane
    Mar 12, 2017 at 14:27
  • 1
    Is there any sort of situation in particular you're interested in? If you're sending a predictable type of data (e.g. just an integer or something), you could forgo a markup language altogether, but it does limit the amount of information you can express. If you're just interested in any situation where you'd normally use JSON/HTML/XML, that's fine too.
    – Aurora0001
    Mar 12, 2017 at 16:48
  • 1
    @Aurora0001 I actually don't have a scenario particularly, but it is worth thinking about. I think for compatibility to Web based networks (IP dominated) which might be connected to Cellular Networks markup languages are the best form of data format. But since the field of IoT is generally lifting off it might worth be giving different formats are try.
    – Shan-Desai
    Mar 12, 2017 at 16:52
  • 1
    Sorry to mix the picture a little bit: Messaging on any network have several layers, where data layer is only one on top. All of them are under optimization, or at least could be. 5G for example enhances the used signalling and thus more data fits in.Even 5G enhances the spectral efficiency of signals in air, so efficiency is taggled from many sides.
    – mico
    Mar 12, 2017 at 19:39

2 Answers 2

Reset to default
6

Are you asking about the protocol or the message format? We often incorrectly use the term protocol when we mean the format of the data. I do this myself, often because the distinction isn't clear to everyone.

Messaging protocols used in IoT tend to be fairly compact, at least more so than http and offer significant features that are important in messaging (sessions, flow control, reliability, etc). The message format is the of data in the message that get sent. I assume that this is what you are asking about.

The most compact message format is a carefully considered hand-rolled binary format. It is frequently used when in low-bandwidth scenarios when you want to send a few bytes, and know exactly what those bytes look like. For larger messages the disadvantages are significant and, in general, should be avoided at all costs.

I went through a detailed assessment on many different data serialisation options. I expected protobuf, messagepack to be fairly compact, which they were. However, my second problem was finding libraries that were maintained and available on a number of different platforms, including C on the device.

The format that I settled on, surprisingly, was gzip compressed JSON. It is easy to implement and understand, runs everywhere, and, with the data that I was using, was about the same, or smaller, than other methods.

Also beware that if you have a secure channel such as TLS, you're going to consume a chunk of data (>6KB) in TLS handshakes anyway.

A few years ago, I expected a format like protocol buffers to dominate, but not much really happened. Probably because of the ease at which json can be written out and parsed (and compressed). I like the look of Flatbuffers, but the advantage is more on parsing speed than being compact.

Since you are at the investigation stage, I suggest you write a bit of code on each, using data that is typical to you situation, and do some comparisons. Having hard data when you start helps confirm your choices.

Improve this answer
4

The big advantage of a markup based format is that you retain flexibility in the choice of what data you transmit. This is hugely important in an evolving ecosystem where you anticipate a service evolving over several years of development.

Although a tightly coded binary data structure will be efficient to transmit, you need to decide up-front at a minimum what the structure will look like. When, later you realise that even one field needs expansion, you're stuck. Even rolling out an update to the protocol is hard, since you can't obsolete an old encoding till every endpoint is updated.

This suggests that the optimum approach is to mix minimalist packets and markup based encoding (using the latter as a fallback). The value of this depends on the highest bandwidth payloads. If you're already transferring frequent video sized chunks, optimising the infrequent control data is less worthwhile. If you have frequent small transfers (a temperature maybe), it makes sense to minimise the overhead in transmission -but maybe just batching the transfers is as good.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.