10

I've been reading about XMPP as a potential communications protocol for IoT devices but, after reading one source, I'm unsure whether it's really an appropriate protocol if you're concerned about overhead for each message.

This source states:

However, XMPP has a number of problems that make it somewhat undesirable for EMBEDDED IOT PROTOCOLS. As an XML-based protocol, XMPP is very verbose, even more so than HTTP, and has heavy data overhead. A single request/response exchange to send one byte of data from an IOT CONNECTED DEVICE to the server is more than 0.5 kB.

There is a draft specification that would compress XMPP using an XML encoding called efficient XML Interchange (EXI). But even with EXI, the same one byte of data gets hundreds of bytes of protocol overhead from XMPP alone. EXI is also a much harder format to process than other options now available. Because of these inherent problems, it is generally recommended to avoid using XMPP in embedded IoT applications.

However, XMPP promotes itself as suitable for IoT applications (although it doesn't specifically say that it's low-overhead), so it seems odd that such a large, seemingly verbose protocol would be recommended/promoted for IoT devices.

Is the overhead of XMPP really as large as the source suggests for small amounts of data? For example, how much overhead would there be when sending an 8-byte message?

Also, is the overhead so great if EXI compression is used (as mentioned in the source)? Would this also come with some pitfalls?

  • 4
    Interesting question. While I am unfamiliar with XMPP it is important to note that EXI requires both end points to have a schema that has to by synchronized. Also the IoT device has to en-/decode that xml which seems in itself awfully complex for sending 8-byte messages. – Helmar Dec 8 '16 at 13:16
  • 1
    @Helmar indeed, with XMPP it looks like what you gain in packet size, you lose in computational complexity. – Aurora0001 Dec 8 '16 at 13:17
  • 1
    I think this question is generally fine but: "For example, how much overhead would there be when sending an 8-byte message every 2 minutes?" -> The "two minutes" is tangential and prone to leading things astray. What you are really asking is how much overhead an 8 byte message would have (I would guess, if it is just one piece of data, the same as a 1 byte message). In relation to a time component this is simply dependant on bandwidth and for anyone contemplating the use of a network protocol that must be dead simple math. – goldilocks Dec 8 '16 at 13:19
  • 1
    @delicateLatticeworkFever I will edit it out if you don't think it's relevant (I wasn't entirely convinced but I thought more detail is better than less) – Aurora0001 Dec 8 '16 at 13:21
  • 2
    It's a suggestion, yes. Just reading that I went, "Doesn't that depend on how long it takes a completely unspecified device to send a determinate number of bytes?" E.g., if the answer is that the message is ~0.5 kB, there's no element of time in the units ;) That's in the unspecified device's bandwidth. – goldilocks Dec 8 '16 at 13:25
7

While it's fair to say that XML is verbose, that should be tempered with the awareness that this verbosity is not all "overhead" in relation to content since it encapsulates semantics; it's overhead that's symptomatic of any protocol that emphasises a dynamic as opposed to static structure. For example, HTML is really a relaxed form of XML that conveys content with a dynamic structure, structure that could be considered an aspect of the content. You can distinguish the content of a table from the table itself, but the fact that the content is tabular data with specific relations is integral to the content; if I just took each cell and transmitted it all as one long string, that structure and those relationships are gone, and so I have lost information and isn't that content?

Let's consider an 8 byte message that might constitute some tabluar data. If I use a very static protocol, I could, minimally, transmit that with no additional overhead simply by defining a protocol like this:

  • Each message is exactly 8 bytes, so we don't need to indicate the length or include any terminating sequence.
  • The eight bytes are always taken to refer to a 2 x 2 grid where each cell contains a 16-bit value.

If all my messages are exactly like that, using XML, HTML or XMPP might be considered silly — I am wasting bandwidth on structural components that are always the same and predetermined anyway, and wasting corresponding computation time at both ends creating and parsing it. A minimal, proper HTML page that contains just a 2 x 2 table with a couple of characters in each cell is probably going to be at least 100 bytes to accommodate the formatting and protocol overhead.

However, if not all my messages are exactly that, then specifying what kind of message it is may not be a literal part of the "payload" but it is a necessary component, content-wise. I could do that with just an extra two bytes and introduce much more dynamism:

  • Messages are now variable length, 0-255 bytes, and the first byte indicates the length.
  • There are (up to) 256 codes for different predefined message types, one of which is "2 x 2 table", that's the second byte.

Now my 8 bytes of table content require 2 bytes of overhead, but there is a much wider range of possibilities in terms of what kinds of messages can be sent with this custom protocol.

It's still nowhere close to the possibilities of an HTML page or XML namespace specification (or set thereof, which is what XMPP essentially is).

So, based on that, if mostly what you are doing is sending simple 8 byte messages, XMPP is probably overkill. However, not necessarily that much. The claim that "a single request/response exchange to send one byte of data from an IOT CONNECTED DEVICE to the server is more than 0.5 kB" seems to me, glancing at the relevant RFC, to be a potential exaggeration (but n.b., all I did is glance at it, I've never implemented or used XMPP). I don't doubt you could construct an example of such, but that is probably not a minimal example.

Since the protocol is TCP oriented, establishing "an XML stream qualified by the 'jabber:client' namespace" only needs to be considered part of the message if we are doing one off things — device contacts a server to send 8 bytes to, sends the data, disconnects. If the relationship is more persistent, which it would often be in an IoT context, then we can assume the device already has an established connection to the destination. In this case, if the ultimate destination of the message is the server (as opposed to another client the server is going to pass the message on to), then the protocol overhead is potentially minimal.

<message><body>8 bytes.</body></message>

A measly 33 bytes of "overhead". It is worth pointing out here that XML is text, and so if your messages are often binary, then it is going to become much less appropriate, because that data needs to be encoded (e.g., to base64), which adds further overhead and computational requirements.

So, ultimately:

Does XMPP have a large overhead for IoT devices sending short, frequent messages?

If there is a persistent connection and the messages are largely unstructured, I don't think so. However, if you don't need what it offers (the dynamism with regard to structure), then there are probably more appropriate methodologies.

Pursuit to that, if we have a context where a single central server is processing and/or relying messages between a variety of devices, even though what any one of those devices is doing may always be simple and straightforward, a protocol which can encapsulate a variety of messages would still be useful. If a client device has limited resources, we can hardcode much of the protocol, and wrapping each message from that end becomes a very simple task; I believe many IoT devices which deploy HTTP servers do that (which is the inverse of "simple clients, complex server"). Those servers cannot handle any kind of HTTP request (except via preformatted rejection) and have a very well defined, focused set of things thing will do and responses they will send, but since they none-the-less function correctly as HTTP servers, implementing clients across a very broad range of commonplace devices (including smartphones and PCs) is simple.

3

First of all, I should say that XML has been used for encapsulating realtime messages with some success, and at large scale, in particular for communicating IM and presence in XMPP. There also seem to be some companies that are intent on leveraging their XML knowledge to try to find yet another application area for this data representation system.

However, not everyone is convinced that XML is the answer to everything in messaging systems. For example, there's been a noticeable shift in recent years to online systems that use JSON as a way to serialize data rather than XML, and if I put my developer hat on for a moment I would say that the tools to encode/decode from native representation (e.g. in Python, PHP, Javascript) seem a lot easier to use than for JSON than their equivalents for XML, even if XML has had more time to get those answers right.

XML is a difficult representation for computers to handle, as it needs a relatively complex text parser, and then some kind of hierarchical representation to allow its data to be extracted in a program. Because there's so much text involved, you need quite a bit of memory available for encoding/decoding.

Often it seems unclear that XML is adding much value to the representation of the data: if the core message is not deeply hierarchical, then adding a lot of textual chaff seems unnecessary, but paradoxically if there is a lot of hierarchy then decoding the message from its textual representation is going to be hard work and need a lot of resources. Also, the benefit of representing in text are not clear to me: When we first write and debug communication systems it is common to use monitoring/decoding tools (for example Wireshark) to help us to figure out what is going wrong. In the long term, everything is working fine, and no human will need to look in detail at the messages going back and forth (only computers). Computers prefer binary representation. The textual representation benefits no-one involved at any stage of deployment.

XML is hard for humans to read (and manually construct) while being hard work for computers at the same time; It is therefore a system that is neither suitable for computers and nor is it suitable for humans.

Importantly, IoT has some special constraints that make it desirable to be efficient. IoT devices will typically have limited processing power and storage (usually no large scale secondary storage, only some RAM and EEPROM). An IoT device might have the simplest possible communication links, maybe not even a TCP/IP stack. There will be a wide variety of microcontroller designs, not even at the level of sophistication where a standard operating system (e.g. Linux, Android) would be in use; this limits the number of off-the-shelf tools that will be lying around offering easy XML interfaces to use.

In summary, I suspect that with IoT, data representation is better left on a case-by-case basis, depending on hardware constraints, style of communication (e.g. broadcast, datagram, reliable etc), frequency of communication and so on. XML certainly shouldn't be thought of as a sine qua non for IoT.

3
  1. Many years ago I did analyse difference for using

    XML in payment network for payment transaction representation (card_number, date, time, terminal_id, and list of additional elements) in comparation with traditional

    bit-maped ISO8583

  2. XML has huge overhead. If you consider impact in networks with 10000+ nodes each of them sending 10+ messages hourly/daily to the central host then XML goes out and you really need something more efficient.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.