0

I'm designing an IoT system, and seeking to adopt a serialization scheme for communications across the entities (Cloud to Device communication, Local Communication,...) taken into consideration some important criteria:

  • Variability: Support both strings and bytes data types.
  • Good programming languages support: C++ is the primary language, but will require support for other platforms to ease/enable the development of the cloud platform and UI applications.
  • Future proof: It's important to design the system with future-proof schemes, where maintenance, backward compatibility, and further upgrades with languages standards where needed.
  • No copy to encode/decode messages: When the data is received from the network, there's no need to copy the data, a std::string_view or std::span<std::uint8_t> or even a std::pair<uint8_t*,size_t> is enough for the strings and bytes to further process them. Copying large payloads will contribute to heap fragmentation in the best cases, OOM exceptions in worst.
  • Minimum data size transfer: When missing (optional) data is not provided, it's good idea no to serialize it. (I think in my case there's no much optimization will happen, so this is not mandatory feature)
  • Enables data hierarchy: The hierarchy of the desired scheme is modeled as (in protobuf):
message Payload {
    optional string str = 1;
    optional bytes raw = 2;
}
message Payloads {
    repeated Payload payloads = 1;
}

message Message {
    string path = 1;
    Payloads payloads = 2;
}

message Messages {
    repeated Message messages = 1;
}

This hierarchy is to exploit each communication protocol features, wherein by MQTT the path of Message can be fetched by the "MQTT topic" while writing "Messages Payloads" to "MQTT payload", and wherein an HTTP request to the device can hold multiple "Messages", as the response is able to, and a "Websocket message" would hold exactly one "Message".
A single payload can be either a string or bytes.

The schemes I'm aware of are Protobuf, Flatbuffers, and CAP'n Proto.
Up to my research, Protobuf seems promising, future-proof, and have little overhead bytes, but does require copying data to encode/decode. Unlike CAP'n Proto which promise to not perform a copy.

I'm looking to take an insightful decision. Which one of the available schemes meets the criteria above?

10
  • No-copy can be an issue for integers taking more than a single byte on platforms which requirement alignment. There is also the issue of byte order/endianness. For strings there may be the issue of encoding. Also remember that on-wire protocol and library implementation are two different things. I don't think anything prevents you from reading data directly from a received message without copying it (though of course that data will remain available only as long as you have the message, so depending on what you do, you may need to copy anyway).
    – jcaron
    Commented Aug 24, 2023 at 12:23
  • I can understand and accept the encoding issue, but not decoding. Yes I might need to copy to process a data, but in my case I just feed the payloads to the system with const reference, if a copy was needed it will be done on a particular received/processor function. Commented Aug 24, 2023 at 14:43
  • Did you check the alternative C++ (and possibly C) implementations of protobuf? github.com/protocolbuffers/protobuf/blob/main/docs/… list many. See for instance github.com/mapbox/protozero/blob/master/doc/tutorial.md which provides a get_view() (but you lose a lot of the convenience of the traditional protobuf implementation) or github.com/squidfunk/protobluff for C rather than C++.
    – jcaron
    Commented Aug 24, 2023 at 15:48
  • Good information to know, I'll check the list for a potential one! Commented Aug 24, 2023 at 17:14
  • It's perhaps the protozero is the only one applicable to use from the list (EmbeddedProto is commercial, struct2x is JSON converter, Simple Proto is an old release "Don't know if compatible to proto3") , but as you've said, I'll lose a lot as these are not scheme-aware. I'll start checking C libraries. Commented Aug 24, 2023 at 18:35

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.