I'm designing an IoT system, and seeking to adopt a serialization scheme for communications across the entities (Cloud to Device communication, Local Communication,...) taken into consideration some important criteria:
- Variability: Support both strings and bytes data types.
- Good programming languages support: C++ is the primary language, but will require support for other platforms to ease/enable the development of the cloud platform and UI applications.
- Future proof: It's important to design the system with future-proof schemes, where maintenance, backward compatibility, and further upgrades with languages standards where needed.
- No copy to encode/decode messages: When the data is received from the network, there's no need to copy the data, a
std::string_view
orstd::span<std::uint8_t>
or even astd::pair<uint8_t*,size_t>
is enough for the strings and bytes to further process them. Copying large payloads will contribute to heap fragmentation in the best cases, OOM exceptions in worst. - Minimum data size transfer: When missing (optional) data is not provided, it's good idea no to serialize it. (I think in my case there's no much optimization will happen, so this is not mandatory feature)
- Enables data hierarchy: The hierarchy of the desired scheme is modeled as (in protobuf):
message Payload {
optional string str = 1;
optional bytes raw = 2;
}
message Payloads {
repeated Payload payloads = 1;
}
message Message {
string path = 1;
Payloads payloads = 2;
}
message Messages {
repeated Message messages = 1;
}
This hierarchy is to exploit each communication protocol features, wherein by MQTT the path of Message
can be fetched by the "MQTT topic" while writing "Messages Payloads" to "MQTT payload", and wherein an HTTP request to the device can hold multiple "Messages", as the response is able to, and a "Websocket message" would hold exactly one "Message".
A single payload can be either a string
or bytes
.
The schemes I'm aware of are Protobuf, Flatbuffers, and CAP'n Proto.
Up to my research, Protobuf seems promising, future-proof, and have little overhead bytes, but does require copying data to encode/decode. Unlike CAP'n Proto which promise to not perform a copy.
I'm looking to take an insightful decision. Which one of the available schemes meets the criteria above?
get_view()
(but you lose a lot of the convenience of the traditional protobuf implementation) or github.com/squidfunk/protobluff for C rather than C++.