Are you asking about the protocol or the message format? We often incorrectly use the term protocol when we mean the format of the data. I do this myself, often because the distinction isn't clear to everyone.
Messaging protocols used in IoT tend to be fairly compact, at least more so than http and offer significant features that are important in messaging (sessions, flow control, reliability, etc). The message format is the of data in the message that get sent. I assume that this is what you are asking about.
The most compact message format is a carefully considered hand-rolled binary format. It is frequently used when in low-bandwidth scenarios when you want to send a few bytes, and know exactly what those bytes look like. For larger messages the disadvantages are significant and, in general, should be avoided at all costs.
I went through a detailed assessment on many different data serialisation options. I expected protobuf, messagepack to be fairly compact, which they were. However, my second problem was finding libraries that were maintained and available on a number of different platforms, including C on the device.
The format that I settled on, surprisingly, was gzip compressed JSON. It is easy to implement and understand, runs everywhere, and, with the data that I was using, was about the same, or smaller, than other methods.
Also beware that if you have a secure channel such as TLS, you're going to consume a chunk of data (>6KB) in TLS handshakes anyway.
A few years ago, I expected a format like protocol buffers to dominate, but not much really happened. Probably because of the ease at which json can be written out and parsed (and compressed). I like the look of Flatbuffers, but the advantage is more on parsing speed than being compact.
Since you are at the investigation stage, I suggest you write a bit of code on each, using data that is typical to you situation, and do some comparisons. Having hard data when you start helps confirm your choices.