A few thoughts on my experience with TCP, UDP, and MQTT as well as some additional resources to review.
With UDP I have run into the silent failure problem in which an application on one network node, the client, is seeing only some of the UDP messages that were sent. There are too many reasons why network traffic may go awry. The problem with UDP is that packets can be discarded pretty much whenever any of the nodes in the network path between producer of packets and consumer of packets warrants. See Wikipedia topic Packet loss.
The question is what is your loss rate under whatever the current network context. So if this is communication within a LAN or sub-network your loss rate may be low. On a WAN or across the internet your loss rate may be quite high. UDP packets are discarded for a number of reasons and routed however the network conditions allow with that hop count decrementing. Sending packets out into the great void with no accountability leaves open the possibility of silent failures.
It sounds like in your case just a simple ack with retransmit after a timeout without receiving an ack would suffice.
I have done XML messages over a maintained TCP connection and it worked great. I had a layer that delivered complete messages each in a buffer to the application layer to handle. I used XML to package the message with an XML starting tag for the beginning of the message and an XML ending tag to know when the entire message had been received.
TCP does have a few features such as sequential order of packets, no repeats, and being a connected transport mechanism means you know if the other end disappears or not though it may take a while to find out. Connecting and disconnecting can introduce delays but not burdensome under ordinary conditions though network conditions can cause TCP throughput to slow down.
MQTT is a protocol that is transported by a network transport layer, normally TCP. MQTT uses a publish/subscribe model so there is no message storing. So when a publisher publishes a message if the subscriber is not connected at the time then when it does connect, it will not see the message. MQTT is pretty much real time, I suppose that is what the telemetry part of the acronym is all about. I do like MQTT for small messages and have been doing some experiments with JSON payload through MQTT using Mosquitto. See this article Reliable message delivery with Mosquitto (MQTT) with an overview of MQTT and the quality of service. And see this brief article with links to resources including a sample application IoT – MQTT Publish and Subscriber C Code.
My experiments with MQTT using JSON text and an SQLite3 database in the subscriber to store messages is at https://github.com/RichardChambers/raspberrypi/tree/master/mqtt though the source is in C and is quite messy.
Here is a 13 minute video #144 Internet Protocols: CoAP vs MQTT, Network Sniffing, and preparation for IKEA Tradfri Hacking. This is an interesting article about CoAP, Constrained Application Protocol: CoAP is IoT's 'modern' protocol. There is this summation of CoAP:
Early adopters agree that the Constrained Application Protocol works
extremely well for constrained networks and devices. Something not as
well known: "On a very congested wireless network -- Wi-Fi or cellular
-- CoAP can continue to work where a Transmission Control Protocol-based (TCP) protocol like MQTT can't even manage to complete
a handshake," Vermillard said.
This is because unlike most other IoT protocols, CoAP is built upon
UDP. In other words, it means no protocol handshakes or error
correction as encountered with TCP. "CoAP may not be as reliable as
HTTP or guarantee delivery of messages like MQTT, but it's extremely
fast," Matthieu noted. "If you're okay with some messages not being
received, you can send many more messages within the same timeframe."
There are a few others such as AMQP, STOMP, and CBOR you might look at as well. See the CBOR standard website as well as this thesis, Implementation and evaluation of the CBOR protocol. See this article, Choosing Your Messaging Protocol: AMQP, MQTT, or STOMP which compares and contrasts the AMQP, MQTT, and STOMP and ends with a note about the RabitMQ broker:
Hopefully, this can help many begin to navigate the protocol soup out
there for each of your use cases. Since it is common for companies to
have many applications with different needs, it is certainly possible
you may need all three brokers across different applications. That’s
where a solid multiprotocol, polyglot broker like RabbitMQ comes
in—since it can send STOMP, MQTT, or AMQP in and get one of the other
ones out. You don’t need to be locked-in by one of these protocols—all
three are supported by the RabbitMQ broker, making it an ideal choice
for interoperability between applications. The plugin architecture
also enables RabbitMQ to evolve to support additional or updated
versions of these protocols in the future.
This slide share package of some 60 slides does a compare and contrast between four different IoT protocols looking at the needs of two different IoT groups, Consumers and Industrial, which have differing needs for reliability and robustness. What's the Right Messaging Standard for the IoT?.