Unable to publish MQTT server in an infinite while loop in Python script

Question

I'm curious to know what the performance of an MQTT server is like. So I created this Python script:

import time
import paho.mqtt.client as mqtt
from multiprocessing import Pool


def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))

    client.subscribe("test")
    while True:
        client.publish("test","hello world")


def on_message(client, userdata, msg):
    print(msg.topic+" "+str(msg.payload))


client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect("myserver.example.com", 1883, 60)
client.loop_forever()

When I run this script with python3 myscript.py, the terminal just stops with this message Connected with result code 0.

When I get rid of the while loop and just do this:

def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))

    client.subscribe("test")
    client.publish("test","hello world")
    client.publish("test","hello world")
    client.publish("test","hello world")
    ... print this another 20 times ...

Everything works. I can even receive this messages from another client device subscribed to the MQTT server. Things only stop working if I put the client.publish inside of an infinite while loop. I also tried using asynchronous calls within the while loop and I tried putting a time.sleep() in the while loop, but this still had no change ---- that is, the while loop still hangs.

What am I doing wrong? How do I get my Python script to continuously publish to the MQTT server?

Answer 1

You should not be running long running (infinite loops) in the callbacks.

All the callbacks run on the client network thread's main loop (the one started by client.loop_forever()).

This means if the on_connect() thread never returns it will never get to handling the calls to client.publish() in the loop.

The individual client.publish() calls work because you build up a queue of messages to be published, then the on_connect() returns and the thread can process that backlog.

P.S. your micro benchmark is not likely to give realistic results, but if you really want to run it you can try:

import paho.mqtt.client as mqtt
 
def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))
    client.subscribe("test")


def on_message(client, userdata, msg):
    print(msg.topic+" "+str(msg.payload))


client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect("myserver.example.com", 1883, 60)
while True:
    client.publish("test","hello world")
    client.loop()