What is MQTT: the protocol, its versions and where it is used

# The short version

MQTT (Message Queuing Telemetry Transport) is a lightweight publish-and-subscribe messaging protocol built for devices with a poor link, little memory and a battery to conserve.

It was created in 1999 by Andy Stanford-Clark of IBM and Arlen Nipper of Arcom, to carry telemetry from oil pipelines over satellite. The conditions were unforgiving: bandwidth was expensive and narrow, the link dropped constantly, and the devices were feeble. Those constraints shaped everything about the protocol — a tiny header, a single long-lived connection, and the ability to survive a disconnect without losing messages.

Today MQTT is an OASIS standard, and version 3.1.1 was additionally adopted as the international standard ISO/IEC 20922. It long ago outgrew the oil industry: smart homes, utility meters, machine tools, cars and medical devices all run on it.

The minimum per-message overhead is two bytes. A single HTTP request, by comparison, spends hundreds of bytes on headers before any payload at all. When a device reports a temperature once a second over a cellular modem in a field, that difference decides the design.

# How it works: publish, subscribe and the broker

In HTTP the client asks and the server answers. In MQTT nobody asks anybody anything: a device publishes a message to a topic, and everyone subscribed to that topic receives it. The sender does not know who is reading, or whether anyone is; the receiver does not know who sent it. The only thing binding them is the name of the topic.

There is always a broker in the middle — a server that holds the connections, keeps the subscription tree and hands the messages out. Without it the model does not work: the broker is what turns “sent into the void” into “delivered to everyone who needs it”.

What the broker does

• Accepts and holds connections — thousands at once, for years without a break.
• Checks who is connecting: a login and password, a certificate or a token.
• Decides what each client may do: publish and subscribe permissions are granted per topic.
• Matches every published message against the subscription tree and fans out the copies.
• Keeps state: last known values (retain), queues for absent clients, unfinished acknowledgements.
• Tracks who is alive, and publishes the last will of anyone who disappears.

Decoupling sender from receiver is not a technical detail but the whole point of the model. You can replace a sensor, add a second consumer, or bolt analytics onto the side without rewriting anything: they all talk to a topic rather than to each other.

# Topics and wildcards

A topic is a string of levels separated by slashes. Nothing needs registering in advance: publishing to a new topic creates it on the spot.

office/floor2/room14/temperature
office/floor2/room14/humidity
office/floor2/room15/temperature

A subscription can capture whole branches with two wildcards:

The # wildcard is only valid as the final level: office/#/temperature is not a legal filter. Topics are case-sensitive, and office/floor2 and office/floor2/ are two different topics.

# Quality of service: QoS 0, 1 and 2

MQTT lets you choose how much effort to spend delivering each message. The level is set separately when publishing and when subscribing; the effective level is the lower of the two.

The cost rises with the guarantee: QoS 2 is four packets instead of one, plus state held on both ends. Resist the urge to “use 2 everywhere just in case”: ten thousand sensors publishing at QoS 2 every second load a broker an order of magnitude harder, to no benefit.

# Retain, Last Will, sessions and keep-alive

Retain — the last known value

An ordinary message reaches only those subscribed at that moment. A message with the retain flag is also remembered by the broker and handed to every new subscriber the instant they subscribe. This is the standard answer to the perennial problem of a dashboard that opens blank and stays blank until a sensor deigns to send its next reading.

A topic holds exactly one retained message — the latest. To clear it, publish an empty payload to the same topic with the retain flag set.

Last Will — a message for when the line drops

On connecting, a client can hand the broker a will: a topic, a payload and its options. If the connection dies impolitely — without a DISCONNECT packet — the broker publishes that message on the client's behalf. The classic pattern is a retained device/42/status topic that gets online on connect and carries offline as the will. The system always knows who is present, with no polling at all.

Sessions: clean and persistent

A clean session lasts exactly as long as the connection: disconnect and the subscriptions are forgotten. A persistent one survives the break: the broker remembers the subscriptions and queues QoS 1 and 2 messages while the client is away, then delivers the backlog on return. For a device that wakes once an hour, this is the only way to miss nothing.

Keep-alive — noticing that the link has died

A broken TCP connection can look perfectly alive to both ends for a long time. So the client declares a keep-alive interval and, with nothing else to say, sends a ping. Hearing nothing for one and a half intervals, the broker declares the client gone, closes the connection and publishes its last will.

# The versions: 3.1, 3.1.1 and 5.0

Three versions turn up in practice, and the differences between them are not cosmetic.

Separate from these is MQTT-SN, a sibling protocol for networks with no TCP: ZigBee, radio links, UDP. It is not “a version of MQTT” but its own specification; such networks get a gateway that translates the traffic into ordinary MQTT.

What MQTT 5.0 actually buys you

Version 5 grew out of accumulated practical annoyances. The most consequential parts:

• Reason codes. In 3.1.1 a refusal looked like a silently closed socket, leaving you to guess. In 5.0 the server says which it was: wrong password, no permission for that topic, packet too large.
• Packet properties. A message can carry metadata: Content Type, Correlation Data, arbitrary user key-value pairs. Previously all of that got stuffed into the payload or the topic name.
• Request and response. The Response Topic and Correlation Data fields make ordinary RPC a first-class pattern rather than a hand-rolled convention.
• Shared subscriptions via $share/group/filter: several instances of a consumer split the stream between them. That is load balancing and failover at the protocol level.
• Session and message lifetimes. Session Expiry and Message Expiry let you say “keep it for a day, then forget it” instead of the binary clean-or-not-clean choice.
• Flow control. Receive Maximum caps the number of unacknowledged messages in flight, and Maximum Packet Size protects a small device from a packet it cannot digest.
• Topic aliases. A long name is sent once and replaced by a two-byte number thereafter — a real saving on deep hierarchies.
• Subscription identifiers, No Local, Retain As Published, Retain Handling. Small things that remove long-standing traps: hearing the echo of your own messages, an avalanche of retained messages on every reconnect, not knowing which subscription a message arrived through.
• Delayed will. Will Delay Interval stops a momentary blip raising an alarm: the will is published only if the client fails to return in time.

The practical upshot: start new projects on 5.0. Shared subscriptions and intelligible error codes save weeks of debugging, and compatibility with older devices is not lost.

# MQTT against HTTP and the alternatives

“Why do I need MQTT when REST exists?” is a fair question. The difference is not fashion but the direction and the cost of a conversation.

Polling makes the point. To learn about an event within a second over HTTP, a thousand devices must ask a thousand times a second, and nearly every answer will be “nothing new”. In MQTT the event arrives by itself the moment it happens, and costs no traffic in the meantime.

What about CoAP, AMQP and WebSocket

• CoAP is REST over UDP for very small nodes. Lighter than MQTT, but offers neither queueing nor reliable delivery without effort.
• AMQP is heavier and richer: elaborate routing, transactions, enterprise-grade queues. Sensible between servers, excessive for a sensor.
• WebSocket is a transport, not an alternative: MQTT itself runs over WebSocket so that it works inside a browser.
• Sparkplug B is not a rival but a layer on top of MQTT: it dictates how to name topics and encode payloads in industrial systems, so that SCADA software from different vendors understands the same data.

# Transport, ports and payloads

MQTT runs over TCP and asks only for reliable, ordered delivery. The conventional ports:

To the protocol, a payload is just a sequence of bytes. MQTT knows nothing about the contents and imposes nothing: JSON, CBOR, protobuf, a single number or an image all qualify. The formal limit is 256 MB, but in practice anything beyond a few hundred kilobytes belongs on HTTP, with only a link travelling over MQTT.

JSON dominates in practice: a human can read it, every language parses it, and it is compact enough. On a narrow link a binary format earns its keep — and that is exactly where MQTT 5.0's Content Type property helps, by stating honestly what is inside.

# Security

MQTT encrypts nothing by itself: on port 1883 the username and password travel in the clear. Security comes from three independent layers, and none of them is optional.

Encrypting the channel

TLS on port 8883. For devices too weak to manage TLS, the only decent option is to keep them on an isolated network and let nothing out except through a gateway.

Authentication

The classic is a username and password in the CONNECT packet. Stricter is client certificates: the device presents its own and the broker verifies the signature. More flexible is JWT: the device brings a signed, time-limited token, and revoking access needs no change on the broker. Good brokers can also read accounts from an external database, a CSV file or an HTTP service, so the device list does not have to exist in two places.

Topic permissions

Authentication answers “who are you”; authorisation answers “what may you do”. Access control lists say which topics a client may read and which it may write. The correct setting is deny everything except what is explicitly allowed.

On top of that come publish rate limits (so one deranged device cannot bury the broker), a maximum packet size, and ordinary network hygiene: the broker's dashboard has no business facing the open internet.

# Where MQTT is actually used

Smart homes and building automation

The largest field by volume. Home Assistant, Zigbee2MQTT, ESPHome and openHAB all speak through an MQTT broker, which doubles as the glue between hardware from different manufacturers: a socket, a leak sensor and a heating controller from three vendors cooperate perfectly, because each simply writes to its own topic.

Industry and SCADA

Machine readings, line status, runtime counters. The classic arrangement is a gateway that reads Modbus or OPC UA from the equipment and republishes it over MQTT, after which SCADA, a historian and a predictive-maintenance system all consume it at once without interfering. This is where Sparkplug B and MQTT 5.0's shared subscriptions earn their place.

Metering and utilities

Water, gas, electricity and heat meters: battery devices that wake once an hour, report a reading and go back to sleep. Persistent sessions and retain do the heavy lifting here — the server always sees the latest value even if a meter will not check in until the evening.

Transport and telematics

Coordinates, fuel consumption, refrigerator status, driver behaviour. The link is a cellular network that vanishes in tunnels and out of town. Queueing and redelivery after reconnect turn a ragged connection into a continuous stream of data.

Energy, agriculture, healthcare

Solar plants and substations, irrigation systems and weather stations, patient monitors and vaccine refrigerators. What they share: many endpoints, a weak link, events that must not be lost, and a need to hear about a fault immediately.

Smart cities and logistics

Street lighting, parking, bins that report how full they are, cargo tracking and cold-chain temperature. Tens of thousands of devices sending short, infrequent messages — precisely the load profile MQTT was designed for.

# Designing your topics: decisions worth making early

Your topic scheme is your system's API. Changing it later hurts, because firmware and server have to be updated in step. A few rules that save time.

• General to specific. plant/hall3/line2/machine7/temperature lets you subscribe at any level. The reverse order makes wildcards useless.
• Give the device id its own level. That is what lets a single ACL rule confine a device to its branch.
• No leading slash. /office/temp creates an empty first level — legal, but a lasting source of confusion.
• Latin letters, digits, hyphen and underscore only. Spaces and the +, # and $ characters in names cause failures nobody expects.
• Never encode changing data in the topic. sensor/42/temp/21.5 is a mistake: the value belongs in the payload, or the broker ends up with an unbounded topic tree and retain becomes useless.
• Separate state from commands. For instance device/42/state for what the device reports and device/42/cmd for what it is told. Otherwise the echo of your own commands will eventually loop the logic.
• Retain status, not telemetry. Retain is right for “last known state” and pointless for a stream of readings.

# Choosing a broker

There are many brokers, and the choice usually comes down to a handful of questions.

• Is MQTT 5.0 supported in full? Partial support is common, and you usually find out at the worst possible moment.
• Can you see what is happening? Being able to watch who is connected, which topics are live and what is flying past right now saves hours of debugging. A broker without a dashboard turns every problem into log archaeology.
• How do devices get provisioned? With thousands of them, by hand is not an option — you need an API or accounts in an external database.
• Does it survive a restart? Retained messages, persistent sessions and queues must reach disk, or a planned reboot means losing data.
• What limits are available? One deranged device must not take the system down: rate and volume limits matter.
• What does it cost and where does it run? A cloud service is convenient until it starts charging per message; your own server needs administering but keeps the data yours.

For a small or medium installation — from a smart home to a plant with a few thousand endpoints — a single broker on a modest server is normally enough. Clustering is needed far less often than it seems during design; more often it pays to link two independent brokers with a bridge that forwards only the topics that matter.

# Getting started in five minutes

The fastest way to understand any of this is to run a broker and watch the messages yourself. On Debian or Ubuntu that is three commands:

sudo wget -qO /usr/share/keyrings/elx-repo.gpg https://repo.um-d.ru/elx-repo.gpg
echo "deb [signed-by=/usr/share/keyrings/elx-repo.gpg] https://repo.um-d.ru stable main" | sudo tee /etc/apt/sources.list.d/elx-repo.list
sudo apt-get update && sudo apt-get install elxmqttbroker

The dashboard opens at http://your-server:8080. Create a user, then try the exchange with any client — the console mosquitto tools, for example:

# in one window — subscribe to everything the device sends
mosquitto_sub -h localhost -u sensor-42 -P password -t 'sensor-42/#' -v

# in another — publish a reading
mosquitto_pub -h localhost -u sensor-42 -P password -t 'sensor-42/temp' -m '21.5'

# the same, but remembered for future subscribers
mosquitto_pub -h localhost -u sensor-42 -P password -t 'sensor-42/temp' -m '21.5' -r

From there, play with the retain flag, disconnect a subscriber and see what piles up in a persistent session, set a last will and pull the cable. Half an hour of that teaches more than any article, this one included.

# Quick answers