Building a real-time telemetry pipeline for 15,000 edge devices

When you have 15,000 edge routers all reporting telemetry, the data doesn't arrive at a steady rate. It arrives in bursts — firmware updates finish simultaneously, power cycles happen across a region, monitoring intervals align. You need a pipeline that handles the quiet periods cheaply and the spikes without dropping data.

Here's how we built ours.

Why MQTT

MQTT is purpose-built for constrained devices on unreliable networks. It's a pub/sub protocol with three QoS levels (fire-and-forget, at-least-once, exactly-once), small packet overhead, and persistent sessions that survive reconnects without losing messages.

Each router publishes to a topic like devices/{device_id}/telemetry. The broker handles fan-out. Our Go service subscribes once and processes everything.

We used Eclipse Mosquitto as the broker. It's battle-tested and handles the connection load without drama.

The Go ingestion service

The core loop is simple: subscribe to the wildcard topic devices/+/telemetry, parse the payload, write to the database. The complexity is in what happens between subscribe and write.

func (w *Worker) Start(ctx context.Context) {
    token := w.client.Subscribe("devices/+/telemetry", 1, w.handleMessage)
    token.Wait()

    <-ctx.Done()
    w.client.Disconnect(250)
}

func (w *Worker) handleMessage(_ mqtt.Client, msg mqtt.Message) {
    w.queue <- msg // non-blocking channel
}

The handler pushes to a buffered channel. A pool of goroutines drains the channel and writes to TDengine (a time-series database optimised for IoT workloads). This decouples ingestion speed from write speed.

Backpressure

The buffered channel acts as a shock absorber for bursts. But what happens when the channel fills up faster than it drains?

We made a deliberate choice: drop the oldest messages, not the newest. Fresh telemetry is more valuable than stale data from 30 seconds ago. The dashboard cares about the current state of a device, not what it was doing during a spike.

We log the drop rate as a metric. If it stays consistently above zero, that signals we need more workers or a broker-side rate limit per device.

Deduplication

Devices on QoS 1 (at-least-once) can resend messages on reconnect. We deduplicate on (device_id, sequence_number) using a short TTL cache before writing. The cache is in-process — no Redis needed for this. Sequence numbers roll over every 65,536 messages, and two messages with the same sequence number arriving within a 60-second window is the duplicate window we care about.

Fan-out to the dashboard

The Go service writes to the time-series DB and also publishes a lightweight event to an internal NATS subject. The Vue 3 dashboard subscribes via WebSocket and updates device cards in real time. This keeps the dashboard decoupled from the ingestion path — it doesn't query the DB on every device update.

What we got wrong first

We initially wrote directly to Postgres. It worked at 100 devices. At 2,000, write latency started climbing during bursts because row-level locking under heavy concurrent inserts is expensive. Switching to TDengine — which is built for append-only time-series data — dropped write latency by an order of magnitude under the same load.

The lesson: time-series data has access patterns that relational databases weren't designed for. Use the right tool for the write path.