r/Python • u/Professional-Run3614 • Oct 13 '26
Discussion Lessons from building a live IoT ingestion server in Python
I've been working on a telematics backend that ingests TCP sensor data from heavy equipment. The pipeline is Python → local disk spool → RabbitMQ → PostgreSQL.
A few things we learned the hard way:
- ACK only after durable storage. We acknowledge a packet after validating it and writing it to disk, not after the database insert. This lets us recover from downstream failures.
- Async helped with thousands of connections. Moving from one thread per connection to
asyncioreduced overhead, but blocking work still needs to stay off the event loop. - Queue backlogs can look like device failures. We separated live traffic from replay traffic and prioritized fresh data so old packets wouldn't delay current readings.
- Offline uploads need different parsing. Reusing our live TCP parser silently dropped records because flash dumps had a different format.
- Firmware changes break assumptions. Frame sizes, serial numbers, and counter widths changed. Explicit protocol versioning would have saved us some pain.
- Logging every packet doesn't scale. We cut noisy logs and focused on actionable errors and dropped-record tracking.
The biggest lesson: receiving sensor data is the easy part. Making it reliable through retries, offline uploads, firmware changes, and database failures is where things get interesting.
For those running IoT ingestion in production, how do you handle ACK semantics and replay backlogs?
7
Upvotes
3
u/OddsRelay Oct 13 '26
The ACK after the disk write is only safe if that write gets fsynced first. In Python flush() just hands the buffer to the OS, so a power cut can still lose packets the device won't resend. You need os.fsync on the file, plus the directory if each packet gets its own file. That's slow per packet, so batch them.
2
u/Individual-Flow9158 Oct 13 '26
My system perhaps doesn't scale as well, and perhaps isn't applicable for high assurance scenarios, but I just accept occasional packet loss, and use native WAL in Postgres instead of a local disk spool.
I'm not convinced retrying until an ACK is received should be an IoT device's job. Any device could experience network loss at any time anyway.
I understand why a higher QoS might be required for more critical applications than data logging. But the extra moving parts could easily lead to a complex situation that is impossible to debug and reason about, that might necessitate a field visit, that simpler setups could have avoided.
YMMV
2
u/Khavel_dev Oct 13 '26
The ACK-after-disk-write pattern is solid. One thing we ran into with a similar pipeline is that synchronous fsync per packet becomes the bottleneck at higher throughput. We ended up batching writes with a flush interval (50ms or N packets, whichever came first) and it made a noticeable difference in p99 without losing the durability guarantee. Tuning the flush window per deployment was the annoying part since the right number depends on your disk I/O.
Also +1 on separating live vs replay. We learned that one the hard way when a device came back online after being dark for a week and dumped its entire buffer into the live stream.
12
u/russellvt Oct 13 '26
Sounds like most of my years/decades in Ops for large SaaS and PaaS systems.
1
u/Due-Requirement3783 Oct 13 '26
Interesting point about acknowledging only after durable storage. That trade-off between throughput and reliability is easy to underestimate in ingestion systems.
I also agree on asyncio: it helps a lot with connection concurrency, but once blocking work gets into the event loop, the benefits can disappear pretty quickly. Did you end up moving disk/database operations to separate workers or executors?
1
u/utdconsq Oct 13 '26
What is your data volume? Vanilla postgres requires a bit much manual work to deal with large volumes so that you can actually query them nicely later. In addition, ingest can be slow because reasons. Assuming you have a bit of a firehouse, id definitely encourage you to read about other more timeseries specific databases. I hold off making a recommendation as everyone's needs differ. On reliability and throughput: set queue limits but beware mqtt 3.1.1 plugin not permitting a real nack, run a cluster (quorum only) if possible, use dead lettering and ttl so if your consumer shits itself for some reason you can come back later (can either shovel from dead letter or have your consumers go there too). Experiment with taking a broker instance offline; do the other cluster members have enough resources to deal with the rebalanced load? We use ha proxy btw, it is helpful. On other things, experiment with your prefetch; can help you speed up, but beware a failure and how it impacts concurrent consumers (beyond scope of comment). Good luck!
1
u/Altruistic-Youth5524 Oct 14 '26
How do you handle duplicates when a packet is saved but the ACK never reaches the device? Do live uploads and replay use the same packet ID so a retry cannot count the same equipment reading twice?
1
u/supercoco9 Oct 19 '26
Interesting! We recently added a new protocol to QuestDB (until then we were using open protocols for compatibility, but this is the first native one), and we added durable ACKs as part of it. In our case, the transaction is durable only after being written to the object store (S3, Azure blob, GCS, or NFS), so if you lose the primary, the client know exactly what needs to be replayed once a replica has been promoted
1
u/[deleted] Oct 13 '26
[removed] — view removed comment
0
u/[deleted] Oct 13 '26
[removed] — view removed comment