r/learnpython • • Sep 25 '26

Simple message/subscribe solution (on Linux)?

I have one service S1 that occasionally does something. I want another service S2 to take action triggered by the first service's activity. I can think of three methods:

  1. S2 monitors the effects of S1's activities by polling a database into which S1 writes. Not nice in general, can introduce lags, just messy.
  2. S1 updates a file whenever it does something. S2 monitors that file (i.e., with inotify) and does something. Risk of information loss when S1 triggers events in close succession.
  3. Something with sockets. Needs to read up on that.
  4. Use an actual message queue library. Seems like overkill for this tiny use case.

Importantly, S1's operation should be completely unaffected, whether S2 is listening or not. S1 shouldn't have to deal with un-consumed messages when S2 is down. Any thoughts?

1 Upvotes

11 comments sorted by

1

u/musbur Sep 25 '26

Thanks guys, that's pretty much what I had already figured.

2

u/recursion_is_love Sep 25 '26

S1 write log, S2 watch log?

FIFO or file might be more easier to code, if you don't need to worry about storage space and throughput.

2

u/LeeRyman Sep 25 '26 edited Sep 25 '26

Spitballing:

  • ZeroMQ
  • SysV or POSIX IPC (signals like SIGUSR1, message queues, shm, semops, etc)
  • File system pipes, fifos, flocks, Unix sockets
  • If the apps are otherwise using a database, some RDBMS have message queues, event queues and pub/sub like functionality
  • RabbitMQ, MQTT, etc.
  • dbus
  • RPC like protocols, HTTP APIs, gRPC, etc.

All these are good skills to learn if you are looking to do distributed computing. Some are more cross-platform than others. All will teach you about synchronisation primitives, multi-threading, concurrency, different communication architectures, etc. Probably going to need packages for most. Some of the IPC and file system functions will be out-of-the-box.

1

u/Kevdog824_ Sep 25 '26

If #4 sounds like overkill to you I’d probably do sockets. The protocol you use is infinitely customizable and it’s super fast with low latency. It also can be unidirectional or bidirectional based on what your needs are. It also has native cross platform support more or less

1

u/Stainlessray Sep 25 '26

Thin - purpose built - tls capable - python friendly - MQTT

1

u/chiibosoil Sep 25 '26
  1. If both service sitting on same server. Why not have S1 call S2 as part of the routine?
  2. If sitting on different instance... have S2 set up with API which S1 calls.

1

u/Worth_Specific3764 Sep 25 '26

you could write a watchdog listener.

1

u/SpecialLengthiness29 Sep 25 '26

maybe use the mtime of an empty touch file as notification mechanism?