parallel processing - Python producer and consumer system design -


if i'm writing python program execute other 2 executables(c++) in producer , consumer manner, right lib do? asyncio? multiprocessing?

the main execution thread or process is:

  1. monitoring status of producer , consumer processes.
  2. if 1 of them fails, terminate other.
  3. if ctrl-c received, terminate both of them
  4. if both of them finishes, exit , report success.

the producer:

  1. keep generating data temp file. (that temp file might fifo os.mkfifo()?)

the consumer:

  1. keep consuming temp file

both producer , consumer cpu intensive, , need run in parallel. ideally intermediate temp file doesn't need touch real storage , stay in memory. otherwise performance big concern since temp file of size 10gb or so.


Comments

Popular posts from this blog

ZeroMQ on Windows, with Qt Creator -

unity3d - Unity SceneManager.LoadScene quits application -

python - Error while using APScheduler: 'NoneType' object has no attribute 'now' -