Communication between worker processes #14

New Issue

Open

opened 2022-09-28 09:18:09 +00:00 by heartles · 2 comments

heartles commented

2022-09-28 09:18:09 +00:00

Owner

Need communication between processes for horizontal scaling

Communication needed for various tasks

Job queue
Streaming endpoints

How to? ZeroMQ? RabbitMQ? Redis?

Need communication between processes for horizontal scaling Communication needed for various tasks - Job queue - Streaming endpoints How to? ZeroMQ? RabbitMQ? Redis?

heartles added a new dependency 2022-10-13 10:12:02 +00:00

#7 ActivityPub federation

heartles added this to the v0.0.2 milestone 2022-10-13 10:27:31 +00:00

heartles added the

backend

label 2022-10-13 10:27:36 +00:00

heartles commented

2022-10-14 02:53:02 +00:00

Author

Owner

So here's my thought right:

I want to minimize external dependencies
We need communication both between threads/fibers and between parallel processes/nodes

With that in mind, I've got the following ideas:

Horizontal scaling is NOT supported when using SQLite
- SQLite backend is meant for smaller deployments and for testing
- The application can still be scaled out between threads, just not between processes
Threads in a single process can communicate using channels (see std.event.Channel)
Processes (when using the postgres backend) can communicate using postgres's NOTIFY/LISTEN.
- Typical messages would just consist of an opcode and an ID
- Each process has one thread dedicated to listening for messages and sending them over channels to the other threads
Jobs are stored as rows in the database
- New jobs are in "pending" state
  - Jobs queued for the future given a timestamp to wait for
- When a job is picked up, its status is set to "running" and a timestamp is written
- When the job fails, its status is set to "failed" and retry logic is applied (delay + retry counter)
- When the job finishes successfully, it gets deleted from the db
- Jobs that were started more than T ago are assumed to have failed and are treated as failed jobs

If this ends up having issues we can look into alternative job queues later.

So here's my thought right: - I want to minimize external dependencies - We need communication both between threads/fibers and between parallel processes/nodes With that in mind, I've got the following ideas: - Horizontal scaling is NOT supported when using SQLite + SQLite backend is meant for smaller deployments and for testing + The application can still be scaled out between threads, just not between processes - Threads in a single process can communicate using channels (see std.event.Channel) - Processes (when using the postgres backend) can communicate using postgres's [NOTIFY/LISTEN](https://www.postgresql.org/docs/current/sql-notify.html). + Typical messages would just consist of an opcode and an ID + Each process has one thread dedicated to listening for messages and sending them over channels to the other threads - Jobs are stored as rows in the database + New jobs are in "pending" state * Jobs queued for the future given a timestamp to wait for + When a job is picked up, its status is set to "running" and a timestamp is written + When the job fails, its status is set to "failed" and retry logic is applied (delay + retry counter) + When the job finishes successfully, it gets deleted from the db + Jobs that were started more than T ago are assumed to have failed and are treated as failed jobs If this ends up having issues we can look into alternative job queues later.

heartles added a new dependency 2022-10-14 02:57:13 +00:00

#13 Internal Job Queue

heartles self-assigned this 2022-10-14 02:57:27 +00:00

heartles changed title from ~~Decide on IPC strategy~~ to Communication between worker processes

2022-10-16 13:14:31 +00:00

heartles commented

2022-11-11 07:10:24 +00:00

Author

Owner

https://github.com/ziglang/zig/issues/6025

Going to push back this feature to another release

After working on this for a while, I'm running into compiler bugs. Most of them are a result of the stage1 compiler sucking, but it's also partly caused by the self-hosted compiler not supporting async/await. https://github.com/ziglang/zig/issues/6025 Going to push back this feature to another release

heartles added the

waiting for zig 0.11

label 2022-11-11 07:11:30 +00:00

heartles removed this from the v0.0.2 milestone 2022-11-11 07:11:35 +00:00