background job processing for delivering ActivityPub pushes #19

New issue

Open

opened 2020-05-19 19:22:49 +02:00 by schmittlauch · 5 comments

schmittlauch commented

2020-05-19 19:22:49 +02:00

Owner

The relay node role requires nodes to push new posts to subscribing servers. Such actions can be queued and processed asynchronously in the background.

Mastodon uses Sidekiq as a job executor and scheduler. Do similar things exist in the Haskell world, are they necessary and should this project use them? Or are forkIOd threads, channels and maybe some persistence layer in between enough?

The relay node role requires nodes to push new posts to subscribing servers. Such actions can be queued and processed asynchronously in the background. Mastodon uses Sidekiq as a job executor and scheduler. Do similar things exist in the Haskell world, are they necessary and should this project use them? Or are `forkIO`d threads, channels and maybe some persistence layer in between enough?

schmittlauch added the

basic functionality

ActivityPub

labels 2020-05-19 19:27:02 +02:00

schmittlauch commented

2020-05-19 19:28:24 +02:00

Author

Owner

The DHT layer also needs to schedule regular maintenance jobs, but as those won't pile up such an asynchronous background processing worker queue isn't necessary.

Hecate commented

2020-05-19 19:36:40 +02:00

Contributor

@schmittlauch If you wish to use one of the best and full-featured libraries on Hackage, I can only recommend odd-jobs.

@schmittlauch If you wish to use one of the best and full-featured libraries on Hackage, I can only recommend [*odd-jobs*](https://www.haskelltutorials.com/odd-jobs/).

schmittlauch commented

2020-05-19 19:40:18 +02:00

Author

Owner

This one pulls in Postgres. I still have to decide on how to persist data, will probably ask around in the SocialHub thread quite soon.

Note to self for further reading: https://www.haskelltutorials.com/odd-jobs/haskell-job-queues-ultimate-guide.html

This one pulls in Postgres. I still have to decide on how to persist data, will probably ask around in the SocialHub thread quite soon. Note to self for further reading: https://www.haskelltutorials.com/odd-jobs/haskell-job-queues-ultimate-guide.html

Hecate commented

2020-05-19 19:43:05 +02:00

Contributor

I personally think Postgres is far from being "overkill" if you are going to build some kind of highly-connected network service, especially if you wish to do some retention. Of course it's not like sqlite3, but it's a fairly standard tool on the "engineering" side of things. :)

I personally think Postgres is far from being "overkill" if you are going to build some kind of highly-connected network service, especially if you wish to do some retention. Of course it's not like `sqlite3`, but it's a fairly standard tool on the "engineering" side of things. :)

schmittlauch added this to the 1st round of experiments milestone 2020-06-03 00:54:29 +02:00

schmittlauch commented

2020-09-16 01:13:32 +02:00

Author

Owner

in the mock post service, this is done by separate worker threads and in-memory.
Thus removing it from the milestone.

in the mock post service, this is done by separate worker threads and in-memory. Thus removing it from the milestone.

schmittlauch removed this from the 1st round of experiments milestone 2020-09-16 01:13:38 +02:00