improve data migration performance #66

New issue

Open

opened 2020-08-18 11:58:53 +02:00 by schmittlauch · 1 comment

schmittlauch commented

2020-08-18 11:58:53 +02:00

Owner

Currently, the relay subscription list is done as a single transaction.

Disadvantages of that approach:

long running transactions have a bad performance, as they are aborted by concurrent transactions to the same data, have to be rolled back and run again
while the atomic transaction is in progress, it prevents modification of its modified variables like new subscriptions

But some of its properties need to be kept when breaking up the atomic transaction:

check responsibility for tags first before modifying the subscriber list
prevent concurrent migrations from already migrating a tag away before the incoming migration has written the received data
- danger of data loss
- possible approach: locking the migration with an MVar

Currently, the relay subscription list is done as a single transaction. Disadvantages of that approach: - long running transactions have a bad performance, as they are aborted by concurrent transactions to the same data, have to be rolled back and run again - while the atomic transaction is in progress, it prevents modification of its modified variables like new subscriptions But some of its properties need to be kept when breaking up the atomic transaction: - check responsibility for tags first before modifying the subscriber list - prevent concurrent migrations from already migrating a tag away before the incoming migration has written the received data - danger of data loss - possible approach: locking the migration with an MVar

schmittlauch added the

refactoring

label 2020-08-18 11:59:01 +02:00

schmittlauch commented

2020-08-27 11:26:44 +02:00

Author

Owner

possible alternative: use a global MVar for whether a migration is in progress (finer grained might be possible as well), thus serialising the migrations.

problem: possible message loss when not all subscriptions are transferred yet and new incoming messages are only forwarded to already migrated subscriptions, but won't reach yet-to-be-migrated ones.

counter measure: stop semantics, during migration new messages are rejected and need to be queued for retry at the senders.

possible alternative: use a global MVar for whether a migration is in progress (finer grained might be possible as well), thus serialising the migrations. problem: possible message loss when not all subscriptions are transferred yet and new incoming messages are only forwarded to already migrated subscriptions, but won't reach yet-to-be-migrated ones. counter measure: stop semantics, during migration new messages are rejected and need to be queued for retry at the senders.