Ledger db migrations

[rakanalh]

Description

This PR implements a migrator which atomically executes ledger DB migrations. The process runs as follows:

1. Create a temporary directory which contains rocksdb files from the current ledger DB.
2. Execute all migrations on temporary directory
3. If all are successful, mark migrations as such
4. Move successful temporary files into the original ledger DB directory.

This happens of course, in case where no ledger DB instance holds the lock on the directory. Otherwise, the process will fail.

This also implements the possibility to execute different migrations for each of sequencer, full node and prover.

TODO

• Implement an example migration to update table data
• Implement an example migration to update table fields
• Implement an example migration to delete table
• Add tests

Linked Issues

• Fixes LedgerDB supports migrations #1338

[codecov]

Codecov Report

Attention: Patch coverage is 94.47005% with 12 lines in your changes missing coverage. Please review.

Project coverage is 78.4%. Comparing base (07729cc) to head (b947778).
Report is 1 commits behind head on nightly.

Files with missing lines	Patch %	Lines
...dk/full-node/db/sov-db/src/ledger_db/migrations.rs	93.5%	5 Missing ⚠️
...reign-sdk/full-node/db/sov-db/src/ledger_db/mod.rs	85.0%	3 Missing ⚠️
...overeign-sdk/full-node/db/sov-schema-db/src/lib.rs	0.0%	3 Missing ⚠️
...ign-sdk/full-node/db/sov-db/src/ledger_db/tests.rs	98.8%	1 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
bin/citrea/src/rollup/mod.rs	97.1% <100.0%> (+0.2%)	⬆️
crates/fullnode/src/db_migrations/mod.rs	100.0% <100.0%> (ø)
crates/fullnode/src/lib.rs	50.0% <ø> (ø)
crates/prover/src/db_migrations/mod.rs	100.0% <100.0%> (ø)
crates/prover/src/lib.rs	50.0% <ø> (ø)
crates/sequencer/src/db_migrations/mod.rs	100.0% <100.0%> (ø)
crates/sequencer/src/lib.rs	64.2% <ø> (ø)
...reign-sdk/full-node/db/sov-db/src/schema/tables.rs	91.9% <ø> (ø)
...ign-sdk/full-node/db/sov-db/src/ledger_db/tests.rs	98.8% <98.8%> (ø)
...reign-sdk/full-node/db/sov-db/src/ledger_db/mod.rs	79.7% <85.0%> (+0.6%)	⬆️
... and 2 more

... and 2 files with indirect coverage changes

[eyusufatik]

we could also create the new table here right?

[rakanalh]

Yes, TestTableNew is a new table and the call to put_value here is adding entries into this new table.

[eyusufatik]

but isn't that table already in the schema when you start the process? I'm talking about creating a new table that we haven't defined in LEDGER_TABLES previously, but we do now, with the migration

[rakanalh]

So tables are not "created" when you start the process. They are created as soon as you write to them.

When we create the inner: DB part of the LedgerDB, we pass the schema of those tables but it won't actually create them until they're written to.

[rakanalh]

The idea behind the TestTableNew is to demonstrate how we can define the new table's schema and we add it to the code and we get the code to use it. However, it doesn't mean that it has data to be used. That's what migrations are for.

[ercecan]

LGTM