feat(app): non blocking enqueue_batch

[psychedelicious]

Summary

Most service methods in the application are blocking. For example, enqueue_batch blocks the main thread - preventing HTTP and socket traffic. When enqueuing very large or complicated batches, this can cause unresponsiveness.

To make enqueue_batch non-blocking, we can make it async and run it in a thread. However, we have another source of "blocking" - a global mutex on database operations.

This global mutex is required because SQLite, by default, does not support concurrent reads and writes. We painstakingly wrap all DB operations in a re-entrant lock to prevent issues with interacting with a locked database.

SQLite has an alternate mode called WAL (write-ahead logging) mode, which supports concurrent reads and allow us to treat writes as concurrent.

(Technically, in WAL mode, SQLite will queue writes using the WAL log as a buffer, so it's not true concurrency. But we can interact with the DB as it if was concurrent.)

We can also enable the SQLite busy timeout at 5s. If we are waiting on an operation for more than 5s, SQLite will error. This is plenty of time for any SQLite operation we may need to do. If something takes longer than that, we have problems and should fail loudly.

The busy timeout ends up being a better failure mode than we had before w/ the global mutex, which had no timeout and could lock up the app indefinitely if something went awry.

So, to make enqueue_batch non-blocking, we need to:

• Run that function in a thread and await it in the router.
• Enable WAL mode and set a busy timeout.
• Remove the global mutex.
  • As a bonus, this makes every method that touches the DB simpler to read.

I also cleaned up many methods that touched the DB. Many read operations caught errors and did a connection rollback, but this is not necessary. Reads do not have anything to roll back and calling rollback is a noop, so we can remove this nonfunctional logic.

There's one draw-back to WAL mode. You cannot open the DB file over the network. For example, if your host machine is different from your client, and you share the DB via SMB, you cannot connect to the DB over SMB from the client.

Long ago, this drawback was the reason to not use WAL mode. But now, I think it is a fine tradeoff. You can still SSH to the host and connect to the DB directly.

Finally, I made a change to our RTKQ implementation fixing an issue where, during a long-running mutation (e.g. enqueuing a big batch), invalidated queries are not immediately re-fetched (e.g. changing a board's name would wait until the enqueue completes before refreshing).

Related Issues / Discussions

n/a

QA Instructions

Try the app. Everything should work. When enqueuing huge, complicated batches, the app should remain responsive.

Thanks to the massive perf improvements to batch prep in #7688, the benefits from making this change will be less impactful as we should never be blocked for long. Nevertheless, this really cleans up our DB code and reduces footgun potential from mishandled mutexes.

Merge Plan

n/a

Checklist

• [z] The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[psychedelicious]

Note for reviewers - there is very little code added in this PR. The diff is large because of changed indentation. Many extraneous try/except blocks were removed, shifting code a bit. git isn't representing this very well.