This PR adds audit queue observability to tctl. It allows you to view
the audit queue depth and also sort records based on number of events in
each queue.
This PR attempts to drain the queue during a graceful shutdown. It adds
the method Drain() to the Queue interface, which blocks while the queue
still has events in it.
This implements a mechanism that will move events that fail to
deserialize from the audit_queue into the corrupt_events table. This
will ensure that events that fail to deserialize don't "clog" the main
queue. These events do not belong in the dead-letter table because if
they fail to deserialize, then there is no point in retrying them.
Between process restarts, we will attempt to re-send corrupt_events in
case the failed deserialization was due to a version mismatch from a
Teleport downgrade and upgrade.
This implements the configuration options for the audit log queue as
described in the RFD. As requested, the queue implements a fallback
mechanism where the operator may specify an ordered list of audit log
queue backends to use. If we encounter an error constructing a queue
backend, we will fallback to the next option to try.
This adds an in-memory SQLite-backed audit log queue. The intention is
this can act as a fallback if there is an error constructing a disk
backed queue, for example, if writing to the filesystem causes an error.
A follow-up PR will implement the fallback logic.
Kyle Kloberdanz