This commit introduces GRPC API for streaming sessions.
It adds structured events and sync streaming
that avoids storing events on disk.
You can find design in rfd/0002-streaming.md RFD.
This PR improves session recording:
* Nodes and proxies always buffer recorded sessions
to disk during the session what improves performance
and makes the recording more resilient to network failures.
* Async uploader running on proxy or node always uploads the
session tarball to the audit log server.
* Audit log server is the only component uploading
to the S3 or any other API.
Updates #1755
Design
------
This commit adds support for pluggable events and
sessions recordings and adds several plugins.
In case if external sessions recording storage
is used, nodes or proxies depending on configuration
store the session recordings locally and
then upload the recordings in the background.
Non-print session events are always sent to the
remote auth server as usual.
In case if remote events storage is used, auth
servers download recordings from it during playbacks.
DynamoDB event backend
----------------------
Transient DynamoDB backend is added for events
storage. Events are stored with default TTL of 1 year.
External lambda functions should be used
to forward events from DynamoDB.
Parameter audit_table_name in storage section
turns on dynamodb backend.
The table will be auto created.
S3 sessions backend
-------------------
If audit_sessions_uri is specified to s3://bucket-name
node or proxy depending on recording mode
will start uploading the recorded sessions
to the bucket.
If the bucket does not exist, teleport will
attempt to create a bucket with versioning and encryption
turned on by default.
Teleport will turn on bucket-side encryption for the tarballs
using aws:kms key.
File sessions backend
---------------------
If audit_sessions_uri is specified to file:///folder
teleport will start writing tarballs to this folder instead
of sending records to the file server.
This is helpful for plugin writers who can use fuse or NFS
mounted storage to handle the data.
Working dynamic configuration.
SearchEvents and SearchSessionEvents set
default limits of fetched events of 500 up to 10K events
per interval.
No UI changes are required, as users will be able
to refine the search interval to fetch more sessions.
* Allow external audit log plugins
* Add support for auth API server plugins
* Add license file path configuration parameter (not used in open-source)
* Extend audit log with user login events
This is a fix for file leak in audit log server caused
by design issue:
Session file descriptors in audit log were opened on demand
when the session event or byte stream chunk was reported.
AuditLog server relied on SessionEnd event to close the
file descriptors associated with the session.
However, when SessionEnd event does not arrive (e.g.
there is a timeout or disconnect), the file descriptors
were not closed. This commit adds periodic clean up
of inactive sessions.
SessionEnd is now used as an optimization measure
to close the files, but is not used as the only
trigger to close files.
Now, inactive idle sessions, will close file descriptors
after periods of inactivity and will reopen the file
descriptors when the session activity resumes.
SessionLogger was not designed to open/close files
multiple times as it was reseting offsets
every time the session files were opened. This
change fixes this condition as well.
Teleport configuration now has a new field: NoAudit (false by default,
which means audit is always on).
When this option is set, Teleport will not record events and will not
record sessions.
It's implemented by adding "DiscardLogger" which implements the same
interface as teh real logger, and it's plugged into the system instead.
NOTE: this option is not exposed in teleport in any way: no config file,
no switch, etc. I quickly needed it for Telecast.
Andrej Tokarčík