teleport/docs/pages/reference/deployment/backends.mdx

---
title: Storage Backends
description: How to configure Teleport deployment for high-availability using storage backends
tags:
 - conceptual
 - platform-wide
---

A Teleport cluster stores different types of data in different locations. By
default everything is stored in a local directory on the Auth Service host.

For self-hosted Teleport deployments, you can configure Teleport to integrate
with other storage types based on the nature of the stored data (size,
read/write ratio, mutability, etc.).

| Data type               | Description                                                                                                                                 | Supported storage backends                                                                             |
|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| core cluster state      | Cluster configuration (e.g. users, roles, auth connectors) and identity (e.g. certificate authorities, registered nodes, trusted clusters). | Local directory (SQLite), etcd, PostgreSQL, Amazon DynamoDB, GCP Firestore, CockroachDB                |
| audit events            | Events from the audit log (e.g. user logins, RBAC changes)                                                                                  | Local directory, PostgreSQL, CockroachDB, Amazon DynamoDB, Amazon S3, GCP Firestore |
| session recordings      | Raw terminal recordings of interactive user sessions                                                                                        | Local directory, Amazon S3 (and any S3-compatible product), GCP Cloud Storage, Azure Blob Storage         |
| teleport instance state | ID and credentials of a non-auth teleport instance (e.g. node, proxy)                                                                       | Local directory, Kubernetes Secret                                                                     |

## Cluster state

Cluster state is stored in a central storage location configured by the Auth
Service. The cluster state includes:

- Agent and Proxy Service membership information, including offline/online
  status.
- List of active sessions.
- List of locally stored users.
- RBAC configuration (roles and permissions).
- Dynamic configuration.

There are two ways to achieve High Availability. You can "outsource"
this function to the infrastructure. For example, using a highly available
network-based disk volumes (similar to AWS EBS) and by migrating a failed VM to
a new host. In this scenario, there's nothing Teleport-specific to be done.

If High Availability cannot be provided by the infrastructure (perhaps you're
running Teleport on a bare metal cluster), you can still configure Teleport to
run in a highly available fashion.

(!docs/pages/includes/cloud/call-to-action.mdx!)

## Auth Service State

To run multiple instances of the Teleport Auth Service, you must switch to one of
the high-availability storage backends listed below first.

Once you have a high-availability storage backend and multiple instances of
the Auth Service running, you'll need to create a load balancer to evenly
distribute traffic to all Auth Service instances and have a single point of
entry for all components that need to communicate with the Auth Service. Use the
address of the load balancer in the [`auth_server`](config.mdx) field when
configuring other components of Teleport.

Configure your load balancer to use Layer 4 (TCP) load balancing, round-robin
load balancing, and a 300 second idle timeout.

<Admonition type="tip" title="NOTE">
With multiple instances of the Auth Service running, special attention needs to
be paid to keeping their configuration identical. Settings like `cluster_name`,
`tokens`, `storage`, etc. must be the same.
</Admonition>

## Proxy Service State

The Teleport Proxy is stateless which makes running multiple instances trivial.

If using the [default configuration](./networking.mdx), configure your load
balancer to forward port `3080` to the servers that run the Teleport Proxy
Service. If you have configured your Proxy Service to not use TLS Routing
and/or are using non-default ports, you will need to configure your load
balancer to forward the ports you specified for `listen_addr`,
`tunnel_listen_addr`, and `web_listen_addr` in `teleport.yaml`.

Configure your load balancer to use Layer 4 (TCP) load balancing, round-robin
load balancing, and a 300 second idle timeout.

<Admonition type="tip" title="NOTE">
If you terminate TLS with your own certificate for `web_listen_addr` at your
load balancer you'll need to run Teleport with `--insecure-no-tls`
</Admonition>

If your load balancer supports HTTP health checks, configure it to hit the
`/readyz` [diagnostics endpoint](../../zero-trust-access/management/diagnostics/monitoring.mdx) on
machines running Teleport. This endpoint must be enabled by using the
`--diag-addr` flag to teleport start:

```code
$ teleport start --diag-addr=0.0.0.0:3000
```

The `/readyz` endpoint will
reply `{"status":"ok"}` if the Teleport service is running without problems.
The endpoint must be exposed on a proxy interface for the load balancer health checks
to succeed.  You should only do this on the proxy instances and ensure that
port 3000 is not exposed to the public internet, just the load balancers.  For other services, continue to use
the 127.0.0.1 local loopback interface.

We'll cover how to use `etcd`, PostgreSQL, DynamoDB, and Firestore storage
backends to make Teleport highly available below.

## Etcd

Teleport can use [etcd](https://etcd.io/) as a storage backend to
achieve highly available deployments. You must take steps to protect access to
`etcd` in this configuration because that is where Teleport secrets like keys
and user records will be stored.

<Admonition
  type="warning"
  title="IMPORTANT"
>
  `etcd` can only currently be used to store Teleport's internal database in a
  highly-available way. This will allow you to have multiple Auth Service instances in your
  cluster for an High Availability deployment, but it will not also store Teleport audit events
  for you in the same way that [DynamoDB](#dynamodb) or
  [Firestore](#firestore) will. `etcd` is not designed to handle large volumes of time series data like audit events.
</Admonition>

To configure Teleport for using etcd as a storage backend:

- Make sure you are using **etcd versions 3.3** or newer.
- Follow [etcd's cluster hardware recommendations](https://etcd.io/docs/v3.6/op-guide/hardware/). In particular, leverage
  SSD or high-performance virtualized block device storage for best performance.
- Install etcd and configure peer and client TLS authentication using the [etcd
  security guide](https://etcd.io/docs/v3.6/op-guide/security/). See
  [Configuring mTLS](#configuring-mtls-authentication-with-etcd) below for
  details.
- Configure all Teleport Auth Service instances to use etcd in the "storage" section of the config file as shown below.
- Deploy several Auth Service instances connected to etcd backend.
- Deploy several Proxy Service instances that have `auth_server` pointed to the Auth Service to connect to.

### Configuring mTLS authentication with etcd

We recommend using [cfssl](https://github.com/cloudflare/cfssl) and etcd's
[`tls-setup` script](https://github.com/etcd-io/etcd/tree/main/hack/tls-setup)
to configure mTLS between Teleport and etcd. It's helpful for creating
certificates and keys for the entire etcd cluster. After [installing
cfssl](https://github.com/cloudflare/cfssl?tab=readme-ov-file#installation),
clone the `tls-setup` repo and follow the `Readme.md` file to customize the
script configuration. Make sure to tweak the makefile if you change the number
of etcd nodes (and thus the number of `infra` variables).

After you run `make`, the following files will be generated:

- `tls-setup/certs/ca.pem` is the certificate authority and it should be
  referenced in Teleport's `tls_ca_file` option (see [Configuring Auth Service
  instances to use etcd](#configuring-auth-service-instances-to-use-etcd)
  below).
- `tls-setup/certs/peer-*` files are used internally by etcd for communicating
  between nodes.
- `tls-setup/certs/<host>.pem` and `tls-setup/certs/<host>-key.pem` are the
  host certificates and their corresponding private keys. Each corresponding
  etcd host needs to know them, and they are required for providing transport
  security on the etcd side.

This should be enough to configure mTLS on etcd side.

On the Teleport side, you need to configure client certificates. Here's one way
to do it:

1. Generate a default CSR:
   ```code
   $ cfssl print-defaults csr > client.json
   ```

1. Remove all hostnames from the hosts element of the `client.json` file, as
   this CSR doesn't represent any host, but a client. Also customize any other
   options if required.

1. For each Auth Service instance, generate the client certificate:
   ```code
   $ cfssl gencert -ca=certs/ca.pem \
     -ca-key=certs/ca-key.pem \
     -config=config/ca-config.json -profile=client client.json \
     | cfssljson -bare certs/client
   ```
   This assumes that the `tls-setup` tool has been previously run and generated
   the `certs/ca.pem` and `certs/ca-key.pem` files.

   The following files will be generated:

   - `certs/client.pem`, which will be referenced by the `tls_cert_file` option
     in Teleport configuration file.
   - `certs/client-key.pem` which will be referenced by the `tls_key_file`
     option in Teleport configuration file.

   Upload these files to the corresponding Auth Service instance, along with
   `certs/ca.pem`.

After generating all the keys and certificates, run etcd using some variation
of this command, replacing <Var name="etcd-host"/> with your etcd hostname:

```code
$ etcd --cert-file=certs/host.pem \
  --key-file=certs/host-key.pem \
  --client-cert-auth \
  --trusted-ca-file=certs/ca.pem \
  --peer-cert-file=certs/peer-host.pem \
  --peer-key-file=certs/peer-host-key.pem \
  --advertise-client-urls=https://<Var name="etcd-host"/>:2379 --listen-client-urls=https://<Var name="etcd-host"/>:2379
  # ... other applicable options
 ```

### Configuring Auth Service instances to use etcd

Put these settings into the configuration file of all Auth Service instances:

```yaml
teleport:
  storage:
     type: etcd

     # List of etcd peers to connect to:
     peers: ["https://172.17.0.1:4001", "https://172.17.0.2:4001"]

     # Required path to TLS client certificate and key files to connect to
     # etcd.
     #
     # If you used cfssl as described above, these should be the
     # certs/client.pem and certs/client-key.pem files from the previous steps.
     tls_cert_file: /var/lib/teleport/etcd-cert.pem
     tls_key_file: /var/lib/teleport/etcd-key.pem

     # Optional file with trusted CA authority
     # file to authenticate etcd nodes
     #
     # If you used the tls-setup script, this CA certificate should be the
     # certs/ca.pem file.
     tls_ca_file: /var/lib/teleport/etcd-ca.pem

     # Alternative password-based authentication, if not using TLS client
     # certificate.
     #
     # See https://etcd.io/docs/v3.4.0/op-guide/authentication/ for setting
     # up a new user.
     username: username
     password_file: /mnt/secrets/etcd-pass

     # etcd key (location) where teleport will be storing its state under.
     # make sure it ends with a '/'!
     prefix: /teleport/

     # NOT RECOMMENDED: enables insecure etcd mode in which self-signed
     # certificate will be accepted
     insecure: false

     # Optionally sets the limit on the client message size.
     # This is usually used to increase the default which is 2MiB
     # (1.5MiB server's default + gRPC overhead bytes).
     # Make sure this does not exceed the value for the etcd
     # server specified with `--max-request-bytes` (1.5MiB by default).
     # Keep the two values in sync.
     #
     # See https://etcd.io/docs/v3.4.0/dev-guide/limit/ for details
     #
     # This bumps the size to 15MiB as an example:
     etcd_max_client_msg_size_bytes: 15728640
```

### Disk sizing, compaction and defragmentation

For large-scale deployments, plan for high write throughput and allocate sufficient storage to meet the history retention requirements.

A rule of thumb for minimum storage size (excluding metadata) is:
```
minimum_storage ≈ backend_resource_count * average_record_size * auto_compaction_retention
```

- `backend_resource_count`: the total number of resources including replicas.
- `average_record_size`: the average size (in bytes) of a resource.
- `auto_compaction_retention`: the maximum number of revisions per item.

Auto-compaction based on revision count is strongly recommended.
Note that physical disk space is not reclaimed during compaction. To recover space, schedule regular defragmentation
based on available disk space and write throughput.

Use `etcd_mvcc_db_total_size_in_bytes` and `etcd_mvcc_db_total_size_in_use_in_bytes` metrics to tune as needed.

Refer to [etcd's maintenance guide](https://etcd.io/docs/v3.6/op-guide/maintenance/) for auto-compaction and defragmentation.

#### Example disk sizing

For a cluster with 50k resources, with average size of 2 KiB and retention history of 5 revisions we would expect
the dataset to be at least:

```
50000 * 2 KiB * 5 ≈ 488 MiB
```

An additional 20-40% safety margin is recommended for metadata and fragmentation overhead.

Given an existing resource you can estimate its size with the following command,
assigning <Var name="resource" /> to the resource name, e.g., `role/editor`:

```code
tctl get <Var name="resource" /> --format=json | awk '{ total += length($0) } END { print total * 2 }'
```

## PostgreSQL

Teleport can use [PostgreSQL](https://www.postgresql.org/) as a storage backend
to achieve high availability. You must take steps to protect access to
PostgreSQL in this configuration because that is where Teleport secrets like
keys and user records will be stored. The PostgreSQL backend supports two types
of Teleport data:

- Cluster state
- Audit log events

The PostgreSQL backend requires PostgreSQL 13 or later, and, for the cluster
state only, the [`wal2json`](https://github.com/eulerto/wal2json) logical
decoding plugin. The plugin is available in packages for all stable versions in
the PostgreSQL [Apt](https://apt.postgresql.org/) and
[Yum](https://yum.postgresql.org/) repositories for Debian- and RPM-based Linux
distributions respectively, or it can be compiled following [the
instructions](https://github.com/eulerto/wal2json#build-and-install) provided in
its repository. The plugin is pre-installed with no extra steps to take in
[Azure Database for
PostgreSQL](https://azure.microsoft.com/en-us/products/postgresql).

<Admonition type="note">
CockroachDB can be used as a PostgreSQL drop-in replacement to store audit events (requires Teleport version >= 15.4.2).

Teleport can store the cluster state in CockroachDB but this require CockroachDB-specific configuration.
See the [CockroachDB backend section](#cockroachdb) for more details.
</Admonition>

Teleport needs separate databases for the cluster state and the
audit log, and it will attempt to create them if given permissions to do so; it
will also set up the database schemas as needed, so we recommend giving the user
ownership over the databases.

The PostgreSQL backend for cluster state relies on the ability to use [logical
decoding](https://www.postgresql.org/docs/current/logicaldecoding.html) to get a
stream of changes from the database; because of that, the
[`wal_level`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-WAL-LEVEL)
parameter must be set to `logical` and
[`max_replication_slots`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-REPLICATION-SLOTS)
must be set to at least as many Teleport Auth Service instances as you'll be
running (a higher number is recommended, to account for network conditions).

The Teleport Auth Service needs to be able to create a replication slot when
starting and when reestablishing a new connection to the PostgreSQL cluster, and
any long-running transaction will prevent that. It's therefore only advisable to
store the Teleport cluster state on a shared PostgreSQL cluster if the other
workloads on the cluster only consist of short-lived transactions.

`wal_level` can only be set at server start, so it should be set in
`postgresql.conf`:
```
# the default value for wal_level is replica
wal_level = logical

# the default value for max_replication_slots is 10
max_replication_slots = 10
```

In addition, the database user must have the `initiating replication` [role
attribute](https://www.postgresql.org/docs/current/role-attributes.html). In the
`psql` shell:

```
postgres=# CREATE USER new_user WITH REPLICATION;
CREATE ROLE

postgres=# ALTER ROLE existing_user WITH LOGIN REPLICATION;
ALTER ROLE
```

Since replication permissions allow for essentially full read access over the
entire cluster (with a physical replication connection) or to all databases that
the user can connect to, it's recommended to prevent the user from opening
replication connections, or from connecting to databases other than the ones
used for Teleport, if the PostgreSQL cluster is shared between Teleport and
other applications.

For convenience, Teleport will attempt to grant itself the `initiating
replication` role attribute, to accommodate the ability of some managed services
(such as Azure Database for PostgreSQL) to create superuser accounts through
their API; this should only be leveraged if the entire PostgreSQL cluster is
dedicated to Teleport.

To configure Teleport to use PostgreSQL:
- Configure all Teleport Auth Service instances to use the PostgreSQL backend in the
  `storage` section of `teleport.yaml` as shown below.
- Deploy several Auth Service instances connected to the PostgreSQL storage backend.
- Deploy several Proxy Service nodes.
- Make sure that the Proxy Service instances and all Teleport Agent services that
  connect directly to the Auth Service have the `auth_server` configuration
  setting populated with the address of a load balancer for Auth Service
  instances.

<Admonition
  type="warning"
  title="PgBouncer"
>
Teleport must connect directly to the Postgres server. `pgbouncer` is incompatible with the Teleport PostgreSQL storage backend.
</Admonition>

```yaml
teleport:
  storage:
    type: postgresql

    # conn_string is a libpq-compatible connection string (see
    # https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING);
    # pool_max_conns is an additional parameter that determines the maximum
    # number of connections in the connection pool used for the cluster state
    # database (the change feed uses an additional connection), defaulting to
    # a value that depends on the number of available CPUs.
    #
    # If your certificates are not stored at the default ~/.postgresql
    # location, you will need to specify them with the sslcert, sslkey, and
    # sslrootcert parameters.
    conn_string: postgresql://user_name@database-address/teleport_backend?sslmode=verify-full&pool_max_conns=20

    # In certain managed environments it can be necessary or convenient to
    # use a different user or different settings for the connection used
    # to set up and make use of logical decoding. If specified, Teleport
    # will use the connection string in change_feed_conn_string for that,
    # instead of the one in conn_string.
    change_feed_conn_string: postgresql://replication_user_name@database-address/teleport_backend?sslmode=verify-full

    # An audit_events_uri with a scheme of postgresql:// will use the
    # PostgreSQL backend for audit log storage; the URI is a libpq-compatible
    # connection string just like the cluster state conn_string, but cannot be
    # specified as key=value pairs. It's possible to specify completely
    # different PostgreSQL clusters for cluster state and audit log.
    #
    # If your certificates are not stored at the default ~/.postgresql
    # location, you will need to specify them with the sslcert, sslkey, and
    # sslrootcert parameters.
    audit_events_uri:
      - postgresql://user_name@database-address/teleport_audit?sslmode=verify-full
```

Audit log events are periodically deleted after a default retention period of
8766 hours (one year); it's possible to select a different retention period or
to disable the cleanup entirely, by specifying the `retention_period` or the
`disable_cleanup` parameters in the fragment of the URI:

```yaml
teleport:
  storage:
    audit_events_uri:
      - postgresql://user_name@database-address/teleport_audit?sslmode=verify-full#disable_cleanup=false&retention_period=2160h
```

### Authentication

We strongly recommend using [client
certificates](https://www.postgresql.org/docs/current/ssl-tcp.html#SSL-CLIENT-CERTIFICATES)
to authenticate Teleport to PostgreSQL, as well as enforcing the use of TLS and
[verifying the server
certificate](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBQ-SSL-CERTIFICATES)
on the client side.

You will need to update your `pg_hba.conf` file to include the following lines
to ensure connections to Teleport use client certificates. See The
[pg_hba.conf](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html)
file in the PostgreSQL documentation for more details.

```
# TYPE  DATABASE        USER            CIDR-ADDRESS            METHOD
hostssl teleport        all             ::/0                    cert
hostssl teleport        all             0.0.0.0/0               cert
```

If the use of passwords is unavoidable, we recommend configuring them in [the
`~/.pgpass` file](https://www.postgresql.org/docs/current/libpq-pgpass.html)
rather than storing them in Teleport's configuration file.

### Microsoft Entra ID authentication

If you are running Teleport on Azure, Teleport can make use of [Microsoft Entra
ID
authentication](https://learn.microsoft.com/azure/postgresql/flexible-server/concepts-azure-ad-authentication)
to connect to an Azure Database for PostgreSQL server without having to manage
any secrets:

```yaml
teleport:
  storage:
    type: postgresql

    conn_string: postgresql://user_name@database-name.postgres.database.azure.com/teleport_backend?sslmode=verify-full&pool_max_conns=20
    auth_mode: azure

    audit_events_uri:
      - postgresql://user_name@database-name.postgres.database.azure.com/teleport_audit?sslmode=verify-full#auth_mode=azure
```

When `auth_mode` is set to `azure`, Teleport will automatically fetch
short-lived tokens from the credentials available to it, to be used as database
passwords. The database user must be [configured to allow connections using
Microsoft Entra ID](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-configure-sign-in-azure-ad-authentication).

Teleport will make use of the Microsoft Entra ID credentials specified by
[environment
variables](https://learn.microsoft.com/azure/developer/go/azure-sdk-authentication#2-authenticate-with-azure),
[Microsoft Entra Workload
ID](https://learn.microsoft.com/azure/aks/workload-identity-overview)
credentials, or [managed
identity](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview)
credentials.

### Google Cloud IAM authentication

If you are running Teleport on Google Cloud, Teleport can make use of [IAM
Authentication](https://cloud.google.com/sql/docs/postgres/iam-authentication)
to connect to an GCP Cloud SQL for PostgreSQL without having to manage any
secrets:
```yaml
teleport:
  storage:
    type: postgresql
    auth_mode: gcp-cloudsql

    # GCP connection name has the format <project>:<location>:<instance>.
    gcp_connection_name: <Var name="project:location:instance"/>

    # The type of IP address to use for connecting to the Cloud SQL instance. Valid options are:
    # - "" (default to "public")
    # - "public"
    # - "private"
    # - "psc" (for Private Service Connect)
    gcp_ip_type: <Var name="public"/>

    # Leave host and port empty as they are not required.
    conn_string: postgresql://<Var name="service-account@project.iam"/>@/teleport_backend

    audit_events_uri:
      - postgresql://<Var name="service-account@project.iam"/>@/teleport_audit#auth_mode=gcp-cloudsql&gcp_connection_name=<Var name="project:location:instance"/>&gcp_ip_type=<Var name="public"/>
```

To enable IAM authentication and logical replication for Cloud SQL, make sure
flags `cloudsql.iam_authentication` and `cloudsql.logical_decoding` are set to
`on` for the Cloud SQL instance. The database user must also have the
`REPLICATION` role attribute for using the logical decoding features. See [set
up logical replication and
decoding](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication)
for more details.

In order for Teleport to use the [Cloud SQL Go
Connector](https://cloud.google.com/sql/docs/mysql/connect-connectors) with IAM
authentication, the service account of the target database user must have "Cloud
SQL Client"/`roles/cloudsql.client` and "Cloud SQL Instance
User"/`roles/cloudsql.instanceUser` roles assigned to the service account.

Teleport will make use of the credentials specified through the
[`GOOGLE_APPLICATION_CREDENTIALS` environment
variable](https://cloud.google.com/docs/authentication/application-default-credentials#GAC),
[Workload Identity
Federation](https://cloud.google.com/iam/docs/workload-identity-federation)
with service account impersonation, or service account credentials attached to
VMs.

If the service account used in the PostgreSQL connection string is different
from the service account of the default credentials, Teleport will impersonate
the service account used in the connection string as a Service Account Token
Creator using the default credentials.

### Development

If you are not ready to connect Teleport to a production instance of
PostgreSQL, you can use the following instructions to set up a throwaway
instance of PostgreSQL using Docker.

First copy the following script to disk and run it to generate the CA, client
certificate, and server certificate used by Teleport and PostgreSQL to
establish a secure mutually authenticated connection:

```
#!/bin/bash

# Create the certs directory.
mkdir -p ./certs
cd certs/

# Create CA key and self-signed certificate.
openssl genpkey -algorithm RSA -out ca.key
openssl req -x509 -new -key ca.key -out ca.crt -subj "/CN=root"

# Function to create certificates.
create_certificate() {
    local name="$1"
    local dns_name="$2"

    openssl genpkey \
        -algorithm RSA \
        -out "${name}.key"
    openssl req -new \
        -key "${name}.key" \
        -out "${name}.csr" \
        -subj "/CN=${dns_name}"
    openssl x509 -req \
        -in "${name}.csr" \
        -CA ca.crt \
        -CAkey ca.key \
        -out "${name}.crt" \
        -extfile <(printf "subjectAltName=DNS:${dns_name}") \
        -CAcreateserial

    chmod 0600 "${name}.key"
}

# Create client certificate with SAN.
create_certificate "client" "teleport"

# Create server certificate with SAN.
create_certificate "server" "localhost"

echo "Certificates and keys generated successfully."
```

Next, create a `Dockerfile` using the [official PostgreSQL Docker
image](https://hub.docker.com/_/postgres) and add `wal2json` to it:

```
FROM postgres:15.0
RUN apt-get update
RUN apt-get install -y postgresql-15-wal2json
```

Create an `init.sql` file that will ensure the Teleport user is created upon
startup of the container:

```
CREATE USER teleport WITH REPLICATION CREATEDB;
```

Create a `pg_hba.conf` file to enforce certificate-based authentication for
connections to PostgreSQL:

```
# TYPE  DATABASE        USER            CIDR-ADDRESS            METHOD
local   all             all                                     trust
hostssl all             all             ::/0                    cert
hostssl all             all             0.0.0.0/0               cert
```

Create a `postgresql.conf` file that configures the WAL level and certificates
used for authentication:

```
listen_addresses = '*'
port = 5432
max_connections = 20
shared_buffers = 128MB
temp_buffers = 8MB
work_mem = 4MB

wal_level=logical
max_replication_slots=10

ssl=on
ssl_ca_file='/certs/ca.crt'
ssl_cert_file='/certs/server.crt'
ssl_key_file='/certs/server.key'
```

Start the PostgreSQL container with the following command:

```
docker run --rm --name postgres \
    -e POSTGRES_DB=db \
    -e POSTGRES_USER=user \
    -e POSTGRES_PASSWORD=password \
    -v $(pwd)/data:/var/lib/postgresql/data \
    -v $(pwd)/certs:/certs \
    -v $(pwd)/postgresql.conf:/etc/postgresql/postgresql.conf \
    -v $(pwd)/pg_hba.conf:/etc/postgresql/pg_hba.conf \
    -v $(pwd)/init.sql:/docker-entrypoint-initdb.d/init.sql \
    -p 5432:5432 \
    $(docker build -q .) \
    postgres \
    -c hba_file=/etc/postgresql/pg_hba.conf \
    -c config_file=/etc/postgresql/postgresql.conf
```

Lastly, update the storage section in `teleport.yaml` to use PostgreSQL and
start Teleport:

```yaml
teleport:
  storage:
    type: postgresql
    conn_string: "postgresql://teleport@localhost:5432/teleport_backend?sslcert=/path/to/certs/client.crt&sslkey=/path/to/certs/client.key&sslrootcert=/path/to/certs/ca.crt&sslmode=verify-full&pool_max_conns=20"
```

## S3 (Session Recordings)

Teleport supports using S3 as a backend for both session recordings
and audit logs. S3 cannot be used as the cluster state backend.
This section covers the use of S3 as a session recording backend.
For information on using S3 for audit logs, see the [Athena](#Athena)
section.

S3 buckets must have versioning enabled, which ensures that a session log cannot
be permanently altered or deleted. Teleport will always look at the oldest
version of a recording.

### Authenticating to AWS

The Teleport Auth Service must be able to read AWS credentials in order to
authenticate to S3.

(!docs/pages/includes/aws-credentials.mdx service="the Teleport Auth Service"!)

### Configuring the S3 backend

Below is an example of how to configure the Teleport Auth Service to store the
recorded sessions in an S3 bucket.

```yaml
teleport:
  storage:
      # The region setting sets the default AWS region for all AWS services
      # Teleport may consume (DynamoDB, S3)
      region: us-east-1

      # Path to S3 bucket to store the recorded sessions in.
      audit_sessions_uri: "s3://Example_TELEPORT_S3_BUCKET/records"

      # Teleport assumes credentials. Using provider chains, assuming IAM role or
      # standard .aws/credentials in the home folder.
```

You can add optional query parameters to the S3 URL. The Teleport Auth
Service reads these parameters to configure its interactions with S3:

`s3://bucket/path?region=us-east-1&endpoint=mys3.example.com&insecure=false&disablesse=false&acl=private&use_fips_endpoint=true`

- `region=us-east-1` - set the Amazon region to use.

- `endpoint=mys3.example.com` - connect to a custom S3 endpoint. Optional.

- `insecure=true` - set to `true` or `false`. If `true`, TLS will be disabled.
  Default value is `false`.

- `disablesse=true` - set to `true` or `false`. The Auth Service checks this
  value before uploading an object to an S3 bucket.

  If this is `false`, the Auth Service will set the server-side encryption
  configuration of the upload to use AWS Key Management Service and, if
  `sse_kms_key` is set, configure the upload to use this key.

  If this value is `true`, the Auth Service will not set an explicit server-side
  encryption configuration for the object upload, meaning that the upload will
  use the bucket-level server-side encryption configuration.

- `sse_kms_key=kms_key_id` - If set to a valid AWS KMS CMK key ID, all objects
  uploaded to S3 will be encrypted with this key (as long as `disablesse` is
  `false`). Details can be found below.

- `acl=private` - set the [canned
  ACL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl)
  to use. Must be one of the predefined ACL values.

- `use_fips_endpoint=true` -  [Configure S3 FIPS
  endpoints](#configuring-aws-fips-endpoints)

- `use_s3_virtual_style_addressing` - Whether to use virtual-host-style instead of path-style URLs for the
  bucket. Only applies when a custom endpoint is set. Defaults to false when unset. If used
  without a custom endpoint set, this option has no effect.

- `complete_initiators` - When specified, Teleport will only complete uploads
initiated by the specified set of initiators. This is helpful in scenarios where
software other than Teleport is initiating multipart uploads in the recordings
bucket. This should be set to the display name of the initiator(s) you want to
allow.

### S3 IAM policy

(!docs/pages/includes/s3-iam-policy.mdx!)

### S3 Server Side Encryption

Teleport supports using a custom AWS KMS Customer Managed Key for encrypting objects uploaded to S3.
This allows you to restrict who can read objects like session recordings separately from those that have read
access to a bucket by restricting key access.

The `sse_kms_key` parameter above can be set to any valid KMS CMK ID corresponding to a symmetric standard spec KMS key.
Example template KMS key policies are provided below for common usage cases. IAM users do not have access to any
key by default. Permissions have to be explicitly granted in the policy.

#### Encryption/Decryption

This policy allows an IAM user to encrypt and decrypt objects.
This allows a cluster auth to write and play back session recordings.

Replace `[iam-key-admin-arn]` with the IAM ARN of the user(s) that should have
administrative key access and `[auth-node-iam-arn]` with the IAM ARN
of the user the Teleport auth nodes are using.

```json
{
  "Id": "Teleport Encryption and Decryption",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Teleport CMK Admin",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[iam-key-admin-arn]"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Teleport CMK Auth",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[auth-node-iam-arn]"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}
```

### Encryption/Decryption with separate clusters

This policy allows specifying separate IAM users for encryption and decryption.
This can be used to set up a multi cluster configuration where the main cluster
cannot play back session recordings but only write them.
A separate cluster authenticating as a different IAM user with decryption access
can be used for playing back the session recordings.

Replace `[iam-key-admin-arn]` with the IAM ARN of the user(s) that should have
administrative key access, `[iam-node-write-arn]` with the IAM ARN of the user the
main write-only cluster auth nodes are using and `[iam-node-read-arn]` with the
IAM ARN of the user used by the read-only cluster.

For this to work the second cluster has to be connected to the same audit log as the main cluster.
This is needed to detect session recordings.

```json
{
  "Id": "Teleport Separate Encryption and Decryption",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Teleport CMK Admin",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[iam-key-admin-arn]"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Teleport CMK Auth Encrypt",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[auth-node-write-arn]"
      },
      "Action": [
        "kms:Encrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Teleport CMK Auth Decrypt",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[auth-node-read-arn]"
      },
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}
```

### ACL example: transferring object ownership

If you are uploading from AWS account `A` to a bucket owned by AWS account `B` and want `A` to retain ownership of the objects, you can take one of two approaches.

#### Without ACLs

If ACLs are disabled, object ownership will be set to `Bucket owner enforced` and no action will be needed.

#### With ACLs

- Set object ownership to `Bucket owner preferred` (under Permissions in the management console).
- Add `acl=bucket-owner-full-control` to `audit_sessions_uri`.

To enforce the ownership transfer, set `B`'s bucket's policy to only allow uploads that include the `bucket-owner-full-control` canned ACL.

```json
{
    "Version": "2012-10-17",
    "Id": "[id]",
    "Statement": [
        {
            "Sid": "[sid]",
            "Effect": "Allow",
            "Principal": {
                "AWS": "[ARN of account A]"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::BucketName/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}

```

For more information, see the [AWS Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/about-object-ownership.html).

## DynamoDB

If you are running Teleport on AWS, you can use
[DynamoDB](https://aws.amazon.com/dynamodb/) as a storage backend to achieve
High Availability. DynamoDB backend supports two types of Teleport data:

- Cluster state
- Audit log events

Teleport uses DynamoDB and DynamoDB Streams endpoints for its storage
backend management.

DynamoDB cannot store the recorded sessions. You are advised to use AWS S3 for
that as shown above.

<Admonition type="warning">

Teleport supports single-region DynamoDB tables for storage. Using a global table for audit log events is untested and unsupported, and using a global table for the cluster state from multiple regions at once can result in concurrency problems and potential data loss, even when using multi-region strong consistency (MRSC). Using a global table just for replication between regions in multi-region eventual consistency (MREC) mode is also not recommended, due to potential read throttling in streams when running more than one replica of the Teleport Auth service in the one active region.

</Admonition>

### Authenticating to AWS

The Teleport Auth Service must be able to read AWS credentials in order to
authenticate to DynamoDB.

(!docs/pages/includes/aws-credentials.mdx service="the Teleport Auth Service"!)

The IAM role that the Teleport Auth Service authenticates as must have the
policies specified in the next section.

### IAM policies

Make sure that the IAM role assigned to Teleport is configured with sufficient
access to DynamoDB.

(!docs/pages/includes/dynamodb-iam-policy.mdx!)

### Configuring the DynamoDB backend

To configure Teleport to use DynamoDB:

- Configure all Teleport Auth servers to use DynamoDB backend in the "storage"
  section of `teleport.yaml` as shown below.
- Auth servers must be able to reach DynamoDB and DynamoDB Streams endpoints.
- Deploy up to two auth servers connected to DynamoDB storage backend.
- Deploy several proxy nodes.
- Make sure that all Teleport resource services have the `auth_servers` configuration setting
  populated with the addresses of your cluster's Auth Service instances.

<Admonition type="danger">

AWS can throttle DynamoDB if more than two processes are reading from the same
stream's shard simultaneously, so you must not deploy more than two Auth Service
instances that read from a DynamoDB backend. For details on DynamoDB Streams,
read the [AWS
documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html).

</Admonition>

```yaml
teleport:
  storage:
    type: dynamodb
    # Region location of dynamodb instance, https://docs.aws.amazon.com/en_pv/general/latest/gr/rande.html#ddb_region
    region: us-east-1

    # Name of the DynamoDB table. If it does not exist, Teleport will create it.
    table_name: Example_TELEPORT_DYNAMO_TABLE_NAME

    # This setting configures Teleport to send the audit events to three places:
    # To keep a copy in DynamoDB, a copy on a local filesystem, and also output the events to stdout.
    # NOTE: The DynamoDB events table has a different schema to the regular Teleport
    # database table, so attempting to use the same table for both will result in errors.
    # When using highly available storage like DynamoDB, you should make sure that the list always specifies
    # the High Availability storage method first, as this is what the Teleport web UI uses as its source of events to display.
    audit_events_uri:  ['dynamodb://events_table_name', 'file:///var/lib/teleport/audit/events', 'stdout://']

    # This setting configures Teleport to save the recorded sessions in an S3 bucket:
    audit_sessions_uri: s3://Example_TELEPORT_S3_BUCKET/records

    # By default, Teleport stores audit events with an AWS TTL of 1 year.
    # This value can be configured as shown below. If set to 0 seconds, TTL is disabled.
    retention_period: 365d

    # Enables either Pay Per Request or Provisioned billing for the DynamoDB table. Set when Teleport creates the table.
    # Possible values: "pay_per_request" and "provisioned"
    # default: "pay_per_request"
    billing_mode: "pay_per_request"

    # continuous_backups is used to optionally enable continuous backups.
    # default: false
    continuous_backups: true
```

- Replace `us-east-1` and `Example_TELEPORT_DYNAMO_TABLE_NAME`
  with your own settings.  Teleport will create the table automatically.
- `Example_TELEPORT_DYNAMO_TABLE_NAME` and `events_table_name` **must** be different
  DynamoDB tables. The schema is different for each. Using the same table name for both will result in errors.
- Audit log settings above are optional. If specified, Teleport will store the
  audit log in DynamoDB and the session recordings **must** be stored in an S3
  bucket, i.e. both `audit_xxx` settings must be present. If they are not set,
  Teleport will default to a local file system for the audit log, i.e.
  `/var/lib/teleport/log` on an Auth Service instance.

The optional `GET` parameters shown below control how Teleport interacts with a DynamoDB endpoint.

`dynamodb://events_table_name?region=us-east-1&endpoint=dynamo.example.com&use_fips_endpoint=true`

- `region=us-east-1` - set the Amazon region to use.
- `endpoint=dynamo.example.com` - connect to a custom S3 endpoint.
- `use_fips_endpoint=true` -  [Configure DynamoDB FIPS endpoints](#configuring-aws-fips-endpoints).


**DynamoDB Continuous Backups**

When setting up DynamoDB it's important to enable backups so that cluster state can be restored
if needed from a snapshot in the past.

- [AWS Blog Post - Amazon DynamoDB Continuous Backup](https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-continuous-backups-and-point-in-time-recovery-pitr/)

**DynamoDB On-Demand**

For best performance it is recommended to use On-Demand mode instead of configuring capacity manually via Provisioned
mode. This helps prevent any DynamoDB throttling due to underestimated usage or increased usage from impacting Teleport.

- [AWS Docs - Amazon DynamoDB On-Demand and Pay-Per-Request](https://aws.amazon.com/blogs/aws/amazon-dynamodb-on-demand-no-capacity-planning-and-pay-per-request-pricing/)

### Configuring AWS FIPS endpoints

This config option applies to [Amazon S3](#s3-session-recordings) and [Amazon DynamoDB](#dynamodb).

Set `use_fips_endpoint` to `true` or `false`. If `true`, [FIPS](https://aws.amazon.com/compliance/fips/) Dynamo endpoints will be used.
If `false`, normal Dynamo endpoints will be used. If unset, the AWS Environment Variable `AWS_USE_FIPS_ENDPOINT` will determine which endpoint is used.
FIPS endpoints will also be used if Teleport is run with the `--fips` flag.

Config option priority is applied in the following order:

- Setting the `use_fips_endpoint` query parameter as shown above
- Using the `--fips` flag when running Teleport
- Using the AWS environment variable

<Admonition
  type="warning"
  title="A warning about AWS_USE_FIPS_ENDPOINT"
>
  Setting this environment variable to true will enable FIPS endpoints for all AWS resource types. Some FIPS endpoints
  are not supported in certain regions or environments or are only supported in GovCloud.
</Admonition>

## Athena

If you are running Teleport on AWS, you can use an
[Athena](https://aws.amazon.com/athena/)-based audit log system that manages
[Parquet files](https://parquet.apache.org/) stored on
[S3](https://aws.amazon.com/s3/) as a storage backend to achieve high
availability. The Athena backend supports only one type of Teleport data, audit
events.

The Athena audit backend is better at scale and search than DynamoDB.

The Athena audit logs are eventually consistent. It may take up to one minute
(depending on the `batchMaxInterval` setting and event load) until you can view
events in the Teleport Web UI.

### Infrastructure setup

The Auth Service uses an SQS queue subscribed to an SNS topic for event
publishing. A single Auth Service instance reads events in batches from SQS,
converts them into Parquet format, and sends the resulting data to S3. During
queries, the Athena engine searches for events on S3, reading metadata from a
Glue table.

You can set up the required infrastructure to support the Athena backend with
the following Terraform script:

<details>
<summary>Terraform script"</summary>
```
(!examples/athena/variables.tf!)
(!examples/athena/athena.tf!)
```
</details>

### Configuring the Athena audit log backend

To configure Teleport to use Athena:

- Prepare infrastructure
- Specify an Athena URL inside the `audit_events_uri` array in your Teleport
  configuration file:

```yaml
teleport:
  storage:
    # This setting configures Teleport to keep a copy of the audit log in Athena
    # and a copy on a local filesystem, and also to output the events to stdout.
    audit_events_uri:
      # More details about the full Athena URL are shown below.
      - 'athena://database.table?params'
      - 'file:///var/lib/teleport/audit/events'
      - 'stdout://'
```

Here is an example of an Amazon Athena URL within the `audit_events_uri` configuration field:

```
athena://db.table?topicArn=arn:aws:sns:region:account_id:topic_name&largeEventsS3=s3://transient/large_payloads&locationS3=s3://long-term/events&workgroup=workgroup&queueURL=https://sqs.region.amazonaws.com/account_id/queue_name&queryResultsS3=s3://transient/query_results
```

The URL hostname consist of `database.table`, which points to the Glue database
and a table which will be used by the Athena audit logger.

Other parameters are specified as query parameters within the Athena URL.

The following parameters are required:

| Parameter name   | Example value                                            | Description                                            |
|------------------|----------------------------------------------------------|--------------------------------------------------------|
| `topicArn`       | `arn:aws:sns:region:account_id:topic_name`               | ARN of SNS topic where events are published            |
| `locationS3`     | `s3://long-term/events`                                  | S3 bucket used for long-term storage                   |
| `largeEventsS3`  | `s3://transient/large_payloads`                          | S3 bucket used for transient storage for large events  |
| `queueURL`       | `https://sqs.region.amazonaws.com/account_id/queue_name` | SQS URL used for a subscription to an SNS topic        |
| `workgroup`      | `workgroup_name`                                         | Athena workgroup used for queries                      |
| `queryResultsS3` | `s3://transient/results`                                 | S3 bucket used for transient storage for query results |


The following parameters are optional:

| Parameter name     | Example value | Description                                                                                           |
|--------------------|---------------|-------------------------------------------------------------------------------------------------------|
| `region`           | `us-east-1`   | AWS region. If empty, defaults to one from the AuditConfig or ambient AWS credentials                 |
| `batchMaxItems`    | `20000`       | defines the maximum number of events allowed for a single Parquet file (default 20000)                |
| `batchMaxInterval` | `1m`          | defines the maximum interval used to buffer incoming data before creating a Parquet file (default 1m) |

### Authenticating to AWS

The Teleport Auth Service must be able to read AWS credentials in order to
authenticate to Athena.

(!docs/pages/includes/aws-credentials.mdx service="the Teleport Auth Service"!)

The IAM role that the Teleport Auth Service authenticates as must have the
policies specified in the next section.

### IAM policies

Make sure that the IAM role assigned to Teleport is configured with sufficient
access to Athena. Below you can find the IAM permissions that the Auth Service
requires to use Athena Audit logs as an audit event backend.

You'll need to replace these values in the policy example below:

| Placeholder value | Replace with                                             |
|-------------------|----------------------------------------------------------|
| `eu-central-1`    | AWS region                                               |
| `1234567890`      | AWS account ID                                           |
| `audit-long-term` | S3 bucket used for long-term storage                     |
| `audit-transient` | S3 bucket used for transient storage                     |
| `audit-sqs`       | SNS topic name                                           |
| `audit-sns`       | SQS name                                                 |
| `kms_id`          | KMS key ID used for server-side encryption of SNS/SQS/S3 |
| `audit_db`        | Glue database used for audit logs                        |
| `audit_table`     | Glue table used for audit logs                           |
| `audit_workgroup` | Athena workgroup used for audit logs                     |

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:ListBucketVersions",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::audit-transient",
                "arn:aws:s3:::audit-long-term"
            ],
            "Sid": "AllowListingMultipartUploads"
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:ListMultipartUploadParts",
                "s3:GetObjectVersion",
                "s3:GetObject",
                "s3:AbortMultipartUpload"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::audit-transient/results/*",
                "arn:aws:s3:::audit-transient/large_payloads/*",
                "arn:aws:s3:::audit-long-term/events/*"
            ],
            "Sid": "AllowMultipartAndObjectAccess"
        },
        {
            "Action": "sns:Publish",
            "Effect": "Allow",
            "Resource": "arn:aws:sns:eu-central-1:1234567890:audit-sns",
            "Sid": "AllowPublishSNS"
        },
        {
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:eu-central-1:1234567890:audit-sqs",
            "Sid": "AllowReceiveSQS"
        },
        {
            "Action": [
                "glue:GetTable",
                "athena:StartQueryExecution",
                "athena:GetQueryResults",
                "athena:GetQueryExecution"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:glue:eu-central-1:1234567890:table/audit_db/audit_table",
                "arn:aws:glue:eu-central-1:1234567890:database/audit_db",
                "arn:aws:glue:eu-central-1:1234567890:catalog",
                "arn:aws:athena:eu-central-1:1234567890:workgroup/audit_workgroup"
            ],
            "Sid": "AllowAthenaQuery"
        },
        {
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kms:eu-central-1:1234567890:key/kms_id",
            "Sid": "AllowAthenaKMSUsage"
        }
    ]
}
```

### Migration from Dynamo to the Athena audit logs backend

<Admonition
  type="tip"
  title="Tip"
>
  Migration is only needed if you used Amazon DynamoDB for audit logs and you
  want to keep old data.
</Admonition>

Migration consist of following steps:

1. Set up Athena infrastructure
1. Dual write to both DynamoDB and Athena, and query from DynamoDB
1. Migrate old data from DynamoDB to Athena
1. Dual write to both DynamoDB and Athena, and query from Athena
1. Disable writing to DynamoDB

In the Teleport storage configuration, `audit_events_uri` accepts multiple
URLs. Those URLs are used to configure connections to the different audit
loggers. If more than 1 is used, then events are written to each audit system,
and queries are executed from first one.

<Admonition
  type="tip"
  title="Tip"
>
  If anything goes wrong during migration steps 1-4, roll back to the Amazon
  DynamoDB solution by making sure its URL is the first value in the
  `audit_events_uri` field and removing the Athena URL.
</Admonition>

Each of these steps is explained in more detail below.

#### Dual write to both DynamoDB and Athena, and query from DynamoDB

The second step of migration requires setting the following configuration:

```yaml
teleport:
  storage:
    audit_events_uri:
    - 'dynamodb://events_table_name'
    - 'athena://db.table?otherQueryParams'
```

When an Auth Service instance is restarted, you should verify that Parquet files
are stored in the S3 bucket specified using the `locationS3` parameter.

#### Migrate old data from DynamoDB to Athena

This step requires using the client machine to export data from Amazon DynamoDB
and publish it to the Athena logger. We recommend using, for example, an EC2
instance with a disk size at least 2x bigger than the table size in Amazon
DynamoDB.

Instructions for how to use the migration tool can be found
[on GitHub](https://github.com/gravitational/teleport/blob/branch/v(=teleport.major_version=)/examples/dynamoathenamigration/README.md).

You should set `exportTime` to the time when dual writing began.

We recommend running your first migration with the `-dry-run` flag because it
validates the exported data. If no errors are reported, proceed to a real
migration without the `-dry-run` flag.

#### Dual write to both DynamoDB and Athena, and query from Athena

Change the order of the `audit_events_uri` values in your Teleport
configuration file:

```yaml
teleport:
  storage:
    audit_events_uri:
    - 'athena://db.table?otherQueryParams'
    - 'dynamodb://events_table_name'
```

When the Auth Service is restarted, you should verify that events are visible
on the Audit Logs page.

#### Disable writing to DynamoDB

Disabling writing to DynamoDB means that you won't be able to roll back to
DynamoDB without losing data. Dual writing to both Athena and DynamoDB does not
have a significant performance impact, and it's recommended to keep dual
writing for some time, even if your system already executes queries from
Athena.

To disable writing to DynamoDB, remove the DynamoDB URL from the
`audit_events_uri` array.

## GCS

Google Cloud Storage (GCS) can be used as storage for recorded
sessions. GCS cannot store the audit log or the cluster state. Below is an
example of how to configure a Teleport Auth Service to store the recorded
sessions in a GCS bucket.

```yaml
teleport:
  storage:
      # Path to GCS to store the recorded sessions in.
      audit_sessions_uri: 'gs://$BUCKET_NAME/records?projectID=$PROJECT_ID&credentialsPath=$CREDENTIALS_PATH'
```

We recommend creating a bucket in `Dual-Region` mode with the `Standard` storage class to ensure cluster performance and high availability.
Replace the following variables in the above example with your own values:
- `$BUCKET_NAME` with the name of the desired GCS bucket. If the bucket does not exist it will be created.
  Please ensure the following permissions are granted for the given bucket:
  - `storage.buckets.get`
  - `storage.objects.create`
  - `storage.objects.get`
  - `storage.objects.list`
  - `storage.objects.update`
  - `storage.objects.delete`

  `storage.objects.delete` is required in order to clean up multipart files after they have been assembled
  into the final blob.

  If the bucket does not exist, please also ensure that the `storage.buckets.create` permission is granted.

- `$PROJECT_ID` with a GCS-enabled GCP project.

- `$CREDENTIALS_PATH` with the path to a JSON-formatted GCP credentials file configured for a service account applicable to the project.

## Firestore

If you are running Teleport on GCP, you can use
[Firestore](https://cloud.google.com/firestore/) as a storage backend to achieve
high availability. Firestore backend supports two types of Teleport data:

- Cluster state
- Audit log events

Firestore cannot store the recorded sessions. You are advised to use Google
Cloud Storage (GCS) for that as shown above. To configure Teleport to use
Firestore:

- Configure all Teleport Auth servers to use Firestore backend in the "storage"
  section of `teleport.yaml` as shown below.
- Deploy several auth servers connected to Firestore storage backend.
- Deploy several proxy nodes.
- Make sure that all Teleport resource services have the `auth_servers`
  configuration setting populated with the addresses of your cluster's Auth
  Service instances or use a load balancer for Auth Service instances in high
  availability mode.

```yaml
teleport:
  storage:
    type: firestore
    # Project ID https://support.google.com/googleapi/answer/7014113?hl=en
    project_id: Example_GCP_Project_Name

    # Name of the Firestore table.
    collection_name: Example_TELEPORT_FIRESTORE_TABLE_NAME

    # An optional database id to use. If not provided the default
    # database for the project is used.
    database_id: Example_TELEPORT_FIRESTORE_DATABASE_ID

    credentials_path: /var/lib/teleport/gcs_creds

    # This setting configures Teleport to send the audit events to three places:
    # To keep a copy in Firestore, a copy on a local filesystem, and also write the events to stdout.
    # NOTE: The Firestore events table has a different schema to the regular Teleport
    # database table, so attempting to use the same table for both will result in errors.
    # When using highly available storage like Firestore, you should make sure that the list always specifies
    # the High Availability storage method first, as this is what the Teleport web UI uses as its source of events to display.
    audit_events_uri:  ['firestore://Example_TELEPORT_FIRESTORE_EVENTS_TABLE_NAME?projectID=$PROJECT_ID&credentialsPath=$CREDENTIALS_PATH&databaseID=$DATABASE_ID', 'file:///var/lib/teleport/audit/events', 'stdout://']

    # This setting configures Teleport to save the recorded sessions in GCP storage:
    audit_sessions_uri: gs://Example_TELEPORT_GCS_BUCKET/records
```

- Replace `Example_GCP_Project_Name` and `Example_TELEPORT_FIRESTORE_TABLE_NAME`
  with your own settings. Teleport will create the table automatically.
- `Example_TELEPORT_FIRESTORE_TABLE_NAME` and `Example_TELEPORT_FIRESTORE_EVENTS_TABLE_NAME`
  **must** be different Firestore tables. The schema is different for each.
  Using the same table name for both will result in errors.
- The GCP authentication setting above can be omitted if the machine itself is
  running on a GCE instance with a Service Account that has access to the
  Firestore table.
- Audit log settings above are optional. If specified, Teleport will store the audit log in Firestore
  and the session recordings **must** be stored in a GCS bucket, i.e. both `audit_xxx` settings must
  be present. If they are not set, Teleport will default to a local filesystem for the audit log, i.e.
  `/var/lib/teleport/log` on an Auth Service instance.

## Azure Blob Storage

Azure Blob Storage for session storage is available starting from Teleport
`13.3`.

Azure Blob Storage can be used as storage for recorded sessions. Azure Blob
Storage cannot store the audit log or the cluster state. Below is an example of
how to configure a Teleport Auth Service instance to store the recorded sessions in an
Azure Blob Storage storage account.

```yaml
teleport:
  storage:
    audit_sessions_uri: azblob://account-name.blob.core.windows.net
```

Teleport makes use of two containers in the account, whose names default to
`inprogress` and `session`, but they can be configured with parameters in the
fragment of the URI.

```yaml
teleport:
  storage:
    audit_sessions_uri: azblob://account-name.core.blob.windows.net#session_container=session_container_name&inprogress_container=inprogress_container_name
```

### Permissions

Teleport needs the following permissions on the `inprogress` container:
- `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read`
- `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write`
- `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete` (only on the `inprogress` container)

In addition, Teleport will check if the containers exist at startup, and it will
attempt to create them if they can't be confirmed to exist; giving Teleport
`Microsoft.Storage/storageAccounts/blobServices/containers/read` will allow for
checking and `Microsoft.Storage/storageAccounts/blobServices/containers/write`
will allow for creating them.

It's highly recommended to set up a [time-based retention
policy](https://learn.microsoft.com/azure/storage/blobs/immutable-time-based-retention-policy-overview)
for the `session` container, as well as a [lifecycle management
policy](https://learn.microsoft.com/azure/storage/blobs/lifecycle-management-overview),
so that recordings are kept in an immutable state for a given period, then
deleted. Teleport will not delete recordings automatically.

With a time-based retention policy in place, it's safe to give Teleport the
"Blob Storage Data Contributor" role scoped to the containers, instead of having
to define a custom role for it.

### Authentication

Teleport will make use of the Microsoft Entra ID credentials specified by [environment
variables](https://learn.microsoft.com/azure/developer/go/azure-sdk-authentication#2-authenticate-with-azure),
[Microsoft Entra Workload
ID](https://learn.microsoft.com/azure/aks/workload-identity-overview)
credentials, or [managed
identity](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview)
credentials.

## SQLite

The Auth Service uses the SQLite backend when no `type` is specified in the
storage section in the Teleport configuration file, or when `type` is set to
`sqlite` or `dir`. The SQLite backend is not designed for high throughput and
it's not capable of serving the needs of Teleport's High Availability configurations.

If you are planning to use SQLite as your backend, scale your cluster slowly and
monitor the number of warning messages in the Auth Service's logs that say
`SLOW TRANSACTION`, as that's a sign that the cluster has outgrown the capabilities
of the SQLite backend.

As a stopgap measure until it's possible to migrate the cluster to use a
HA-capable backend, you can configure the SQLite backend to reduce the amount of
disk synchronization, in exchange for less resilience against system crashes or
power loss. For an explanation on what the options mean, see [the official
SQLite docs](https://www.sqlite.org/pragma.html#pragma_synchronous). No matter
the configuration, we recommend you take regular backups of your cluster state.

To reduce disk synchronization:
```yaml
teleport:
  storage:
    type: sqlite
    sync: NORMAL
```

To disable disk synchronization altogether:
```yaml
teleport:
  storage:
    type: sqlite
    sync: "OFF"
```

When running on a filesystem that supports file locks (i.e. a local filesystem,
not a networked one) it's possible to also configure the SQLite database to use
Write-Ahead Logging (see [the official docs on WAL
mode](https://www.sqlite.org/wal.html)) for significantly improved performance
without sacrificing reliability:

```yaml
teleport:
  storage:
    type: sqlite
    sync: NORMAL
    journal: WAL
```

The SQLite backend and other required data will be written to the Teleport data directory.
By default, Teleport's data directory is `/var/lib/teleport`. To modify
the location set the `data_dir` value within the Teleport configuration file.

```yaml
teleport:
  data_dir: /var/lib/teleport_data
```

## CockroachDB

<Admonition type="warning" title="Enterprise">
  Use of the CockroachDB storage backend requires Teleport Enterprise.
</Admonition>

Teleport can use [CockroachDB](https://www.cockroachlabs.com/) as a storage backend
to achieve high availability and survive regional failures. You must take steps to
protect access to CockroachDB in this configuration because that is where Teleport
secrets like keys and user records will be stored.

At a minimum you must configure CockroachDB to allow Teleport to create tables.
Teleport will create the database if given permission to do so but this is not
required if the database already exists.

```
CREATE DATABASE database_name;
CREATE USER database_user;
GRANT CREATE ON DATABASE database_name TO database_user;
```

You must also enable change feeds in CockroachDB's cluster settings. Teleport
will configure this setting itself if granted `SYSTEM MODIFYCLUSTERSETTING`.

```
SET CLUSTER SETTING kv.rangefeed.enabled = true;
```

There are several ways to deploy and configure CockroachDB,
the details of which are not in scope for this guide. To learn about deploying
CockroachDB, see [CockroachDB's deployment options](https://www.cockroachlabs.com/docs/stable/choose-a-deployment-option).
To learn about how to configure multi-region survival goals, see
[multi-region survival goals](https://www.cockroachlabs.com/docs/stable/multiregion-survival-goals).

To configure Teleport to use CockroachDB as a storage backend:
- Configure all Teleport Auth Service instances to use the CockroachDB backend in the
  `storage` section of `teleport.yaml` as shown below.
- Deploy several Auth Service instances connected to the CockroachDB storage backend.
- Deploy several Proxy Service instances.
- Make sure that the Proxy Service instances and all Teleport Agent services that
  connect directly to the Auth Service have the `auth_server` configuration
  setting populated with the address of a load balancer for Auth Service
  instances.

```yaml
teleport:
  storage:
    type: cockroachdb

    # conn_string is a required parameter. It is a PostgreSQL connection string used
    # to connect to CockroachDB using the PostgreSQL wire protocol. Client
    # parameters may be specified using the URL. For a detailed list of available
    # parameters see https://www.cockroachlabs.com/docs/v25.3/connection-parameters.html
    #
    # If your certificates are not stored at the default ~/.postgresql
    # location, you will need to specify them with the sslcert, sslkey, and
    # sslrootcert parameters.
    #
    # pool_max_conns is an additional parameter that determines the maximum
    # number of connections in the connection pool used for the cluster state
    # database (the change feed uses an additional connection), defaulting to
    # a value that depends on the number of available CPUs.
    conn_string: postgresql://user_name@database-address/teleport_backend?sslmode=verify-full&pool_max_conns=20

    # change_feed_conn_string is an optional parameter. When unspecified Teleport
    # will default to using the same value specified for conn_string. It may be used
    # to configure Teleport to use a different user or connection parameters when
    # establishing a change feed connection.
    #
    # If your certificates are not stored at the default ~/.postgresql
    # location, you will need to specify them with the sslcert, sslkey, and
    # sslrootcert parameters.
    change_feed_conn_string: postgresql://user_name@database-address/teleport_backend?sslmode=verify-full

    # ttl_job_cron is an optional parameter which configures the interval at which CockroachDB will expire backend
    # items based on their time to live. By default this is configured to run every
    # 20 minutes. This is used by Teleport to clean up old resources that are no longer
    # connected to or needed by Teleport. Note that configuring this to run more
    # frequently may have performance implications for CockroachDB.
    ttl_job_cron: '*/20 * * * *'
```