teleport/docs/pages/reference/backends.mdx

---
title: Storage backends
description: How to configure Teleport deployment for high-availability using storage backends
---

A Teleport cluster stores different types of data in different locations. By
default everything is stored in a local directory on the Auth Service host.

For self-hosted Teleport deployments, you can configure Teleport to integrate
with other storage types based on the nature of the stored data (size,
read/write ratio, mutability, etc.).

| Data type | Description | Supported storage backends |
| - | - | - |
| core cluster state | Cluster configuration (e.g. users, roles, auth connectors) and identity (e.g. certificate authorities, registered nodes, trusted clusters). | Local directory (SQLite), etcd, PostgreSQL, AWS DynamoDB, GCP Firestore |
| audit events | JSON-encoded events from the audit log (e.g. user logins, RBAC changes) | Local directory, PostgreSQL, AWS DynamoDB, GCP Firestore |
| session recordings | Raw terminal recordings of interactive user sessions | Local directory, AWS S3 (and any S3-compatible product), GCP Cloud Storage, Azure Blob Storage |
| teleport instance state | ID and credentials of a non-auth teleport instance (e.g. node, proxy) | Local directory |

<Admonition
  type="tip"
  title="Tip"
>
  Before continuing, please make sure to take a look at the [Cluster State section](../architecture/nodes.mdx#cluster-state)
  in the Teleport Architecture documentation.
</Admonition>

There are two ways to achieve High Availability. You can "outsource"
this function to the infrastructure. For example, using a highly available
network-based disk volumes (similar to AWS EBS) and by migrating a failed VM to
a new host. In this scenario, there's nothing Teleport-specific to be done.

If High Availability cannot be provided by the infrastructure (perhaps you're
running Teleport on a bare metal cluster), you can still configure Teleport to
run in a highly available fashion.

(!docs/pages/includes/cloud/call-to-action.mdx!)

## Auth Server State

To run multiple instances of the Teleport Auth Service, you must switch to one of
the high-availability secrets backend listed below first.

Once you have a high-availability secrets backend and multiple instances of
the Auth Service running, you'll need to create a load balancer to evenly
distribute traffic to all Auth Service instances and have a single point of
entry for all components that need to communicate with the Auth Service. Use the
address of the load balancer in the [`auth_server`](./config.mdx) field when
configuring other components of Teleport.

Configure your load balancer to use Layer 4 (TCP) load balancing, round-robin
load balancing, and a 300 second idle timeout.

<Admonition type="tip" title="NOTE">
With multiple instances of the Auth Service running, special attention needs to
be paid to keeping their configuration identical. Settings like `cluster_name`,
`tokens`, `storage`, etc. must be the same.
</Admonition>

## Proxy Server State

The Teleport Proxy is stateless which makes running multiple instances trivial.

If using the [default configuration](./networking.mdx), configure your load
balancer to forward port `3080` to the servers that run the Teleport Proxy
Service. If you have configured your Proxy Service to not use TLS Routing
and/or are using non-default ports, you will need to configure your load
balancer to forward the ports you specified for `listen_addr`,
`tunnel_listen_addr`, and `web_listen_addr` in `teleport.yaml`.

Configure your load balancer to use Layer 4 (TCP) load balancing, round-robin
load balancing, and a 300 second idle timeout.

<Admonition type="tip" title="NOTE">
If you terminate TLS with your own certificate for `web_listen_addr` at your
load balancer you'll need to run Teleport with `--insecure-no-tls`
</Admonition>

If your load balancer supports HTTP health checks, configure it to hit the
`/readyz` [diagnostics endpoint](../management/diagnostics/monitoring.mdx) on
machines running Teleport. This endpoint must be enabled by using the
`--diag-addr` flag to teleport start: `teleport start
--diag-addr=127.0.0.1:3000` The
[http://127.0.0.1:3000/readyz](http://127.0.0.1:3000/readyz) endpoint will
reply `{"status":"ok"}` if the Teleport service is running without problems.

We'll cover how to use `etcd`, PostgreSQL, DynamoDB, and Firestore storage
back-ends to make Teleport highly available below.

## Etcd

Teleport can use [etcd](https://etcd.io/) as a storage backend to
achieve highly available deployments. You must take steps to protect access to
`etcd` in this configuration because that is where Teleport secrets like keys
and user records will be stored.

<Admonition
  type="warning"
  title="IMPORTANT"
>
  `etcd` can only currently be used to store Teleport's internal database in a
  highly-available way. This will allow you to have multiple auth servers in your
  cluster for an High Availability deployment, but it will not also store Teleport audit events
  for you in the same way that [DynamoDB](#dynamodb) or
  [Firestore](#firestore) will. `etcd` is not designed to handle large volumes of time series data like audit events.
</Admonition>

To configure Teleport for using etcd as a storage backend:

- Make sure you are using **etcd versions 3.3** or newer.
- Follow [etcd's cluster hardware recommendations](https://etcd.io/docs/v3.5/op-guide/hardware/). In particular, leverage
  SSD or high-performance virtualized block device storage for best performance.
- Install etcd and configure peer and client TLS authentication using the [etcd
  security guide](https://etcd.io/docs/v3.5/op-guide/security/).
  - You can use [this script provided by
    etcd](https://github.com/etcd-io/etcd/tree/master/hack/tls-setup) if you
    don't already have a TLS setup.
- Configure all Teleport Auth servers to use etcd in the "storage" section of the config file as shown below.
- Deploy several auth servers connected to etcd backend.
- Deploy several Proxy Service instances that have `auth_server` pointed to the auth server to connect to.

```yaml
teleport:
  storage:
     type: etcd

     # List of etcd peers to connect to:
     peers: ["https://172.17.0.1:4001", "https://172.17.0.2:4001"]

     # Required path to TLS client certificate and key files to connect to etcd.
     #
     # To create these, follow
     # https://coreos.com/os/docs/latest/generate-self-signed-certificates.html
     # or use the etcd-provided script
     # https://github.com/etcd-io/etcd/tree/master/hack/tls-setup.
     tls_cert_file: /var/lib/teleport/etcd-cert.pem
     tls_key_file: /var/lib/teleport/etcd-key.pem

     # Optional file with trusted CA authority
     # file to authenticate etcd nodes
     #
     # If you used the script above to generate the client TLS certificate,
     # this CA certificate should be one of the other generated files
     tls_ca_file: /var/lib/teleport/etcd-ca.pem

     # Alternative password-based authentication, if not using TLS client
     # certificate.
     #
     # See https://etcd.io/docs/v3.4.0/op-guide/authentication/ for setting
     # up a new user.
     username: username
     password_file: /mnt/secrets/etcd-pass

     # etcd key (location) where teleport will be storing its state under.
     # make sure it ends with a '/'!
     prefix: /teleport/

     # NOT RECOMMENDED: enables insecure etcd mode in which self-signed
     # certificate will be accepted
     insecure: false

     # Optionally sets the limit on the client message size.
     # This is usually used to increase the default which is 2MiB
     # (1.5MiB server's default + gRPC overhead bytes).
     # Make sure this does not exceed the value for the etcd
     # server specified with `--max-request-bytes` (1.5MiB by default).
     # Keep the two values in sync.
     #
     # See https://etcd.io/docs/v3.4.0/dev-guide/limit/ for details
     #
     # This bumps the size to 15MiB as an example:
     etcd_max_client_msg_size_bytes: 15728640
```

## PostgreSQL

PostgreSQL cluster state and audit log storage is available starting from
Teleport `13.3`.

<Admonition
  type="tip"
  title="Tip"
>
  Before continuing, please make sure to take a look at the [Cluster State section](../architecture/nodes.mdx#cluster-state)
  in Teleport Architecture documentation.
</Admonition>

Teleport can use [PostgreSQL](https://www.postgresql.org/) as a storage back-end
to achieve high availability. You must take steps to protect access to
PostgreSQL in this configuration because that is where Teleport secrets like
keys and user records will be stored. The PostgreSQL backend supports two types
of Teleport data:

- Cluster state
- Audit log events

The PostgreSQL backend requires PostgreSQL 13 or later, and, for the cluster
state only, the [`wal2json`](https://github.com/eulerto/wal2json) logical
decoding plugin. The plugin is available in packages for all stable versions in
the PostgreSQL [Apt](https://apt.postgresql.org/) and
[Yum](https://yum.postgresql.org/) repositories for Debian- and RPM-based Linux
distributions respectively, or it can be compiled following [the
instructions](https://github.com/eulerto/wal2json#build-and-install) provided in
its repository. The plugin is pre-installed with no extra steps to take in
[Azure Database for
PostgreSQL](https://azure.microsoft.com/en-us/products/postgresql).

Teleport needs separate databases for the cluster state and the
audit log, and it will attempt to create them if given permissions to do so; it
will also set up the database schemas as needed, so we recommend giving the user
ownership over the databases.

The PostgreSQL backend for cluster state relies on the ability to use [logical
decoding](https://www.postgresql.org/docs/current/logicaldecoding.html) to get a
stream of changes from the database; because of that, the
[`wal_level`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-WAL-LEVEL)
parameter must be set to `logical` and
[`max_replication_slots`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-REPLICATION-SLOTS)
must be set to at least as many Teleport Auth Service instances as you'll be
running (a higher number is recommended, to account for network conditions).

The Teleport Auth Service needs to be able to create a replication slot when
starting and when reestablishing a new connection to the PostgreSQL cluster, and
any long-running transaction will prevent that. It's therefore only advisable to
store the Teleport cluster state on a shared PostgreSQL cluster if the other
workloads on the cluster only consist of short-lived transactions.

`wal_level` can only be set at server start, so it should be set in
`postgresql.conf`:
```
# the default value for wal_level is replica
wal_level = logical

# the default value for max_replication_slots is 10
max_replication_slots = 10
```

In addition, the database user must have the `initiating replication` [role
attribute](https://www.postgresql.org/docs/current/role-attributes.html). In the
`psql` shell:

```
postgres=# CREATE USER new_user WITH REPLICATION;
CREATE ROLE

postgres=# ALTER ROLE existing_user WITH LOGIN REPLICATION;
ALTER ROLE
```

Since replication permissions allow for essentially full read access over the
entire cluster (with a physical replication connection) or to all databases that
the user can connect to, it's recommended to prevent the user from opening
replication connections, or from connecting to databases other than the ones
used for Teleport, if the PostgreSQL cluster is shared between Teleport and
other applications.

For convenience, Teleport will attempt to grant itself the `initiating
replication` role attribute, to accommodate the ability of some managed services
(such as Azure Database for PostgreSQL) to create superuser accounts through
their API; this should only be leveraged if the entire PostgreSQL cluster is
dedicated to Teleport.

To configure Teleport to use PostgreSQL:
- Configure all Teleport Auth Service instances to use the PostgreSQL back-end in the
  `storage` section of `teleport.yaml` as shown below.
- Deploy several Auth Service instances connected to the PostgreSQL storage back-end.
- Deploy several Proxy Service nodes.
- Make sure that the Proxy Service instances and all Teleport agent services that
  connect directly to the the Auth Service have the `auth_server` configuration
  setting populated with the address of a load balancer for Auth Service
  instances.

```yaml
teleport:
  storage:
    type: postgresql

    # conn_string is a libpq-compatible connection string (see
    # https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING);
    # pool_max_conns is an additional parameter that determines the maximum
    # number of connections in the connection pool used for the cluster state
    # database (the change feed uses an additional connection), defaulting to
    # a value that depends on the number of available CPUs.
    #
    # If your certificates are not stored at the default ~/.postgresql
    # location, you will need to specify them with the sslcert, sslkey, and
    # sslrootcert parameters.
    conn_string: postgresql://user_name@database-address/teleport_backend?sslmode=verify-full&pool_max_conns=20

    # In certain managed environments it can be necessary or convenient to
    # use a different user or different settings for the connection used
    # to set up and make use of logical decoding. If specified, Teleport
    # will use the connection string in change_feed_conn_string for that,
    # instead of the one in conn_string. Available in Teleport 13.4 and later.
    change_feed_conn_string: postgresql://replication_user_name@database-address/teleport_backend?sslmode=verify-full

    # An audit_events_uri with a scheme of postgresql:// will use the
    # PostgreSQL backend for audit log storage; the URI is a libpq-compatible
    # connection string just like the cluster state conn_string, but cannot be
    # specified as key=value pairs. It's possible to specify completely
    # different PostgreSQL clusters for cluster state and audit log.
    #
    # If your certificates are not stored at the default ~/.postgresql
    # location, you will need to specify them with the sslcert, sslkey, and
    # sslrootcert parameters.
    audit_events_uri:
      - postgresql://user_name@database-address/teleport_audit?sslmode=verify-full
```

Audit log events are periodically deleted after a default retention period of
8766 hours (one year); it's possible to select a different retention period or
to disable the cleanup entirely, by specifying the `retention_period` or the
`disable_cleanup` parameters in the fragment of the URI:

```yaml
storage:
  teleport:
    audit_events_uri:
      - postgresql://user_name@database-address/teleport_audit?sslmode=verify-full#disable_cleanup=false&retention_period=2160h
```

### Authentication

We strongly recommend using [client
certificates](https://www.postgresql.org/docs/current/ssl-tcp.html#SSL-CLIENT-CERTIFICATES)
to authenticate Teleport to PostgreSQL, as well as enforcing the use of TLS and
[verifying the server
certificate](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBQ-SSL-CERTIFICATES)
on the client side.

You will need to update your `pg_hba.conf` file to include the following lines
to ensure connections to Teleport use client certificates. See The
[pg_hba.conf](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html)
file in the PostgreSQL documentation for more details.

```
# TYPE  DATABASE        USER            CIDR-ADDRESS            METHOD
hostssl teleport        all             ::/0                    cert
hostssl teleport        all             0.0.0.0/0               cert
```

If the use of passwords is unavoidable, we recommend configuring them in [the
`~/.pgpass` file](https://www.postgresql.org/docs/current/libpq-pgpass.html)
rather than storing them in Teleport's configuration file.

### Azure AD authentication

If you are running Teleport on Azure, Teleport can make use of [Azure AD
authentication](https://learn.microsoft.com/azure/postgresql/flexible-server/concepts-azure-ad-authentication)
to connect to an Azure Database for PostgreSQL server without having to manage
any secrets:

```yaml
teleport:
  storage:
    type: postgresql

    conn_string: postgresql://user_name@database-name.postgres.database.azure.com/teleport_backend?sslmode=verify-full&pool_max_conns=20
    auth_mode: azure

    audit_events_uri:
      - postgresql://user_name@database-name.postgres.database.azure.com/teleport_audit?sslmode=verify-full#auth_mode=azure
```

When `auth_mode` is set to `azure`, Teleport will automatically fetch
short-lived tokens from the credentials available to it, to be used as database
passwords. The database user must be [configured to allow connections using
Azure
AD](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-configure-sign-in-azure-ad-authentication).

Teleport will make use of the Azure AD credentials specified by [environment
variables](https://learn.microsoft.com/azure/developer/go/azure-sdk-authentication#2-authenticate-with-azure),
[Azure AD Workload
Identity](https://learn.microsoft.com/azure/aks/workload-identity-overview)
credentials, or [managed
identity](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview)
credentials.

### Development

If you are not ready to connect Teleport to a production instance of
PostgreSQL, you can use the following instructions to set up a throwaway
instance of PostgreSQL using Docker.

First copy the following script to disk and run it to generate the CA, client
certificate, and server certificate used by Teleport and PostgreSQL to
establish a secure mutually authenticated connection:

```
#!/bin/bash

# Create the certs directory.
mkdir -p ./certs
cd certs/

# Create CA key and self-signed certificate.
openssl genpkey -algorithm RSA -out ca.key
openssl req -x509 -new -key ca.key -out ca.crt -subj "/CN=root"

# Function to create certificates.
create_certificate() {
    local name="$1"
    local dns_name="$2"

    openssl genpkey \
        -algorithm RSA \
        -out "${name}.key"
    openssl req -new \
        -key "${name}.key" \
        -out "${name}.csr" \
        -subj "/CN=${dns_name}"
    openssl x509 -req \
        -in "${name}.csr" \
        -CA ca.crt \
        -CAkey ca.key \
        -out "${name}.crt" \
        -extfile <(printf "subjectAltName=DNS:${dns_name}") \
        -CAcreateserial

    chmod 0600 "${name}.key"
}

# Create client certificate with SAN.
create_certificate "client" "teleport"

# Create server certificate with SAN.
create_certificate "server" "localhost"

echo "Certificates and keys generated successfully."
```

Next, create a `Dockerfile` using the [official PostgreSQL Docker
image](https://hub.docker.com/_/postgres) and add `wal2json` to it:

```
FROM postgres:15.0
RUN apt-get update
RUN apt-get install -y postgresql-15-wal2json
```

Create an `init.sql` file that will ensure the Teleport user is created upon
startup of the container:

```
CREATE USER teleport WITH REPLICATION CREATEDB;
```

Create a `pg_hba.conf` file to enforce certificate-based authentication for
connections to PostgreSQL:

```
# TYPE  DATABASE        USER            CIDR-ADDRESS            METHOD
local   all             all                                     trust
hostssl all             all             ::/0                    cert
hostssl all             all             0.0.0.0/0               cert
```

Create a `postgresql.conf` file that configures the WAL level and certificates
used for authentication:

```
listen_addresses = '*'
port = 5432
max_connections = 20
shared_buffers = 128MB
temp_buffers = 8MB
work_mem = 4MB

wal_level=logical
max_replication_slots=10

ssl=on
ssl_ca_file='/certs/ca.crt'
ssl_cert_file='/certs/server.crt'
ssl_key_file='/certs/server.key'
```

Start the PostgreSQL container with the following command:

```
docker run --rm --name postgres \
    -e POSTGRES_DB=db \
    -e POSTGRES_USER=user \
    -e POSTGRES_PASSWORD=password \
    -v $(pwd)/data:/var/lib/postgresql/data \
    -v $(pwd)/certs:/certs \
    -v $(pwd)/postgresql.conf:/etc/postgresql/postgresql.conf \
    -v $(pwd)/pg_hba.conf:/etc/postgresql/pg_hba.conf \
    -v $(pwd)/init.sql:/docker-entrypoint-initdb.d/init.sql \
    -p 5432:5432 \
    $(docker build -q .) \
    postgres \
    -c hba_file=/etc/postgresql/pg_hba.conf \
    -c config_file=/etc/postgresql/postgresql.conf
```

Lastly, update the storage section in `teleport.yaml` to use PostgreSQL and
start Teleport:

```yaml
teleport:
  storage:
    type: postgresql
    conn_string: "postgresql://teleport@localhost:5432/teleport_backend?sslcert=/path/to/certs/client.crt&sslkey=/path/to/certs/client.key&sslrootcert=/path/to/certs/ca.crt&sslmode=verify-full&pool_max_conns=20"
```

## S3

<Admonition
  type="tip"
  title="Tip"
>
  Before continuing, please make sure to take a look at the [Cluster State section](../architecture/nodes.mdx#cluster-state)
  in Teleport Architecture documentation.
</Admonition>

S3 buckets can only be used as storage for the recorded sessions. S3 cannot
store the audit log or the cluster state.

S3 buckets must have versioning enabled, which ensures that a session log cannot
be permanently altered or deleted. Teleport will always look at the oldest
version of a recording.

### Authenticating to AWS

The Teleport Auth Service must be able to read AWS credentials in order to
authenticate to S3.

(!docs/pages/includes/aws-credentials.mdx service="the Teleport Auth Service"!)

### Configuring the S3 backend

Below is an example of how to configure the Teleport Auth Service to store the
recorded sessions in an S3 bucket.

```yaml
teleport:
  storage:
      # The region setting sets the default AWS region for all AWS services
      # Teleport may consume (DynamoDB, S3)
      region: us-east-1

      # Path to S3 bucket to store the recorded sessions in.
      audit_sessions_uri: "s3://Example_TELEPORT_S3_BUCKET/records"

      # Teleport assumes credentials. Using provider chains, assuming IAM role or
      # standard .aws/credentials in the home folder.
```

You can add optional query parameters to the S3 URL. The Teleport Auth
Service reads these parameters to configure its interactions with S3:

`s3://bucket/path?region=us-east-1&endpoint=mys3.example.com&insecure=false&disablesse=false&acl=private&use_fips_endpoint=true`

- `region=us-east-1` - set the Amazon region to use.

- `endpoint=mys3.example.com` - connect to a custom S3 endpoint. Optional.

- `insecure=true` - set to `true` or `false`. If `true`, TLS will be disabled.
  Default value is `false`.

- `disablesse=true` - set to `true` or `false`. The Auth Service checks this
  value before uploading an object to an S3 bucket.

  If this is `false`, the Auth Service will set the server-side encryption
  configuration of the upload to use AWS Key Management Service and, if
  `sse_kms_key` is set, configure the upload to use this key.

  If this value is `true`, the Auth Service will not set an explicit server-side
  encryption configuration for the object upload, meaning that the upload will
  use the bucket-level server-side encryption configuration.

- `sse_kms_key=kms_key_id` - If set to a valid AWS KMS CMK key ID, all objects
  uploaded to S3 will be encrypted with this key (as long as `disablesse` is
  `false`). Details can be found below.

- `acl=private` - set the [canned
  ACL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl)
  to use. Must be one of the predefined ACL values.

- `use_fips_endpoint=true` -  [Configure S3 FIPS
  endpoints](#configuring-aws-fips-endpoints)

### S3 IAM policy

(!docs/pages/includes/s3-iam-policy.mdx!)

### S3 Server Side Encryption

Teleport supports using a custom AWS KMS Customer Managed Key for encrypting objects uploaded to S3.
This allows you to restrict who can read objects like session recordings separately from those that have read
access to a bucket by restricting key access.

The `sse_kms_key` parameter above can be set to any valid KMS CMK ID corresponding to a symmetric standard spec KMS key.
Example template KMS key policies are provided below for common usage cases. IAM users do not have access to any
key by default. Permissions have to be explicitly granted in the policy.

#### Encryption/Decryption

This policy allows an IAM user to encrypt and decrypt objects.
This allows a cluster auth to write and play back session recordings.

Replace `[iam-key-admin-arn]` with the IAM ARN of the user(s) that should have
administrative key access and `[auth-node-iam-arn]` with the IAM ARN
of the user the Teleport auth nodes are using.

```json
{
  "Id": "Teleport Encryption and Decryption",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Teleport CMK Admin",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[iam-key-admin-arn]"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Teleport CMK Auth",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[auth-node-iam-arn]"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}
```

### Encryption/Decryption with separate clusters

This policy allows specifying separate IAM users for encryption and decryption.
This can be used to set up a multi cluster configuration where the main cluster
cannot play back session recordings but only write them.
A separate cluster authenticating as a different IAM user with decryption access
can be used for playing back the session recordings.

Replace `[iam-key-admin-arn]` with the IAM ARN of the user(s) that should have
administrative key access, `[iam-node-write-arn]` with the IAM ARN of the user the
main write-only cluster auth nodes are using and `[iam-node-read-arn]` with the
IAM ARN of the user used by the read-only cluster.

For this to work the second cluster has to be connected to the same audit log as the main cluster.
This is needed to detect session recordings.

```json
{
  "Id": "Teleport Separate Encryption and Decryption",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Teleport CMK Admin",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[iam-key-admin-arn]"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Teleport CMK Auth Encrypt",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[auth-node-write-arn]"
      },
      "Action": [
        "kms:Encrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Teleport CMK Auth Decrypt",
      "Effect": "Allow",
      "Principal": {
        "AWS": "[auth-node-read-arn]"
      },
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}
```

### ACL example: transferring object ownership

If you are uploading from AWS account `A` to a bucket owned by AWS account `B` and want `A` to retain ownership of the objects, you can take one of two approaches.

#### Without ACLs

If ACLs are disabled, object ownership will be set to `Bucket owner enforced` and no action will be needed.

#### With ACLs

- Set object ownership to `Bucket owner preferred` (under Permissions in the management console).
- Add `acl=bucket-owner-full-control` to `audit_sessions_uri`.

To enforce the ownership transfer, set `B`'s bucket's policy to only allow uploads that include the `bucket-owner-full-control` canned ACL.

```json
{
    "Version": "2012-10-17",
    "Id": "[id]",
    "Statement": [
        {
            "Sid": "[sid]",
            "Effect": "Allow",
            "Principal": {
                "AWS": "[ARN of account A]"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::BucketName/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}

```

For more information, see the [AWS Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/about-object-ownership.html).

## DynamoDB

<Admonition
  type="tip"
  title="Tip"
>
  Before continuing, please make sure to take a look at the [Cluster State section](../architecture/nodes.mdx#cluster-state)
  in the Teleport Architecture documentation.
</Admonition>

If you are running Teleport on AWS, you can use
[DynamoDB](https://aws.amazon.com/dynamodb/) as a storage back-end to achieve
High Availability. DynamoDB backend supports two types of Teleport data:

- Cluster state
- Audit log events

Teleport uses DynamoDB and DynamoDB Streams endpoints for its storage
back-end management.

DynamoDB cannot store the recorded sessions. You are advised to use AWS S3 for
that as shown above.

### Authenticating to AWS

The Teleport Auth Service must be able to read AWS credentials in order to
authenticate to DynamoDB.

(!docs/pages/includes/aws-credentials.mdx service="the Teleport Auth Service"!)

The IAM role that the Teleport Auth Service authenticates as must have the
policies specified in the next section.

### IAM policies

Make sure that the IAM role assigned to Teleport is configured with sufficient
access to DynamoDB.

(!docs/pages/includes/dynamodb-iam-policy.mdx!)

### Configuring the DynamoDB backend

To configure Teleport to use DynamoDB:

- Configure all Teleport Auth servers to use DynamoDB back-end in the "storage"
  section of `teleport.yaml` as shown below.
- Auth servers must be able to reach DynamoDB and DynamoDB Streams endpoints.
- Deploy several auth servers connected to DynamoDB storage back-end.
- Deploy several proxy nodes.
- Make sure that all Teleport resource services have the `auth_servers` configuration setting
  populated with the addresses of your cluster's Auth Service instances.

```yaml
teleport:
  storage:
    type: dynamodb
    # Region location of dynamodb instance, https://docs.aws.amazon.com/en_pv/general/latest/gr/rande.html#ddb_region
    region: us-east-1

    # Name of the DynamoDB table. If it does not exist, Teleport will create it.
    table_name: Example_TELEPORT_DYNAMO_TABLE_NAME

    # This setting configures Teleport to send the audit events to three places:
    # To keep a copy in DynamoDB, a copy on a local filesystem, and also output the events to stdout.
    # NOTE: The DynamoDB events table has a different schema to the regular Teleport
    # database table, so attempting to use the same table for both will result in errors.
    # When using highly available storage like DynamoDB, you should make sure that the list always specifies
    # the High Availability storage method first, as this is what the Teleport web UI uses as its source of events to display.
    audit_events_uri:  ['dynamodb://events_table_name', 'file:///var/lib/teleport/audit/events', 'stdout://']

    # This setting configures Teleport to save the recorded sessions in an S3 bucket:
    audit_sessions_uri: s3://Example_TELEPORT_S3_BUCKET/records

    # By default, Teleport stores audit events with an AWS TTL of 1 year.
    # This value can be configured as shown below. If set to 0 seconds, TTL is disabled.
    retention_period: 365d

    # Enables either Pay Per Request or Provisioned billing for the DynamoDB table. Set when Teleport creates the table.
    # Possible values: "pay_per_request" and "provisioned"
    # default: "pay_per_request"
    billing_mode: "pay_per_request"
```

- Replace `us-east-1` and `Example_TELEPORT_DYNAMO_TABLE_NAME`
  with your own settings.  Teleport will create the table automatically.
- `Example_TELEPORT_DYNAMO_TABLE_NAME` and `events_table_name` **must** be different
  DynamoDB tables. The schema is different for each. Using the same table name for both will result in errors.
- Audit log settings above are optional. If specified, Teleport will store the
  audit log in DynamoDB and the session recordings **must** be stored in an S3
  bucket, i.e. both `audit_xxx` settings must be present. If they are not set,
  Teleport will default to a local file system for the audit log, i.e.
  `/var/lib/teleport/log` on an auth server.

The optional `GET` parameters shown below control how Teleport interacts with a DynamoDB endpoint.

`dynamodb://events_table_name?region=us-east-1&endpoint=dynamo.example.com&use_fips_endpoint=true`

- `region=us-east-1` - set the Amazon region to use.
- `endpoint=dynamo.example.com` - connect to a custom S3 endpoint.
- `use_fips_endpoint=true` -  [Configure DynamoDB FIPS endpoints](#configuring-aws-fips-endpoints).

### DynamoDB autoscaling

When setting up DynamoDB it's important to set up backups. Autoscaling is
optional with Teleport, and it is advised to collect some production usage data
before enabling autoscaling.

Autoscaling and backup can be configured on DynamoDB directly. We also make
setup simpler by allowing AWS DynamoDB settings to be set automatically during
Teleport startup. The configuration is described below.

**DynamoDB Continuous Backups**

- [AWS Blog Post - Amazon DynamoDB Continuous Backup](https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-continuous-backups-and-point-in-time-recovery-pitr/)

**DynamoDB Autoscaling Options**

- [AWS Docs - Managing Throughput Capacity Automatically with DynamoDB Auto Scaling](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html)

<Notice type="danger">

AWS can throttle DynamoDB if more than two processes are reading from the same
stream's shard simultaneously, so you must not deploy more than two Auth Service
instances that read from a DynamoDB backend. For details on DynamoDB Streams,
read the [AWS
documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html).

</Notice>

```yaml
# ...
teleport:
  storage:
    type: "dynamodb"
    [...]

    # continuous_backups is used to optionally enable continuous backups.
    # default: false
    continuous_backups: true

    # auto_scaling is used to optionally enable (and define settings for) auto scaling.
    # default: false
    auto_scaling: true
    # Enables either Pay-Per-Request or Provisioned billing for the DynamoDB table. Set when Teleport creates the table.
    # If billing_mode is set to "pay_per_request", read/write capacity and auto_scaling settings will be ignored.
    # Possible values: "pay_per_request" and "provisioned"
    # default: "pay_per_request"
    billing_mode: ["pay_per_request"|"provisioned"]
    # Minimum/maximum read capacity in units
    read_min_capacity: int
    read_max_capacity: int
    read_target_value: float
    # Minimum/maximum write capacity in units
    write_min_capacity: int
    write_max_capacity: int
    write_target_value: float
```

To enable these options you will need to update the IAM Policy for Teleport.

```json
{
    "Action": [
        "application-autoscaling:PutScalingPolicy",
        "application-autoscaling:RegisterScalableTarget"
    ],
    "Effect": "Allow",
    "Resource": "*"
},
{
    "Action": [
        "iam:CreateServiceLinkedRole"
    ],
    "Condition": {
        "StringEquals": {
            "iam:AWSServiceName": [
                "dynamodb.application-autoscaling.amazonaws.com"
            ]
        }
    },
    "Effect": "Allow",
    "Resource": "*"
}
```

<Admonition
  type="warning"
  title="pay_per_request with autoscaling"
>
  If billing mode is set to `pay_per_request`, the settings for read/write capacity and auto scaling will be ignored.
</Admonition>

### Configuring AWS FIPS endpoints

This config option applies to [AWS S3](#s3) and [AWS DynamoDB](#dynamodb).

Set `use_fips_endpoint` to `true` or `false`. If `true`, [FIPS](https://aws.amazon.com/compliance/fips/) Dynamo endpoints will be used.
If `false`, normal Dynamo endpoints will be used. If unset, the AWS Environment Variable `AWS_USE_FIPS_ENDPOINT` will determine which endpoint is used.
FIPS endpoints will also be used if Teleport is run with the `--fips` flag.

Config option priority is applied in the following order:

- Setting the `use_fips_endpoint` query parameter as shown above
- Using the `--fips` flag when running Teleport
- Using the AWS environment variable

<Admonition
  type="warning"
  title="A warning about AWS_USE_FIPS_ENDPOINT"
>
  Setting this environment variable to true will enable FIPS endpoints for all AWS resource types. Some FIPS endpoints
  are not supported in certain regions or environments or are only supported in GovCloud.
</Admonition>

## Athena

The Athena audit log back-end is available starting from Teleport v14.0.

If you are running Teleport on AWS, you can use an
[Athena](https://aws.amazon.com/athena/)-based audit log system that manages
[Parquet files](https://parquet.apache.org/) stored on
[S3](https://aws.amazon.com/s3/) as a storage back-end to achieve high
availability. The Athena back-end supports only one type of Teleport data, audit
events.

The Athena audit back-end is better at scale and search than DynamoDB.

The Athena audit logs are eventually consistent. It may take up to one minute
(depending on the `batchMaxInterval` setting and event load) until you can view
events in the Teleport Web UI.

### Infrastructure setup

The Auth Service uses an SQS queue subscribed to an SNS topic for event
publishing. A single Auth Service instance reads events in batches from SQS,
converts them into Parquet format, and sends the resulting data to S3. During
queries, the Athena engine searches for events on S3, reading metadata from a
Glue table.

You can set up the required infrastructure to support the Athena backend with
the following Terraform script:

<Details title="Terraform script" opened={false}>
```
(!examples/athena/variables.tf!)
(!examples/athena/athena.tf!)
```
</Details>

### Configuring the Athena audit log backend

To configure Teleport to use Athena:

- Make sure you are using **Teleport version 14.0.0** or newer.
- Prepare infrastructure
- Specify an Athena URL inside the `audit_events_uri` array in your Teleport
  configuration file:

```yaml
teleport:
  storage:
    # This setting configures Teleport to keep a copy of the audit log in Athena
    # and a copy on a local filesystem, and also to output the events to stdout.
    audit_events_uri:
      # More details about the full Athena URL are shown below.
      - 'athena://database.table?params'
      - 'file:///var/lib/teleport/audit/events'
      - 'stdout://'
```

Here is an example of an Amazon Athena URL within the `audit_events_uri` configuration field:

```
athena://db.table?topicArn=arn:aws:sns:region:account_id:topic_name&largeEventsS3=s3://transient/large_payloads&locationS3=s3://long-term/events&workgroup=workgroup&queueURL=https://sqs.region.amazonaws.com/account_id/queue_name&queryResultsS3=s3://transient/query_results
```

The URL hostname consist of `database.table`, which points to the Glue database
and a table which will be used by the Athena audit logger.

Other parameters are specified as query parameters within the Athena URL.

The following parameters are required:

| Parameter name   | Example value                                            | Description                                            |
|------------------|----------------------------------------------------------|--------------------------------------------------------|
| `topicArn`       | `arn:aws:sns:region:account_id:topic_name`               | ARN of SNS topic where events are published            |
| `locationS3`     | `s3://long-term/events`                                  | S3 bucket used for long-term storage                   |
| `largeEventsS3`  | `s3://transient/large_payloads`                          | S3 bucket used for transient storage for large events  |
| `queueURL`       | `https://sqs.region.amazonaws.com/account_id/queue_name` | SQS URL used for a subscription to an SNS topic        |
| `workgroup`      | `workgroup_name`                                         | Athena workgroup used for queries                      |
| `queryResultsS3` | `s3://transient/results`                                 | S3 bucket used for transient storage for query results |


The following parameters are optional:

| Parameter name     | Example value | Description                                                                                           |
|--------------------|---------------|-------------------------------------------------------------------------------------------------------|
| `region`           | `us-east-1`   | AWS region. If empty, defaults to one from the AuditConfig or ambient AWS credentials                 |
| `batchMaxItems`    | `20000`       | defines the maximum number of events allowed for a single Parquet file (default 20000)                |
| `batchMaxInterval` | `1m`          | defines the maximum interval used to buffer incoming data before creating a Parquet file (default 1m) |

### Authenticating to AWS

The Teleport Auth Service must be able to read AWS credentials in order to
authenticate to Athena.

(!docs/pages/includes/aws-credentials.mdx service="the Teleport Auth Service"!)

The IAM role that the Teleport Auth Service authenticates as must have the
policies specified in the next section.

### IAM policies

Make sure that the IAM role assigned to Teleport is configured with sufficient
access to Athena. Below you can find the IAM permissions that the Auth Service
requires to use Athena Audit logs as an audit event backend.

You'll need to replace these values in the policy example below:

| Placeholder value | Replace with                                             |
|-------------------|----------------------------------------------------------|
| `eu-central-1`    | AWS region                                               |
| `1234567890`      | AWS account ID                                           |
| `audit-long-term` | S3 bucket used for long-term storage                     |
| `audit-transient` | S3 bucket used for transient storage                     |
| `audit-sqs`       | SNS topic name                                           |
| `audit-sns`       | SQS name                                                 |
| `kms_id`          | KMS key ID used for server-side encryption of SNS/SQS/S3 |
| `audit_db`        | Glue database used for audit logs                        |
| `audit_table`     | Glue table used for audit logs                           |
| `audit_workgroup` | Athena workgroup used for audit logs                     |

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:ListBucketVersions",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::audit-transient",
                "arn:aws:s3:::audit-long-term"
            ],
            "Sid": "AllowListingMultipartUploads"
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:ListMultipartUploadParts",
                "s3:GetObjectVersion",
                "s3:GetObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::audit-transient/results/*",
                "arn:aws:s3:::audit-transient/large_payloads/*",
                "arn:aws:s3:::audit-long-term/events/*"
            ],
            "Sid": "AllowMultipartAndObjectAccess"
        },
        {
            "Action": "sns:Publish",
            "Effect": "Allow",
            "Resource": "arn:aws:sns:eu-central-1:1234567890:audit-sqs",
            "Sid": "AllowPublishSNS"
        },
        {
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:eu-central-1:1234567890:audit-sns",
            "Sid": "AllowReceiveSQS"
        },
        {
            "Action": [
                "glue:GetTable",
                "athena:StartQueryExecution",
                "athena:GetQueryResults",
                "athena:GetQueryExecution"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:glue:eu-central-1:1234567890:table/audit_db/audit_table",
                "arn:aws:glue:eu-central-1:1234567890:database/audit_db",
                "arn:aws:glue:eu-central-1:1234567890:catalog",
                "arn:aws:athena:eu-central-1:1234567890:workgroup/audit_workgroup"
            ],
            "Sid": "AllowAthenaQuery"
        },
        {
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kms:eu-central-1:1234567890:key/kms_id",
            "Sid": "AllowAthenaKMSUsage"
        }
    ]
}
```

### Migration from Dynamo to the Athena audit logs backend

<Admonition
  type="tip"
  title="Tip"
>
  Migration is only needed if you used Amazon DynamoDB for audit logs and you
  want to keep old data.
</Admonition>

Migration consist of following steps:

1. Set up Athena infrastructure
1. Dual write to both DynamoDB and Athena, and query from DynamoDB
1. Migrate old data from DynamoDB to Athena
1. Dual write to both DynamoDB and Athena, and query from Athena
1. Disable writing to DynamoDB

In the Teleport storage configuration, `audit_events_uri` accepts multiple
URLs. Those URLs are used to configure connections to the different audit
loggers. If more than 1 is used, then events are written to each audit system,
and queries are executed from first one.

<Admonition
  type="tip"
  title="Tip"
>
  If anything goes wrong during migration steps 1-4, roll back to the Amazon
  DynamoDB solution by making sure its URL is the first value in the
  `audit_events_uri` field and removing the Athena URL.
</Admonition>

Each of these steps is explained in more detail below.

#### Dual write to both DynamoDB and Athena, and query from DynamoDB

The second step of migration requires setting the following configuration:

```yaml
teleport:
  storage:
    audit_events_uri:
    - 'dynamodb://events_table_name'
    - 'athena://db.table?otherQueryParams'
```

When an Auth Service instance is restarted, you should verify that Parquet files
are stored in the S3 bucket specified using the `locationS3` parameter.

#### Migrate old data from DynamoDB to Athena

This step requires using the client machine to export data from Amazon DynamoDB
and publish it to the Athena logger. We recommend using, for example, an EC2
instance with a disk size at least 2x bigger than the table size in Amazon
DynamoDB.

Instructions for how to use the migration tool can be found
[on GitHub](https://github.com/gravitational/teleport/blob/master/examples/dynamoathenamigration/README.md).

You should set `exportTime` to the time when dual writing began.

We recommend running your first migration with the `-dry-run` flag because it
validates the exported data. If no errors are reported, proceed to a real
migration without the `-dry-run` flag.

#### Dual write to both DynamoDB and Athena, and query from Athena

Change the order of the `audit_events_uri` values in your Teleport
configuration file:

```yaml
teleport:
  storage:
    audit_events_uri:
    - 'athena://db.table?otherQueryParams'
    - 'dynamodb://events_table_name'
```

When the Auth Service is restarted, you should verify that events are visible
on the Audit Logs page.

#### Disable writing to DynamoDB

Disabling writing to DynamoDB means that you won't be able to roll back to
DynamoDB without losing data. Dual writing to both Athena and DynamoDB does not
have a significant performance impact, and it's recommended to keep dual
writing for some time, even if your system already executes queries from
Athena.

To disable writing to DynamoDB, remove the DynamoDB URL from the
`audit_events_uri` array.

## GCS

<Admonition
  type="tip"
  title="Tip"
>
  Before continuing, please make sure to take a look at the [Cluster State section](../architecture/nodes.mdx#cluster-state)
  in Teleport Architecture documentation.
</Admonition>

Google Cloud Storage (GCS) can be used as storage for recorded
sessions. GCS cannot store the audit log or the cluster state. Below is an
example of how to configure a Teleport auth server to store the recorded
sessions in a GCS bucket.

```yaml
teleport:
  storage:
      # Path to GCS to store the recorded sessions in.
      audit_sessions_uri: 'gs://$BUCKET_NAME/records?projectID=$PROJECT_ID&credentialsPath=$CREDENTIALS_PATH'
```

We recommend creating a bucket in `Dual-Region` mode with the `Standard` storage class to ensure cluster performance and high availability.
Replace the following variables in the above example with your own values:
- `$BUCKET_NAME` with the name of the desired GCS bucket. If the bucket does not exist it will be created.
  Please ensure the following permissions are granted for the given bucket:
  - `storage.buckets.get`
  - `storage.objects.create`
  - `storage.objects.get`
  - `storage.objects.list`
  - `storage.objects.update`
  - `storage.objects.delete`

  `storage.objects.delete` is required in order to clean up multipart files after they have been assembled
  into the final blob.

  If the bucket does not exist, please also ensure that the `storage.buckets.create` permission is granted.

- `$PROJECT_ID` with a GCS-enabled GCP project.

- `$CREDENTIALS_PATH` with the path to a JSON-formatted GCP credentials file configured for a service account applicable to the project.

## Firestore

<Admonition
  type="tip"
  title="Tip"
>
  Before continuing, please make sure to take a look at the [Cluster State section](../architecture/nodes.mdx#cluster-state)
  in Teleport Architecture documentation.
</Admonition>

If you are running Teleport on GCP, you can use
[Firestore](https://cloud.google.com/firestore/) as a storage back-end to achieve
high availability. Firestore backend supports two types of Teleport data:

- Cluster state
- Audit log events

Firestore cannot store the recorded sessions. You are advised to use Google
Cloud Storage (GCS) for that as shown above. To configure Teleport to use
Firestore:

- Configure all Teleport Auth servers to use Firestore back-end in the "storage"
  section of `teleport.yaml` as shown below.
- Deploy several auth servers connected to Firestore storage back-end.
- Deploy several proxy nodes.
- Make sure that all Teleport resource services have the `auth_servers`
  configuration setting populated with the addresses of your cluster's Auth
  Service instances or use a load balancer for Auth Service instances in high
  availability mode.

```yaml
teleport:
  storage:
    type: firestore
    # Project ID https://support.google.com/googleapi/answer/7014113?hl=en
    project_id: Example_GCP_Project_Name

    # Name of the Firestore table.
    collection_name: Example_TELEPORT_FIRESTORE_TABLE_NAME

    credentials_path: /var/lib/teleport/gcs_creds

    # This setting configures Teleport to send the audit events to three places:
    # To keep a copy in Firestore, a copy on a local filesystem, and also write the events to stdout.
    # NOTE: The Firestore events table has a different schema to the regular Teleport
    # database table, so attempting to use the same table for both will result in errors.
    # When using highly available storage like Firestore, you should make sure that the list always specifies
    # the High Availability storage method first, as this is what the Teleport web UI uses as its source of events to display.
    audit_events_uri:  ['firestore://Example_TELEPORT_FIRESTORE_EVENTS_TABLE_NAME', 'file:///var/lib/teleport/audit/events', 'stdout://']

    # This setting configures Teleport to save the recorded sessions in GCP storage:
    audit_sessions_uri: gs://Example_TELEPORT_GCS_BUCKET/records
```

- Replace `Example_GCP_Project_Name` and `Example_TELEPORT_FIRESTORE_TABLE_NAME`
  with your own settings. Teleport will create the table automatically.
- `Example_TELEPORT_FIRESTORE_TABLE_NAME` and `Example_TELEPORT_FIRESTORE_EVENTS_TABLE_NAME`
  **must** be different Firestore tables. The schema is different for each.
  Using the same table name for both will result in errors.
- The GCP authentication setting above can be omitted if the machine itself is
  running on a GCE instance with a Service Account that has access to the
  Firestore table.
- Audit log settings above are optional. If specified, Teleport will store the audit log in Firestore
  and the session recordings **must** be stored in a GCS bucket, i.e. both `audit_xxx` settings must
  be present. If they are not set, Teleport will default to a local filesystem for the audit log, i.e.
  `/var/lib/teleport/log` on an auth server.

## Azure Blob Storage

Azure Blob Storage for session storage is available starting from Teleport
`13.3`.

<Admonition
  type="tip"
  title="Tip"
>
  Before continuing, please make sure to take a look at the [Cluster State section](../architecture/nodes.mdx#cluster-state)
  in Teleport Architecture documentation.
</Admonition>

Azure Blob Storage can be used as storage for recorded sessions. Azure Blob
Storage cannot store the audit log or the cluster state. Below is an example of
how to configure a Teleport auth server to store the recorded sessions in an
Azure Blob Storage storage account.

```yaml
teleport:
  storage:
    audit_sessions_uri: azblob://account-name.core.blob.windows.net
```

Teleport makes use of two containers in the account, whose names default to
`inprogress` and `session`, but they can be configured with parameters in the
fragment of the URI.

```yaml
teleport:
  storage:
    audit_sessions_uri: azblob://account-name.core.blob.windows.net#session_container=session_container_name&inprogress_container=inprogress_container_name
```

### Permissions

Teleport needs the following permissions on the `inprogress` container:
- `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read`
- `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write`
- `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete` (only on the `inprogress` container)

In addition, Teleport will check if the containers exist at startup, and it will
attempt to create them if they can't be confirmed to exist; giving Teleport
`Microsoft.Storage/storageAccounts/blobServices/containers/read` will allow for
checking and `Microsoft.Storage/storageAccounts/blobServices/containers/write`
will allow for creating them.

It's highly recommended to set up a [time-based retention
policy](https://learn.microsoft.com/azure/storage/blobs/immutable-time-based-retention-policy-overview)
for the `session` container, as well as a [lifecycle management
policy](https://learn.microsoft.com/azure/storage/blobs/lifecycle-management-overview),
so that recordings are kept in an immutable state for a given period, then
deleted. Teleport will not delete recordings automatically.

With a time-based retention policy in place, it's safe to give Teleport the
"Blob Storage Data Contributor" role scoped to the containers, instead of having
to define a custom role for it.

### Authentication

Teleport will make use of the Azure AD credentials specified by [environment
variables](https://learn.microsoft.com/azure/developer/go/azure-sdk-authentication#2-authenticate-with-azure),
[Azure AD Workload
Identity](https://learn.microsoft.com/azure/aks/workload-identity-overview)
credentials, or [managed
identity](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview)
credentials.

## SQLite

The Auth Service uses the SQLite backend when no `type` is specified in the
storage section in the Teleport configuration file, or when `type` is set to
`sqlite` or `dir`. The SQLite backend is not designed for high throughput and
it's not capable of serving the needs of Teleport's High Availability configurations.

If you are planning to use SQLite as your backend, scale your cluster slowly and
monitor the number of warning messages in the Auth Service's logs that say
`SLOW TRANSACTION`, as that's a sign that the cluster has outgrown the capabilities
of the SQLite backend.

As a stopgap measure until it's possible to migrate the cluster to use a
HA-capable backend, you can configure the SQLite backend to reduce the amount of
disk synchronization, in exchange for less resilience against system crashes or
power loss. For an explanation on what the options mean, see [the official
SQLite docs](https://www.sqlite.org/pragma.html#pragma_synchronous). No matter
the configuration, we recommend you take regular backups of your cluster state.

To reduce disk synchronization:
```yaml
teleport:
  storage:
    type: sqlite
    sync: NORMAL
```

To disable disk synchronization altogether:
```yaml
teleport:
  storage:
    type: sqlite
    sync: "OFF"
```

When running on a filesystem that supports file locks (i.e. a local filesystem,
not a networked one) it's possible to also configure the SQLite database to use
Write-Ahead Logging (see [the official docs on WAL
mode](https://www.sqlite.org/wal.html)) for significantly improved performance
without sacrificing reliability:

```yaml
teleport:
  storage:
    type: sqlite
    sync: NORMAL
    journal: WAL
```