teleport/docs/pages/upgrading/cloud-linux.mdx

---
title: Upgrade Teleport Cloud Agents on Linux
description: Provides instructions for upgrading Teleport Cloud agents that run on Linux servers. 
---

This guide explains how to upgrade Teleport Cloud agents running on Linux
servers. On Teleport Cloud, Auth Service and Proxy Service upgrades are managed
for you. To keep agents up to date, you can either enroll them in automatic
upgrades or upgrade them manually using the method you used to install Teleport.

## Prerequisites

- Familiarity with the [Upgrading Compatibility Overview](./overview.mdx) guide,
  which describes the sequence in which to upgrade components in your cluster.

- A Teleport Enterprise Cloud account. You can determine the current version of
  the Auth Service and Proxy Service by running the following command, where
  `mytenant` is the name of your Teleport Enterprise Cloud account. This
  requires the [`jq` CLI tool](https://jqlang.github.io/jq/):

  ```code
  $ curl -s https://mytenant.teleport.sh/webapi/ping | jq '.server_version'
  ```

- (!docs/pages/includes/cloud/tctl-tsh-prerequisite.mdx!)

- (!docs/pages/includes/tctl.mdx!)

Teleport supports automatic agent updates for systemd-based Linux distributions
using `apt`, `yum`, and `zypper` package managers. The [Automatic Update
Architecture](../architecture/agent-update-management.mdx) guide describes
how agent updating works. Automatic agent upgrades require:

- A Teleport Cloud account that supports automatic updates. To determine if
  your account supports automatic updates, run the following command, replacing
  `mytenant.teleport.sh` with the address of your Teleport Cloud account:

  ```code
  $ curl -s https://mytenant.teleport.sh/webapi/ping | jq '.automatic_upgrades'
  true
  ```

- At least one Teleport Enterprise agent, started via systemd on a distribution
  using the `apt` or `yum` package managers. 

  So you can access each agent, we assume that each agent runs the Teleport SSH
  Service in addition to any other services. See the [Deploy Teleport Agents
  with Terraform](../agents/deploy-agents-terraform.mdx) guide for an example.

## Step 1/2. Find agents to upgrade

To list agents that are not enrolled in automatic updates, use the `tctl
inventory ls` command with the `--upgrader=none` flag. This command also uses
the `--services` flag to constrain the search to Teleport processes running
agent services:

```code
$ tctl inventory ls \
  --services=node,kube,db,app,windows_desktop,discovery,okta \
  --upgrader=none
Server ID                            Hostname      Services Version Upgrader
------------------------------------ ------------- -------- ------- --------
00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node     v13.4.5 none
```

If you have a lot of agents on different versions and want to prioritize enrolling
your oldest agents, you can limit your search using the `--older-than` filter:

```code
$ tctl inventory ls --upgrader=none --older-than=v1.2.3
Server ID                            Hostname        Services       Version Upgrader
------------------------------------ --------------- -------------- ------- --------
00000000-0000-0000-0000-000000000000 old.example.com Node           v1.1.1  none
00000000-0000-0000-0000-000000000001 teleport-proxy  Proxy          v14.3.6 none
00000000-0000-0000-0000-000000000002 teleport-auth   Auth,Discovery v14.3.6 none
...
```

Note that the `inventory ls` command will also list `teleport-auth` and `teleport-proxy` services.
You do not need to update these services or enroll them in automatic updates.
Teleport Cloud manages updates for these services.

When you enroll each agent in automatic updates in the next section, you can
run the following commands to fetch the hostname for each agent and access it
via Teleport. 

For each agent ID returned by the `tctl inventory ls` command, copy the ID and
run the following `tctl` command to access the host via `tsh`:

```code
$ HOST=00000000-0000-0000-0000-000000000000
$ USER=root
$ tsh ssh "${USER?}@${HOST?}"
```

You can then follow the steps in the next section on each agent to enroll it in
automatic updates.

## Step 2/2. Enroll agents in automatic updates

Complete the following instructions on each agent you would like to enroll into
automatic updates:

1. Ensure the Teleport repository is added and Teleport Enterprise is installed.

   To verify if the Teleport repository was added to the system, check if either of
   the follow files exists:
   
   ```code
   $ ls /etc/apt/sources.list.d/teleport.list
   # or
   $ ls /etc/yum.repos.d/teleport.repo
   ```

   The upgrader checks the repository for available releases, so make sure that
   it is up to date.

1. If the repository was added, make sure the Teleport binary installed on the
   agent can run the automatic updater:

   ```code
   $ which teleport-upgrade || echo "Install the upgrader"
   Install the upgrader
   ```

1. If the Teleport repository is not found, or the Teleport binary you installed
   does not include the upgrader, add the appropriate repository and reinstall
   Teleport:

   (!docs/pages/includes/cloud/install-linux-cloud.mdx!)

1. If you changed the agent user to run as non-root, create
   `/etc/teleport-upgrade.d/schedule` and grant ownership to your Teleport user.
   Otherwise, you can skip this step:

   ```code
   $ sudo mkdir -p /etc/teleport-upgrade.d/
   $ sudo touch /etc/teleport-upgrade.d/schedule
   $ sudo chown <your-teleport-user> /etc/teleport-upgrade.d/schedule
   ```

1. Verify that the upgrader can see your version endpoint by checking for
   updates:

   ```code
   $ sudo teleport-upgrade dry-run
   ```
   
   You should see one of the following messages, depending on the target version
   you are currently serving:
   
   ```text
   no upgrades available (1.2.3 == 1.2.3)
   an upgrade is available (1.2.3 -> 2.3.4)
   ```

   The upgrader may display warnings about not having a valid upgrade schedule.
   This is expected immediately after install as the maintenance schedule might
   not be exported yet.

1. Ensure that the upgrader is running:

   ```code
   $ sudo systemctl status teleport-upgrade
   ```

## Version locking

As of Teleport `15.1.10`, a version locking mechanism is enabled by the updater.
This mechanism locks the version of Teleport and prevents manual updates of the Teleport
package. This prevents unintentional updates during routine system maintenance, or
an accidental update by a user. The updater still has the capability to update the
Teleport package, and all updates of Teleport are expected to be performed by the
updater.

The version locking mechanism is implemented using the features of the package managers.
In case a user would like to manually update the Teleport version, this can be done
with the following commands.

With the APT package manager CLI, the `--allow-change-held-packages` flag must be provided
to bypass the version lock.
```code
$ apt-get install --allow-change-held-packages "teleport-ent=<target-version>"
```

With the YUM package manager CLI, the `--disableexcludes="teleport"` flag must be provided
to bypass the version lock.
```code
$ yum install --disablerepo="*" --enablerepo="teleport" --disableexcludes="teleport" "teleport-ent-<target-version>"
```

With the Zypper package manager CLI, the lock must be disabled and then re-enabled
after the update.
```code
$ zypper removelock "teleport-ent"
$ zypper install --repo="teleport" "teleport-ent-<target-version>"
$ zypper addlock "teleport-ent"
```

## Updating the updater

The updater is designed to be minimal and relatively stable, but the updater will
receive updates on occasion. Currently, the updater does not have the capability
to update itself. Updates to the `teleport-ent-updater` package must be done manually.

The `teleport-ent-updater` will be backwards compatible with older versions of Teleport,
so you can always update the `teleport-ent-updater` package to the latest available
version.

## Troubleshooting

The `teleport-upgrade` tool provides some basic commands to verify and perform an
update of the Teleport agent.

```code
$ teleport-upgrade help
USAGE: /usr/sbin/teleport-upgrade <command>

Tool for automatic upgrades of Teleport agents.

Commands:
  run           check for and potentially apply a teleport upgrade.
  dry-run       check for new teleport version but do not upgrade.
  version       print the current version of /usr/sbin/teleport-upgrade.
  help          show this help text.
```

The `dry-run` command can be used to check for an available update without performing
an update.
```code
# Example output when teleport is already on the latest compatible version.
$ teleport-upgrade dry-run
[i] no upgrades available (14.3.14 == 14.3.14) [ 582 ]

# Example output when an update is available.
$ teleport-upgrade dry-run
[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ]
[i] within maintenance window, upgrade will be attempted. [ 596 ]
```

The `run` command performs an update if available.
```code
# Successful teleport update from 13.4.14 to 14.3.14.
$ teleport-upgrade run
[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ]
[i] within maintenance window, upgrade will be attempted. [ 596 ]
[i] attempting apt install teleport-ent=14.3.14... [ 480 ]
[...]
[i] gracefully restarting Teleport (if already running) [ 449 ]

# Teleport updates are not attempted when outside the maintenance window.
$ teleport-upgrade run
[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ]
[i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ]
```

Teleport agents are not updated immediately when a new version of Teleport is released,
and agent updates can lag behind the cluster by a few days.

If the Teleport agent has not been automatically updating for several weeks, you
can view the logs of the `teleport-upgrade.service` to help troubleshoot the problem.

```code
$ journalctl -t teleport-upgrade
Apr 16 16:44:32 example teleport-upgrade[168983]: [i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ]
Apr 16 16:50:32 example teleport-upgrade[169084]: [i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ]
Apr 16 16:50:33 example teleport-upgrade[169084]: [i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ]
Apr 16 16:56:32 example teleport-upgrade[169187]: [i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ]
...
```

To suspend automatic updates, disable the `teleport.upgrade.timer` systemd timer:

```code
$ sudo systemctl disable --now teleport-upgrade.timer
```

To enable and start the systemd timer after suspending:

```code
$ sudo systemctl enable --now teleport-upgrade.timer
```