From 8ec2a3bea507cce013ee65f0c90844eb747ba65d Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Mon, 10 Nov 2025 11:46:38 -0500 Subject: [PATCH] Tbot Managed Updates & Canary Documentation (#60845) * wip * wip2 * wip3 * fix * validate * wip4 * linting * tbot install docs * remove v2 * linting * lint * more * lint * reduce ampersands * reduce ampersands more * lint --- docs/pages/includes/machine-id/daemon.mdx | 27 ++- .../resources/autoupdate_agent_rollout.mdx | 20 ++ .../reference/resources/autoupdate_config.mdx | 6 + .../machine-id/deployment/linux.mdx | 2 +- docs/pages/reference/cli/teleport-update.mdx | 26 +- .../deployment/managed-updates-v2.mdx | 43 ++++ .../upgrading/agent-managed-updates-v1.mdx | 8 +- .../agent-managed-updates.mdx | 223 ++++++++++++------ 8 files changed, 269 insertions(+), 86 deletions(-) diff --git a/docs/pages/includes/machine-id/daemon.mdx b/docs/pages/includes/machine-id/daemon.mdx index 82a45e72590..909c039411b 100644 --- a/docs/pages/includes/machine-id/daemon.mdx +++ b/docs/pages/includes/machine-id/daemon.mdx @@ -1,7 +1,29 @@ By default, `tbot` will run in daemon mode. However, this must then be configured as a service within the service manager on the Linux host. The service manager will start `tbot` on boot and ensure it is restarted if it -fails. For this guide, systemd will be demonstrated but `tbot` should be +fails. + +**If tbot was installed using the Teleport install script or `teleport-update` +command, the `tbot` systemd service is automatically created for you.** + +After `tbot.yaml` is created, enable and start the service:: + +```code +$ sudo systemctl enable tbot --now +``` + +Check the service has started successfully: + +```code +$ sudo systemctl status tbot +``` + +Service properties like `User` and `Group` may be configured using `systemctl edit tbot`.` + +**If tbot was installed manually, service configuration will need to be +performed manually as well.** + +For this guide, systemd will be demonstrated but `tbot` should be compatible with all common alternatives. Use `tbot install systemd` to generate a systemd service file: @@ -32,8 +54,7 @@ service: ```code $ sudo systemctl daemon-reload -$ sudo systemctl enable tbot -$ sudo systemctl start tbot +$ sudo systemctl enable tbot --now ``` Check the service has started successfully: diff --git a/docs/pages/includes/reference/resources/autoupdate_agent_rollout.mdx b/docs/pages/includes/reference/resources/autoupdate_agent_rollout.mdx index 20af4b9bb88..2334b53fd45 100644 --- a/docs/pages/includes/reference/resources/autoupdate_agent_rollout.mdx +++ b/docs/pages/includes/reference/resources/autoupdate_agent_rollout.mdx @@ -80,6 +80,26 @@ status: # group may execute after, from autoupdate_config. config_wait_hours: 24 + # canary_count is the number of agents selected to update and verify before the rest + # of the group. Only present for the halt-on-error schedule. + canary_count: 1 + + # canaries describes the status of the selected canaries for this group. + canaries: + + # updater_id is the unique ID of the updater that is managing the canary. + - updater_id: 3c6bcc1b-1992-4abc-a6f1-43f8f1b9ff05 + + # host_id is the unique host ID of the agent that is being updated. + host_id: c757ad82-95f2-4416-ae9d-a20ffd9b5a54 + + # hostname is the hostname of the agent that is being updated. + hostname: prod43 + + # success is true if the canary was updated successfully. + success: true + + # start_time of the rollout start_time: 0001-01-01T00:00:00Z diff --git a/docs/pages/includes/reference/resources/autoupdate_config.mdx b/docs/pages/includes/reference/resources/autoupdate_config.mdx index 43428ef9d9a..08e82e70820 100644 --- a/docs/pages/includes/reference/resources/autoupdate_config.mdx +++ b/docs/pages/includes/reference/resources/autoupdate_config.mdx @@ -56,6 +56,12 @@ spec: # Default: 0 wait_hours: 24 + # canary_count specifies the number of agents to update and verify before the rest + # of the group. Only valid for the halt-on-error schedule. + # Possible values: 0-5 + # Default: 0 + canary_count: 5 + tools: # mode allows users to enable or disable client tool updates at the # cluster level. Disable client tool automatic updates only if self-managed diff --git a/docs/pages/machine-workload-identity/machine-id/deployment/linux.mdx b/docs/pages/machine-workload-identity/machine-id/deployment/linux.mdx index 374ce9472b2..ba619eb7198 100644 --- a/docs/pages/machine-workload-identity/machine-id/deployment/linux.mdx +++ b/docs/pages/machine-workload-identity/machine-id/deployment/linux.mdx @@ -38,7 +38,7 @@ is not possible to use the same token multiple times. First, `tbot` needs to be installed on the VM that you wish to use Machine ID on. -Download the appropriate Teleport package for your platform: +Install Teleport as appropriate for your platform: (!docs/pages/includes/install-linux.mdx!) diff --git a/docs/pages/reference/cli/teleport-update.mdx b/docs/pages/reference/cli/teleport-update.mdx index 96027679d2e..e6eda3fd3ca 100644 --- a/docs/pages/reference/cli/teleport-update.mdx +++ b/docs/pages/reference/cli/teleport-update.mdx @@ -7,7 +7,7 @@ tags: - platform-wide --- -`teleport-update` is a CLI tool that is used to update Teleport Agents installed on Linux servers. +`teleport-update` is a CLI tool that is used to update Teleport Agents and Bots installed on Linux servers. See [Teleport Agent Managed Updates](../../upgrading/agent-managed-updates/agent-managed-updates.mdx) for more details. @@ -29,7 +29,7 @@ The primary commands for `teleport-update` are as follows: ## teleport-update enable -Enables agent Managed Updates and performs an initial installation of the Teleport Agent. +Enables Managed Updates and performs an initial installation of the Teleport Agent and tbot. This command also creates a systemd timer that periodically runs the update subcommand. If Teleport is already installed, `enable` will update to the cluster-advertised version @@ -41,8 +41,9 @@ Existing tarball-based, static installations may require `--overwrite`. Files are installed to the following paths (with the default `install-suffix`): -- `/usr/local/bin/{teleport,tsh,...}` - Symbolic links into `/opt/teleport/default/versions/X.Y.Z/bin/` +- `/usr/local/bin/{teleport,tbot,tsh,...}` - Symbolic links into `/opt/teleport/default/versions/X.Y.Z/bin/` - `/lib/systemd/system/teleport.service` - Teleport SystemD service +- `/etc/systemd/system/tbot.service` - Tbot SystemD service (not replaced if already present) - `/opt/teleport/default` - Storage for Teleport versions and updater configuration - `/etc/systemd/system/teleport-update.{service,timer}` - Updater SystemD timer and service - `/etc/systemd/system/teleport.service.d/teleport-update.conf` - Environment variables that configure Teleport @@ -66,7 +67,7 @@ To change these flags, run `enable` again with the new flags. ### Examples -**Example for a new installation.** +**Example for a new Teleport Agent installation.** Install Teleport with Managed Updates enabled on a fresh system. @@ -76,6 +77,17 @@ $ sudo teleport-update enable $ sudo systemctl enable teleport --now ``` +**Example for a new Teleport Agent and tbot installation.** + +Install Teleport with Managed Updates enabled on a fresh system. + +```code +# create /etc/teleport.yaml and /etc/tbot.yaml +$ sudo teleport-update enable +$ sudo systemctl enable teleport --now +$ sudo systemctl enable tbot --now +``` + **Example for an existing installation.** Install Teleport with Managed Updates enabled on a system with a running Teleport version. @@ -100,7 +112,7 @@ $ export PATH=/opt/teleport/mycluster/bin:$PATH ## teleport-update disable -Disable Managed Updates for the installed agent. +Disable Managed Updates for the installed Teleport Agent and/or tbot. This command does not remove or change the active installation of Teleport. Unlike `pin`, this command will not touch the current installation, and version lookup requests will stop entirely. @@ -126,7 +138,7 @@ $ sudo teleport-update disable ## teleport-update pin -Pin the installed agent to a specific version of Teleport. +Pin the installed Teleport Agent and/or tbot to a specific version of Teleport. This command updates Teleport to latest version (or a version specified with `--force-version`), and ensures the local installation of Teleport remains at that version. New versions will continue to be reported in SystemD `teleport-update.service` logs, they but will not be installed. @@ -238,7 +250,7 @@ Link the system installation of Teleport from the Teleport package, if Managed U This command is used to link the system package installation by: - Creating symbolic links from `/opt/teleport/system/bin/*` into `/usr/local/bin/`. -- Copying the Teleport systemd service file from `/opt/teleport/system/lib/systemd/system/teleport.service` into `/ib/systemd/system/teleport.service`. +- Copying the Teleport systemd service file from `/opt/teleport/system/lib/systemd/system/teleport.service` into `/lib/systemd/system/teleport.service`. This command is executed automatically when the Teleport package is installed, and does not need to be manually executed. diff --git a/docs/pages/reference/deployment/managed-updates-v2.mdx b/docs/pages/reference/deployment/managed-updates-v2.mdx index c00145b30cd..2f222deb42e 100644 --- a/docs/pages/reference/deployment/managed-updates-v2.mdx +++ b/docs/pages/reference/deployment/managed-updates-v2.mdx @@ -43,6 +43,10 @@ your clients receive security patches and remain compatible with your cluster. Those resources are generated automatically by Teleport. Users should not edit them, reading them can be helpful to track the autoupdate state and understand Teleport's update decisions. +These resources only track Teleport Agent installations. +However, if tbot is deployed alongside a Teleport Agent, a tbot update failure will cause the agent update to revert, +and the agent update will be marked as failed. + #### `autoupdate_agent_rollout` `autoupdate_agent_rollout` describes the rollout of the version across agent groups. @@ -195,6 +199,45 @@ intervention to resume (`tctl autoupdate agents mark-done $GROUP_NAME`). If `dev` finishes updating Friday at 16:30, `prod` will not start updating immediately as update windows are 1-hour-long and `prod`'s update window is over. `prod` update will start on Monday at 15:00. +#### Canaries + +With the `halt-on-error` strategy, the `canary_count` field can be set on each group to specify +a number of randomly selected agents (fewer than five) to update and verify before +proceeding to the rest of the agents in the group. This can be used to reduce the impact +of a failed update that might not be caught by earlier groups due to environment differences. + +**Key characteristics:** + +- Up to five Linux agent hosts are selected automatically and updated first. +- If any canary fails to update, the group is marked as failed and not updated. +- Subsequent groups will not be updated until the group successfully updates. +- Kubernetes agents are not currently selected as canaries. +- tbot-only installations are not selected as canaries, but agents may be deployed + alongside tbot to detect tbot update failures. + +**Example configuration:** + +```yaml +kind: autoupdate_config +metadata: + name: autoupdate-config +spec: + agents: + mode: enabled + strategy: halt-on-error + schedules: + regular: + - name: stage + days: ["Mon", "Tue", "Wed", "Thu"] + start_hour: 18 + canary_count: 5 + - name: prod + days: ["Mon", "Tue", "Wed", "Thu"] + start_hour: 20 + wait_hours: 24 + canary_count: 5 +``` + ### `time-based` Strategy Version Behavior diff --git a/docs/pages/upgrading/agent-managed-updates-v1.mdx b/docs/pages/upgrading/agent-managed-updates-v1.mdx index 9d848e84e67..e193c6219fe 100644 --- a/docs/pages/upgrading/agent-managed-updates-v1.mdx +++ b/docs/pages/upgrading/agent-managed-updates-v1.mdx @@ -1,5 +1,5 @@ --- -title: Managed Updates for Teleport Agents (v1) +title: Managed Updates for Teleport Agents (legacy) description: Describes how to set up Managed Updates for Teleport Agents (v1) tags: - how-to @@ -7,8 +7,8 @@ tags: --- -This document describes Managed Updates for Agents (v1), which -is currently supported but may be removed in future versions of Teleport. +This document describes Managed Updates for Agents v1, which +is a legacy system that may be removed in future versions of Teleport. For Managed Updates v2 instructions, see [Managed Updates for Agents (v2)](./agent-managed-updates/agent-managed-updates.mdx). @@ -23,7 +23,7 @@ this document. Only Enterprise versions of Teleport can use Managed Updates v1. -Please consider using [Managed Updates for Agents (v2)](./agent-managed-updates/agent-managed-updates.mdx), +Please consider using [Managed Updates for Agents v2](./agent-managed-updates/agent-managed-updates.mdx), as it provides a safer, simpler, more flexible, compatible, and reliable update experience compared to Managed Updates v1. diff --git a/docs/pages/upgrading/agent-managed-updates/agent-managed-updates.mdx b/docs/pages/upgrading/agent-managed-updates/agent-managed-updates.mdx index 33a64488688..3ddddd1a21e 100644 --- a/docs/pages/upgrading/agent-managed-updates/agent-managed-updates.mdx +++ b/docs/pages/upgrading/agent-managed-updates/agent-managed-updates.mdx @@ -1,37 +1,41 @@ --- -title: Managed Updates (v2) for Teleport Agents -description: Describes how to set up Managed Updates (v2) for Teleport Agents +title: Managed Updates for Teleport Agents and Bots +description: Describes how to set up Managed Updates for Teleport Agents and Bots tags: - - conceptual - - platform-wide + - conceptual + - platform-wide --- -This document describes Managed Updates for Agents (v2), which replaces Managed Updates v1. +This document describes Managed Updates for Agents and Bots (v2), +which replaces Managed Updates for Agents (v1). For Managed Updates v1 instructions, see [Managed Updates for Agents (v1)](../agent-managed-updates-v1.mdx). -In Managed Updates v2, a binary called `teleport-update` is distributed in -all Teleport packages, alongside the `teleport` binary. Admins configure updates -by managing the `autoupdate_version` and `autoupdate_config` dynamic resources. +For Managed Updates, a binary called `teleport-update` is distributed in +all Teleport packages, alongside the `teleport`, `tbot`, and other binaries. +Admins configure updates by managing the `autoupdate_version` and +`autoupdate_config` dynamic resources. This document covers how to use `teleport-update` and the `autoupdate_*` -resources to manage your agent updates from Teleport. It describes: +resources to manage automated agent and bot updates from Teleport. It describes: + - [The agent architecture](#how-it-works) -- [How to enroll existing agents](#quick-setup-for-existing-connected-linux-servers) -- [How to enroll new agents](#quick-setup-for-new-linux-servers) -- [How to configure Managed Updates v2](#configuring-managed-agent-updates) ( +- [How to enroll existing agents](#quick-setup-for-existing-linux-agent-and-bot-installations) +- [How to enroll new agents](#quick-setup-for-new-linux-agents-and-bot-installations) +- [How to configure Managed Updates v2](#configuring-managed-agent-and-tbot-updates) ( [when updates happen](#configuring-the-schedule) and for self-hosted users, [which version to update to](#setting-the-version-self-hosted-only)) - [How to migrate to Managed Updates v2](#migrating-agents-on-linux-servers-to-managed-updates) `teleport-update` supports: -- Both Teleport Enterprise and Teleport Community Edition + +- Teleport Enterprise and Teleport Community Edition - Both cloud and self-hosted Teleport Enterprise deployments - Regular and FIPS variants of Teleport -- amd64 and arm64 CPU architectures +- amd64, arm64, and other supported CPU architectures - systemd-based operating systems, regardless of the package manager used @@ -46,37 +50,47 @@ migration between Managed Updates v1 and v2. If `autoupdate_config` is not prese and `autoupdate_version` is present, the `autoupdate_config` settings are implicitly derived from `cluster_maintenance_config`. -Users of cloud-hosted Teleport Enterprise will be migrated to Managed Updates v2 -in the first half of 2025 and should plan to migrate their agents to `teleport-update`. +Regardless of how the cluster is configured, `teleport-update` is capable of managing +both Teleport Agent and tbot installations, while `teleport-upgrade` is only capable +of managing Teleport Agents. + +Users of cloud-hosted Teleport Enterprise have been migrated to Managed Updates v2 +and should migrate their agents to `teleport-update` as soon as possible. ## How it works -When Managed Updates are enabled, a Teleport updater is installed alongside -each new Teleport Agent. The updater communicates with the Teleport Proxy Service to -determine when an update is available and if it should perform the update now. +Managed Updates for Agents and Bots are designed to manage long-running, unattended Teleport +clients, such as Teleport Agents and tbot. This is different from Managed Updates for +Client Tools, which are designed to manage interactive Teleport clients, such as `tsh` and `tctl`. -Each agent belongs to an update group. The update schedule specifies when each +When Managed Updates are enabled, a Teleport updater is installed alongside +each new Teleport Agent or tbot. The updater communicates with the Teleport Proxy +Service to determine when an update is available and if it should perform the update now. + +Each installation belongs to an update group. The update schedule specifies when each group is updated. The schedule is stored in the `autoupdate_config` resource and -can be edited via `tctl`. +can be edited via `tctl`. The `tctl autoupdate agents` subcommands are used to interact +with the rollout for both Teleport Agents and long-running tbot installations. For Linux server-based installations, `teleport-update` command configures -Managed Updates locally on the server. +Managed Updates for Teleport Agents and tbot locally on the server. For Kubernetes-based installations, the `teleport-kube-agent` Helm chart deploys a controller that automatically updates the main Teleport container. -Existing agents must be manually enrolled into Managed Updates. +Agents and bots that were installed before Managed Updates was enabled on the +cluster usually need to be manually enrolled into Managed Updates. ## Prerequisites - Familiarity with the [Upgrading Compatibility Overview](../overview.mdx) guide, which describes the sequence in which to upgrade components in your cluster. -- Teleport Agents that are not yet enrolled in Managed Updates. +- Teleport Agent or tbot installations that are not yet enrolled in Managed Updates. - (!docs/pages/includes/edition-prereqs-tabs.mdx!) - (!docs/pages/includes/tctl.mdx!) -## Quick setup for existing connected Linux servers +## Quick setup for existing Linux Agent and Bot installations Users can enable Managed Updates v2 on Linux servers that are already running a Teleport Agent by running the following command on every server: @@ -110,20 +124,55 @@ If Teleport was installed via the apt or yum package, `teleport-update uninstall` will revert the running version of Teleport back to the version provided by the package. -## Quick setup for new Linux servers +### Migrating Bots -The [Install Script](../../installation/linux.mdx) is the -fastest way to onboard new Linux servers. However, you may also use -`teleport-update` by itself to set up a Teleport Agent manually. +Existing tbot installations require additional steps to be converted to Managed Updates. -Users can create a new installation of Teleport using any version of the -`teleport-update` binary. First, download copy of the Teleport tarball from -the downloads page. Next, invoke `teleport-update` to install the correct version -for your cluster. +When `teleport-update enable` is run, a disabled systemd service is created at `/etc/systemd/system/tbot.service` +if a service does not already exist at that location. + +If a custom tbot systemd service is already installed at `/etc/systemd/system/tbot.service`, +a warning will be displayed when `teleport-update enable` is run. To overwrite that custom service +and replace it with an updater-managed service, run the following command: ```code -$ tar xf teleport-[version].tgz -$ cd teleport-[version] +$ sudo teleport-update enable --overwrite +``` + +If a custom tbot systemd service is installed with a different name (e.g., `/etc/systemd/system/machineid.service`), +it must be stopped if it shares the same configuration and data directories as the updater-managed service: + +```code +$ sudo systemctl disable machineid --now +``` + +After `teleport-update enable` has successfully created the service, its output will recommend that you run +the following command to enable `tbot.service`: + +```code +$ sudo systemctl enable tbot --now +``` + +Note that you must have a valid `/etc/tbot.yaml` file to use tbot. +See [Deploying tbot on Linux](../../machine-workload-identity/machine-id/deployment/linux.mdx) for more information. + +## Quick setup for new Linux Agents and Bot installations + +The [Web UI onboarding and Install Script](../../installation/linux.mdx) are the +fastest ways to onboard new Linux servers. However, you may also use +`teleport-update` by itself to set up a Teleport Agent and/or tbot manually. +Note that web-based agent enrollment does not automatically configure tbot. +See the end of this section for information on how to configure and enable tbot. + +Users can create a new installation of Teleport using any version of the +`teleport-update` binary. First, download copy of the `teleport-update` tarball from +the [Agent Installer & Updater section](https://goteleport.com/download/all-downloads/?kind=agentInstaller&os=linux) of +the downloads page. +Next, invoke `teleport-update` to install the correct version for your cluster. + +```code +$ tar xf teleport-update-[version].tgz +$ cd teleport-update-[version] $ sudo ./teleport-update enable --proxy example.teleport.sh ``` @@ -135,9 +184,19 @@ started via the `systemctl` command: $ sudo systemctl enable teleport --now ``` -## Configuring managed agent updates +Similarly, you can create an `/etc/tbot.yaml` file, either manually or using `tbot configure`. +See [Deploying tbot on Linux](../../machine-workload-identity/machine-id/deployment/linux.mdx) for more information. + +After, tbot can be enabled and started via the `systemctl` command: + +```code +$ sudo systemctl enable tbot --now +``` + +## Configuring managed agent and tbot updates + +Managed agent and bot updates are configured via two Teleport resources: -Managed agent updates are configured via two Teleport resources: - `autoupdate_config` controls the update schedule - `autoupdate_version` controls the desired version @@ -202,20 +261,14 @@ spec: wait_hours: 24 ``` -This schedule would update agents in the `staging` group at 4 UTC, and then update +This schedule would update agents and bots in the `staging` group at 4 UTC, and then update the `production` group at 5 UTC the next day. The `production` group will not execute update until the staging group has updated. The `wait_hours` field sets a minimum duration between groups, ensuring that `production` happens the day after `staging`, and not one hour after. - -While failed installations will revert automatically on the client-side, -server-side healthchecks are still in development. To prevent the `production` -group above from updating after `staging` has failed, you must manually suspend -the schedule by setting the `spec.agents.mode` to `suspended`. - - Two update rollout strategies are available: + - The `halt-on-error` strategy provides predictable, sequential updates across environments. It's ideal for traditional development pipelines where you want to ensure that development environments are successfully updated @@ -226,7 +279,13 @@ Two update rollout strategies are available: for a group, regardless of the status of other groups. This strategy does not provide ordering guarantees across groups. -You can find more information in [the Managed Updates v2 resource reference](../../reference/deployment/managed-updates-v2.mdx) +With the `halt-on-error` strategy, the `canary_count` field can be set on each group to specify +a number of randomly selected agents (fewer than five) to update and verify before +proceeding to the rest of the agents in the group. This can be used to reduce the impact +of a failed update that might not be caught by earlier groups due to environment differences. + +You can find more information +in [the Managed Updates v2 resource reference](../../reference/deployment/managed-updates-v2.mdx) Except for `autoupdate_config.agents.mode`, changes to `autoupdate_config` fields take effect during the next version rollout. A new rollout happens when @@ -243,9 +302,9 @@ This ensures your agents are always up-to-date and running the best version for your Teleport cluster. -Self-hosted Teleport users must specify which version their agents should update -to via the `autoupdate_version` resource. -If the resource does not exist, agents will not update. +Self-hosted Teleport users must specify which version their agents and bots should +update to via the `autoupdate_version` resource. +If the resource does not exist, agents and bots will not update. Create a file called `autoupdate_version.yaml` containing: @@ -262,12 +321,12 @@ spec: mode: enabled ``` -This resource is used to deploy new versions of Teleport to your agents. -The cluster will update agents to `target_version` according to the update +This resource is used to deploy new versions of Teleport to your agents and bots. +The cluster will update agents and bots to `target_version` according to the update schedule specified in the `autoupdate_config`. The `start_version` is only used to determine the version used for newly -connected agents when their update window has not occurred yet. +connected agents and bots when their update window has not occurred yet. This is useful to prevent version drift within groups, but some users may prefer to set both version fields to the same version. @@ -296,6 +355,21 @@ stage Unstarted previous_groups_not_done prod Unstarted previous_groups_not_done ``` +### Monitoring tbot updates + +Unlike the Teleport Agent, tbot does not have a persistent connection to the cluster and cannot be monitored directly +during upgrades. + +However, tbot installation failures are still tracked if tbot is installed alongside a running Teleport Agent. + +If a tbot upgrade fails and tbot is installed alongside an agent, both tbot and the agent will be rolled back to the +previous version, and the update group may be marked as failed. +If tbot is installed without an agent, tbot will still be rolled back to the previous version, but the upgrade may still +progress to further groups. + +Similarly, tbot installations are only considered candidates for canary installations if they are deployed alongside a +running Teleport Agent. + ## Migrating agents on Linux servers to Managed Updates ### Finding unmanaged agents @@ -452,7 +526,7 @@ This section assumes that the name of your `teleport-kube-agent` release is ## GitOps tools Managed updates for Kubernetes agents requires workarounds when used with GitOps tools for -continuous deployment. The `teleport-kube-agent` Helm chart owns the version of the +continuous deployment. The `teleport-kube-agent` Helm chart owns the version of the `teleport-agent` resource, so when the `teleport-agent-updater` modifies the image version of the `teleport-agent` resource, the GitOps tool will detect a drift or a diff in the `teleport-agent` resource. @@ -462,7 +536,8 @@ The sections below describe workarounds for various GitOps tools. ### ArgoCD deployments After a managed update, ArgoCD reports the `teleport-agent` resource as `OutOfSync`. -As a workaround to this problem use a [Diff Customization](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#diffing-customization) +As a workaround to this problem use +a [Diff Customization](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#diffing-customization) to ignore the difference in image version. Here is an example deployment using the name `teleport-agent` and namespace `teleport`. @@ -474,12 +549,12 @@ metadata: namespace: teleport spec: ignoreDifferences: - - group: apps - kind: StatefulSet - name: teleport-agent - namespace: teleport - jqPathExpressions: - - .spec.template.spec.containers[] | select(.name == "teleport").image + - group: apps + kind: StatefulSet + name: teleport-agent + namespace: teleport + jqPathExpressions: + - .spec.template.spec.containers[] | select(.name == "teleport").image ... ``` @@ -500,17 +575,18 @@ spec: driftDetection: mode: enabled ignore: - - paths: ["/spec/template/spec/containers/0/image"] - target: - kind: StatefulSet - name: teleport-agent - namespace: teleport + - paths: [ "/spec/template/spec/containers/0/image" ] + target: + kind: StatefulSet + name: teleport-agent + namespace: teleport ... ``` ## Troubleshooting -You can inspect the current agent autoupdate status by running: +You can inspect the current autoupdate status by running: + ```code $ tctl autoupdate agents status @@ -526,7 +602,8 @@ Group Name State Start Time State Reason default Unstarted outside_window ``` -This rollout state is computed by each Auth Service instance every minute. An `autoupdate_config` or `autoupdate_version` +This rollout state is computed by each Auth Service instance every minute. An `autoupdate_config` or +`autoupdate_version` change might take up to a minute to be reflected and applied. Teleport Agents are not updated immediately when a new version of Teleport is @@ -620,9 +697,11 @@ Here are a couple of potential solutions to this issue: #### Use an HTTP CONNECT proxy -If you configure the `HTTPS_PROXY` variable in the `teleport-update` process's environment, it will use this proxy to pull updates. +If you configure the `HTTPS_PROXY` variable in the `teleport-update` process's environment, it will use this proxy to +pull updates. -The easiest way to configure a proxy with a default install is to add this variable to `/etc/systemd/system/teleport-update.service.d/override.conf`: +The easiest way to configure a proxy with a default install is to add this variable to +`/etc/systemd/system/teleport-update.service.d/override.conf`: ```bash $ sudo mkdir -p /etc/systemd/system/teleport-update.service.d @@ -636,7 +715,8 @@ You can view the `teleport-update` process logs with `sudo journalctl -u telepor #### Mirror the Teleport tarball packages and change the base-url -If you can mirror the Teleport tarball installers somewhere that your agents are able to access, you can change the `base-url` +If you can mirror the Teleport tarball installers somewhere that your agents are able to access, you can change the +`base-url` used by Teleport updaters so they can pull them directly. To change the `base-url`, you should add the `-b` or `--base-url` flag to the `teleport-update enable` command: @@ -648,4 +728,5 @@ $ sudo teleport-update enable --base-url https://teleport.artifactory.company.lo It is safe to re-run `sudo teleport-update enable` to modify the base URL. Existing updater settings will be preserved if not explicitly overridden by flags. -More information about flags that can be used with `teleport-update enable` can be found [here](../../reference/cli/teleport-update.mdx#teleport-update-enable) +More information about flags that can be used with `teleport-update enable` can be +found [here](../../reference/cli/teleport-update.mdx#teleport-update-enable)