Add documentation for proxy peering (#15790)

docs/config.json

@@ -418,6 +418,11 @@
               "title": "TLS Routing Migration",
               "slug": "/management/operations/tls-routing/",
               "forScopes": ["oss", "enterprise"]
+            },
+            {
+              "title": "Proxy Peering Migration",
+              "slug": "/management/operations/proxy-peering/",
+              "forScopes": ["enterprise"]
             }
           ]
         },
@@ -1040,6 +1045,10 @@
         {
           "title": "TLS Routing",
           "slug": "/architecture/tls-routing/"
+        },
+        {
+          "title": "Proxy Peering",
+          "slug": "/architecture/proxy-peering/"
         }
       ]
     },
@@ -1784,6 +1793,11 @@
       "destination": "/management/operations/ca-rotation/",
       "permanent": true
     },
+    {
+      "source": "/setup/operations/proxy-peering/",
+      "destination": "/management/operations/proxy-peering/",
+      "permanent": true
+    },
     {
       "source": "/setup/operations/scaling/",
       "destination": "/management/operations/scaling/",

docs/pages/architecture/proxy-peering.mdx

@@ -0,0 +1,61 @@
+---
+title: Proxy Peering
+description: How Teleport implements more efficient networking with Proxy Peering.
+---
+
+<Details
+  title="Version warning"
+  opened={true}
+  scope={["oss", "enterprise"]}
+  scopeOnly={true}
+  min="10.0"
+>
+  Proxy Peering is available starting from Teleport `10.0`.
+</Details>
+
+Proxy Peering enables Teleport agents to be reachable without connecting to
+every Teleport Proxy Service. This allows Teleport Proxy instances to scale
+horizontally without increasing the number of connections created by agents.
+
+<Admonition title="Note" type="tip">
+  A Teleport agent is a Teleport instance that provides access to resources in your infrastructure, i.e., by running the SSH,
+  Kubernetes, Database, Application, or Desktop Services.
+</Admonition>
+
+By default, Teleport agents need to create a reverse tunnel to every Teleport Proxy
+to ensure a client is able to reach the agent. With Proxy Peering this is no
+longer a requirement. When Proxy Peering is enabled agents will automatically
+change their behavior to connect to the configured number of Teleport Proxy
+instances.
+
+## How it works
+
+### Proxy Service
+
+A gRPC service on each Teleport Proxy Service instance provides an API for establishing a
+bi-directional connection to the agents connected to that Teleport Proxy. Teleport
+Proxy instances manage a gRPC client to all other Teleport Proxy instances in the
+cluster.
+
+Routing information on which Teleport Proxy instances each agent is connected to
+is stored in Teleport's backend and propagated to each Teleport Proxy instance.
+
+The routing information and gRPC service allow a Teleport Proxy to identify which
+Proxy Service instance an agent is connected to and create an end-to-end connection
+from a client to that agent. This allows for access to the agent without connecting
+to the same Teleport Proxy instance initially.
+
+### Reverse tunnel agents
+
+Agents will check whether you have enabled Proxy Peering before
+attempting to create a reverse tunnel to a Teleport Proxy instance.
+
+By default, in Proxy Peering mode, agents are configured to connect to a single
+Teleport Proxy instance. For high availability a cluster administrator may
+configure agents to connect to 2 or more Teleport Proxy instances.
+
+![Teleport Proxy Peering](../../img/architecture/proxy-peering@1.2x.svg)
+
+## Next Steps
+- See the [migration guide](../management/operations/proxy-peering.mdx) to learn how to upgrade an existing cluster to use
+Proxy Peering

docs/pages/management/operations.mdx

@@ -15,3 +15,4 @@ the [Cluster Administration Guides](./admin.mdx) section.
 - [Backup and Restore](./operations/backup-restore.mdx): Backing up and restoring the cluster.
 - [CA Rotation](./operations/ca-rotation.mdx): Rotating Teleport certificate authorities.
 - [TLS Routing Migration](./operations/tls-routing.mdx): Migrating your Teleport cluster to single-port TLS routing mode.
+- [Proxy Peering Migration](./operations/proxy-peering.mdx): Migrating your Teleport cluster to Proxy Peering mode.

docs/pages/management/operations/proxy-peering.mdx

@@ -0,0 +1,48 @@
+---
+title: Proxy Peering Migration
+description: How to upgrade an existing Teleport cluster to Proxy Peering mode.
+---
+
+This guide shows you how to migrate your Teleport
+cluster to use Proxy Peering, which enables you to scale
+your Proxy Service instances horizontally by reducing the
+number of connections created between Teleport Proxy instances
+and Teleport services like the Database Service and Application Service.
+
+## Prerequisites
+
+An existing Teleport Enterprise cluster. See our [getting started guide](../../deploy-a-cluster/teleport-enterprise/getting-started.mdx) for help
+setting up a Teleport cluster.
+
+All components in the cluster should be running Teleport `10.0` or later. See our
+[upgrade procedure](./upgrading.mdx) for help upgrading a Teleport cluster.
+
+Teleport Proxy Service instances must be able to reach each other over the network on port
+`3021` by default. Ensure there are no firewall policies that would block communication
+between instances.
+
+## Step 1/3. Enable Proxy Peering
+Update your cluster's Auth Service configuration to set the tunnel strategy type
+to `proxy_peering`.
+
+```yaml
+auth_service:
+    tunnel_strategy:
+        type: proxy_peering
+        agent_connection_count: 1
+```
+
+This setting will indicate to agents that they are only required to connect to 1
+Teleport Proxy instance as specified by the `agent_connection_count` field.
+
+For high availability, an `agent_connection_count` greater than 1 can be configured.
+This ensures an agent is still reachable if one of the Proxy Service instances it is connected to is not available.
+
+## Step 2/3. Restart the Auth Services
+Restart all Teleport Auth Services running in the cluster to apply the new Auth Service
+configuration.
+
+## Step 3/3. Restart the Proxy Services
+Restart all Teleport Proxy Service instances running in the cluster in order to
+start the services required for Proxy Peering.
+

docs/pages/reference/config.mdx

@@ -505,6 +505,24 @@ auth_service:
     # See "TLS Routing" in Architecture section for additional information.
     proxy_listener_mode: multiplex
 
+    # Determines the strategy that the cluster uses for connecting clients to
+    # agents through the Teleport Proxy Service.
+    tunnel_strategy:
+      # Possible tunnel strategy types are:
+      #
+      # "agent_mesh":    The default behavior, where agents will connect to every
+      #                  Teleport Proxy Service instance.
+      # "proxy_peering": Agents will connect to a subset of Proxy Service instances
+      #                  and clients will be routed between Proxy Service instances
+      #                  for end-to-end connectivity.
+      type: proxy_peering
+
+      # The number of reverse tunnel connections agents will attempt to create.
+      # This field is only available when using the "proxy_peering" tunnel
+      # strategy type. For high availability we recommend setting this value to
+      # 2 or more.
+      agent_connection_count: 1
+
 # This section configures the 'node service':
 ssh_service:
     # Turns 'ssh' role on. Default is 'yes'
@@ -631,6 +649,10 @@ proxy_service:
     # "v1": defaults to 0.0.0.0:3024
     tunnel_listen_addr: 0.0.0.0:3024
 
+    # Proxy Peering listening address. Teleport Proxy Services will advertise this address
+    # for dialing agents in Proxy Peering mode.
+    peer_listen_addr: 0.0.0.0:3021
+
     # The HTTPS listen address to serve the Web UI and authenticate users.
     # Handles the PostgreSQL proxy if Database Access is enabled.
     web_listen_addr: 0.0.0.0:3080

docs/pages/reference/networking.mdx

@@ -116,7 +116,7 @@ numbers for each service.
   Service, use the following command:
 
   ```code
-  $ curl https://teleport.example.com:443/webapi/ping | jq 
+  $ curl https://teleport.example.com:443/webapi/ping | jq
   ```
 
   Note that if `auth_service.proxy_listener_mode` is set to `multiplex` in your
@@ -135,6 +135,7 @@ Read more in our [TLS Routing](../architecture/tls-routing.mdx) guide.
 | Port | Service | Description |
 | - | - | - |
 | 443 | Proxy | In TLS Routing mode, the Proxy handles all protocols, including Web UI, HTTPS, Kubernetes, SSH, and all databases on a single port. |
+| 3021 | Proxy | Port used by Teleport Proxy Service instances to dial agents in Proxy Peering mode. |
 | 3022 | Node | SSH port. This is Teleport's equivalent of port `#22` for SSH. Only used when Teleport Node is replacing SSH.|
 | 3025 | Auth | TLS port used by the Auth Service to serve its API to other Nodes in a cluster.|
 | 3028 | Desktop | When using Desktop Service `windows_desktop_service.listen_addr` |
@@ -146,6 +147,7 @@ In those cases, they can set up separate listeners in the config file.
 
 | Port | Service | Description |
 | - | - | - |
+| 3021 | Proxy | Port used by Teleport Proxy Service instances to dial agents in Proxy Peering mode. |
 | 3022 | Node | SSH port. This is Teleport's equivalent of port `#22` for SSH. |
 | 3023 | Proxy | SSH port clients connect to. The Proxy Service will forward this connection to port `#3022` on the destination Node. |
 | 3024 | Proxy | SSH port used to create "reverse SSH tunnels" from behind-firewall environments into a trusted proxy server. |