Sets `http.Server.IdleTimeout` and `http.Client.IdleConnTimeout`
on clients and servers which didn't have them set. A default of
360s was chosen to be on par with the default of an NLB without
being identical.
This was added as another safety measure to prevent leaking any
idle connections indefinitely as seen in #22757.
`apidefaults.DefaultDialTimeout` was also renamed to
`apidefaults.DefaultIOTimeout` to better reflect its usage.
This PR introduces the static label's key validation to Kubernetes Service.
Services with invalid labels fail to hearbeat their clusters but the
service is kept working. This change forces the service to fail fast
during startup if any key is invalid.
Fixes#22696
`auth.Middleware` to make use of them. This allows for less redundancy
per RPC and provides the ability to perform session control when grpc
connections are established.
The `auth.Middleware.withAuthenticatedUser` function was called from all
interceptors of any gRPC server that leveraged `auth.Middleware`. This
was constructing the identity of the client from the TLS certificate
per RPC. Since the identity of a connection cannot change between RPCs
there is no need to construct the identity in an interceptor. Instead,
the new `auth.TransportCredentials` can build the identity once after the
TLS handshake completes. When interceptors or gRPC handlers need to
acquire the identity they can do so using the same
mechanism via `peer.Peer.AuthInfo` and assert on the new `auth.IdentityInfo`
type:
```go
peerInfo, ok := peer.FromContext(ctx)
if !ok {
return
}
connInfo, ok := peerInfo.AuthInfo.(auth.IdentityInfo)
if !ok {
return
}
identity := connInfo.IdentityGetter.GetIdentity()
connState := connInfo.TLSInfo.State
```
The gRPC server that is served on the Proxy ssh port needs to hold the
same invariants as the ssh server available on the same port. This
means that connection limits need to be enforced when new connections
are established. To achieve this the new `auth.TransportCredentials` can
be provided with an `auth.ConnectionEnforcer` and an `auth.Authorizer`
to first build an identity, then ensures that the user is authorized and
finally that the user has not exceeded their connection limits.
Other minor changes include providing the cluster name to the
`auth.Middleware` instead of it looking the value up from the cache every
time an identity is created. Some interceptors were also moved around or
removed to eliminate the chances of having duplicate interceptors end up in
the interceptor chain.
Part of #19812
* Change formatting for "/version" endpoint
This PR changes the formatting for "/version" endpoint since Kubernetes
clients do not expect a JSON response on that endpoint.
This PR returns the error message directly into the body without any
formatting so that `kubectl` is able to present the desired error.
```
kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.4
Kustomize Version: v4.5.7
Error from server (InternalError): an error on the server ("this user cannot request kubernetes access, has no assigned groups or users") has prevented the request from succeeding
```
Fixes#20900
* change message wording
* add validation when generating kube creds
* fix typo
During binary upgrades, Teleport forks the agents and calls the shutdown
method for each one of them.
Kubernetes Agent was calling the `http.Server` shutdown method and which
wasn't stopping the heartbeats neither handling the listeners.
This PR fixes the shutdown behaviour for Kube agent and avoids deleting
the heartbeated resources when the agent is reloading.
Increases deadline until the agent receives the first byte to 10s.
It's required to accommodate setups with high latency and where the time
between the TCP being accepted and the time for the first byte is longer
than the default value - 1s.
Fixes#20442
* Add secure IP propagation from proxy to auth server when using ALPN
We're using PROXY protocol extensions called TLVs to send
signed JWT and proxy's certificate to the auth server. Auth
validates JWT using provided signing certificate and host CA
to make sure that IP information comes from our internal proxy.
With the arrival of Teleport 12, we stop supporting v10.x.x clients, and we no longer require the `types.KindKubernetesService` legacy heartbeat.
This change removes the legacy heartbeat from Kubernetes Service but keeps the legacy Auth Server CRUD methods and heartbeat support to maintain compatibility with Teleport 11 Kubernetes Service - Teleport 11 still heartbeats the legacy type.
We postponed the related `DELETES` to Teleport 13.
This PR presents a watcher for automatic `kube_cluster` discovery for Azure AKS clusters. Given a user with access to the Azure cloud, the auto-discovery service will scan the cloud and register all clusters available in AKS .
Once the discovery service creates a `kube_cluster` in Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster depend on the different AKS clusters configurations:
# Authentication
## Local Accounts
If the AKS cluster auth is based on local accounts created during the provisioning phase of the cluster, the agent will use the [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint.
This endpoint returns a `kubeconfig` fully populated with user credentials that Teleport can use to access the cluster.
## AZ Active Directory
When AZ active directory integration is enabled, Azure allows login with AD users. Azure forces the login to happen with dynamic short-lived user tokens. These tokens are generated by calling `credentials.GetToken` with a fixed Scope: `6dae42f8-4368-4678-94ff-3960e28e3630` and with the cluster's `tenant_id`. The token contains the user details as well as `group_ids` to match with authorization rules.
```go
// getAzureToken generates an authentication token for clusters with AD enabled.
func (a *aKSClient) getAzureToken(ctx context.Context, tentantID string, clientCfg *rest.Config) (time.Time, error) {
const (
azureManagedClusterScope = "6dae42f8-4368-4678-94ff-3960e28e3630"
)
cred, err := a.azIdentity(&azidentity.DefaultAzureCredentialOptions{
TenantID: tentantID,
})
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
cliAccessToken, err := cred.GetToken(ctx, policy.TokenRequestOptions{
// azureManagedClusterScope is a fixed scope that identifies azure AKS managed clusters.
Scopes: []string{azureManagedClusterScope},
},
)
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
// reset the old exec provider credentials
clientCfg.ExecProvider = nil
clientCfg.BearerToken = cliAccessToken.Token
return cliAccessToken.ExpiresOn, nil
}
```
# Authorization
## Local Accounts
The [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint returns credentials with enough permissions for Teleport to enroll the cluster.
## AZ AD
### Azure RBAC
When Azure RBAC mode is enabled, the cluster authorization is based on rules specified in the Azure Identity permissions.
The AZ group associated with the AZ identity the Teleport Process is running has to define the following permissions:
```json
{
"Name": "AKS Teleport Discovery Permissions",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [],
"NotActions": [],
"DataActions": [
"Microsoft.ContainerService/managedClusters/pods/read",
"Microsoft.ContainerService/managedClusters/users/impersonate/action",
"Microsoft.ContainerService/managedClusters/groups/impersonate/action",
"Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
```
If correctly specified, the Azure authentication service automatically grants access to any cluster within `subscription_id`
without any other definition. On the other hand, if it's incorrectly configured, an error is triggered but Teleport cannot gain access to the cluster.
### Kubernetes RBAC
If AZ RBAC integration is disabled, the authorization to the cluster is processed by Kubernetes RBAC. This is done by matching the Az Identity principals (`group_ids`) with `Role`, `ClusterRole` objects that live in the AKS cluster. This mode requires that the `ClusterRole` and `ClusterRoleBinding` must exist and must be well configured for each cluster to enroll.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport-role
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- "authorization.k8s.io"
resources:
- selfsubjectaccessreviews
- selfsubjectrulesreviews
verbs:
- create
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teleport-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-role
subjects:
- kind: Group
name: {group_name}
apiGroup: rbac.authorization.k8s.io
```
#### `ClusterRole` and `ClusterRoleBinding` configured
If cluster operators or previous Teleport run has configured access to the cluster, no further action is required since Teleport already has access to the cluster.
#### Cluster `aks:ListClusterAdminCredentials` returns valid credentials
If the Teleport process has access to [`aks:ListClusterAdminCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-admin-credentials?tabs=HTTP) and the endpoint returns valid cluster admin credentials, Teleport will automatically create the `ClusterRole` and `ClusterRoleBinding` objects in the cluster configured to the `group_id` that is listed in the access token. In order to extract the `group_id` from the token, Teleport parses the JWT claims and extracts the first element.
If the object creation was successful, Teleport can access it, otherwise, it will use the `aks:BeginRunCommand` method to try to configure access to itself.
#### Cluster `aks:BeginRunCommand` returns valid credentials
When we reach this mode, Teleport tries to run a `kubectl` command against the cluster to configure the `ClusterRole` and `ClusterRoleBinding`. `aks:BeginRunCommand` allows any user with access to that endpoint to run arbitrary commands in the cluster (commands cannot be validated). Teleport will use it as the last resource to configure the access to itself.
If the command failed, Teleport cannot grant access to the cluster and an error is returned.
# UX
Currently, to discover AKS resources created and to have them dynamically served by the `kubernetes_service`one can define the following configuration.
```yaml
discovery_service:
enabled: true
azure:
- subscriptions: ["*"]
types: ["aks"]
regions: ["*"]
tags:
'*': '*'
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- Support AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276, #16281
This PR presents a watcher for dynamic `kube_cluster` resources. The cluster credentials can be set using a `kubeconfig` whose payload is defined in `KubernertesClusterV3.Spec.Kubeconfig`. These credentials are used to connect to the Kubernetes API server and if invalid, the cluster is not served and returns an error.
Currently, `kube_cluster` resources created via the API with a kubeconfig payload can be dynamically served by the `kubernetes_service` by enabling dynamic resources.
```yaml
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- The discovery service will have to create these resources.
- Support Azure and AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276
Tracing handlers consume a large amount of memory over time, which
makes using one per route suboptimal (See https://github.com/open-telemetry/opentelemetry-go-contrib/issues/2413).
Wrapping all `http.Server` handlers with a tracing handler provides
the same tracing abilities for a fraction of the memory usage. A new
span formatter is added which allows servers to still contain the
request span in the span name.
## What
First part of the Kubernetes [Discovery RFD](https://github.com/gravitational/teleport/pull/13376/) to introduce a Kubernetes server per cluster.
This PR introduces a separate Kubernetes server that uses the already introduced `KubernetesClusterV3`.
## Compatibility
In previous versions, Kubernetes Clusters were part of regular `ServerV2` resource and this refactoring deprecates the `ServerV2` usage but keeps them for compatibility with previous version.
Everything is backward compatible, so v10 kubernetes agents and trusted clusters can connect fine.
## Next steps
Once this is merged, a new PR will introduce dynamic registration for Kubernetes Clusters discovered through EKS Discovery.
This adds proxy peering support. A configurable setting that allows for agents
to connect to a subset of proxies and be reachable through any proxy in the
cluster. This is achieved by creating grpc connections between each proxy
server. Client connections can then be passed between proxies to the desired
agent.
* Revert "Make `PortList.Pop()` thread-safe (#11799)"
This reverts commit a17337d1a1.
* Revert "Ensure stateOK is reported only when all components have sent updates (#11249)"
This reverts commit b749302e2c.
* Revert "Throw startup error if `TeleportReadyEvent` is not emitted (#11725)"
This reverts commit 933e247287.
* Revert "Fix ProxyKube not reporting its readiness (#12150)"
This reverts commit 6cdcfe7721.
Fixesgravitational/teleport-private#78
LAT-APP21-3
Change the multiplexer from opt-out to opt-in for protocol listeners.
The multiplexer previously always created new listeners for each protocol (SSH, TLS, DB) and its Config contained opt-out configurations (DisableTLS, DisableSSH, DisablePostgres). When callers didn't explicitly disable a protocol, new connections for that protocol would never close and leak a goroutine. This exposed a 3-line DoS whereby multiple connections could be passed to the multiplexer for a protocol that was not being serviced, eventually resulting in file descriptor limits being hit, which then prevented the teleport process from operating (see example in issue).
Changing to opt-in means new protocols can be added to the multiplexer without requiring all existing callers to be changed (to opt-out of the new protocol). Forgetting to opt-out would expose new code to DoS without compile-time, test, or operational notification.
Rather than rely on opt-out flags in the config, this change creates listeners only when explicitly requested by callers. The existing getter methods on multiplexer were changed to create listeners when called. And multiplexer protocol detection now closes connections when a listener hasn't been created. This also allowed for the protocol detection routine to be simplified.
Some integration tests modify global "constants" to speed up test
execution (e.g. shortening polling intervals). This is occasionally
tripping the Go data race detector, so I have added explicit
serialisation to reading and writing these global settings.
These values are only ever changed in a test environment, and there
should be zero contention for them in a non-test environment.
* kube: handle large number of trusted clusters in mTLS handshake
Same as https://github.com/gravitational/teleport/issues/3870 but for
k8s endpoints. There is a hard limit on how many CAs we can put into a
client CertPool, usually several hundred (depending on Subject length).
The solution here is to fall back to only using the current cluster's CA
for validation if the limit is reached. This is almost always the case
in root clusters. There, the client certificate will be signed by the
root cluster and validation will pass.
In the unlikely case that you have a leaf cluster which in turn has
hundreds of trusted leaf clusters itself, the validation will fail. The
client cert will still be signed by the root cluster (not the leaf).
However, that's better than a panic. And I'm not aware of any real
setups like that.
Also in this PR, add the wildcard `*.teleport.cluster.local` SAN to
proxy and k8s TLS certificates, which was missing before. This SAN is
used for clients to encode the cluster name and pass it in SNI. The
client (kubectl) is not updated to set this SNI yet, it would break
existing clusters without the SAN change.
* add SNI tests for k8s
Test that mTLS works with large numbers of CAs.
Proxy protocol is sometimes used by load balancers to communicate the
real client IP address. Re-use the detection/parsing code from
lib/multiplexer on all k8s listeners (proxy and kubernetes_service)
* kube: emit audit events using process context
Using the request context can prevent audit events from getting emitted,
if client disconnected and request context got closed.
We shouldn't be losing audit events like that.
Also, log all response errors from exec handler.
* kube: cleanup forwarder code
Rename a few config fields to be more descriptive.
Avoid embedding unless necessary, to keep the package API clean.
* kube: cache only user certificates, not the entire session
The expensive part that we need to cache is the client certificate.
Making a new one requires a round-trip to the auth server, plus entropy
for crypto operations.
The rest of clusterSession contains request-specific state, and only
adds problems if cached.
For example: clusterSession stores a reference to a remote teleport
cluster (if needed); caching requires extra logic to invalidate the
session when that cluster disappears (or tunnels drop out). Same problem
happens with kubernetes_service tunnels.
Instead, the forwarder now picks a new target for each request from the
same user, providing a kind of "load-balancing".
* Init session uploader in kubernetes service
It's started in all other services that upload sessions (app/proxy/ssh),
but was missing here. Because of this, the session storage directory for
async uploads wasn't created on disk and caused interactive sessions to
fail.
Without this, deleted kube_services linger in the backend and show up as
obsolete kubernetes clusters in tsh.
Ideally, this TTL logic should be enforced centrally, but I'd like to
fix the bug first, and do a larger refactoring later.
Added fields:
- kube users/groups
- pod name/namespace
- container name/image
- node name
Container image and node name need to be fetched from the k8s API, they
are not known from just the client request. This fetch is optional, and
if it fails (like due to permission errors), those fields will be
missing.
Since kubernetes_service can talk to k8s API and proxy_service can't,
all session events are now emitted by kubernetes_service and skipped by
the proxy (used to be the other way around).
A proxy running in pre-5.0 mode (e.g. with local kubeconfig) should
register an entry in `tsh kube clusters`.
After upgrading to 5.0, without migration to kubernetes_service, all the
new `tsh kube` commands will work as expected.
This change has several parts: cluster registration, cache updates,
routing and a new tctl flag.
> cluster registration
Cluster registration means adding `KubernetesClusters` to `ServerSpec`
for servers with `KindKubeService`.
`kubernetes_service` instances will parse their kubeconfig or local
`kube_cluster_name` and add them to their `ServerSpec` sent to the auth
server. They are effectively declaring that "I can serve k8s requests
for k8s cluster X".
> cache updates
This is just cache plumbing for `kubernetes_service` presence, so that
other teleport processes can fetch all of kube services. It was missed
in the previous PR implementing CRUD for `kubernetes_service`.
> routing
Now the fun part - routing logic. This logic lives in
`/lib/kube/proxy/forwarder.go` and is shared by both `proxy_service`
(with kubernetes integration enabled) and `kubernetes_service`.
The target k8s cluster name is passed in the client cert, along with k8s
users/groups information.
`kubernetes_service` only serves requests for its direct k8s cluster
(from `Forwarder.creds`) and doesn't route requests to other teleport
instances.
`proxy_service` can serve requests:
- directly to a k8s cluster (the way it works pre-5.0)
- to a leaf teleport cluster (also same as pre-5.0, based on
`RouteToCluster` field in the client cert)
- to a `kubernetes_service` (directly or over a tunnel)
The last two modes require the proxy to generate an ephemeral client TLS
cert to do an outbound mTLS connection.
> tctl flag
A flag `--kube-cluster-name` for `tctl auth sign --format=kubernetes`
which allows generating client certs for non-default k8s cluster name
(as long as it's registered in a cluster).
I used this for testing, but it could be used for automation too.
* Implement kubernetes_service registration and sratup
The new service now starts, registers (locally or via a join token) and
heartbeats its presence to the auth server.
This service can handle k8s requests (like a proxy) but not to remote
teleport clusters. Proxies will be responsible for routing those.
The client (tsh) will not yet go to this service, until proxy routing is
implemented. I manually tweaked server addres in kubeconfig to test it.
You can also run `tctl get kube_service` to list all registered
instances. The self-reported info is currently limited - only listening
address is set.
* Address review feedback
This commit expands the usage of the caching layer
for auth server API:
* Introduces in-memory cache that is used to serve all
Auth server API requests. This is done to achieve scalability
on 10K+ node clusters, where each node fetches certificate authorities,
roles, users and join tokens. It is not possible to scale
DynamoDB backend or other backends on 10K reads per seconds
on a single shard or partition. The solution is to introduce
an in-memory cache of the backend state that is always used
for reads.
* In-memory cache has been expanded to support all resources
required by the auth server.
* Experimental `tctl top` command has been introduced to display
common single node metrics.
Replace SQLite Memory Backend with BTree
SQLite in memory backend was suffering from
high tail latencies under load (up to 8 seconds
in 99.9%-ile on load configurations).
This commit replaces the SQLite memory caching
backend with in-memory BTree backend that
brought down tail latencies to 2 seconds (99.9%-ile)
and brought overall performance improvement.
This commit switches Teleport proxy to use impersonation
API instead of the CSR API.
This allows Teleport to work on EKS clusters, GKE and all
other CNCF compabitble clusters.
This commit updates helm chart RBAC as well.
It introduces extra configuration flag to proxy_service
configuration parameter:
```yaml
proxy_service:
# kubeconfig_file is used for scenarios
# when Teleport Proxy is deployed outside
# of the kubernetes cluster
kubeconfig_file: /path/to/kube/config
```
It deprecates similar flag in auth_service:
```yaml
auth_service:
# DEPRECATED. THIS FLAG IS IGNORED
kubeconfig_file: /path/to/kube/config
```
rosstimothy