The Okta service and client have been introduced. This will allow for the
maintenance of Okta objects via our gRPC server and auth client. Some
additional improvements have been made so that the local okta service now
better reflects the gRPC definition.
Additionally, the gRPC authorization mechanism has been lifted out of
`lib/auth/grpcserver.go` and moved into a common authorizer. Finally, the
ability to check access to verbs has been lifted out of the `pluginsv1`
service into that same authorizer. This will hopefully provide a good base
for separate gRPC services in the future.
Adds a new `GetCertAuthorities` unary RPC to
`teleport.v1.trust.TrustService` and deprecates the equivalent
HTTP endpoint. All associated interfaces have been updated to remove
the marshal options as well.
This PR removes the `types.KindKubeService` objects that were
deprecated in Teleport 11.
Teleport 12 removed the double heartbeat and for Teleport 13 we can
safely drop the support of the old hearbeat based on `types.Server`
objects.
Clients older than Teleport 12 connecting to a Teleport 13 server will
fail since they cannot heartbeat using the old mechanism.
An assignment before checking the error was replacing the session event
emitter with a `nil` value which eventually panic when the caller
function tries to close the session.
Sets `http.Server.IdleTimeout` and `http.Client.IdleConnTimeout`
on clients and servers which didn't have them set. A default of
360s was chosen to be on par with the default of an NLB without
being identical.
This was added as another safety measure to prevent leaking any
idle connections indefinitely as seen in #22757.
`apidefaults.DefaultDialTimeout` was also renamed to
`apidefaults.DefaultIOTimeout` to better reflect its usage.
The `http.Client.Transport` created in `newWebClient` wraps a
chain of `http.RoundTripper` over an underlying `http.Transport`.
Since we cannot guarantee that each `http.RoundTipper` has a
`CloseIdleConnections` method the usage of
`defer clt.CloseIdleConnections()` does not guarantee that the http
connections created during Find/Ping/etc are closed.
To prevent leaking connections implementations of `http.RoundTripper`
have added a `CloseIdleConnections` method added that forwards the
request on to the wrapped interface. The `otelhttp.Transport` does
not implement this method either so care has been taken to wrap it
in a `http.RoundTripper` which will call the root transport in
`enforceCloseIdleConnections`. An upstream issue has been filed
with otelhttp: https://github.com/open-telemetry/opentelemetry-go-contrib/issues/3543
to get the method added to their `Transport` implementation.
This wasn't noticed prior to upgrading to go1.20 becuase prior to
[this](https://github.com/golang/go/commit/4e7e7ae1406c70d9cc0809ec11105a55a60a0b70)
commit the `ReadHeaderTimeout` set on the Proxy web api http.Server
would cause the connection to appear idle and get terminated by the
server.
`TestWebClientClosesIdleConnections` was added to capture the leak
in connections as reported in #22757 and prevent any regressions.
The HTTP client created by `auth.NewClient` has an idle timeout of
30s and each instance periodically gets cert authorities in
`TeleportProcess.syncRotationStateCycle`. If there has been no
activity when the ticker expires then the request will result in a
brand new connection to Auth to be established. This can cause a
spikes in open file descriptors on the Proxy when there are a large
number of instances connected via a tunnel.
To prevent periodic dialing when retrieving cert authorities the
endpoint has been migrated to the new Trust gRPC service. For now
only the get endpoint has been migrated but the rest of the CRUD
operations for cert authorities should be migrated from the HTTP
api to the gRPC service as well.
* Move configuration from lib/service to lib/service/servicecfg
The new servicecfg package will hold only configuration for services.
This will allow other packages (like tctl and tsh) to depend on
servicecfg without pulling in all of lib/service (which has a number
of platform-specific details).
This is the first step towards being able to build tctl for Windows.
* Move PAM and BPF config into servicecfg
This breaks a compile-time dependency on BPF/PAM for tctl.
* Move the authorizer into its own package.
The Authorizer and associated functions and constants have been moved into
their own package `authz`. This will allow us to more easily create separate
services for gRPC functionality so that we're not pushing as much into the
`lib/auth` package.
Though this PR is large, the only actual code change here is making
`ContextForLocalUser` exported and changing the `permissions_test.go` so that
it doesn't rely on the `lib/auth` package.
* Use context functions.
* Test and linting fixes.
* Add in a temporary alias file to ensure e still compiles.
* Use functions instead of var aliases for performance reasons.
* Adding in godocs, cleaning up tests.
* Fix tests.
* Add a few more aliases.
This PR introduces the static label's key validation to Kubernetes Service.
Services with invalid labels fail to hearbeat their clusters but the
service is kept working. This change forces the service to fail fast
during startup if any key is invalid.
Fixes#22696
* Fix Kube impersonation header overwrite when dealing with remote clusters
When forwarding exec requests to remote clusters I introduced an empty
header that failed the validation in `setupImpersonationHeaders`. This
results in the request being denied.
This problem only affects Teleport 11 and 10, but I am fixing the behaviour
on master as well.
Fixes#22226
* add tests
Kube
This PR fixes the inheritance of `kubernetes_resources` from the user's roles when the user has active access requests to a `kube_cluster`.
If the user had an active request to a `kube_cluster`, Teleport ignored the role's `kubernetes_resources` and returned an empty set for the allowed and denied pods. This resulted in the user being unable to see any pods in the cluster.
Fixes#22126
`auth.Middleware` to make use of them. This allows for less redundancy
per RPC and provides the ability to perform session control when grpc
connections are established.
The `auth.Middleware.withAuthenticatedUser` function was called from all
interceptors of any gRPC server that leveraged `auth.Middleware`. This
was constructing the identity of the client from the TLS certificate
per RPC. Since the identity of a connection cannot change between RPCs
there is no need to construct the identity in an interceptor. Instead,
the new `auth.TransportCredentials` can build the identity once after the
TLS handshake completes. When interceptors or gRPC handlers need to
acquire the identity they can do so using the same
mechanism via `peer.Peer.AuthInfo` and assert on the new `auth.IdentityInfo`
type:
```go
peerInfo, ok := peer.FromContext(ctx)
if !ok {
return
}
connInfo, ok := peerInfo.AuthInfo.(auth.IdentityInfo)
if !ok {
return
}
identity := connInfo.IdentityGetter.GetIdentity()
connState := connInfo.TLSInfo.State
```
The gRPC server that is served on the Proxy ssh port needs to hold the
same invariants as the ssh server available on the same port. This
means that connection limits need to be enforced when new connections
are established. To achieve this the new `auth.TransportCredentials` can
be provided with an `auth.ConnectionEnforcer` and an `auth.Authorizer`
to first build an identity, then ensures that the user is authorized and
finally that the user has not exceeded their connection limits.
Other minor changes include providing the cluster name to the
`auth.Middleware` instead of it looking the value up from the cache every
time an identity is created. Some interceptors were also moved around or
removed to eliminate the chances of having duplicate interceptors end up in
the interceptor chain.
Part of #19812
* Fix Moderated session on leave pause action.
* Unpause when moderator rejoins session with onLeave=pause
This commite fixes a case when the moderator rejoins a paused session
but Teleport didn't resume the operations and the session was still
locked to everyone.
* Fix panic when re-joining SSH session.
* Update lib/kube/proxy/sess.go
Co-authored-by: Noah Stride <noah.stride@goteleport.com>
* Update lib/kube/proxy/sess.go
Co-authored-by: Noah Stride <noah.stride@goteleport.com>
* Fix typo
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
---------
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
Co-authored-by: Noah Stride <noah.stride@goteleport.com>
* Start removing generateToken
* Refactor `tokens add` to use `CreateToken` RPC
* Correct deprecation version
* Remove calls to GenerateToken
* Start tests for token crud
* Allow user role rules to be provided wth CreateUserAndRole helper
* Add TestCreateToken to tls_test
* Add TestUpsertToken
* Add GetTokens test
* Add TestGetToken
* Add test for DeleteToken RPC
* Dedupe error assertions
* Add support for emitting trusted cluster token creation events to Create/UpsertToken
* Fix imports
* Change parameter order to appease linter
* Fix broken token expiry test
* Fix token expiry time for TestAuth_RegisterUsingToken
* Remove unnecessary cleanup
* Apply suggestions from code review
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Change test comment from deprecation notice
* `make gprc` after code review changes
* Update lib/auth/tls_test.go
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Update lib/auth/tls_test.go
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Update lib/auth/tls_test.go
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Update lib/auth/tls_test.go
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Rename tests
* Update lib/auth/tls_test.go
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* More test cases for GetTokens
* Improve message on duplicate token
---------
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Change formatting for "/version" endpoint
This PR changes the formatting for "/version" endpoint since Kubernetes
clients do not expect a JSON response on that endpoint.
This PR returns the error message directly into the body without any
formatting so that `kubectl` is able to present the desired error.
```
kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.4
Kustomize Version: v4.5.7
Error from server (InternalError): an error on the server ("this user cannot request kubernetes access, has no assigned groups or users") has prevented the request from succeeding
```
Fixes#20900
* change message wording
* add validation when generating kube creds
* fix typo
During binary upgrades, Teleport forks the agents and calls the shutdown
method for each one of them.
Kubernetes Agent was calling the `http.Server` shutdown method and which
wasn't stopping the heartbeats neither handling the listeners.
This PR fixes the shutdown behaviour for Kube agent and avoids deleting
the heartbeated resources when the agent is reloading.
This PR introduces a gRPC service to retrieve the pods available using the `search_as_roles` configured in the user's roles.
This gRPC service receives a TLS identity and manipulates it to access the Teleport Kubernetes Proxy and retrieve the pods available for a hypothetical user that has the `search_as_roles` defined for its roles.
The response does not leak the certificate nor the pod details other than `name`, `namespace` and `labels`.
```mermaid
sequenceDiagram
participant User as User
participant gRPC as grpc Server
participant Auth as Auth Server
participant KubeProxy as Kube Proxy
participant KubeSVC as Kube Service
User->>+gRPC: ListKubeResources(cluster,namespace)
Note left of gRPC: Includes search_as_roles <br/> and preview_as_roles <br/>into identity!
gRPC-->>+Auth: Sign cert with ProcessKubeCSR
Auth-->>-gRPC: Signed certificate
gRPC->>+KubeProxy: list pods "v1.../namespaces/{namespace}/pods"
KubeProxy-->>+KubeSVC: list pods "v1.../namespaces/{namespace}/pods"
KubeSVC-->>-KubeProxy: filtered response
Note left of KubeSVC: Applies Pod RBAC filtering!
KubeProxy-->>-gRPC: response
gRPC->>-User:Return name, namespace and labels
```
Part of #21107 and #19573
This PR fixes the impersonation headers propagation when using the SPDY
roundtrip. SPDY roundtrip creates a new http request and loses the
impersonation headers received from the client. When this happens for
users that define multiple `kubernetes_users`, the requests are denied
because Teleport forces you to select a user.
With the changes introduced we conserve the headers received from the
client.
Fixes#21088
This PR adds support for `gzip` compressed responses received from
the Kubernetes API.
Most Kubernetes vendors run with compression disabled since it causes a lot of CPU
spikes and timeouts but at least IBM cloud has it enabled.
(https://github.com/kubernetes/kubernetes/issues/112296)
Fixes#20980
* Expose device verification methods in lib/devicetrust/authz
* Set device fields in auth.Context.GetAccessState
* Enable role-based device verification for SSH
* Enable role-based device verification for DBs
* Add TODO for better SSH error messages
* Do not override device error on lib/kube/proxy
* Run gci
Increases deadline until the agent receives the first byte to 10s.
It's required to accommodate setups with high latency and where the time
between the TCP being accepted and the time for the first byte is longer
than the default value - 1s.
Fixes#20442
Refactor `services.AccessMFAParams` and the accompanying `MFAParams` methods so
they may be extended to carry additional data (eg, information required to
perform device trust checks).
This PR refactors existing code without adding any new functionality, so it may
be backported to older branches to alleviate eventual conflicts. Follow up
changes will take advantage of it to add support for device trust.
Related to gravitational/teleport.e#514.
This PR introduces support for creating resource access requests to `pod` resources to `tsh` and their approval from `tsh` and `tctl`.
Adds support `tsh request create --resource /<cluster>/pod/<kube_cluster>/<namespace>/<podname>`.
Part of #18434
Running computationally expensive tests causes connection failures that lead to test failures.
This PR tries to overcome this problem by making only the subtests run in parallel.
This PR introduces a mechanism to control access to Pods in Kubernetes clusters based on access criteria defined in user Roles or Resource Access Requests.
New `Role` version - `v6` - introduced a new field `kubernetes_resources` that allows admins to restrict access to a subset of cluster resources based on the configuration provided.
This PR implements the resource control for `pods` and allows
```yaml
kubernetes_resources:
- kind: pod
name: nginx-*
namespace: default
- kind: pod
name: *
namespace: dev
```
It also implements the logic blocks for supporting Resource Access Requests to Kubernetes Pods. The tsh boilerplate is not included in this PR and will be included in future PRs.
An example of usage:
```
tsh request search --kind pod --kube-cluster=minikube
Name Namespace Labels Resource ID
--------------------------------- --------- -------------------------------------- ---------------------------------------------
nginx-deployment-6595874d85-6j2zm default app=nginx,pod-template-hash=6595874d85 /tele.teleport.local/pod/minikube/default/...
nginx-deployment-6595874d85-7xgmb default app=nginx,pod-template-hash=6595874d85 /tele.teleport.local/pod/minikube/default/...
nginx-deployment-6595874d85-c4kz8 default app=nginx,pod-template-hash=6595874d85 /tele.teleport.local/pod/minikube/default/...
test default run=test /tele.teleport.local/pod/minikube/default/...
To request access to these resources, run
> tsh request create --resource /tele.teleport.local/pod/minikube/default/nginx-deployment-6595874d85-6j2zm --resource /tele.teleport.local/pod/minikube/default/nginx-deployment-6595874d85-7xgmb --resource /tele.teleport.local/pod/minikube/default/nginx-deployment-6595874d85-c4kz8 --resource /tele.teleport.local/pod/minikube/default/test \
--reason <request reason>
```
Part of #18434
Improve UserMetadata audit logging by using `tlsca.Identity.GetUserMetadata()`
more consistently in audit logs.
There are still a handful of logs that don't include the data above for lack of
context (in the general sense and in the Go context sense), but many more
entries now call back to identity in some way.
Related to gravitational/teleport.e#514.
When users access a cluster, Teleport caches their credentials to avoid generating a new cert key pair each time they run a command on the cluster. If the user's certificate includes an active access request that was later discarded, the Teleport Kubernetes Proxy continues to use the cached credentials - which include the dropped access request - resulting in subsequent requests being denied by Teleport. The problem persists even if the user assumes another access request that grants him access to the cluster.
This situation happens because Kubernetes Proxy stores in a TTL map the user's certificate to avoid generating and signing it each time the user hits the proxy. The lookup in cache happens using a key that includes the `kubeCluster`, `username`, `certificate_expiration`, `kube_users`, and `kube_groups` but does not include the `active_requests`.
This PR adds the `active_requests` into the cache's key to distinguish different certificate requests for the same user.
Fixes#19884
This PR includes a new Role resource version that is compatible with V5 spec.
The new resource introduces the `kubernetes_resources` definition that allows operators to limit the Kubernetes resources that each member can access. The `kubernetes_resources` entries must follow the following format: `{"kind":"<kind>", "namespace":"<namespace>","name":"<pod>"}`. Currently, it only supports objects of `kind` `pod`. Valid examples `<namespace>/<name>:
- `*/*`: matches all pods in all namespaces.
- `default/*`: matches all pods in the `default` namespace.
- `*/nginx-*`: matches every pod prefixed with `nginx-` in every namespace.
For older resource versions - V5, V4, V3 - `kubernetes_resources` is automatically populated with `{"kind":"pod","namespace":"*","name":"*"}` to keep compatibility. For the newest version, it's mandatory to define its value otherwise access to pods will be denied.
Part of #18434
The `fileStreamer` continues to write events after the server shuts down and races against the `os.RemoveAll` call during the test cleanup causing the test to fail.
Using `node-sync` recording mode to write the events and session recordings directly to AuthClient solves the issue.
Fixes#19847
Enable device authorization by plugging into auth.Authorizer and selectively
disabling it for processes that don't (yet) want device authz.
`GenerateUserCerts` is modified to issue device-aware certificates (DB/k8s
access), as well as `CreateAppSession`. The latter is not necessary for DB
access, but it does enable App Access to issue device-aware certs - commands
such as `tsh apps login` and `tsh proxy app` can benefit from those.
DB access is now ready to benefit from trusted devices. k8s access is likely
supported with these changes as well, but I've postponed enabling it after I've
done more testing.
Both `GenerateUserCerts` and `GenerateUserSingleUseCerts` now do early
device-aware authorization; this creates a better UX, as it allows us to return
error messages directly via `tsh`, instead of having to pipe them through
database-specific protocols. Further PRs could improve errors for scenarios
where the existing certificate became lacking due to higher server-side authz
enforcement.
gravitational/teleport.e#514
This PR fixes multiple goroutine and memory leaks when interactive sessions are used.
- When the session terminates, the `multiResizeQueue` never returns, and the resize stream goroutine blocks.
- A goroutine leak exited when the server received resizing events after the connection terminated - this happens with fast exec requests.
- A memory leak existed when users tried to leave the session after the `session.tracker` was closed.
This PR also releases the connection monitor earlier. When the server is under heavy load, it might take a while for the connection to return an `EOF` - which triggers the service monitor automatic release - and the service monitor resources were leaking until the server resumed normal operation.
It also fixes reloads when new parties join and leave the `multiResizeQueue`.
This PR replaces the following PRs opened by dependabot:
- #19678
- #19677
It also bumps:
- Bump k8s.io/api from v0.25.4 to v0.26.0
- Bump k8s.io/apiextensions-apiserver from v0.25.4 to v0.26.0
- Bump k8s.io/apimachinery from v0.25.4 to v0.26.0
- Bump k8s.io/apiserver from v0.25.4 to v0.26.0
- Bump k8s.io/cli-runtime from v0.25.4 to v0.26.0
- Bump k8s.io/kubectl from v0.25.4 to v0.26.0
- Bump k8s.io/utils v0.0.0-20220728103510-ee6ede2d64ed to v0.0.0-20221128185143-99ec85e7a448
- Bump sigs.k8s.io/controller-runtime from v0.13.1 to v0.14.1
* Add secure IP propagation from proxy to auth server when using ALPN
We're using PROXY protocol extensions called TLVs to send
signed JWT and proxy's certificate to the auth server. Auth
validates JWT using provided signing certificate and host CA
to make sure that IP information comes from our internal proxy.
* Move logic out of RoundTrip and into ServeHTTP as a middleware before handing off to oxy forwarder
* Move AWS signing service code into lib/utils/aws/signing.go
* use app server close context for audit event emitting
* add go doc comments.
* refactor request rewriting to make the copy in a more robust way.
* pass status code as uint32 rather than casting in audit emitter
* clone request instead of making a new request, and rewrite url to force https
* update header handling
* Set oxy forwarder to PassHostHeader=false to ensure the host header is
the URL being sought.
* Remove code that deleted forwarding headers previously, we should keep
those (X-Forwarded-*).
* Audit log the AWS Host sought rather than the incoming request Host
header (prior behavior maintained, we just rewrite the request
differently using Clone).
* Remove obsolete header copying helper func
* Enable nolintlint linter
* Fix nolint comments in the api package
* Fix RDP client comment
* Address review comment
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Allow unused for nolintlint linter
* Remove redundant casting
* Add comment on why allowed unused is enabled
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
This PR extends the Kubernetes Service to support the WebSocket
protocol in Kubernetes Porforward calls.
The Websocket protocol is required so that Kubernetes clients
like C#, Python, and Javascript can create portforward requests
to Kubernetes PODs. Although Teleport supports clients using WebSocket
protocol, it still uses SPDY protocol when connecting to upstreams -
Teleport Services or Kubernetes API.
Contrary to requests through SPDY, WebSocket requests do not support any
concurrency or multiplexing. This is a protocol limitation since
it only creates a single stream pair per port. When using it
concurrently, the requests will be mixed and fail.
Fixes#18783
Many of our tests (db package, I'm looking at you) generate many RSA keys. This has two main side effects; makes our tests slow and flaky as CPU usage spikes in random moments when the tests are run in parallel.
This change pre-generates RSA keys at the beginning of each test module and reuse them in randomized order to reduce the situation that one key has been used multiple times in one test.
I had to move a few files to avoid circular dependencies.
Michael Wilson