A new `api/client/proxy/Client` has been added to interact with
the SSH and gRPC servers that the Proxy serves on its SSH port.
The client will first try connecting to the gRPC server and if
that fails it will fall back to the SSH server.
Much of the SSH functionality mimics the existing behavior of the
`ProxyClient` in `lib/client`. This is the first part of phasing
out that client in favor of the new client. There will be a follow
up PR that migrates `lib/client` to make use of the new client instead.
Part of #19812
* Fix TestDiagnoseSSHConnection to be less strict on the exact unknown user error message, which may vary between OS's.
* Decouple/refactor SkipLocalAuth, UseKeyPrincipal, and static auth methods.
* Resolve comments.
* * Add tests for new methods
* Update proxy ssh principal test to catch regression
* Bump predicate error in switch case priority
The `http.Client.Transport` created in `newWebClient` wraps a
chain of `http.RoundTripper` over an underlying `http.Transport`.
Since we cannot guarantee that each `http.RoundTipper` has a
`CloseIdleConnections` method the usage of
`defer clt.CloseIdleConnections()` does not guarantee that the http
connections created during Find/Ping/etc are closed.
To prevent leaking connections implementations of `http.RoundTripper`
have added a `CloseIdleConnections` method added that forwards the
request on to the wrapped interface. The `otelhttp.Transport` does
not implement this method either so care has been taken to wrap it
in a `http.RoundTripper` which will call the root transport in
`enforceCloseIdleConnections`. An upstream issue has been filed
with otelhttp: https://github.com/open-telemetry/opentelemetry-go-contrib/issues/3543
to get the method added to their `Transport` implementation.
This wasn't noticed prior to upgrading to go1.20 becuase prior to
[this](https://github.com/golang/go/commit/4e7e7ae1406c70d9cc0809ec11105a55a60a0b70)
commit the `ReadHeaderTimeout` set on the Proxy web api http.Server
would cause the connection to appear idle and get terminated by the
server.
`TestWebClientClosesIdleConnections` was added to capture the leak
in connections as reported in #22757 and prevent any regressions.
* Move configuration from lib/service to lib/service/servicecfg
The new servicecfg package will hold only configuration for services.
This will allow other packages (like tctl and tsh) to depend on
servicecfg without pulling in all of lib/service (which has a number
of platform-specific details).
This is the first step towards being able to build tctl for Windows.
* Move PAM and BPF config into servicecfg
This breaks a compile-time dependency on BPF/PAM for tctl.
* Add Plugins feature flag
* Add config for plugins runtime
* Allow reading OAuth secrets from files
* Rename Plugins to HostedPlugins
Make it clear that this is separate from the upcoming
self-hosted, standalone plugins that will be part of the Teleport
binary.
* Only allow to read client credentials from files
* Improve error messages for tsh login connectivity and ssh port connection errors
* spell fix
* lint fix
* lint fix
* suggestion applied
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Moved message to helper function for ssh proxy connection error
* format function
* Include error message on non-specified port login
* Give error message on attempted proxy hosts
* style coding
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* change to string construction
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* bytes buffer update
* lint fix
* Update message
* Update lib/client/api.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Update lib/client/api.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Update tool/tsh/tsh.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
---------
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
While loading a key from storage, Teleport client transverses the
storage directory to read the secrets and reads each file one by one.
This has huge impacts on `tsh kube credentials` because when calling
`kubectl` in parallel, the disk operations become overloaded and the
operation time increases exponentially.
This PR removes the transverse when reading credentials for Kubernetes
clusters and adds a quick read operation that reads the following files:
- `$TSH_HOME/current_profile`
- `$TSH_HOME/$profile.yaml`
- `$TSH_HOME/keys/$PROXY/$USER-kube/$TELEPORT_CLUSTER/$KUBE_CLUSTER-x509.pem`
- `$TSH_HOME/keys/$PROXY/$USER`
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old direct | 9.90s user 3.14s system 64% cpu 20.284 total | 11.08s user 3.60s system 496% cpu 2.955 total | 12.46s user 4.36s system 820% cpu 2.050 total | 12.81s user 4.76s system 811% cpu 2.164 total | 13.02s user 5.07s system 861% cpu 2.100 total|
| old teleport | 18.45s user 7.80s system 51% cpu 51.085 total | 21.58s user 10.03s system 434% cpu 7.270 total | 22.33s user 10.52s system 178% cpu 18.402 total | 22.04s user 10.83s system 96% cpu 33.977 total | 22.57s user 11.45s system 55% cpu 1:01.34 total|
| new direct | 9.51s user 2.90s system 146% cpu 8.460 total | 13.75s user 4.95s system 718% cpu 2.603 total | 12.21s user 4.43s system 832% cpu 1.999 total | 11.72s user 4.50s system 839% cpu 1.933 total | 12.06s user 4.62s system 830% cpu 2.009 total |
| new teleport | 16.10s user 5.97s system 42% cpu 52.104 total | 21.62s user 9.01s system 381% cpu 8.023 total | 23.22s user 10.11s system 540% cpu 6.162 total | 21.64s user 9.81s system 603% cpu 5.208 total | 20.70s user 9.13s system 602% cpu 4.948 total |
---
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old no tsh kube credential | 8.61s user 2.99s system 31% cpu 36.336 total | 9.57s user 3.39s system 283% cpu 4.565 total | 10.22s user 4.03s system 553% cpu 2.574 total | 9.32s user 3.82s system 559% cpu 2.351 total | 11.49s user 4.85s system 624% cpu 2.617 total
| new no tsh kube credential | 8.27s user 2.94s system 24% cpu 45.041 total | 10.85s user 4.31s system 195% cpu 7.767 total | 11.32s user 4.64s system 469% cpu 3.401 total | 10.83s user 4.51s system 553% cpu 2.773 total | 10.90s user 4.84s system 537% cpu 2.930 total |
---
| instances | old total | old avg | **new total** | **total avg** |
| --- | --- | --- | --- | --- |
| 1 | 0.145 | 0.145 | 0.074 | 0.074 |
| 10 | 0.430 | 0.043 | 0.146 | 0.0146 |
| 20 | 0.740 | 0.037 | 0.284 | 0.0142 |
| 30 | 1.108 | 0.037 | 0.417 | 0.0139 |
| 40 | 1.483 | 0.037 | 0.580 | 0.0145 |
| 60 | 13.604 | 0.228 | 0.825 | 0.0137 |
| 80 | 30.236 | 0.378 | 1.143 | 0.0142 |
| 106 | 46.681 | 0.440 | 1.413 | 0.0133 |
Fixes#19313
Co-authored-by: Brian Joerger <bjoerger@goteleport.com>
* Add plugin exchange service
* Add Plugin methods to auth
* Add gRPC-layer methods for Plugin
* Add RBAC presets for Plugin
* Test GetPlugin()/NoSecrets access
* Make error assertions more correct in role test
* Deny setting credentials if user can not read them
* gofmt
* Apply minor suggestions from code review
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Move dependency into an existing block in go.mod
* Improve error messages for failed type assertions
* DRY WithSecrets access checks for Plugins
* Run new tests in parallel
* Improve error assertions in auth_with_roles_test
* TestGetPluginWithSecrets: split cases to subtests
* Clean up test servers and clients
* Add proto for plugin service
* Remove Plugin methods from auth service
Moved to a dedicated service
* Remove plugin-related auth methods
Moved to a dedicated service in Enterprise
* Remove CreatePlugin test from auth_with_roles_test
Moved to a dedicated service in Enterprise
* Pass "backend getter" to local plugins service
This pattern is used in Enterprise to set up secondary services
before auth (and backend) are created.
* Rename InitialCredentials to BootstrapCredentials
* Add plugins service to genproto.sh
* Reformat generated proto
* Remove obsolete PluginExchangeService
The equivalent of this is now in Enterprise
* Add kube service to genproto.sh; regenerate
* Add ListPlugins to plugin backend service
* Reimplement GetPlugins on top of ListPlugins
This is a "convenience" implementation for the backend service layer.
* Replace GetPlugins with ListPlugins in gRPC schema
* Fix ListKubernetesResources unit test
* Simplify plugin pagination key to just the name
* Use existing constant for page size
* Make dummy clients return errors instead of panic
* Remove obsolete field
* Ensure go.mod is valid for corresponding e changes
* Fix passing mutex ref
* Move teleport-plugins import to e_imports
* Revert oauth change in go.mod
* Use limit+1 to look-ahead when paginating plugins
* Test plugin pagination with pageSize > numPlugins
* Add descriptive messages to gRPC dummy clients
* Plugin: add RW for editor; remove secrets from gRPC
* Make message more descriptive for dummy gRPC conn
---------
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* check if --cert-file, --key-file are given when tunnel is required
* require tunnel mode for SQL Server and Cassandra
* deprecate --cert-file and --key-file flags in favor of --tunnel
* Add flag to only check tsh binary version and not server version
* Update language
* Update variable comment
* Removed unncessary default flag, changed to --client from --client-only, updated variables used
* Add showing proxy address to tsh version
* Verbiage change for client flag
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* client only flag in docs
* Fix word for tsh —client
* Change flag description
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Updated tsh version comments, --client description
* copy edits
* Remove CLI docs, putting in a sep PR
---------
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
Co-authored-by: alexfornuto <alex.fornuto@goteleport.com>
Currently the `tsh` debug log is polluted with "errors" created by the
[automatic access request feature](https://goteleport.com/docs/access-controls/access-requests/resource-requests/?scope=enterprise#automatically-request-access-for-ssh)
even in completely expected scenarios, e.g. when the user has no
permission to create Resource Access Requests.
Before this change:
```
$ tsh ssh -d alice@one-auth
...<omitted>...
2023-02-17T15:30:16-08:00 DEBU [TSH] unable to request access to node error:[
ERROR REPORT:
Original Error: *trace.BadParameterError user attempted a resource request but does not have any "search_as_roles"
Stack Trace:
github.com/gravitational/teleport/api@v0.0.0/client/client.go:880 github.com/gravitational/teleport/api/client.(*Client).CreateAccessRequest
github.com/gravitational/teleport/tool/tsh/tsh.go:2896 main.accessRequestForSSH.func1
github.com/gravitational/teleport/lib/client/api.go:1351 github.com/gravitational/teleport/lib/client.(*TeleportClient).WithRootClusterClient
github.com/gravitational/teleport/tool/tsh/tsh.go:2895 main.accessRequestForSSH
github.com/gravitational/teleport/tool/tsh/tsh.go:2916 main.retryWithAccessRequest
github.com/gravitational/teleport/tool/tsh/tsh.go:2993 main.onSSH
github.com/gravitational/teleport/tool/tsh/tsh.go:1086 main.Run
github.com/gravitational/teleport/tool/tsh/tsh.go:482 main.main
runtime/proc.go:250 runtime.main
runtime/asm_amd64.s:1598 runtime.goexit
User Message: user attempted a resource request but does not have any "search_as_roles"] tsh/tsh.go:2920
ERROR REPORT:
Original Error: *trace.AccessDeniedError access denied to alice connecting to one-auth:0@default@cluster-one
Stack Trace:
github.com/gravitational/teleport/lib/client/client.go:1633 github.com/gravitational/teleport/lib/client.NewNodeClient
github.com/gravitational/teleport/lib/client/client.go:1563 github.com/gravitational/teleport/lib/client.(*ProxyClient).ConnectToNode
github.com/gravitational/teleport/lib/client/api.go:1451 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToNode
github.com/gravitational/teleport/lib/client/api.go:1525 github.com/gravitational/teleport/lib/client.(*TeleportClient).runShellOrCommandOnSingleNode
github.com/gravitational/teleport/lib/client/api.go:1408 github.com/gravitational/teleport/lib/client.(*TeleportClient).SSH
github.com/gravitational/teleport/tool/tsh/tsh.go:2995 main.onSSH.func1.1
github.com/gravitational/teleport/lib/client/api.go:504 github.com/gravitational/teleport/lib/client.RetryWithRelogin
github.com/gravitational/teleport/tool/tsh/tsh.go:2994 main.onSSH.func1
github.com/gravitational/teleport/tool/tsh/tsh.go:2907 main.retryWithAccessRequest
github.com/gravitational/teleport/tool/tsh/tsh.go:2993 main.onSSH
github.com/gravitational/teleport/tool/tsh/tsh.go:1086 main.Run
github.com/gravitational/teleport/tool/tsh/tsh.go:482 main.main
runtime/proc.go:250 runtime.main
runtime/asm_amd64.s:1598 runtime.goexit
User Message: access denied to alice connecting to one-auth:0@default@cluster-one
```
After:
```
$ tsh ssh -d alice@one-auth
...<omitted>...
2023-02-17T16:42:29-08:00 DEBU [TSH] Not attempting to automatically request access, reason: Resource Access Requests require usable "search_as_roles", none found for user "nklaassen" tsh/tsh.go:2922
ERROR REPORT:
Original Error: *trace.AccessDeniedError access denied to alice connecting to one-auth:0@default@cluster-one
Stack Trace:
github.com/gravitational/teleport/lib/client/client.go:1633 github.com/gravitational/teleport/lib/client.NewNodeClient
github.com/gravitational/teleport/lib/client/client.go:1563 github.com/gravitational/teleport/lib/client.(*ProxyClient).ConnectToNode
github.com/gravitational/teleport/lib/client/api.go:1451 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToNode
github.com/gravitational/teleport/lib/client/api.go:1525 github.com/gravitational/teleport/lib/client.(*TeleportClient).runShellOrCommandOnSingleNode
github.com/gravitational/teleport/lib/client/api.go:1408 github.com/gravitational/teleport/lib/client.(*TeleportClient).SSH
github.com/gravitational/teleport/tool/tsh/tsh.go:2997 main.onSSH.func1.1
github.com/gravitational/teleport/lib/client/api.go:504 github.com/gravitational/teleport/lib/client.RetryWithRelogin
github.com/gravitational/teleport/tool/tsh/tsh.go:2996 main.onSSH.func1
github.com/gravitational/teleport/tool/tsh/tsh.go:2907 main.retryWithAccessRequest
github.com/gravitational/teleport/tool/tsh/tsh.go:2995 main.onSSH
github.com/gravitational/teleport/tool/tsh/tsh.go:1086 main.Run
github.com/gravitational/teleport/tool/tsh/tsh.go:482 main.main
runtime/proc.go:250 runtime.main
runtime/asm_amd64.s:1598 runtime.goexit
User Message: access denied to alice connecting to one-auth:0@default@cluster-one
```
Fixes#19293
The error message "missing parameter hostname" wasn't very clear what hostname
was missing. Fail faster further up the stack and print a better error message
explaining what is happening.
This applies some checks aroung argument parsing while also adding some
belt-and-suspenders validation into `NewClientStoreFromIdentityFile`
* Allow a tsh aws user to specify a command to run with the AWS environment variables set
* Update name of the argument per feedback
* Address nits
---------
Co-authored-by: Daniel Ballenger <daniel@otter.ai>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Co-authored-by: STeve (Xin) Huang <xin.huang@goteleport.com>
* Change formatting for "/version" endpoint
This PR changes the formatting for "/version" endpoint since Kubernetes
clients do not expect a JSON response on that endpoint.
This PR returns the error message directly into the body without any
formatting so that `kubectl` is able to present the desired error.
```
kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.4
Kustomize Version: v4.5.7
Error from server (InternalError): an error on the server ("this user cannot request kubernetes access, has no assigned groups or users") has prevented the request from succeeding
```
Fixes#20900
* change message wording
* add validation when generating kube creds
* fix typo
This PR removes the requirement of providing a cluster name to login
into if the user provides the `--all` flag.
If the user does not provide a cluster, Teleport does not select any
cluster by default.
Fixes#21713
The `tsh appps` family of commands is aliased to `tsh apps`, so both
invocations work correctly. The command itself is defined as `tsh apps`,
so this is what appears in the help message.
Update references to `tsh app` to recommend `tsh apps` instead so that
there isn't confusion when browsing `tsh help` and looking for a missing
`app` subcommand.
Fixes#21367
Previously doing an `ls -R` would list all resources per profile in
parallel. This had little benefit when one profile needed to list
multiple clusters as those were still done serially. To mitigate this
things are broken up into two separate steps:
1) Connect to all clusters the user has credentials for in parallel
2) List resources for each cluster in parallel
This ensures that by the time a client is established for all available
clusters that resource listing can happen in parallel regardless of
profile. All operations are still restricted to an upper bound, however
those limits were also bumped to help performance.
* Save private key policy in profile.
* Fix tsh status/logout with yubikey unplugged; fix cluster alert on tsh status with hardware_key_touch.
* Set private key policy from ping response.
* Skip cluster alerts on tsh status.
* Set SkipLocalAuth on identity file use; Simplify logic for SkipLocalAuth.
* Fix TrustedCertsFromCACerts to work with known hosts without proxy host set in host name; Fix HostKeyCallback for keys.
* Add regression test where known hosts are not written with proxy host.
* Fix client key default host key callback.
* Return empty list, not not found error, when there are no trusted certs.
* Restore option to filter trusted host keys in host key callback.
* Update lib/client/identityfile/identity_test.go
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
---------
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
This PR implements a `kubectl` wrapper inside `tsh` that creates resource access requests, waits for their approval and retries the command when it detects that access to a pod was denied due to missing role or Kubernetes RBAC principals permissions.
Part of #18434
Updates #19573
This PR disables the update of kubeconfig when `tsh login` does not have the `--kube-cluster` flag.
The new behavior will update the kubeconfig only if the user provided a non-empty value in `--kube-cluster`.
This PR also introduces a mechanism to restore the current context on `tsh logout`. On `tsh login --kube-cluster=<cluster>` or `tsh kube login <cluster>`, Teleport updates the selected context - if it's not from Teleport - to add an extension whose key is: `selected_context`. Once `tsh logout` is called and the active context was a Teleport cluster, we restore the current context to the cluster that has the `selected_context` key in extensions.
Fixes#9718
- Add a generalized client store made up of a key, profile, and trusted certs store. Each sub store can support different backends (~/.tsh, identity_file, in-memory).
- Replace custom identity file handling with in-memory client store.
- Fix issues with trusted certs handling.
This PR includes a new Role resource version that is compatible with V5 spec.
The new resource introduces the `kubernetes_resources` definition that allows operators to limit the Kubernetes resources that each member can access. The `kubernetes_resources` entries must follow the following format: `{"kind":"<kind>", "namespace":"<namespace>","name":"<pod>"}`. Currently, it only supports objects of `kind` `pod`. Valid examples `<namespace>/<name>:
- `*/*`: matches all pods in all namespaces.
- `default/*`: matches all pods in the `default` namespace.
- `*/nginx-*`: matches every pod prefixed with `nginx-` in every namespace.
For older resource versions - V5, V4, V3 - `kubernetes_resources` is automatically populated with `{"kind":"pod","namespace":"*","name":"*"}` to keep compatibility. For the newest version, it's mandatory to define its value otherwise access to pods will be denied.
Part of #18434
Brian Joerger