A new `api/client/proxy/Client` has been added to interact with
the SSH and gRPC servers that the Proxy serves on its SSH port.
The client will first try connecting to the gRPC server and if
that fails it will fall back to the SSH server.
Much of the SSH functionality mimics the existing behavior of the
`ProxyClient` in `lib/client`. This is the first part of phasing
out that client in favor of the new client. There will be a follow
up PR that migrates `lib/client` to make use of the new client instead.
Part of #19812
Sets `http.Server.IdleTimeout` and `http.Client.IdleConnTimeout`
on clients and servers which didn't have them set. A default of
360s was chosen to be on par with the default of an NLB without
being identical.
This was added as another safety measure to prevent leaking any
idle connections indefinitely as seen in #22757.
`apidefaults.DefaultDialTimeout` was also renamed to
`apidefaults.DefaultIOTimeout` to better reflect its usage.
* Allow node to handle old and new way of client IP propagation on same listener
With addition of signed PROXY headers, node was listening on multiplexer, but because
of that it couldn't processing incoming connection from older proxies
when ProxyHelloSignature was used, because
both ends were waiting for the other side to send data first.
Here we integrate ability to handle PROXY headers into connection itself,
so we can start ssh server without waiting for multiplexer to detect connection
* Remove unneeded code
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Fix imports
* Unexport function
* Add godoc
* Remove unneeded comment
* Move ProxyHelloSignature to constants
* Check that proxyline is verified
* Use Warn() instead of Warnf()
* Move ProxyHelloSignature to api/constants
* Add timeout for getting host CA during proxyline verification.
* Rearrange conditions
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Clarify comment.
---------
Co-authored-by: Michael Wilson <mike@mdwn.dev>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
While loading a key from storage, Teleport client transverses the
storage directory to read the secrets and reads each file one by one.
This has huge impacts on `tsh kube credentials` because when calling
`kubectl` in parallel, the disk operations become overloaded and the
operation time increases exponentially.
This PR removes the transverse when reading credentials for Kubernetes
clusters and adds a quick read operation that reads the following files:
- `$TSH_HOME/current_profile`
- `$TSH_HOME/$profile.yaml`
- `$TSH_HOME/keys/$PROXY/$USER-kube/$TELEPORT_CLUSTER/$KUBE_CLUSTER-x509.pem`
- `$TSH_HOME/keys/$PROXY/$USER`
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old direct | 9.90s user 3.14s system 64% cpu 20.284 total | 11.08s user 3.60s system 496% cpu 2.955 total | 12.46s user 4.36s system 820% cpu 2.050 total | 12.81s user 4.76s system 811% cpu 2.164 total | 13.02s user 5.07s system 861% cpu 2.100 total|
| old teleport | 18.45s user 7.80s system 51% cpu 51.085 total | 21.58s user 10.03s system 434% cpu 7.270 total | 22.33s user 10.52s system 178% cpu 18.402 total | 22.04s user 10.83s system 96% cpu 33.977 total | 22.57s user 11.45s system 55% cpu 1:01.34 total|
| new direct | 9.51s user 2.90s system 146% cpu 8.460 total | 13.75s user 4.95s system 718% cpu 2.603 total | 12.21s user 4.43s system 832% cpu 1.999 total | 11.72s user 4.50s system 839% cpu 1.933 total | 12.06s user 4.62s system 830% cpu 2.009 total |
| new teleport | 16.10s user 5.97s system 42% cpu 52.104 total | 21.62s user 9.01s system 381% cpu 8.023 total | 23.22s user 10.11s system 540% cpu 6.162 total | 21.64s user 9.81s system 603% cpu 5.208 total | 20.70s user 9.13s system 602% cpu 4.948 total |
---
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old no tsh kube credential | 8.61s user 2.99s system 31% cpu 36.336 total | 9.57s user 3.39s system 283% cpu 4.565 total | 10.22s user 4.03s system 553% cpu 2.574 total | 9.32s user 3.82s system 559% cpu 2.351 total | 11.49s user 4.85s system 624% cpu 2.617 total
| new no tsh kube credential | 8.27s user 2.94s system 24% cpu 45.041 total | 10.85s user 4.31s system 195% cpu 7.767 total | 11.32s user 4.64s system 469% cpu 3.401 total | 10.83s user 4.51s system 553% cpu 2.773 total | 10.90s user 4.84s system 537% cpu 2.930 total |
---
| instances | old total | old avg | **new total** | **total avg** |
| --- | --- | --- | --- | --- |
| 1 | 0.145 | 0.145 | 0.074 | 0.074 |
| 10 | 0.430 | 0.043 | 0.146 | 0.0146 |
| 20 | 0.740 | 0.037 | 0.284 | 0.0142 |
| 30 | 1.108 | 0.037 | 0.417 | 0.0139 |
| 40 | 1.483 | 0.037 | 0.580 | 0.0145 |
| 60 | 13.604 | 0.228 | 0.825 | 0.0137 |
| 80 | 30.236 | 0.378 | 1.143 | 0.0142 |
| 106 | 46.681 | 0.440 | 1.413 | 0.0133 |
Fixes#19313
Co-authored-by: Brian Joerger <bjoerger@goteleport.com>
Provides a client that can be used to connect to and interact with
the transport service in `lib/srv/transport`. The client abstracts
the fact that a gRPC stream is being used for the `net.Conn` created
by `DialCluster` and `DialHost`.
This also moves `lib/utils/grpc/stream` to `api/utils/grpc/stream`
so that the client and server can make use of the same stream
abstractions.
Part of #19812
* Save private key policy in profile.
* Fix tsh status/logout with yubikey unplugged; fix cluster alert on tsh status with hardware_key_touch.
* Set private key policy from ping response.
* Skip cluster alerts on tsh status.
* Set SkipLocalAuth on identity file use; Simplify logic for SkipLocalAuth.
* Fix TrustedCertsFromCACerts to work with known hosts without proxy host set in host name; Fix HostKeyCallback for keys.
* Add regression test where known hosts are not written with proxy host.
* Fix client key default host key callback.
* Return empty list, not not found error, when there are no trusted certs.
* Restore option to filter trusted host keys in host key callback.
* Update lib/client/identityfile/identity_test.go
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
---------
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
- Add a generalized client store made up of a key, profile, and trusted certs store. Each sub store can support different backends (~/.tsh, identity_file, in-memory).
- Replace custom identity file handling with in-memory client store.
- Fix issues with trusted certs handling.
* Add a new db engine
* Add tests for new engine
* Update tsh db subcommands
* Refactor error message and suggestions for unsupported tsh commands
* Add dynamodb to test plan
* Add AWS external ID to db config and update protos
When the AWS Account ID is provided, we must check that it is a valid
one.
AWS Account ID is a well documented field: 12-digit string
This check is meant to prevent typos when creating Databases mainly from
the Web UI in the context of Teleport Discover.
* Fix web ssh session with proxy recording mode
The agent was not being propagated when establishing an ssh session
via the web which resulted in the error described in #18850. Providing
the agent was straightforward, however, due to the changes from #18656
when using the forward server, as is required with proxy recording mode,
the ssh connection is now being performed directly over a `net.Pipe`.
Due to the synchronous nature of `net.Pipe` this causes a deadlock when
performing the ssh handshake. To mitigate the deadlocks, `DualPipeNetConn`
was changed to leverage `syscall.Socketpair` instead of `net.Pipe`.
`TestTerminal` now has two cases, one for node recording mode and another
for proxy recording mode. In order for the proxy recording mode test to pass
the fake clock used in the test needed to be properly propagated to the
`ssh.CertChecker` and `forward.ServerConfig`.
Fixes#18850
We were inconsistent throughout the codebase and would sometimes
use the slices package and other times use our own equivalents
in api/.
This removes our versions in favor of the golang.org/x package that
does the same, which has the added benefit of reducing the surface
area of the public API module.
Note: despite existing uses of the slices package, for some reason
it didn't show up in go.mod or go.sum. Fixed that too.
* Ensure invalid tunnel agent connections get closed
Connections from reverse tunnel agents were being marked
as invalid by the proxy under certain conditions but would
ultimately never be closed. This could lead to scenarios where
the agent thought things were fine but the proxy considered
that agent unhealthy and unroutable.
Pruning of invalid connections used to occur when a proxy
tried to retrieve a connection for that tunnel. This also
further muddied the point in time at which the proxy could
close a connection as it never explicitly stopped tracking
the connection and closed it at the same time.
To remedy this, connections are explicitly closed by the proxy
and removed from the mapping to stop tracking immediately. In order
to prevent a connection that is servicing an active connection
from being closed the proxy now tracks which connections have
sessions. Closing does not occur when there are any active
sessions to prevent them from being force terminated.
When the proxy receives a heartbeat from an agent it now restores
the connection to a valid state. In the event that too many heart
beats have been missed for an agent, the proxy will now terminate
the connection, again only if it is not serving any sessions.
Fixes#15911
- When user registers or resets, when running into
policy errors, we don't send back an error, but instead
a 200 (to indicate user has successfully registered/resetted)
and a flag to determine if policy was enabled
- Send back policy configurations for cluster config,
this will allow the web UI login page to just display
redirection messages without login attempts
- For role configured policy, login attempts will be
required and a specific error will be returned so that
the UI can then better redirect the user
* Add Yubikey PrivateKey implementation for use by Teleport clients.
- Add yubikey login logic, reusing previously stored private keys.
- Fix identity file decoding with PIV keys, which sign ecdsa certificates.
- Add libpcsclite-dev pre-req for building on linux.
- Remove unnecessary keys.Signer interface and move its functionality to keys.PrivateKey.
- Move retry and jitter utils to new api/utils/retryutils package.
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
Primary Changes:
- Remove reliance on Private Key PEM:
- Update native and keygen packages to return PrivateKey instead of PEM key
- Add new PrivateKey interface which implements crypto.Signer
- Replace PEM encoded private key usage where possible
- Replace calls to tls.(Load)X509KeyPair with keys.(Load)X509KeyPair in
client packages
Minor Changes:
- Remove unused agent.AddedKey return from LoadKey
- Simplify sshutils and removed unused code paths
- Add ecdsa and ed25519 key support
* Add additional debug logging for instantiation of ssh server conn
* Rough out introducing mux to ssh listener
* Catch errors from multiplexer serve
* Adapt multiplexer to support ProxyHelloSignature
* Adjust logging for ssh conn creation
* Adjust tests to use semantically correct assert/require helpers
* Address PR comments
* Check Client addr is properly propagated in mux test
* SSH request tracing
Add tracing support for ssh global requests and channels. Wrappers
for `ssh.Client`, `ssh.Channel`, and `ssh.NewChannel` provide a
mechanism for tracing context to be propagated via a `context.Context`.
In order to maintain backwards compatibility the ssh.Client wrapper
tries to open a TracingChannel when constructed. Any servers that
don't support tracing will reject the unknown channel. The client
will only provide tracing context to servers which do NOT reject
the TracingChannel request.
In order to include pass tracing context along all ssh payloads
are wrapped in an Envelope that includes the original payload
AND any tracing context. Servers now try to unmarshal all payloads
into said Envelope when processing messages. If there is an Envelope
provided, a new span will be created and the original payload will
be pass along to handlers.
Part of #12241
* Update api/types/fuzz_test.go
* do not fail if the file is missing
* missing go-118-fuzz-build fix
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
rosstimothy