* Don't loop over authMethods in (*agentDialer)DialContext
* Make HostKeyCallbackConfig.OnCheckCert fallible
* Add a way to check if a proxy is claimed
* Abort reverse tunnel conns early if the proxy is already claimed
* Document the pre-closed global request channel
* feat: support authenticating with AWS IAM for MongoDB Atlas
* chore(lint): fix errors
* test(tsh): add missing database field
* refactor(mongodb): check for error on each authenticator branch
* refactor(mongodb): update log messages and atlas check
* refactor(auth): use IsRoleARN helper instead of IsARN
* chore(db): remove unused line
* chore(mongodb): split authenticator func
* refactor(db): rename get atlas token function
* tests(db): reuse already existent auth property
* chore(mongodb): add docs reference
* refactor(db): support role chaining
* feat(types): "require" iam role for atlas users
* refactor(db): use external id only on the first session
* refactor(services): add new database matcher for regular users and aws
* chore(db): rename functions to be more assertive
* chore(types): fix lint
* test(db): remove duplicated test
The test being removed here is covered by `TestMongoDBAtlas`
(lib/srv/db/auth_test.go).
This commit updates all `err == io.EOF` comparisons to use
`errors.Is(err, io.EOF)`. This is necessary when the error may have been
wrapped and fixes at least one current breakage (in `tsh request ls`).
`golang.org/x/tools/refactor/eg` was very handy for this, I used the
following template:
```go
package teleport
import (
"errors"
"io"
)
func before(err error) bool { return err == io.EOF }
func after(err error) bool { return errors.Is(err, io.EOF) }
```
"github.com/gogo/protobuf/proto.Clone" has multiple bugs that cause
panics, so create a wrapper function that uses a thoroughly tested
cloning function from "google.golang.org/protobuf".
* Add full IP pinning enforcement
We're adding IP pinning check to `authorizer.Authorize` which is used for every call,
so now all communications with teleport should enforce IP pinning.
Also making sure we always provide login IP for user certificate creation
and correct client IP propagation everywhere.
* Add integration test for App IP pinning.
* Fix wording
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Wrap error
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Add godocs
* Clone TLS config
* Improve proxyHeaderSigner usage
* Wider use proxyHeaderDialer and remove adhoc writing of singed header
* Add helper function TLSDial
* Use proxyHeader dialer in authConnect
* Simplify tlsConfig manipulation
Co-authored-by: Przemko Robakowski <przemko.robakowski@goteleport.com>
* Remove redundant channels processing in TLSDial
* Reduce nesting
* Update generated protobufs
* Remove ignoring of bad IP on signed PROXY header generation
* Provide logger to CheckIPPinning function
---------
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
Co-authored-by: Przemko Robakowski <przemko.robakowski@goteleport.com>
* ALPN connect test improvements
* fix typos
* remove extra period
* simplify error check
* moving things over
* tsh dials
* reverse tunnel
* fix auth connect
* move ping
* add ssh support
* add HTTP client support
* Move ALPN dialer, ALPN conn upgrade, Ping conn to api
* beatify
* add test
* beautify round 2
* fix timeout
* Implement alpn-ping upgrade for reversetunnel and ssh
* clean up
* fix proxy test
* minor refactor
* remove WebProxyAddr
* require IsALPNConnUpgradeRequiredFunc
* add tlsRoutingWithConnUpgradeConnect
* fix lint
* simplify
* remove debug log and change unknown upgrade type to 404
* Force new proxy client to use web proxy when TLS routing is enabled
* AWS-hosted OpenSearch support.
* Review: comments, case, cleanup.
* Review: comments, region discovery, typos, shared func, fix error type, fix empty CA case.
* Review: address comments.
- add docs
- refactor opensearch engine code
- make `opensearchsql` default client
- drop utils.CloneRequest function, it is not suitable for general use
- drop "extra args" functionality, it should be a separate PR and more generic
- minor refactorings
* Fix: linting, refactoring issues.
* Review: update the comment for accuracy.
* Correct merge issue.
* Review: reuse transport, use different context, rename tests
* Review: unexport internal error types, explicitly test the error serialization.
* Convert tsh ssh to use the proxy transport service instead of ssh
In an effort to reduce latency establishing sessions `tsh ssh` is
migrating away from connecting to the Proxy via SSH in favor of
using gRPC. The SSH handshakes with the Proxy increase latency in
situations where the distance between geolocations of the client
and Proxy are large. TLS handshakes used by the gRPC service have
proven to reduce latency by ~20% in the same scenario.
A new `lib/client.ClusterClient` has been introduced that should
be used instead of `lib/client.ProxyClient` to connect to a Teleport
cluster. Most of the functionality within the `ClusterClient` was
a direct copy from the `ProxyClient`.
The `lib/client.TeleportClient` now has a `ConnectToCluster` method
which will connect to both the Proxy and Auth service via the
`api/client.ProxyClient` which first attempts to use gRPC and reverts
back to SSH to preserve backwards compatability. The `ClusterClient`
should be passed around and reused instead of following the
established pattern of `tc.ConnectToProxy` followed by a
`proxy.ConnectToCluster` to get an `auth.ClientI`.
Additionally some of the `agentless` package was refactored to reduce
dependencies and allow it to work with connections to the Proxy that
originated via gRPC instead of SSH.
Changes to the integration tests are mostly to accomodate IP Pinning
and ensure that it works for both connections established via SSH
and gPRC.
This is the final PR needed to complete #19812.
* fix typos and unify span attributes
* pass node name to ConnectToNode
* simplify jump host resetting
* Implement `tsh proxy kube`
* add --exec and test
* use proxy-url
* minor cleanup
* Fix not reusing previously saved kube cert.
* Add UT
* fix lint
* use real kube client for tsh test
* add integration test
* review comments
* fix function name
* make a kube test pack to setup once
The interface smuggling within `stream.ReadWriter.Close` now checks
for `io.Closer` instead of `grpc.ClientStream` to allow clients
that need to perform additoinal closing logic besides sending a
`CloseSend` message to do so. All existing client implementations
of `stream.Source` now implement `io.Closer` so they are still
cleaned up properly.
The multiplexed ssh streams are now tied together to prevent attempts
to send a message at the same time. In addition, the client ssh
streams have been updated to only send a single `CloseSend` on which
ever protocol is terminated first.
* log a warning when an invalid AWS IAM role ARN is filtered out
* add tests for nested role arn path and remove superfluous test parallelism
* add extra godoc info for BuildRoleARN
* improve mock IAM error messages
- Failing tests' error messages were not specific enough to know what
happened.
* refactor code to call GetAWS() once
- Avoids constantly fetching a full copy of the AWS metadata.
- Reduces verbosity of code.
* reuse MetadataFromRedshiftCluster in meta fetcher
- Avoids repeating the same logic inside the fetcher.
* improve database-access DynamoDB config checking
- Don't early return from database CheckAndSetDefaults so AWS account ID
is checked properly, and any other future checks added will apply to
DynamoDB as well.
* add aws utils for parsing AWS IAM roles
- ParseRoleARN will parse an AWS ARN and verify that it is a valid
AWS IAM Role resource
- CheckARNPartitionAndAccount is a helper that verifies an ARN partition
and account ID matches the caller's expectations.
* use cmp.Diff instead of require.Equal in db tests
- require.Equal spams a lot of protobuf noise.
- cmp.Diff yields a precise, readable diff on test failure.
A new `api/client/proxy/Client` has been added to interact with
the SSH and gRPC servers that the Proxy serves on its SSH port.
The client will first try connecting to the gRPC server and if
that fails it will fall back to the SSH server.
Much of the SSH functionality mimics the existing behavior of the
`ProxyClient` in `lib/client`. This is the first part of phasing
out that client in favor of the new client. There will be a follow
up PR that migrates `lib/client` to make use of the new client instead.
Part of #19812
Sets `http.Server.IdleTimeout` and `http.Client.IdleConnTimeout`
on clients and servers which didn't have them set. A default of
360s was chosen to be on par with the default of an NLB without
being identical.
This was added as another safety measure to prevent leaking any
idle connections indefinitely as seen in #22757.
`apidefaults.DefaultDialTimeout` was also renamed to
`apidefaults.DefaultIOTimeout` to better reflect its usage.
* Allow node to handle old and new way of client IP propagation on same listener
With addition of signed PROXY headers, node was listening on multiplexer, but because
of that it couldn't processing incoming connection from older proxies
when ProxyHelloSignature was used, because
both ends were waiting for the other side to send data first.
Here we integrate ability to handle PROXY headers into connection itself,
so we can start ssh server without waiting for multiplexer to detect connection
* Remove unneeded code
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Fix imports
* Unexport function
* Add godoc
* Remove unneeded comment
* Move ProxyHelloSignature to constants
* Check that proxyline is verified
* Use Warn() instead of Warnf()
* Move ProxyHelloSignature to api/constants
* Add timeout for getting host CA during proxyline verification.
* Rearrange conditions
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Clarify comment.
---------
Co-authored-by: Michael Wilson <mike@mdwn.dev>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
While loading a key from storage, Teleport client transverses the
storage directory to read the secrets and reads each file one by one.
This has huge impacts on `tsh kube credentials` because when calling
`kubectl` in parallel, the disk operations become overloaded and the
operation time increases exponentially.
This PR removes the transverse when reading credentials for Kubernetes
clusters and adds a quick read operation that reads the following files:
- `$TSH_HOME/current_profile`
- `$TSH_HOME/$profile.yaml`
- `$TSH_HOME/keys/$PROXY/$USER-kube/$TELEPORT_CLUSTER/$KUBE_CLUSTER-x509.pem`
- `$TSH_HOME/keys/$PROXY/$USER`
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old direct | 9.90s user 3.14s system 64% cpu 20.284 total | 11.08s user 3.60s system 496% cpu 2.955 total | 12.46s user 4.36s system 820% cpu 2.050 total | 12.81s user 4.76s system 811% cpu 2.164 total | 13.02s user 5.07s system 861% cpu 2.100 total|
| old teleport | 18.45s user 7.80s system 51% cpu 51.085 total | 21.58s user 10.03s system 434% cpu 7.270 total | 22.33s user 10.52s system 178% cpu 18.402 total | 22.04s user 10.83s system 96% cpu 33.977 total | 22.57s user 11.45s system 55% cpu 1:01.34 total|
| new direct | 9.51s user 2.90s system 146% cpu 8.460 total | 13.75s user 4.95s system 718% cpu 2.603 total | 12.21s user 4.43s system 832% cpu 1.999 total | 11.72s user 4.50s system 839% cpu 1.933 total | 12.06s user 4.62s system 830% cpu 2.009 total |
| new teleport | 16.10s user 5.97s system 42% cpu 52.104 total | 21.62s user 9.01s system 381% cpu 8.023 total | 23.22s user 10.11s system 540% cpu 6.162 total | 21.64s user 9.81s system 603% cpu 5.208 total | 20.70s user 9.13s system 602% cpu 4.948 total |
---
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old no tsh kube credential | 8.61s user 2.99s system 31% cpu 36.336 total | 9.57s user 3.39s system 283% cpu 4.565 total | 10.22s user 4.03s system 553% cpu 2.574 total | 9.32s user 3.82s system 559% cpu 2.351 total | 11.49s user 4.85s system 624% cpu 2.617 total
| new no tsh kube credential | 8.27s user 2.94s system 24% cpu 45.041 total | 10.85s user 4.31s system 195% cpu 7.767 total | 11.32s user 4.64s system 469% cpu 3.401 total | 10.83s user 4.51s system 553% cpu 2.773 total | 10.90s user 4.84s system 537% cpu 2.930 total |
---
| instances | old total | old avg | **new total** | **total avg** |
| --- | --- | --- | --- | --- |
| 1 | 0.145 | 0.145 | 0.074 | 0.074 |
| 10 | 0.430 | 0.043 | 0.146 | 0.0146 |
| 20 | 0.740 | 0.037 | 0.284 | 0.0142 |
| 30 | 1.108 | 0.037 | 0.417 | 0.0139 |
| 40 | 1.483 | 0.037 | 0.580 | 0.0145 |
| 60 | 13.604 | 0.228 | 0.825 | 0.0137 |
| 80 | 30.236 | 0.378 | 1.143 | 0.0142 |
| 106 | 46.681 | 0.440 | 1.413 | 0.0133 |
Fixes#19313
Co-authored-by: Brian Joerger <bjoerger@goteleport.com>
Provides a client that can be used to connect to and interact with
the transport service in `lib/srv/transport`. The client abstracts
the fact that a gRPC stream is being used for the `net.Conn` created
by `DialCluster` and `DialHost`.
This also moves `lib/utils/grpc/stream` to `api/utils/grpc/stream`
so that the client and server can make use of the same stream
abstractions.
Part of #19812
* Save private key policy in profile.
* Fix tsh status/logout with yubikey unplugged; fix cluster alert on tsh status with hardware_key_touch.
* Set private key policy from ping response.
* Skip cluster alerts on tsh status.
* Set SkipLocalAuth on identity file use; Simplify logic for SkipLocalAuth.
* Fix TrustedCertsFromCACerts to work with known hosts without proxy host set in host name; Fix HostKeyCallback for keys.
* Add regression test where known hosts are not written with proxy host.
* Fix client key default host key callback.
* Return empty list, not not found error, when there are no trusted certs.
* Restore option to filter trusted host keys in host key callback.
* Update lib/client/identityfile/identity_test.go
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
---------
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
- Add a generalized client store made up of a key, profile, and trusted certs store. Each sub store can support different backends (~/.tsh, identity_file, in-memory).
- Replace custom identity file handling with in-memory client store.
- Fix issues with trusted certs handling.
* Add a new db engine
* Add tests for new engine
* Update tsh db subcommands
* Refactor error message and suggestions for unsupported tsh commands
* Add dynamodb to test plan
* Add AWS external ID to db config and update protos
When the AWS Account ID is provided, we must check that it is a valid
one.
AWS Account ID is a well documented field: 12-digit string
This check is meant to prevent typos when creating Databases mainly from
the Web UI in the context of Teleport Discover.
* Fix web ssh session with proxy recording mode
The agent was not being propagated when establishing an ssh session
via the web which resulted in the error described in #18850. Providing
the agent was straightforward, however, due to the changes from #18656
when using the forward server, as is required with proxy recording mode,
the ssh connection is now being performed directly over a `net.Pipe`.
Due to the synchronous nature of `net.Pipe` this causes a deadlock when
performing the ssh handshake. To mitigate the deadlocks, `DualPipeNetConn`
was changed to leverage `syscall.Socketpair` instead of `net.Pipe`.
`TestTerminal` now has two cases, one for node recording mode and another
for proxy recording mode. In order for the proxy recording mode test to pass
the fake clock used in the test needed to be properly propagated to the
`ssh.CertChecker` and `forward.ServerConfig`.
Fixes#18850
We were inconsistent throughout the codebase and would sometimes
use the slices package and other times use our own equivalents
in api/.
This removes our versions in favor of the golang.org/x package that
does the same, which has the added benefit of reducing the surface
area of the public API module.
Note: despite existing uses of the slices package, for some reason
it didn't show up in go.mod or go.sum. Fixed that too.
* Ensure invalid tunnel agent connections get closed
Connections from reverse tunnel agents were being marked
as invalid by the proxy under certain conditions but would
ultimately never be closed. This could lead to scenarios where
the agent thought things were fine but the proxy considered
that agent unhealthy and unroutable.
Pruning of invalid connections used to occur when a proxy
tried to retrieve a connection for that tunnel. This also
further muddied the point in time at which the proxy could
close a connection as it never explicitly stopped tracking
the connection and closed it at the same time.
To remedy this, connections are explicitly closed by the proxy
and removed from the mapping to stop tracking immediately. In order
to prevent a connection that is servicing an active connection
from being closed the proxy now tracks which connections have
sessions. Closing does not occur when there are any active
sessions to prevent them from being force terminated.
When the proxy receives a heartbeat from an agent it now restores
the connection to a valid state. In the event that too many heart
beats have been missed for an agent, the proxy will now terminate
the connection, again only if it is not serving any sessions.
Fixes#15911
- When user registers or resets, when running into
policy errors, we don't send back an error, but instead
a 200 (to indicate user has successfully registered/resetted)
and a flag to determine if policy was enabled
- Send back policy configurations for cluster config,
this will allow the web UI login page to just display
redirection messages without login attempts
- For role configured policy, login attempts will be
required and a specific error will be returned so that
the UI can then better redirect the user
Marco André Dinis