* Move gRPC error intercetpors to api/utils/grpc/interceptors.
* Use error interceptors in api client and mock server.
* Apply suggestions from CR.
* Unwrap FromGRPC errors in middleware.
* Use gRPC auth service in tests instead of external example service.
* It's gRPC!!!
* Fix unit test.
* Add error interceptor to proxy client.
* Fix merge conflict.
* url checker
* add ut
* add comment in test plan
* warnOnError only when err is not nil
* fix typos etc.
* enable metadata service permission for discovered databases
* add a debug log when azure db url is validated
It was required for compatibilty reasons - v13 proxy could connect to v12 agents,
which didn't support signed PROXY headers. But we don't need it in v14.
* assist: support recording non-interactive forwarded sessions
* assist: add integration tests for assist command recording on agentless
This also fixes a bugged openssh integration test check.
* Mock OpenAI API in integration tests.
Also fixes the OpenAI mock handler that was not supporting the "/v1/*"
routes used in some web/ tests.
* Docstrings + adressing again a feedback that got lost during a rebase
* restore signer function signature
* Address Jakub's feedback + diverse improvements
- improve comments
- simpler file creation
- more efficient rsa key generation
- use assert instead of require in other goroutines
- only save env vars if they are Teleport-related
- move more logic into newTestCredentials for readability
* Update lib/srv/forward/sshserver.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Add a timeout in the ssh handler coroutine
---------
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Don't loop over authMethods in (*agentDialer)DialContext
* Make HostKeyCallbackConfig.OnCheckCert fallible
* Add a way to check if a proxy is claimed
* Abort reverse tunnel conns early if the proxy is already claimed
* Document the pre-closed global request channel
* feat: support authenticating with AWS IAM for MongoDB Atlas
* chore(lint): fix errors
* test(tsh): add missing database field
* refactor(mongodb): check for error on each authenticator branch
* refactor(mongodb): update log messages and atlas check
* refactor(auth): use IsRoleARN helper instead of IsARN
* chore(db): remove unused line
* chore(mongodb): split authenticator func
* refactor(db): rename get atlas token function
* tests(db): reuse already existent auth property
* chore(mongodb): add docs reference
* refactor(db): support role chaining
* feat(types): "require" iam role for atlas users
* refactor(db): use external id only on the first session
* refactor(services): add new database matcher for regular users and aws
* chore(db): rename functions to be more assertive
* chore(types): fix lint
* test(db): remove duplicated test
The test being removed here is covered by `TestMongoDBAtlas`
(lib/srv/db/auth_test.go).
This commit updates all `err == io.EOF` comparisons to use
`errors.Is(err, io.EOF)`. This is necessary when the error may have been
wrapped and fixes at least one current breakage (in `tsh request ls`).
`golang.org/x/tools/refactor/eg` was very handy for this, I used the
following template:
```go
package teleport
import (
"errors"
"io"
)
func before(err error) bool { return err == io.EOF }
func after(err error) bool { return errors.Is(err, io.EOF) }
```
"github.com/gogo/protobuf/proto.Clone" has multiple bugs that cause
panics, so create a wrapper function that uses a thoroughly tested
cloning function from "google.golang.org/protobuf".
* Add full IP pinning enforcement
We're adding IP pinning check to `authorizer.Authorize` which is used for every call,
so now all communications with teleport should enforce IP pinning.
Also making sure we always provide login IP for user certificate creation
and correct client IP propagation everywhere.
* Add integration test for App IP pinning.
* Fix wording
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Wrap error
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Add godocs
* Clone TLS config
* Improve proxyHeaderSigner usage
* Wider use proxyHeaderDialer and remove adhoc writing of singed header
* Add helper function TLSDial
* Use proxyHeader dialer in authConnect
* Simplify tlsConfig manipulation
Co-authored-by: Przemko Robakowski <przemko.robakowski@goteleport.com>
* Remove redundant channels processing in TLSDial
* Reduce nesting
* Update generated protobufs
* Remove ignoring of bad IP on signed PROXY header generation
* Provide logger to CheckIPPinning function
---------
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
Co-authored-by: Przemko Robakowski <przemko.robakowski@goteleport.com>
* ALPN connect test improvements
* fix typos
* remove extra period
* simplify error check
* moving things over
* tsh dials
* reverse tunnel
* fix auth connect
* move ping
* add ssh support
* add HTTP client support
* Move ALPN dialer, ALPN conn upgrade, Ping conn to api
* beatify
* add test
* beautify round 2
* fix timeout
* Implement alpn-ping upgrade for reversetunnel and ssh
* clean up
* fix proxy test
* minor refactor
* remove WebProxyAddr
* require IsALPNConnUpgradeRequiredFunc
* add tlsRoutingWithConnUpgradeConnect
* fix lint
* simplify
* remove debug log and change unknown upgrade type to 404
* Force new proxy client to use web proxy when TLS routing is enabled
* AWS-hosted OpenSearch support.
* Review: comments, case, cleanup.
* Review: comments, region discovery, typos, shared func, fix error type, fix empty CA case.
* Review: address comments.
- add docs
- refactor opensearch engine code
- make `opensearchsql` default client
- drop utils.CloneRequest function, it is not suitable for general use
- drop "extra args" functionality, it should be a separate PR and more generic
- minor refactorings
* Fix: linting, refactoring issues.
* Review: update the comment for accuracy.
* Correct merge issue.
* Review: reuse transport, use different context, rename tests
* Review: unexport internal error types, explicitly test the error serialization.
* Convert tsh ssh to use the proxy transport service instead of ssh
In an effort to reduce latency establishing sessions `tsh ssh` is
migrating away from connecting to the Proxy via SSH in favor of
using gRPC. The SSH handshakes with the Proxy increase latency in
situations where the distance between geolocations of the client
and Proxy are large. TLS handshakes used by the gRPC service have
proven to reduce latency by ~20% in the same scenario.
A new `lib/client.ClusterClient` has been introduced that should
be used instead of `lib/client.ProxyClient` to connect to a Teleport
cluster. Most of the functionality within the `ClusterClient` was
a direct copy from the `ProxyClient`.
The `lib/client.TeleportClient` now has a `ConnectToCluster` method
which will connect to both the Proxy and Auth service via the
`api/client.ProxyClient` which first attempts to use gRPC and reverts
back to SSH to preserve backwards compatability. The `ClusterClient`
should be passed around and reused instead of following the
established pattern of `tc.ConnectToProxy` followed by a
`proxy.ConnectToCluster` to get an `auth.ClientI`.
Additionally some of the `agentless` package was refactored to reduce
dependencies and allow it to work with connections to the Proxy that
originated via gRPC instead of SSH.
Changes to the integration tests are mostly to accomodate IP Pinning
and ensure that it works for both connections established via SSH
and gPRC.
This is the final PR needed to complete #19812.
* fix typos and unify span attributes
* pass node name to ConnectToNode
* simplify jump host resetting
* Implement `tsh proxy kube`
* add --exec and test
* use proxy-url
* minor cleanup
* Fix not reusing previously saved kube cert.
* Add UT
* fix lint
* use real kube client for tsh test
* add integration test
* review comments
* fix function name
* make a kube test pack to setup once
The interface smuggling within `stream.ReadWriter.Close` now checks
for `io.Closer` instead of `grpc.ClientStream` to allow clients
that need to perform additoinal closing logic besides sending a
`CloseSend` message to do so. All existing client implementations
of `stream.Source` now implement `io.Closer` so they are still
cleaned up properly.
The multiplexed ssh streams are now tied together to prevent attempts
to send a message at the same time. In addition, the client ssh
streams have been updated to only send a single `CloseSend` on which
ever protocol is terminated first.
* log a warning when an invalid AWS IAM role ARN is filtered out
* add tests for nested role arn path and remove superfluous test parallelism
* add extra godoc info for BuildRoleARN
* improve mock IAM error messages
- Failing tests' error messages were not specific enough to know what
happened.
* refactor code to call GetAWS() once
- Avoids constantly fetching a full copy of the AWS metadata.
- Reduces verbosity of code.
* reuse MetadataFromRedshiftCluster in meta fetcher
- Avoids repeating the same logic inside the fetcher.
* improve database-access DynamoDB config checking
- Don't early return from database CheckAndSetDefaults so AWS account ID
is checked properly, and any other future checks added will apply to
DynamoDB as well.
* add aws utils for parsing AWS IAM roles
- ParseRoleARN will parse an AWS ARN and verify that it is a valid
AWS IAM Role resource
- CheckARNPartitionAndAccount is a helper that verifies an ARN partition
and account ID matches the caller's expectations.
* use cmp.Diff instead of require.Equal in db tests
- require.Equal spams a lot of protobuf noise.
- cmp.Diff yields a precise, readable diff on test failure.
A new `api/client/proxy/Client` has been added to interact with
the SSH and gRPC servers that the Proxy serves on its SSH port.
The client will first try connecting to the gRPC server and if
that fails it will fall back to the SSH server.
Much of the SSH functionality mimics the existing behavior of the
`ProxyClient` in `lib/client`. This is the first part of phasing
out that client in favor of the new client. There will be a follow
up PR that migrates `lib/client` to make use of the new client instead.
Part of #19812
Sets `http.Server.IdleTimeout` and `http.Client.IdleConnTimeout`
on clients and servers which didn't have them set. A default of
360s was chosen to be on par with the default of an NLB without
being identical.
This was added as another safety measure to prevent leaking any
idle connections indefinitely as seen in #22757.
`apidefaults.DefaultDialTimeout` was also renamed to
`apidefaults.DefaultIOTimeout` to better reflect its usage.
* Allow node to handle old and new way of client IP propagation on same listener
With addition of signed PROXY headers, node was listening on multiplexer, but because
of that it couldn't processing incoming connection from older proxies
when ProxyHelloSignature was used, because
both ends were waiting for the other side to send data first.
Here we integrate ability to handle PROXY headers into connection itself,
so we can start ssh server without waiting for multiplexer to detect connection
* Remove unneeded code
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Fix imports
* Unexport function
* Add godoc
* Remove unneeded comment
* Move ProxyHelloSignature to constants
* Check that proxyline is verified
* Use Warn() instead of Warnf()
* Move ProxyHelloSignature to api/constants
* Add timeout for getting host CA during proxyline verification.
* Rearrange conditions
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Clarify comment.
---------
Co-authored-by: Michael Wilson <mike@mdwn.dev>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
While loading a key from storage, Teleport client transverses the
storage directory to read the secrets and reads each file one by one.
This has huge impacts on `tsh kube credentials` because when calling
`kubectl` in parallel, the disk operations become overloaded and the
operation time increases exponentially.
This PR removes the transverse when reading credentials for Kubernetes
clusters and adds a quick read operation that reads the following files:
- `$TSH_HOME/current_profile`
- `$TSH_HOME/$profile.yaml`
- `$TSH_HOME/keys/$PROXY/$USER-kube/$TELEPORT_CLUSTER/$KUBE_CLUSTER-x509.pem`
- `$TSH_HOME/keys/$PROXY/$USER`
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old direct | 9.90s user 3.14s system 64% cpu 20.284 total | 11.08s user 3.60s system 496% cpu 2.955 total | 12.46s user 4.36s system 820% cpu 2.050 total | 12.81s user 4.76s system 811% cpu 2.164 total | 13.02s user 5.07s system 861% cpu 2.100 total|
| old teleport | 18.45s user 7.80s system 51% cpu 51.085 total | 21.58s user 10.03s system 434% cpu 7.270 total | 22.33s user 10.52s system 178% cpu 18.402 total | 22.04s user 10.83s system 96% cpu 33.977 total | 22.57s user 11.45s system 55% cpu 1:01.34 total|
| new direct | 9.51s user 2.90s system 146% cpu 8.460 total | 13.75s user 4.95s system 718% cpu 2.603 total | 12.21s user 4.43s system 832% cpu 1.999 total | 11.72s user 4.50s system 839% cpu 1.933 total | 12.06s user 4.62s system 830% cpu 2.009 total |
| new teleport | 16.10s user 5.97s system 42% cpu 52.104 total | 21.62s user 9.01s system 381% cpu 8.023 total | 23.22s user 10.11s system 540% cpu 6.162 total | 21.64s user 9.81s system 603% cpu 5.208 total | 20.70s user 9.13s system 602% cpu 4.948 total |
---
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old no tsh kube credential | 8.61s user 2.99s system 31% cpu 36.336 total | 9.57s user 3.39s system 283% cpu 4.565 total | 10.22s user 4.03s system 553% cpu 2.574 total | 9.32s user 3.82s system 559% cpu 2.351 total | 11.49s user 4.85s system 624% cpu 2.617 total
| new no tsh kube credential | 8.27s user 2.94s system 24% cpu 45.041 total | 10.85s user 4.31s system 195% cpu 7.767 total | 11.32s user 4.64s system 469% cpu 3.401 total | 10.83s user 4.51s system 553% cpu 2.773 total | 10.90s user 4.84s system 537% cpu 2.930 total |
---
| instances | old total | old avg | **new total** | **total avg** |
| --- | --- | --- | --- | --- |
| 1 | 0.145 | 0.145 | 0.074 | 0.074 |
| 10 | 0.430 | 0.043 | 0.146 | 0.0146 |
| 20 | 0.740 | 0.037 | 0.284 | 0.0142 |
| 30 | 1.108 | 0.037 | 0.417 | 0.0139 |
| 40 | 1.483 | 0.037 | 0.580 | 0.0145 |
| 60 | 13.604 | 0.228 | 0.825 | 0.0137 |
| 80 | 30.236 | 0.378 | 1.143 | 0.0142 |
| 106 | 46.681 | 0.440 | 1.413 | 0.0133 |
Fixes#19313
Co-authored-by: Brian Joerger <bjoerger@goteleport.com>
Provides a client that can be used to connect to and interact with
the transport service in `lib/srv/transport`. The client abstracts
the fact that a gRPC stream is being used for the `net.Conn` created
by `DialCluster` and `DialHost`.
This also moves `lib/utils/grpc/stream` to `api/utils/grpc/stream`
so that the client and server can make use of the same stream
abstractions.
Part of #19812
* Save private key policy in profile.
* Fix tsh status/logout with yubikey unplugged; fix cluster alert on tsh status with hardware_key_touch.
* Set private key policy from ping response.
* Skip cluster alerts on tsh status.
* Set SkipLocalAuth on identity file use; Simplify logic for SkipLocalAuth.
* Fix TrustedCertsFromCACerts to work with known hosts without proxy host set in host name; Fix HostKeyCallback for keys.
* Add regression test where known hosts are not written with proxy host.
* Fix client key default host key callback.
* Return empty list, not not found error, when there are no trusted certs.
* Restore option to filter trusted host keys in host key callback.
* Update lib/client/identityfile/identity_test.go
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
---------
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
- Add a generalized client store made up of a key, profile, and trusted certs store. Each sub store can support different backends (~/.tsh, identity_file, in-memory).
- Replace custom identity file handling with in-memory client store.
- Fix issues with trusted certs handling.
* Add a new db engine
* Add tests for new engine
* Update tsh db subcommands
* Refactor error message and suggestions for unsupported tsh commands
* Add dynamodb to test plan
* Add AWS external ID to db config and update protos
When the AWS Account ID is provided, we must check that it is a valid
one.
AWS Account ID is a well documented field: 12-digit string
This check is meant to prevent typos when creating Databases mainly from
the Web UI in the context of Teleport Discover.
* Fix web ssh session with proxy recording mode
The agent was not being propagated when establishing an ssh session
via the web which resulted in the error described in #18850. Providing
the agent was straightforward, however, due to the changes from #18656
when using the forward server, as is required with proxy recording mode,
the ssh connection is now being performed directly over a `net.Pipe`.
Due to the synchronous nature of `net.Pipe` this causes a deadlock when
performing the ssh handshake. To mitigate the deadlocks, `DualPipeNetConn`
was changed to leverage `syscall.Socketpair` instead of `net.Pipe`.
`TestTerminal` now has two cases, one for node recording mode and another
for proxy recording mode. In order for the proxy recording mode test to pass
the fake clock used in the test needed to be properly propagated to the
`ssh.CertChecker` and `forward.ServerConfig`.
Fixes#18850
Brian Joerger