* fix: return OpaqueAccessDenied for NotFound errors
This commit modifies OpaqueAccessDenied to return an identical generic
NotFound error whether the input error is NotFound or AccessDenied.
The commit also updates all call sites of OpaqueAccessDenied to use it in
the paths where there is any error fetching the resource as well as when
access is denied.
It doesn't do much good to return AccessDenied errors as NotFound, if
they don't match the NotFound error you would get if the resource really
didn't exist.
It's trivial to tell the errors apart and discover the existence of a
resource you shouldn't be allowed to access.
The commit attempts to mitigate that issue and properly hide the
existence of resources the user should not be allowed to list.
* remove changes unrelated to remote clusters
* improve doc comment
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* make sure gRPC errors match
* fix integration test
---------
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* Remove logging from circuit breaker
The logging is never output, and if it was would be very verbose.
Instead of dropping the level down to trace the logger is removed
entirely.
* Add constants from trace library
The logging functionality in trace is being removed in an upcoming
release. To prepare the logging component keys are copied from trace
into teleport.
* Convert trace.ComponentFields to teleport.ComponentFields
* Replace trace.Component with teleport.ComponentKey
* Replace trace logging constants
* Remove marshal options from GetClusterNetworkingConfig
* Add ClusterNetworkingConfig support to cluster configuration RPC service
Extends the clusterconfigv1.Service to implement RPCs for interacting
with ClusterNetworkingConfig. The various GetClusterNetworkingConfig
interfaces were also all updated to remove the marshal options since
they were never used and served no purpose.
During refactoring performed in #18370 the counter was moved but
never used outside of registration. This restores incrementing the
counter for dial attempts and improves error reporting in traces
in the router.
Originally there was a default limit of 100 max concurrent streams, however in 2017 the GRPC team removed this default: https://github.com/grpc/grpc-go/pull/1624
With the recent HTTP/2 Rapid Reset DoS, it is now being encouraged to re-introduce a limit. The fix requires this value to be configured in fact: https://github.com/grpc/grpc-go/pull/6703
Ensure that `proxy_ssh_sessions_total` is incremented in both the
root and the leaf cluster if a session to a node in the leaf cluster
is established through the root cluster. This now also prevents
leaf clusters from terminating during a graceful shutdown if there
is an active connection that was created from the root.
Fixes#32605.
* Dont allow directly dialing to servers not in inventory
add direct dial escape hatch
* Fix failing unit test
* Fix TestProxySSH
* Fix TestTraitsPropagation
* resolve comment
* fix non-multiplexed trusted cluster setup in tsh test suite
* Fix TestProxySSH
* wait on nodes
* Skip the flaky check for TestSSHLoadAllCAs
---------
Co-authored-by: Forrest Marshall <forrest@goteleport.com>
* Move gRPC error intercetpors to api/utils/grpc/interceptors.
* Use error interceptors in api client and mock server.
* Apply suggestions from CR.
* Unwrap FromGRPC errors in middleware.
* Use gRPC auth service in tests instead of external example service.
* It's gRPC!!!
* Fix unit test.
* Add error interceptor to proxy client.
* Fix merge conflict.
* Use Amazon EICE to connect into EC2 instances
This PR adds a new method to SSH into EC2 instances.
It uses the Amazon EC2 Instance Connect Endpoint to do so.
You can read more about the capabilities/limitations of this feature
here
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-using-eice.html
From the Teleport point of view, this uses the AWS OIDC Integration to
call the AWS APIs.
This connection mode is similar to the Agentless mode, where Teleport
connects to the OpenSSH daemon on the target server.
Except, that with this mode there's no required set up on the target
machine (no CAs are being used).
An overview of the flow when the user tries to SSH into a node:
- an SSH Public Key is sent to the target EC2 Instance
- a connection is made to the EC2 Instance Connect Endpoint using
websockets, which in turn pipes the connection to the Private IP of
the EC2 Instance
* review pt1
* pass CA for tests
* add godocs to OpenTunnelEC2
* fetch proxy public address jit
* add docs and move key generation to native package
* remove tcp socket
It was required for compatibilty reasons - v13 proxy could connect to v12 agents,
which didn't support signed PROXY headers. But we don't need it in v14.
#24864 added a dependency of lib/web into tsh which broke windows
tsh builds because lib/web transiently depends on lib/srv which
has linux specific code. This shuffles around a few things so
that lib/web is no longer importing lib/srv at all by:
- Indirectly using the srv.SessionController to apply session
control for web ssh sessions
- Moving the common reversetunnel interfaces into
reversetunnelclient since lib/reversetunnel imports
lib/srv/forward which imports lib/srv.
- Directly converting mysql client errors in the connection
tester instead of calling a common function.
* Add a no-op syslog impl for Windows
* Break tctl's dependency on lib/srv
tctl's auth client depends on the lib/reversetunnel package in order
to dial the auth server via reverse tunnel. This package in turn
depends on lib/srv/forward and lib/srv, which is a bunch of
Linux-specific code for Teleport services (re-exec, etc).
By extracting the client side code for dialing via reverse tunnels
into its own package, we can now build tctl for Windows with the
following incantation:
CGO_ENABLED=1 GOOS=windows GOARCH=amd64 CC=x86_64-w64-mingw32-gcc go build -o tctl.exe ./tool/tctl
* fix connecting to agentless leaf nodes
Pass user and role information to Auth server signing OpenSSH cert
so avoid user/role lookup errors if the target node is on a leaf
cluster. Also use role and trait information from the current
connection rather than the backend to support role impersonation.
* make protobuf changes backwards-compatible
* Convert tsh ssh to use the proxy transport service instead of ssh
In an effort to reduce latency establishing sessions `tsh ssh` is
migrating away from connecting to the Proxy via SSH in favor of
using gRPC. The SSH handshakes with the Proxy increase latency in
situations where the distance between geolocations of the client
and Proxy are large. TLS handshakes used by the gRPC service have
proven to reduce latency by ~20% in the same scenario.
A new `lib/client.ClusterClient` has been introduced that should
be used instead of `lib/client.ProxyClient` to connect to a Teleport
cluster. Most of the functionality within the `ClusterClient` was
a direct copy from the `ProxyClient`.
The `lib/client.TeleportClient` now has a `ConnectToCluster` method
which will connect to both the Proxy and Auth service via the
`api/client.ProxyClient` which first attempts to use gRPC and reverts
back to SSH to preserve backwards compatability. The `ClusterClient`
should be passed around and reused instead of following the
established pattern of `tc.ConnectToProxy` followed by a
`proxy.ConnectToCluster` to get an `auth.ClientI`.
Additionally some of the `agentless` package was refactored to reduce
dependencies and allow it to work with connections to the Proxy that
originated via gRPC instead of SSH.
Changes to the integration tests are mostly to accomodate IP Pinning
and ensure that it works for both connections established via SSH
and gPRC.
This is the final PR needed to complete #19812.
* fix typos and unify span attributes
* pass node name to ConnectToNode
* simplify jump host resetting
Closes#23230
Proxy peer clients now validate that any connections established to
a peer have the expected UUID. This prevents clients from connecting
to incorrect peers if the `peer_public_addr` is shared by multiple
proxies, or is a load balancer.
In the event that a connection is established to an incorrect peer
the connection is closed and a error is written to the logs. The
gRPC connection should be attempted multiple times and eventually
land on the correct peer. However there may be some degradation
of service until the peer connections are all properly established.
The interface smuggling within `stream.ReadWriter.Close` now checks
for `io.Closer` instead of `grpc.ClientStream` to allow clients
that need to perform additoinal closing logic besides sending a
`CloseSend` message to do so. All existing client implementations
of `stream.Source` now implement `io.Closer` so they are still
cleaned up properly.
The multiplexed ssh streams are now tied together to prevent attempts
to send a message at the same time. In addition, the client ssh
streams have been updated to only send a single `CloseSend` on which
ever protocol is terminated first.
* lazily forward SSH agents when connecting to agentless nodes
When connecting to unregistered OpenSSH nodes, the SSH agent is always
forwarded. When connecting to registered OpenSSH (agentless) nodes
however, the SSH agent doesn't *need* to be forwarded, so only do so
if the user explicitly asks to.
* test SSH agent forwarding in agentless integration test
To prevent users from being able to directly connect to registered
OpenSSH (Agentless) nodes, the OpenSSH CA was introduced. When
connecting to Agentless nodes, the Proxy will forward the connection
so the session can be recorded. The forwarding SSH server is configured
to authenticate to the target node with a certificate signed by OpenSSH
CA, which users do not have access to.
Prior to the NodeWatcher being successfully initialized, or when
it became stale any calls to `GetNodes` would return no data or
stale data. This behavior results in the problems described in
#22672.
The NodeWatcher will now fallback to pulling data from the cache
in the event that it has no data or stale data when callers attempt
to retrieve nodes. This should reduce the time that it takes the
a new Proxy or a Proxy with intermittent connectivity issues to
Auth be able to route to nodes.
The impact of this change should be noticeable in tests that
create a Proxy and several Nodes and then immediately try to
open a session on the Nodes.
Closes#22672
The RBAC logic assumes that SSH connections to registered OpenSSH nodes
will always be proxied and terminated, which will come in a future PR.
Part of implementing #19261.
Completes the service implementation of the transportv1.Service.
Ensures that both SSH and SSH Agent protocols are multiplexed
correctly over the gRPC stream via TestService_ProxySSH.
Part of #19812
Provides a client that can be used to connect to and interact with
the transport service in `lib/srv/transport`. The client abstracts
the fact that a gRPC stream is being used for the `net.Conn` created
by `DialCluster` and `DialHost`.
This also moves `lib/utils/grpc/stream` to `api/utils/grpc/stream`
so that the client and server can make use of the same stream
abstractions.
Part of #19812
The first step in implementing the teleport.proxy.v1.ProxyServic
as defined in RFD 100. This includes part of the service
implementation and a stream abstraction to allow transporting
payloads for frames over a gRPC stream.
The proxy peering specific implementation of the stream abstraction
was refactored into `lib/utils/grpc/stream` so that other services
can make use of it.
Part of #19812
Migrates metrics counting proxied connections and connection attempts
from `regular/proxy.go` to `proxy.Router` so that connections via
the web UI, tsh ssh via the Proxy ssh and grpc servers are all
accounted for.
Many of our tests (db package, I'm looking at you) generate many RSA keys. This has two main side effects; makes our tests slow and flaky as CPU usage spikes in random moments when the tests are run in parallel.
This change pre-generates RSA keys at the beginning of each test module and reuse them in randomized order to reduce the situation that one key has been used multiple times in one test.
I had to move a few files to avoid circular dependencies.
* create package to contain proxy peering code
* Refactor proxy routing logic into a reusable object
Routing logic existed within an unexported handler of ssh subsystem
requests, which prevented it from being reused by other components
within the proxy, like the webapi server. This causes significant
latency issues for web sessions because the web apiserver is required
to dial the proxy ssh server to determine how to route to the host.
Since the web apiserver and the proxy ssh server exist in the same
process this is an entirely unnecesarry step that could be avoided
if the routing and ability to established connections were shared
throughout the proxy.
A new `proxy.Router` object is introduced which contains all the
logic that used to exist in `regular.proxySubsys` for determining
how to connect to servers and clusters. All routing within the
`regular.proxySubsys` now leverages the `proxy.Router` to dial
the target.
This is step 1 in addressing #15167. Now that the `proxy.Router`
exists `web.APIServer` will be able to make use of it to avoid
dialing the same process to establish connections.
* Previously additional peer proxies were only attempted if the stream
failed to be created. Now any failure for example a missing reverse
tunnel will also attempt additional peer proxies.
* Shuffle the order in which proxy connections are used.
* configure golangci-lint misspell to check for anglicized spellings
* Americanize spellings
* fix aws constant value with british spelling 🇬🇧
* update api types with americanized spellings
* use american spellings .cloudbuild/scripts
This adds proxy peering support. A configurable setting that allows for agents
to connect to a subset of proxies and be reachable through any proxy in the
cluster. This is achieved by creating grpc connections between each proxy
server. Client connections can then be passed between proxies to the desired
agent.
rosstimothy