* prevent SSH session id reuse
If a user sets `TELEPORT_SESSION` to a session ID
that isn't running, the sent session ID is ignored
and a new one is used to prevent the user
controlling the value. When parsing a session ID
the parsed result is used to ensure all session
IDs will have the same format.
`CreateSessionTracker` is updated to create
instead of upsert to ensure `SessionTracker`
resources cannot be overwritten by accident.
Clients often need to know the current session ID,
and before when they were in control of it that
wasn't an issue. To fix this servers will send the
current session ID in a SSH channel request on the
channel the SSH session was created on. This
should be fully backwards compatible, as when an
unknown SSH request type is received clients will
just ignore it/reply with false.
* add test for wrappedSSHConn
* Wire context.Context into `auth/register.go`
* Thread `context.Context` through `auth.Register` and add tracing
* Unwrap error before recording it in the span
Since all supported versions of Teleport now support tracing we can
drop opening a specific channel to evaluate if tracing is supported
and just rely on the SSH server version.
* Add Prometheus metrics to Kubernetes Access
This PR adds several prometheus metrics to the Kubernetes Access
package.
The new metrics are:
| Name | Type | Component | Description |
| ----------------------------------------------------------------- | --------- | --------------------------- | --------------------------------------------------------------------------------------------------------- |
| `teleport_kubernetes_client_in_flight_requests` | gauge | Teleport Kubernetes Service | In-flight requests waiting for the upstream response. |
| `teleport_kubernetes_client_requests_total` | counter | Teleport Kubernetes Service | Total number of requests sent to the upstream teleport proxy, kube_service or Kubernetes Cluster servers. |
| `teleport_kubernetes_client_tls_duration_seconds` | histogram | Teleport Kubernetes Service | Latency distribution of TLS handshakes. |
| `teleport_kubernetes_client_got_conn_duration_seconds` | histogram | Teleport Kubernetes Service | Latency distribution of time to dial to the upstream server - using reversetunnel or direct dialer. |
| `teleport_kubernetes_client_first_byte_response_duration_seconds` | histogram | Teleport Kubernetes Service | Latency distribution of time to receive the first response byte from the upstream server. |
| `teleport_kubernetes_client_request_duration_seconds` | histogram | Teleport Kubernetes Service | Latency distribution of the upstream request time. |
| Name | Type | Component | Description |
| ----------------------------------------------------------- | --------- | --------------------------- | --------------------------------------------------- |
| `teleport_kubernetes_server_in_flight_requests` | gauge | Teleport Kubernetes Service | In-flight requests currently handled by the server. |
| `teleport_kubernetes_server_api_requests_total` | counter | Teleport Kubernetes Service | Total number of requests handled by the server. |
| `teleport_kubernetes_server_request_duration_seconds` | histogram | Teleport Kubernetes Service | Latency distribution of the total request time. |
| `teleport_kubernetes_server_response_size_bytes` | histogram | Teleport Kubernetes Service | Distribution of the response size. |
| `teleport_kubernetes_server_exec_in_flight_sessions` | gauge | Teleport Kubernetes Service | Number of active kubectl exec sessions. |
| `teleport_kubernetes_server_exec_sessions_total` | counter | Teleport Kubernetes Service | Total number of kubectl exec sessions. |
| `teleport_kubernetes_server_portforward_in_flight_sessions` | gauge | Teleport Kubernetes Service | Number of active kubectl portforward sessions. |
| `teleport_kubernetes_server_portforward_sessions_total` | counter | Teleport Kubernetes Service | Number of active kubectl portforward sessions. |
| `teleport_kubernetes_server_join_in_flight_sessions` | gauge | Teleport Kubernetes Service | Number of active joining sessions, |
| `teleport_kubernetes_server_join_sessions_total` | counter | Teleport Kubernetes Service | Total number of joining sessions. |
Closes#29362
Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
* replace prometheus.MustRegister with metrics.RegisterPrometheusCollectors
* Apply suggestions from code review
Co-authored-by: Alex Fornuto <alex.fornuto@goteleport.com>
---------
Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
Co-authored-by: Alex Fornuto <alex.fornuto@goteleport.com>
* Reduce time spent setting ssh session envs
`tsh` sets a number of environment variables when setting up the
users session. Each key value pair is transmitted one at a time
in a "env" ssh request, which adds a num envs * RTT of additional
latency per session.
This introduces a new `envs@goteleport.com` request which sets
multiple environment variables in a single ssh request, which
reduces the amount of time spent setting envs down to the RTT of
a single ssh request. In order to ensure backward compat and
interoperability with OpenSSH, if the server does not recognize
the `envs@goteleport.com` request the ssh client will resort to
sending individual "env" requests.
* address feedback
* fix: use a single timer for fallback requests in tests
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* fix: remove extra whitespace
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* fix: gci
---------
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Add Headless Authn service.
* Add/fix 3 minute headless login timeout.
* * Prevent repeated updates to headless authentication state
* Prevent user lock out from headless authentication failure
* Delete headless authentication on failed attempts
* Add auth_with_roles test.
* Extend timeout in test to reduce flakiness.
* Fix error typo.
* Add context timeouts, remove initial GetHeadlessAuthentication call.
* Resolve comments.
* Move http client to it's own file; Add ability to clone HTTP client for per-request configuration changes.
* Fix flaky test.
* Remove shared state from test.
* Update error handling and testing for auth_with_roles.
* Fix rebase misshap.
* Fix race condition in test.
* update e ref
* Fix ctx missing.
* Extend test timeout to prevent flakiness.
* Fix issue with roundtrip.ClientParams not being applied due to roundtripper wrapping.
---------
Co-authored-by: Tim Ross <tim.ross@goteleport.com>
The `http.Client.Transport` created in `newWebClient` wraps a
chain of `http.RoundTripper` over an underlying `http.Transport`.
Since we cannot guarantee that each `http.RoundTipper` has a
`CloseIdleConnections` method the usage of
`defer clt.CloseIdleConnections()` does not guarantee that the http
connections created during Find/Ping/etc are closed.
To prevent leaking connections implementations of `http.RoundTripper`
have added a `CloseIdleConnections` method added that forwards the
request on to the wrapped interface. The `otelhttp.Transport` does
not implement this method either so care has been taken to wrap it
in a `http.RoundTripper` which will call the root transport in
`enforceCloseIdleConnections`. An upstream issue has been filed
with otelhttp: https://github.com/open-telemetry/opentelemetry-go-contrib/issues/3543
to get the method added to their `Transport` implementation.
This wasn't noticed prior to upgrading to go1.20 becuase prior to
[this](https://github.com/golang/go/commit/4e7e7ae1406c70d9cc0809ec11105a55a60a0b70)
commit the `ReadHeaderTimeout` set on the Proxy web api http.Server
would cause the connection to appear idle and get terminated by the
server.
`TestWebClientClosesIdleConnections` was added to capture the leak
in connections as reported in #22757 and prevent any regressions.
* Allow node to handle old and new way of client IP propagation on same listener
With addition of signed PROXY headers, node was listening on multiplexer, but because
of that it couldn't processing incoming connection from older proxies
when ProxyHelloSignature was used, because
both ends were waiting for the other side to send data first.
Here we integrate ability to handle PROXY headers into connection itself,
so we can start ssh server without waiting for multiplexer to detect connection
* Remove unneeded code
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Fix imports
* Unexport function
* Add godoc
* Remove unneeded comment
* Move ProxyHelloSignature to constants
* Check that proxyline is verified
* Use Warn() instead of Warnf()
* Move ProxyHelloSignature to api/constants
* Add timeout for getting host CA during proxyline verification.
* Rearrange conditions
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Clarify comment.
---------
Co-authored-by: Michael Wilson <mike@mdwn.dev>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Ensure ssh connection rejection errors are returned
#18302 altered the connection flow which changed the communication
via ssh. Prior to the change an ssh channel was opened first, then
global requests were sent. After the change a global request is sent
prior to opening a channel. This caused users to no longer receive a
human readable rejection message a la: `ssh: rejected: administratively
prohibited (too many concurrent ssh connections for user`, instead only
an `EOF` was returned.
To ensure that the correct message is provided to users two changes are
made, although only the client side change is strictly required.
1) the tracing ssh client now returns any rejected errors cached from
opening the tracing channel
2) the ssh server now processes global requests during it's wait period
waiting for a channel to reject and send the replies to any requests
with said rejection
`TestLockInForce` was updated to include attempting to send a global
request so both scenarios are covered as order should not matter.
* Forward traces from the web UI
Adds a `/webapi/traces` endpoint to the proxy web handler
to allow receiving traces from the UI so that they can
be forwarded to the configured exporter in the proxy_service.
To accomodate traces coming in via websockets the tracing
handler is updated to pull tracing context first from the
standard headers and fallback to retrieving it from a query
parameter as web sockets from the UI cannot alter headers.
Additionally update `web.Terminal` to propagate the tracing
context properly and instruments some of the functions
to ensure spans from the UI are properly correlated to spans
on the backend.
Tracing handlers consume a large amount of memory over time, which
makes using one per route suboptimal (See https://github.com/open-telemetry/opentelemetry-go-contrib/issues/2413).
Wrapping all `http.Server` handlers with a tracing handler provides
the same tracing abilities for a fraction of the memory usage. A new
span formatter is added which allows servers to still contain the
request span in the span name.
* Prevent ssh.Session SendRequest from wrapping payload twice
Payloads were inadvertently being wrapped twice by sessions making
the ssh server unable to parse the request payload. This is largely
only an issue for window-change requests, as they are the only request
being sent directly via a session that contains a payload. All window-change
requests now use ssh.Session WindowChange instead or sending the request
manually.
* Add spans to TeleportClient/ProxyClient/NodeSession
* Improve dialer tracing
* Alter default http client span name to include path
* propagate tracing options in NewClientConn
Adds a wrapper around `ssh.Session` which injects tracing context
in a similar manner to the `ssh.Client` wrapper. All usages of
`ssh.Session` have now been replaced and have the appropriate
`context.Context` passed along
Part of #12241
* SSH request tracing
Add tracing support for ssh global requests and channels. Wrappers
for `ssh.Client`, `ssh.Channel`, and `ssh.NewChannel` provide a
mechanism for tracing context to be propagated via a `context.Context`.
In order to maintain backwards compatibility the ssh.Client wrapper
tries to open a TracingChannel when constructed. Any servers that
don't support tracing will reject the unknown channel. The client
will only provide tracing context to servers which do NOT reject
the TracingChannel request.
In order to include pass tracing context along all ssh payloads
are wrapped in an Envelope that includes the original payload
AND any tracing context. Servers now try to unmarshal all payloads
into said Envelope when processing messages. If there is an Envelope
provided, a new span will be created and the original payload will
be pass along to handlers.
Part of #12241
* Prevent forwarding traces to servers which don't support tracing
Tracing clients can detect if a server doesn't support tracing by
checking for a trace.NotImplented error in response to an
UploadTraces request. Since the grpc.Conn used by the client is
likely to be bound to that server for the duration of its life
it doesn't make sense to keep trying to forward traces. Instead
the client now remembers that a server doesn't support tracing
and will drop any spans.
Part of #12241
* Span forwarding
Modifies the auth grpc server to implement the OTLP Collector
RegisterTraceServiceServer API (https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/trace/v1/trace_service.proto).
This allows the auth server to receive spans from other services
like `tsh`, `tctl`, and `tbot`. Any spans received by the auth
server will be forwarded to the exporter configured via the
`tracing_service` if it is enabled. All received spans will be
dropped in the event that the`tracing_service` disabled. By
forwarding spans to the auth server, `tsh` doesn't need to
be provided with any of the telemetry backend information
to have its spans exported.
Adds a new `--trace` flag to `tsh` to enable collecting and
forwarding spans to the auth server. When set, the tracing
provider is initialized with a sampling rate of 1.0 to force
all spans to be recorded. Teleport respects the sampling rate
from remote spans, which means that when `--trace` is set, all
spans from `tsh` and any downstream Teleport services will be
recorded and exported regardless of the sampling rate that each
Teleport service is configured with.
* Add tracing instrumentation for ssh clients/servers
Add tracing context to the existing ProxyHelloSignature to provide
span information across ssh connections. To add span context per
ssh session on top of new connections, the same tracing context is
passed in the first global request of the session.
In order to ensure that tracing context is pulled from and inserted
into the proper context.Context, some interfaces and methods were
changed to take one as the first argument.
Nic Klaassen