Tracing handlers consume a large amount of memory over time, which
makes using one per route suboptimal (See https://github.com/open-telemetry/opentelemetry-go-contrib/issues/2413).
Wrapping all `http.Server` handlers with a tracing handler provides
the same tracing abilities for a fraction of the memory usage. A new
span formatter is added which allows servers to still contain the
request span in the span name.
* Prevent ssh.Session SendRequest from wrapping payload twice
Payloads were inadvertently being wrapped twice by sessions making
the ssh server unable to parse the request payload. This is largely
only an issue for window-change requests, as they are the only request
being sent directly via a session that contains a payload. All window-change
requests now use ssh.Session WindowChange instead or sending the request
manually.
* Add spans to TeleportClient/ProxyClient/NodeSession
* Improve dialer tracing
* Alter default http client span name to include path
* propagate tracing options in NewClientConn
Adds a wrapper around `ssh.Session` which injects tracing context
in a similar manner to the `ssh.Client` wrapper. All usages of
`ssh.Session` have now been replaced and have the appropriate
`context.Context` passed along
Part of #12241
* SSH request tracing
Add tracing support for ssh global requests and channels. Wrappers
for `ssh.Client`, `ssh.Channel`, and `ssh.NewChannel` provide a
mechanism for tracing context to be propagated via a `context.Context`.
In order to maintain backwards compatibility the ssh.Client wrapper
tries to open a TracingChannel when constructed. Any servers that
don't support tracing will reject the unknown channel. The client
will only provide tracing context to servers which do NOT reject
the TracingChannel request.
In order to include pass tracing context along all ssh payloads
are wrapped in an Envelope that includes the original payload
AND any tracing context. Servers now try to unmarshal all payloads
into said Envelope when processing messages. If there is an Envelope
provided, a new span will be created and the original payload will
be pass along to handlers.
Part of #12241
* Prevent forwarding traces to servers which don't support tracing
Tracing clients can detect if a server doesn't support tracing by
checking for a trace.NotImplented error in response to an
UploadTraces request. Since the grpc.Conn used by the client is
likely to be bound to that server for the duration of its life
it doesn't make sense to keep trying to forward traces. Instead
the client now remembers that a server doesn't support tracing
and will drop any spans.
Part of #12241
* Span forwarding
Modifies the auth grpc server to implement the OTLP Collector
RegisterTraceServiceServer API (https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/trace/v1/trace_service.proto).
This allows the auth server to receive spans from other services
like `tsh`, `tctl`, and `tbot`. Any spans received by the auth
server will be forwarded to the exporter configured via the
`tracing_service` if it is enabled. All received spans will be
dropped in the event that the`tracing_service` disabled. By
forwarding spans to the auth server, `tsh` doesn't need to
be provided with any of the telemetry backend information
to have its spans exported.
Adds a new `--trace` flag to `tsh` to enable collecting and
forwarding spans to the auth server. When set, the tracing
provider is initialized with a sampling rate of 1.0 to force
all spans to be recorded. Teleport respects the sampling rate
from remote spans, which means that when `--trace` is set, all
spans from `tsh` and any downstream Teleport services will be
recorded and exported regardless of the sampling rate that each
Teleport service is configured with.
* Add tracing instrumentation for ssh clients/servers
Add tracing context to the existing ProxyHelloSignature to provide
span information across ssh connections. To add span context per
ssh session on top of new connections, the same tracing context is
passed in the first global request of the session.
In order to ensure that tracing context is pulled from and inserted
into the proper context.Context, some interfaces and methods were
changed to take one as the first argument.
rosstimothy