* Prevent forwarding traces to servers which don't support tracing
Tracing clients can detect if a server doesn't support tracing by
checking for a trace.NotImplented error in response to an
UploadTraces request. Since the grpc.Conn used by the client is
likely to be bound to that server for the duration of its life
it doesn't make sense to keep trying to forward traces. Instead
the client now remembers that a server doesn't support tracing
and will drop any spans.
Part of #12241
* Span forwarding
Modifies the auth grpc server to implement the OTLP Collector
RegisterTraceServiceServer API (https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/trace/v1/trace_service.proto).
This allows the auth server to receive spans from other services
like `tsh`, `tctl`, and `tbot`. Any spans received by the auth
server will be forwarded to the exporter configured via the
`tracing_service` if it is enabled. All received spans will be
dropped in the event that the`tracing_service` disabled. By
forwarding spans to the auth server, `tsh` doesn't need to
be provided with any of the telemetry backend information
to have its spans exported.
Adds a new `--trace` flag to `tsh` to enable collecting and
forwarding spans to the auth server. When set, the tracing
provider is initialized with a sampling rate of 1.0 to force
all spans to be recorded. Teleport respects the sampling rate
from remote spans, which means that when `--trace` is set, all
spans from `tsh` and any downstream Teleport services will be
recorded and exported regardless of the sampling rate that each
Teleport service is configured with.
rosstimothy