* Deduplicate multiplexer detection errors over 1-minute windows
This commit deduplicates errors returned by `Mux.detect` (which detects
the type of a new connection just established) over 1-minute windows.
This ensures that health checkers / malicious actors cannot pollute /
overpower the logs with warnings when such connections are invalid or
unknown to the multiplexer.
* Improve error message
* Fix lint
* Add `TestErrorDeduplicator`
* Multi-line Sprintf
* ErrorDeduplicator -> LogLimiter
* Improve `LogLimiter.Log` API
* Fix typo
* Remove `windowInfo`
* Improve test assertions
* Use new log limiter API
* Remove unnecessary append
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Fix `toLogMessages`
* Start log limiter in `Mux.New`
* Use public API in tests
* Make `Config` field private
* Improve godoc
* Remove channel size from log limiter config
* Use `assert` instead of `require` for log assertions
* Revert "Start log limiter in `Mux.New`"
This reverts commit 3554daa6db.
* Use mutex instead of `entryCh`
* Start log limiter goroutine in `loglimit.New`
* Use `context.Background` as default.
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
---------
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Allow node to handle old and new way of client IP propagation on same listener
With addition of signed PROXY headers, node was listening on multiplexer, but because
of that it couldn't processing incoming connection from older proxies
when ProxyHelloSignature was used, because
both ends were waiting for the other side to send data first.
Here we integrate ability to handle PROXY headers into connection itself,
so we can start ssh server without waiting for multiplexer to detect connection
* Remove unneeded code
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Fix imports
* Unexport function
* Add godoc
* Remove unneeded comment
* Move ProxyHelloSignature to constants
* Check that proxyline is verified
* Use Warn() instead of Warnf()
* Move ProxyHelloSignature to api/constants
* Add timeout for getting host CA during proxyline verification.
* Rearrange conditions
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Clarify comment.
---------
Co-authored-by: Michael Wilson <mike@mdwn.dev>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Add secure IP propagation from proxy to auth server when using ALPN
We're using PROXY protocol extensions called TLVs to send
signed JWT and proxy's certificate to the auth server. Auth
validates JWT using provided signing certificate and host CA
to make sure that IP information comes from our internal proxy.
Many of our tests (db package, I'm looking at you) generate many RSA keys. This has two main side effects; makes our tests slow and flaky as CPU usage spikes in random moments when the tests are run in parallel.
This change pre-generates RSA keys at the beginning of each test module and reuse them in randomized order to reduce the situation that one key has been used multiple times in one test.
I had to move a few files to avoid circular dependencies.
* Recognize GSSEncRequest as postgres proto
* Respond to GSSEncRequest with 'N' for 'not supported' in postgres engine
* Add test for GSSEncRequest to postgres via proxy
* Save gssencmode=disable in .pg_service.conf and test it
* Check for repeated startup encryption request
* Refactor postgres startup test code into its own test
* Test covers all code paths in the proxy startup handler
* Make postgres proxy tests run in parallel
* Reduced test execution time from 15 seconds to 4.5 seconds.
* Ran with race detector and all tests passed.
* Make sure we read at least one byte
* Fix lint
* Fix lint
* Change err to bad parameter
* Update test
Moves from github.com/golang/protobuf protoc-gen-go plugin to google.golang.org/
plugins.
This change was a long-time coming, but is now possible to do since our
dependencies are up-to-date.
* Move away from deprecated protoc-gen-go plugin
* Embed unimplemented server in handler.Handler
* Embed unimplemented server in multiplexer_test.go
* Update generated protos
Add definitions for Device Trust RPCs.
DeviceTrustService is fairly isolated from other services, so it is generated
using plain `protoc-gen-go` instead of Gogo.
teleport.e#514
* Add Device Trust proto definitions
* Allow proto generation without gogo
* Drop Gogo from lib/multiplexer protos
* Update generated protos
* Tidy modules
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
Similarly to #15856, moves lib/ protos (except for Connect) under the proto/ tree.
The intended layout is a mirror of api:
```
teleport-root
proto/ <- root for protos and proto imports
teleport/
lib/...
gen/proto/go/ <- ideal root for generated protos
```
#15187
* Move lib/ protos under the proto/ tree
* Reformat protos
* Update generated protos
* Add additional debug logging for instantiation of ssh server conn
* Rough out introducing mux to ssh listener
* Catch errors from multiplexer serve
* Adapt multiplexer to support ProxyHelloSignature
* Adjust logging for ssh conn creation
* Adjust tests to use semantically correct assert/require helpers
* Address PR comments
* Check Client addr is properly propagated in mux test
Previously, 8 bytes was always passed into 'detectProto'. But assumptions
were made that the input was only 3 bytes, causing 'detectProto' to match
against inputs that were prefixed with a part of a magic byte sequence.
Magic byte sequences by definition need to be fully present in order to be
effective, so don't match against truncated magic byte sequences, and
do an extra read if necessary to match the ProxyV2 protocol. The length of
the magic bytes of that protocol is 12 bytes, but many other protocols will
only send 8 bytes at first, so only do an extra read if we're reasonably sure
the input is using ProxyV2 to avoid EOF errors.
#11684 added support for proxy protocol v2 for SSH and Postgres but MySQL uses different code path and it was missing. This change fixes that.
It also adds tests for v2 protocol support for MySQL, Postgres, Mongo and Redis
* Remove unused backend wrapper from Cache
* Remove double printShutdownStatus
* Fix readyz race condition
* Test coverage for the readyz.monitor fix
* Close listeners immediately in proxy.shutdown
* Use and handle net.ErrClosed correctly
This adapts utils.IsUseOfClosedNetworkError to check for net.ErrClosed
even inside trace.Aggregate errors, makes it so that we always return
something that would pass errors.Is(err, net.ErrClosed) when returning
from a (net.Listener).Accept(), and handles closed listeners within our
various Serve() loops so that we don't hit spurious backoff waits while
shutting down.
* Close listeners early and emitters late
* Test coverage for the proxy listener changes
* Revert some errors back to trace.ConnectionProblem
* Reduce PR scope to just the proxy, add comments
* Improve error logging.
Some tests in multiplexer passed connections to Mux before registering
to handle a specific type of connection (TLS, Postgres, etc). When the
Mux Serve function handles the connection before the type of connection
is registered to be handled, it will close the connection.
Race condition:
1. create Mux
2. send new connection to Mux listener
3. register DB listener and receive conn
Fixed race:
1. create Mux
2. register DB listener
3. send new connection to Mux listener
4. receive conn from DB listener
Fixesgravitational/teleport-private#78
LAT-APP21-3
Change the multiplexer from opt-out to opt-in for protocol listeners.
The multiplexer previously always created new listeners for each protocol (SSH, TLS, DB) and its Config contained opt-out configurations (DisableTLS, DisableSSH, DisablePostgres). When callers didn't explicitly disable a protocol, new connections for that protocol would never close and leak a goroutine. This exposed a 3-line DoS whereby multiple connections could be passed to the multiplexer for a protocol that was not being serviced, eventually resulting in file descriptor limits being hit, which then prevented the teleport process from operating (see example in issue).
Changing to opt-in means new protocols can be added to the multiplexer without requiring all existing callers to be changed (to opt-out of the new protocol). Forgetting to opt-out would expose new code to DoS without compile-time, test, or operational notification.
Rather than rely on opt-out flags in the config, this change creates listeners only when explicitly requested by callers. The existing getter methods on multiplexer were changed to create listeners when called. And multiplexer protocol detection now closes connections when a listener hasn't been created. This also allowed for the protocol detection routine to be simplified.
Introduce new make targets to check and add license headers to files
("make lint-license" and "make fix-license"). License checking is now a part of
"make lint" as well.
Initial attempts used goheader, but it caused "make lint-go" to become about 9x
slower (if not more), plus it only targets go files. Google's addlicense is fast
enough and targets however many file types we want.
Existing files that were missing licenses got the header added, using the
current year as the license date.
* Introduce lint-license and fix-license make targets
* Ignore generated files
* Add license to go files
* Replace irregular licenses with standard copyright/license
* Add license to proto files
* Install addlicense in build.assets Dockerfile
Updated TLS handshake timeout. During some operations, Teleport can
flood the network with traffic which causes the TLS handshake to occur
slower than 1 second.
One example is during SSO login. The initial connection is an
unauthenticated connection, and upon successful SSO login a "types.User"
is created and replicated to all nodes. For large clusters this can mean
10k+ "types.User" objects getting replicated at the same time the user
attempts to re-establishing another connection to Auth this time with
valid identity credentials. This connection sometimes can take longer
than the original 1 second timeout.
An extra dockerfile for gRPC generation is extra maintenance burden. It
was also using a really old base image that has a ton of known vulns.
Also, update GOGO_PROTO_TAG to match the version we have vendored via
go.mod.
Each user can now have multiple devices. This commit only changes the
backend structure to support it, the client and API haven't been updated
yet.
Also added a migration for existing MFA data on auth server startup.
`require` is a sister package to `assert` that terminates the test on
failure. `assert` records the failure but lets the test proceed, which
is un-intuitive.
Also update all existing tests to match.
This commit introduces GRPC API for streaming sessions.
It adds structured events and sync streaming
that avoids storing events on disk.
You can find design in rfd/0002-streaming.md RFD.
Changes agent channel setup behavior to be consistent
openssh by having servers lazily request agent channels
when they are needed, rather than immediately starting a
single connection-wide channel as soon as forwarding is
requested. Fixes an issue introduced in #3613 which
caused openssh clients to hang on exit due to persistent
agent channel.
Changes the lifetime of agent forwarding to be scoped
to the underlying ssh connection, instead of the
specific ssh channel which initially passed the agent
forwarding request.
