The use of a re-badged buildbox image (rebadging quay.io/gravitational/teleport-buildbox to us-docker.pkg.dev/ci-account/teleport/buildbox-root) is holdover from when CI builds required separate "root user" and "non-root user" images to execute tests under different contexts.
This requirement has since been lifted by using a Go-based build script in CI, and the having to manually invoke a process to re-badge the buildboxes is, tedious, confusing and causing stoppages.
This patch restores the use of buildboxes from quay.io in CI.
* Enable map key sorting in `utils.FastMarshal`
This enables map key sorting during JSON serialization to ensure
`CompareAndSwap` opertions work consistently if the resource in
question has embedded map values (e.g. labels).
Note that this is hypothetically a breaking change: it makes
comparisons consistent, so if it saves a variant that consistently
fails a `CompareAndSwap()`, it will... fail consistently rather
than sometimes work. In practice, today only `CertificateAuthority`
resources use `CompareAndSwap()` and under normal circumstances they
never have labels attached.
In the unlikely event that some resource is affected, a load/store
should save it with sorted key ordering and fix the problem.
* Log a warning if CAs fail a compare-and-swap with multiple labels
This attempts to mitigate compare-and-swap failures if CAs have
multiple labels by logging a warning instructing users to re-save
their CertificateAuthority resource.
* Fix copied comment typo and make the logged warning actionable
* Fix log.Warn() call with formatting directives
* Dynamically resolve reverse tunnel address
The reverse tunnel address is currently a static string that is
retrieved from config and passed around for the duration of a
services lifetime. When the `tunnel_public_address` is changed
on the proxy and the proxy is then restarted, all established
reverse tunnels over the old address will fail indefinintely.
As a means to get around this, #8102 introduced a mechanism
that would cause nodes to restart if their connection to the
auth server was down for a period of time. While this did
allow the nodes to pickup the new address after the nodes
restarted it was meant to be a stop gap until a more robust
solution could be applid.
Instead of using a static address, the reverse tunnel address
is now resolved via a `reversetunnel.Resolver`. Anywhere that
previoulsy relied on the static proxy address now will fetch
the actual reverse tunnel address via the webclient by using
the Resolver. In addition this builds on the refactoring done
in #4290 to further simplify the reversetunnel package. Since
we no longer track multiple proxies, all the left over bits
that did so have been removed to accomodate using a dynamic
reverse tunnel address.
Prior to this change, the entire Teleport process would crash if
it is unable to connect to LDAP on startup.
Instead, we perform retries so that the service doesn't fail if
an LDAP client cannot be initialized. Initialization failures could
occur due to temporary outages of the LDAP server, misconfiguration,
or because Teleport Enterprise is running and the auth server is not
licensed to generate Windows certs.
This has the added side effect of re-publishing the Windows CRL every
time we re-issue the LDAP client certificate, which will ensure it stays
up to date, even in the face of CA rotations.
Fixes#8527
Actually tracking down the cause of a failure in the integration tests can
be hard:
* It's hard to get an overall summary of what failed
* The tests sometimes emit no output before timing out, meaning any
diagnostic info is lost
* The emitted logs are too voluminous for a human to parse
* The emitted logs can present information out of order
* It's often hard to tell where the output from one test ends
and the next one begins
This patch attempts to address these concerns without attempting to rewrite
any of the underlying teleport logging.
* It improves the render-tests script to (optionally) report progress per-
test, rather than on a per-package basis. My working hypothesis on the
tests that time out with no output is that go test ./integration is
waiting for the entire set of integration tests tests to be complete
before reporting success or failure. Reporting on a per-test cycle gives
faster feedback and means that any timed-out builds should give at least
some idea of where they are stuck.
* Adds the render-tests filter to the integration and integration-root make
targets. This will show an overall summary of test results, as well as
- Discarding log output from passing tests to increase signal-to-noise
ratio, and
- Strongly delimiting the output from each failed test, making failures
easier to find.
* Removes the notion of a failure-only logger in favour of post-processing
the log events with render-tests. The failure-only logger catches log
output from the tests and only forwards it to the console if the test
fails. Unfortunately, not all log output is guaranteed to pass through
this logger (some teleport packages do not honour the configured logger,
and reports from the go race detector certainly don't), meaning some
output is presented at the time it happens, and other output is batched
and displayed at the end of the test. This makes working out what
happened where harder than it need be.
In addition, this patch also promotes the render-tests script into a fully-
fledged program, with appropriate makefile targets, make clean support, etc.
It is now also more robust in the face on non-JSON output from go test
(which happens if a package fails to compile).
This PR likely won't fix any of the flakiness, but should help in
debugging:
- Wait for TeleInstance's process to close
- Don't delete the data directory before the process is shut down
- Include site name in logging to make it easier to distinguish which
site is shutting down
- Don't call StopAll twice (it was deferred and run manually)
- Include cluster name in log output
- Remove unused TestWrapper left over from the Check framework
Move cache and resourceWatcher watchers from a 10s retry to a jittered backoff retry up to ~1min. Replace the
reconnectToAuthService interval with a retry to add jitter and backoff there as well for when a node restarts due to
changes introduced in #8102.
Fixes#6889.
Prior to this change, if the Duration of a Linear Retry was greater
than the configured Max, then the Max value would be returned - regardless
of whether or not a Jitter was provided. This modification applies any provided
Jitter to the Max value in the event that the calculated Duration is greater
than the Max.
Fixes https://github.com/gravitational/teleport-private/issues/79
LAT-APP21-4: DOS - Goroutine leak in app server
Prevent the app server's HandleConnection from blocking for
every connection until the server closes. This change blocks
only until the connection is closed.
Fixes https://github.com/gravitational/teleport-private/issues/80
LAT-APP21-5: Insecure random number generation
- updated rand call from rand.Reader.Read to rand.Read
- changed length parameter name from len to l.
- changed byte slice var name to b.
- updated godoc
This new feature in Go 1.17 automatically restores the environment
variable to its previous value when a test ends, making it simpler
to set up the environment for tests and less likely that we accidentally
leave behind global state.
Also convert some of the remaining uses of check to standard Go tests.
Implements RFD 45 / "where" conditions for active sessions[1].
In few words, the purpose of the RFD is to allow the creation of roles that
permits users to only join a subset of active sessions (for example, only their
own sessions).
Implementation goes a bit further than the RFD, allowing the conditions to be
applied to `update` and `delete` verbs as well.
Originally implemented by @andrejtokarcik (#8568), tweaks by @codingllama.
[1] https://github.com/gravitational/teleport/blob/master/rfd/0045-ssh_session-where-condition.md
* Implement where conditions for active sessions list/read
* actionWithConditionForList => actionForListWithCondition
* Make Context-exposed sessions follow the RFD API
* Add tests for "where" conditions on active sessions
* Fix typos
* Fix typos and spacing
* Rename "parties" to "participants" in the context session
* Update RFD to reflect PR changes
Update RFD to reflect PR changes
Specifically, mark as implemented and rename `parties` to `participants`.
* Push list authz logic to ServerWithRoles, obsolete cond
* Remove cond from GetSessions signature
* Simplify cast in lib.utils.Fields.GetString
* Add TODO to refactor SearchSessionEvents / stored sessions
Co-authored-by: Andrej Tokarčík <andrej@goteleport.com>
Fixes#7606, where a node doesn't notice when the tunnel port changes.
Imagine you have a cluster with a node connected in via a tunnel through a proxy `proxy.example.com` on port `3024`
Now change the proxy config so that `tunnel_public_address` is `proxy.example.com:4024`. You either restart the proxy, or reload the proxy config with a `SIGHUP`.
...and then the node
a) loses its connection to auth (because the tunnel is gone), and
b) _doesn't reconnect_, because even though the proxy address hasn't changed,
the node has cached the old tunnel_public_address and keeps trying to connect
to that.
You can always manually restart the node to have it reconnect, but that would be a pain if you have thousands of nodes.
In order to not have to manually restart all nodes, this change implements a check for a connection failures to the auth server, and re-starts the node if there are multiple connection failures in a given period of time. The check as-implemented piggybacks on the node's "common.rotate" service, which can already restart the node in certain circumstances, and uses the success of the periodic rotation sync as a proxy for the health of the node's connection to the auth server.
See-Also: #7606
The race condition detector is being tripped by a concurrent `Write` and
`Close` in the `PipeNetCon` in several integration tests. This is a naive
fix to serialize the write and close operations to resolve the race
condition.
The affected tests were also not handling asynchronous error reporting
correctly (i.e. it's not legal to call `require.XYZ()` from a goroutine
other than the one executing the test function.). This patch introduces
some plumbing to marshal asynchronous errors back into the main test
routine before failing the test.
- Don't assume an explicit $GOPATH is set
- Remove golint from linters - it's been deprecated for over a year
and golangci-lint prints a warning instead of running it.
Prior to this patch, the `LoadBalancer` method `nextBackend()` returned
a pointer to an element of the `LoadBalancer`'s `backends` slice.
It is possible for the element referenced by that pointer to be overwritten
during backend removal (see loadbalancer.go:120). Even though the backend
removal itself is carried out under a lock, it is possible for code
dereferencing a pointer returned by `nextBackend() to get an unexpectedly
modified value.
This patch changes `nextBackend()` to return by value, severing the link
between the returned value and the backing store.
This is a collection of a few small changes related to user presentation of
WebAuthn/MFA in tsh. The intent is to make tsh language match ongoing Web UI
changes.
* Make use of preferred MFA in `tsh mfa add`
* Tweak prompt error message
Old:
ERROR: "U2F\n" is not a valid option, please specify one of [TOTP, WEBAUTHN]
New:
ERROR: "U2F" is not a valid option, please specify one of [TOTP, WEBAUTHN]
* Directly mention WebAuthn when prompting for challenges
* Fix typo on godoc
* Print devices sorted by name on `tsh mfa ls`
* Address origin validation TODOs
For registration and a few other use cases the original error is relayed
back to the client, so there is already a good indicator that it failed
due to origin woes.
For login we purposefully obsfucate errors. To address that I've added a
few debug-level server-side log statements; it seems best to not make
further changes in this case.
* Amend preferred device type logic
* Adjust PromptMFAChallenge message
This patch will allow the web api to (optionally) fall back from HTTPS
to HTTP under very particular circumstances:
1. The appropriate `insecure` flag was set, and
2. The target host is the loopback
If both conditions are met, this patch will allow the WebAPI client to
fall back to plain HTTP when attempting to login to the auth server.
* Disable colorized error formatting on Windows
This disables colorized error formatting on Windows. While recent
versions of Windows can render ANSI sequences, this functionality
needs to be explicitly enabled at runtime and then reset before
exiting. As it stands today, Windows clients see the raw ANSI
escapes, for example:
←[31mERROR: ←[0mfailed connecting to node example [...]
Due to compatibility concerns and `tsh`'s lack of a single exit
point, this just disables ANSI escapes when
`runtime.GOOS == "windows"`. The `ERROR:` text won't be colorized,
but also won't be surrounded by garbage characters.
* Add TODO note about why we disable colors on Windows.
Introduce new make targets to check and add license headers to files
("make lint-license" and "make fix-license"). License checking is now a part of
"make lint" as well.
Initial attempts used goheader, but it caused "make lint-go" to become about 9x
slower (if not more), plus it only targets go files. Google's addlicense is fast
enough and targets however many file types we want.
Existing files that were missing licenses got the header added, using the
current year as the license date.
* Introduce lint-license and fix-license make targets
* Ignore generated files
* Add license to go files
* Replace irregular licenses with standard copyright/license
* Add license to proto files
* Install addlicense in build.assets Dockerfile
Marek Smoliński