Sets `http.Server.IdleTimeout` and `http.Client.IdleConnTimeout`
on clients and servers which didn't have them set. A default of
360s was chosen to be on par with the default of an NLB without
being identical.
This was added as another safety measure to prevent leaking any
idle connections indefinitely as seen in #22757.
`apidefaults.DefaultDialTimeout` was also renamed to
`apidefaults.DefaultIOTimeout` to better reflect its usage.
When performing a KeepAlive the key was first retrieved from
etcd to verify that it existed before updating the item. However,
a Put operation with the `ignore_value` set will return a not
found error if the key does not exist
(https://etcd.io/docs/v3.4/learning/api/#put). So we can safely
remove the get and rely on etcd to prevent updating a key that
does not exist.
Also converts `concurrentqueue.Queue` to be generic over the given
Input and Output of the worker function.
Since #16636 etcd leases have been bucketed, yet when watching events
we still determine the ttl for each particular item in `fromEvent`
by retrieving the ttl from etcd. To reduce the number of ttl requests
sent to etcd we now cache the ttl per lease id.
Closes#21380
If all etcd nodes are unreachable, Teleport exits with the following
message:
```
ERROR: initialization failed
context deadline exceeded
```
With this commit we get instead:
```
ERROR: timed out dialing etcd endpoints: [https://127.0.0.1:2381https://127.0.0.1:12381]
initialization failed
context deadline exceeded
```
Note that if only some of the etcd nodes are unreachable, the etcd
client already provides a useful error message:
```
ERROR: initialization failed
failed to dial endpoint https://127.0.0.1:12381 with maintenance client: context deadline exceeded
```
From the etcd docs:
> To obtain lower latencies and higher throughput for read requests
> clients can configure a request’s consistency mode to serializable,
> which may access stale data with respect to quorum, but removes the
> performance penalty of linearized accesses' reliance on live consensus.
By leveraging serializable mode it is entirely possible to retrieve
stale data. This means that a Create followed by a Get could result
in a NotFound error, which results in issues like #13276. While this
improves latency it breaks the consistency guarantees required by
Teleport.
To ensure that our consistency requirements are met all etcd operations
are converted to use the default linearized mode by removing the
`WithSerializable` option.
Closes#13276.
* configure golangci-lint misspell to check for anglicized spellings
* Americanize spellings
* fix aws constant value with british spelling 🇬🇧
* update api types with americanized spellings
* use american spellings .cloudbuild/scripts
The Migrate method on the Backend interface was not implemented by any
backends.
Migration should be implemented in the New method of backends so they
can be sure migration happens before any background processes are
started.
* Prepare `etcdbk` to use the newer `go.etcd.io/etcd/v3` lib
* `go get go.etcd.io/etcd/api/v3 go.etcd.io/etcd/client/v3`
* Fix direct/indirect split in go.mod
In order to get better visibility into the backend database tests using
the standard Go tooling, this changeset ports the backend tests away
from `Check`, and into subtests & `testify` for assertions.
This change means that individual sub-tests
1. can be more easily identified in the json test logs, and
2. can be more easily run individually from the command line
During this port I also discovered that some tests are using the fake
clocks incorrectly, which may be a cause of some of our flaky etcd
tests.
* Evaluate watcher events to decide whether keep-alives are effective
instead of relying on arbitrary TTLs (implemented as absolute time which
adds to trouble).
Fixes https://github.com/gravitational/teleport/issues/5346.
* Replace the approximate expire timestamps comparisons with the ordering tests
* Address review comments. Move ordered keep-alive tests back to backend/test/suite
* Use an alternative implementation of FakeClock.Advance for etcd to use real time.Sleep as etcd server cannot use fakeclock
* Address review comments
* Use fake clock in firestore tests
* Add missing import
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
* Fix local etcd test failures when etcd is not running
* Add kubernetes_service to teleport.yaml
This plumbs config fields only, they have no effect yet.
Also, remove `cluster_name` from `proxy_config.kubernetes`. This field
will only exist under `kubernetes_service` per
https://github.com/gravitational/teleport/pull/4455
* Handle IPv6 in kubernetes_service and rename label fields
* Disable k8s cluster name defaulting in user TLS certs
Need to implement service registration first.
When custom prefix is `/foo/`, the migration would move `/teleport/a` to
`/foo/a`, but the backend later tries to read `/foo//a`.
Also added tests to cover these edge cases.
The prefix fetching logic has a bug: it treats everything starting with
`/teleport` as the legacy prefix data, even if it's `/teleport-foo/bar`.
This is an issue if user specifies `/teleport-foo` as their custom
prefix. Each restart will copy the data from `/teleport-foo/...` to
`/teleport-foo-foo/...`.
Set the legacy prefix const to `/teleport/` instead. This avoids
excessive copying during startup.
Prefixes can still be confused later on, with `Watch` and `GetRange`,
but this is harder to migrate with backwards-compatibility.
This code is only intended for 4.3 and will be removed in 4.4.
The goal is to automatically move data to the correct etcd prefix that
the customer has specified in their config. Prior to 4.3, their data was
always written to `/teleport` prefix, ignoring the config.
The copy happens iff:
- /teleport is not empty
- config prefix is empty OR older than /teleport
This double-writing will only exist in 4.3 and the config prefix is
still source-of-truth for reads.
Double-writes are necessary to allow users to downgrade to 4.2 and back
up again without losing data.
rosstimothy