The reqs member of the mock Firestore Server used in the `TestDeleteDocuments`
test was occasionally causing a data race. Accesses to this field have now
been wrapped in a lock.
* Add Yubikey PrivateKey implementation for use by Teleport clients.
- Add yubikey login logic, reusing previously stored private keys.
- Fix identity file decoding with PIV keys, which sign ecdsa certificates.
- Add libpcsclite-dev pre-req for building on linux.
- Remove unnecessary keys.Signer interface and move its functionality to keys.PrivateKey.
- Move retry and jitter utils to new api/utils/retryutils package.
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
Firestore only allows 500 writes per commit. Prior to this change
DeleteRange or purgeExpiredDocuments could result in deleting
more than 500 documents and would result in an error. To comply with
the limits both DeletRange and purgeExpiredDocuments now call
deleteDocuments which properly handles limiting a batch to 500 writes
and commiting multiple times until all documents have been removed.
Fixes#12007
The Migrate method on the Backend interface was not implemented by any
backends.
Migration should be implemented in the New method of backends so they
can be sure migration happens before any background processes are
started.
This change originally intended to improve the reliability of the FirestoreDB unit tests. These tests run against a local fierestore emulator, and after much wailing and gnashing of teeth, I have come to the conclusion that the emulator is not a good match for what we need it to do.
The firetore tests are currently providing negative value, so I have disabled their execution. They can be enabled by defining the `TELEPORT_FIRESTORE_TEST` environment var (in the same manner as the `etcd` tests). This way, the tests will still be compiled so we can at least detect any obviously breaking changes.
Also; previously to this patch the DynamoDB tests were not compiled without the `dynamodb` build tag being set. To bring the DynamoDB backend into line with the others, the DynamoDB tests are now automatically compiled, but are skipped by default during a test run. They can be enabled by defining the TELEPORT_DYNAMODB_TEST
This patch still contains some Firestore test cleanup and additional commentary from the original patch, but that is not the main point of the change.
In order to get better visibility into the backend database tests using
the standard Go tooling, this changeset ports the backend tests away
from `Check`, and into subtests & `testify` for assertions.
This change means that individual sub-tests
1. can be more easily identified in the json test logs, and
2. can be more easily run individually from the command line
During this port I also discovered that some tests are using the fake
clocks incorrectly, which may be a cause of some of our flaky etcd
tests.
Introduce new make targets to check and add license headers to files
("make lint-license" and "make fix-license"). License checking is now a part of
"make lint" as well.
Initial attempts used goheader, but it caused "make lint-go" to become about 9x
slower (if not more), plus it only targets go files. Google's addlicense is fast
enough and targets however many file types we want.
Existing files that were missing licenses got the header added, using the
current year as the license date.
* Introduce lint-license and fix-license make targets
* Ignore generated files
* Add license to go files
* Replace irregular licenses with standard copyright/license
* Add license to proto files
* Install addlicense in build.assets Dockerfile
* Evaluate watcher events to decide whether keep-alives are effective
instead of relying on arbitrary TTLs (implemented as absolute time which
adds to trouble).
Fixes https://github.com/gravitational/teleport/issues/5346.
* Replace the approximate expire timestamps comparisons with the ordering tests
* Address review comments. Move ordered keep-alive tests back to backend/test/suite
* Use an alternative implementation of FakeClock.Advance for etcd to use real time.Sleep as etcd server cannot use fakeclock
* Address review comments
* Use fake clock in firestore tests
* Add missing import
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
This comit fixes#4508
Gogoproto is not compatible with APIv2 protoc-gen-go.
Track the issue here: https://github.com/gogo/protobuf/issues/678
Meanwhile, this commit switches to google protobuf to unmarshal firebase struct.
Add a missing method EmitAuditEvent causing teleport to crash
with firestore events backend.
Same as values, `backend.Item` passes keys as `[]byte` and has no
guarantees about the encoding. GetRange queries need extra care because
of this type mismatch too: `where` clauses are type-sensitive.
Firestore marshaler requires all string data to be valid UTF-8. Our
values are sometimes binary (like QR codes for OTP signup), which would
fail to marhal.
On reads, try using both new and old formats to support existing data.
Added tests for this fallback and for writing binary data into all
backends.
https://github.com/gravitational/teleport/pull/3766 removed a nil
pointer check when creating firestore indexes. This was a legitimate
check, for when indexes already exist.
Tested this manually. Unit testing is trickier because the firestore
emulator in gcloud doesn't support indexes.
List of fixed items:
```
integration/helpers.go:1279:2 gosimple S1000: should use for range instead of for { select {} }
integration/integration_test.go:144:5 gosimple S1009: should omit nil check; len() for nil slices is defined as zero
integration/integration_test.go:173:5 gosimple S1009: should omit nil check; len() for nil slices is defined as zero
integration/integration_test.go:296:28 gosimple S1019: should use make(chan error) instead
integration/integration_test.go:570:41 gosimple S1019: should use make(chan interface{}) instead
integration/integration_test.go:685:40 gosimple S1019: should use make(chan interface{}) instead
integration/integration_test.go:759:33 gosimple S1019: should use make(chan string) instead
lib/auth/init_test.go:62:2 gosimple S1021: should merge variable declaration with assignment on next line
lib/auth/tls_test.go:1658:22 gosimple S1024: should use time.Until instead of t.Sub(time.Now())
lib/backend/dynamo/dynamodbbk.go:420:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/dynamo/dynamodbbk.go:656:12 gosimple S1039: unnecessary use of fmt.Sprintf
lib/backend/etcdbk/etcd.go:458:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/firestore/firestorebk.go:407:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/lite/lite.go:317:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/lite/lite.go:336:6 gosimple S1004: should use !bytes.Equal(value, expected.Value) instead
lib/backend/memory/memory.go:365:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/memory/memory.go:376:5 gosimple S1004: should use !bytes.Equal(existingItem.Value, expected.Value) instead
lib/backend/test/suite.go:327:10 gosimple S1024: should use time.Until instead of t.Sub(time.Now())
lib/client/api.go:1410:9 gosimple S1003: should use strings.ContainsRune(name, ':') instead
lib/client/api.go:2355:32 gosimple S1019: should use make([]ForwardedPort, len(spec)) instead
lib/client/keyagent_test.go:85:2 gosimple S1021: should merge variable declaration with assignment on next line
lib/client/player.go:54:33 gosimple S1019: should use make(chan int) instead
lib/config/configuration.go:1024:52 gosimple S1019: should use make(services.CommandLabels) instead
lib/config/configuration.go:1025:44 gosimple S1019: should use make(map[string]string) instead
lib/config/configuration.go:930:21 gosimple S1003: should use strings.Contains(clf.Roles, defaults.RoleNode) instead
lib/config/configuration.go:931:22 gosimple S1003: should use strings.Contains(clf.Roles, defaults.RoleAuthService) instead
lib/config/configuration.go:932:23 gosimple S1003: should use strings.Contains(clf.Roles, defaults.RoleProxy) instead
lib/service/supervisor.go:387:2 gosimple S1001: should use copy() instead of a loop
lib/tlsca/parsegen.go:140:9 gosimple S1034: assigning the result of this type assertion to a variable (switch generalKey := generalKey.(type)) could eliminate type assertions in switch cases
lib/utils/certs.go:140:9 gosimple S1034: assigning the result of this type assertion to a variable (switch generalKey := generalKey.(type)) could eliminate type assertions in switch cases
lib/utils/certs.go:167:40 gosimple S1010: should omit second index in slice, s[a:len(s)] is identical to s[a:]
lib/utils/certs.go:204:5 gosimple S1004: should use !bytes.Equal(certificateChain[0].SubjectKeyId, certificateChain[0].AuthorityKeyId) instead
lib/utils/parse/parse.go:116:45 gosimple S1003: should use strings.Contains(variable, "}}") instead
lib/utils/parse/parse.go:116:6 gosimple S1003: should use strings.Contains(variable, "{{") instead
lib/utils/socks/socks.go:192:10 gosimple S1025: should use String() instead of fmt.Sprintf
lib/utils/socks/socks.go:199:10 gosimple S1025: should use String() instead of fmt.Sprintf
lib/web/apiserver.go:1054:18 gosimple S1024: should use time.Until instead of t.Sub(time.Now())
lib/web/apiserver.go:1954:9 gosimple S1039: unnecessary use of fmt.Sprintf
tool/tsh/tsh.go:1193:14 gosimple S1024: should use time.Until instead of t.Sub(time.Now())
```
This code is not caught by linters because it's exported and they assume
there's some external users.
Since teleport is relatively self-contained, we can tell for sure
whether something is called or not.
Spring cleaning!
A very mechanical cleanup using several linters (unused, deadcode,
structcheck). Build and tests still pass so no behavior should be
affected.
Michael Wilson