* build.assets Dockerfiles: Remove unnecessary ENV NODE_URL
NODE_URL is being redefined within the RUN instruction anyway. We suspect
it might be causing problems because sometimes the logs from build failures
suggest that the NODE_URL export was either ignored or ${NODE_URL} passed
to curl reads ENV NODE_URL and not the env var set within the shell.
* Pass fsSl flags to curl
Add a couple of parameters for the developer key ID and bundle ID for
signing/notarizing binaries. Keep the hard-coded values as defaults for
now, but we will remove these soon when all the call sites of the tool
have been updated to pass these values.
We want to parameterize these values so we can use different signing
keys in GitHub Actions and to make the tool agnostic to which binaries
it is signing.
The HTTP client created by `auth.NewClient` has an idle timeout of
30s and each instance periodically gets cert authorities in
`TeleportProcess.syncRotationStateCycle`. If there has been no
activity when the ticker expires then the request will result in a
brand new connection to Auth to be established. This can cause a
spikes in open file descriptors on the Proxy when there are a large
number of instances connected via a tunnel.
To prevent periodic dialing when retrieving cert authorities the
endpoint has been migrated to the new Trust gRPC service. For now
only the get endpoint has been migrated but the rest of the CRUD
operations for cert authorities should be migrated from the HTTP
api to the gRPC service as well.
* Emit new `AgentMetadataEvent`
Part of https://github.com/gravitational/cloud/issues/3550.
This commits adds a new `UpstreamInventoryAgentMetadata` that is sent
from a Teleport agent to an auth server.
Once received, it is transformed into an `AgentMetadataEvent` and sent
to PreHog.
Most `UpstreamInventoryAgentMetadata` fields are intentionally kept as
empty in this PR. Follow up PRs will be opened with the mechanisms
required to fill them (as described in #21337).
* server_id -> host_id
* compute OS and host architecture
* Compute OS version and container runtime
* Close stream if agent receives an agent metadata message
* cmd -> exec ; file -> read
* implement fetchOSVersion for linux
* Remove unused import
* Add note about `agentMetadataCh`
* Allow commands with args
* Remove parseFun abstraction
* fetch glibc version
* fetch container orchestrator
* Fix lint
* Fix TODO
* Add note about glibc version
* cmd -> command
* fetch cloud environment
* fetch install methods
* GLibCVersion -> GlibcVersion
* Use `http.NewRequestWithContext`
* Add missing comment
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* GLibCVersion -> GlibcVersion
* Fix lint
* Fix helm unit tests
* Add missing comments
* 5 second timeout on http requests
* Spawn gorountine that fetches metadata on each new stream
* Use `defaults.HTTPClient()`
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Add missing import
* Handle error
* Revert "Spawn gorountine that fetches metadata on each new stream"
This reverts commit fe4f2790be.
* Send agent metadata to auth server once per stream
* Improve note about agentMetadataDone
* Don't process command output & file content on the agent
Since agents cannot be trusted, regex validation and sanitization should
happen in PreHog anyways. So this commit removes such logic in favor of
moving it to PreHog.
* Fix lint
* Trim space
* Move handling of `AgentMetadataEvent` to `handleControlStream`
* Use cached hello message
* Move metadata files to lib/inventory/metadata
* make sending of agent metadata more self contained
* Minimize diff
* Send all system roles to PreHog
* Remove unused import
* Add parsing of command output / file content back
* Usage reporter refactor
* Usage reporter refactor
* Add missing handling of inventory agent metadata msg
* Fix ICS usage reporter
* Improve comments
* Add cached `metadata.Get*` methods
* Use systemctl status instead of is-active
* Add `Metadata` struct
* return pointer in `FetchMetadata`
* Pass context to `GetMetadata`
* metadataFetchConfig -> fetchConfig
* GetMetadata -> Get
* Add note about `Get` result
* Ensure install methods are non-nil
* Exit `metadata.Get` if context is closed
* Replace sync.Once with atomic.Bool.Swap
* Initialize channel
* Fix lint
* Fix lint
* Make `metadata.Get` return an error instead of bool
* Allow multiple true/false values for env vars
* Use `strings.Cut`
* Use /etc/os-release ID instead of NAME
* Improve `autoEmitMetadata`
* Use `gnu_get_libc_version`
* Ubuntu -> ubuntu
* Use GOARCH
* gofmt
* Move import C up
* Variables may include quotes
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Default values for ID and VERSION_ID
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Blank lines are permitted
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Anonymize host id
---------
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Introduce Okta gRPC and client interfaces.
To support the upcoming Okta integration, Okta gRPC calls and client
interfaces have been added. This utilizes a non-legacy proto to try to
get away from our reliance on gogo.
This commit does NOT add the OktaImportRules and OktaAssignments to the
cache, which will occur in a later commit.
* Intermediary.
* Modifications needed in order to move this to enterprise.
* Remove okta service from grpcserver.
An interface has been created that will allow the signing of SAML IdP
requests on the auth server. This will eliminate the need for the SAML IdP
to have private key material.
* Distroless dockerfile and smoke tests for same
This patch adds a Dockerfile to `build.assets/charts` that will construct
a docker image for teleport based on the Distroless Debian images
published by Google. The actual workflows used to construct and publish
these images are defined in `teleport.e` for security reasons.
The Smoke Testing framework exists to make some quick assertions about
the resulting images: will Teleport even start in this context, etc. See
the included README for more details.
* Update README.md
* Linter appeasement
* Revert spurious submodule update
* Rename release component var
* Smoke test docs
* Smoke test docs
* Revert spurious subrepo update
* Added `--target-cloud` flag to OS package repo tool
* Updated OS package repo tool to use "version channel" instead of "artifact version"
* Added help flag examples
Changes the name to better align with the naming used by the rest
of the feature and to prevent ambiguity as the term proxy is quite
overloaded. This has not landed in a release yet so there are no
backward compatability concerns as nothing serves or consumes this
api yet.
* Add plugin exchange service
* Add Plugin methods to auth
* Add gRPC-layer methods for Plugin
* Add RBAC presets for Plugin
* Test GetPlugin()/NoSecrets access
* Make error assertions more correct in role test
* Deny setting credentials if user can not read them
* gofmt
* Apply minor suggestions from code review
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Move dependency into an existing block in go.mod
* Improve error messages for failed type assertions
* DRY WithSecrets access checks for Plugins
* Run new tests in parallel
* Improve error assertions in auth_with_roles_test
* TestGetPluginWithSecrets: split cases to subtests
* Clean up test servers and clients
* Add proto for plugin service
* Remove Plugin methods from auth service
Moved to a dedicated service
* Remove plugin-related auth methods
Moved to a dedicated service in Enterprise
* Remove CreatePlugin test from auth_with_roles_test
Moved to a dedicated service in Enterprise
* Pass "backend getter" to local plugins service
This pattern is used in Enterprise to set up secondary services
before auth (and backend) are created.
* Rename InitialCredentials to BootstrapCredentials
* Add plugins service to genproto.sh
* Reformat generated proto
* Remove obsolete PluginExchangeService
The equivalent of this is now in Enterprise
* Add kube service to genproto.sh; regenerate
* Add ListPlugins to plugin backend service
* Reimplement GetPlugins on top of ListPlugins
This is a "convenience" implementation for the backend service layer.
* Replace GetPlugins with ListPlugins in gRPC schema
* Fix ListKubernetesResources unit test
* Simplify plugin pagination key to just the name
* Use existing constant for page size
* Make dummy clients return errors instead of panic
* Remove obsolete field
* Ensure go.mod is valid for corresponding e changes
* Fix passing mutex ref
* Move teleport-plugins import to e_imports
* Revert oauth change in go.mod
* Use limit+1 to look-ahead when paginating plugins
* Test plugin pagination with pageSize > numPlugins
* Add descriptive messages to gRPC dummy clients
* Plugin: add RW for editor; remove secrets from gRPC
* Make message more descriptive for dummy gRPC conn
---------
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Install deb/yum repos when using node-join script
When a repo is available for the current Linux distro/version, use it
instead of just installing Teleport from the deb/rpm files.
It fallsback to the traditional binary installation when the repo is not
available.
* comment /etc/os-release
* remove sudo; add comment to runners
* improve is_repo_available function
My previous PR https://github.com/gravitational/teleport/pull/21660 broke the git installation path, as `git` compiles in the `--prefix` provided during configuration. Using`DESTDIR` with `make install` instead of `--prefix` copies the file in the intermediate container with the correct path.
* Unify x86/AMD64 build process
Currently, our ARM64 pipeline builds limited subset of Teleport features as none of the 3rd party dependencies (openssh, libbpf etc) are not built on AMR64. This change build all dependencies on AMR64 in the same way as we do on x86.
FIPS changes are not included as we do not support FIPS on ARM64.
* Apply suggestions from code review
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
---------
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* BPF build fix
https://github.com/gravitational/teleport/pull/21745 switched CentOS 7 image to the upstream, but I missed a few other places were we're using our fork.
This change fixes all places.
* Add missing FIPS changes
* Update e
Update to libbpf 1.0.1 and github.com/aquasecurity/libbpfgo v0.4.5-libbpf-1.0.1. As we're building our releases on CentOS 7 anyway we can also switch to mainstream libbpf instead of using our fork.
The `tsh appps` family of commands is aliased to `tsh apps`, so both
invocations work correctly. The command itself is defined as `tsh apps`,
so this is what appears in the help message.
Update references to `tsh app` to recommend `tsh apps` instead so that
there isn't confusion when browsing `tsh help` and looking for a missing
`app` subcommand.
Fixes#21367
Moving our CentOS build assets, aka Clang-10 is the first step to enabling our full Teleport to build on ARM64. This change should also save us some $$ as getting the assets from S3 sounds expensive.
* Update JS grpc-tools to 1.12.4
1.11.2 didn't have support for arm64 so we had to do all this extra stuff
in the Dockerfile.
1.11.3 added support for Darwin arm64 and 1.12.4 finally adds support for
Linux arm64. This means we can completely remove extra cruft and just
install grpc-tools 1.12.4 on all architectures.
* Add comment to ptyHostService.proto
* Adjust go_package of lib/prehog Go protobufs
This makes them follow the pattern set out by api/proto and proto.
* Adjust go_package of lib/teleterm Go protobufs
* Use single buf.gen.yaml to generate JS protos
This also entailed changing the location of lib/teleterm protos and changing
the value of their package specifier to match the conventions in other parts
of the codebase. This is a breaking change but that is fine for Connect
as the protos are used locally only and each build ships with matching
protobufs.
* Make web/packages/teleterm use protobufs from gen-proto-js
We used to copy protobufs over to web/packages/teleterm/src/services/tshd
since webapps used to be in a separate repo.
This is no longer the case, so we can just make teleterm use protobufs
from gen-proto-js.
* Move prehog & teleterm protos into proto/teleport/lib
* Generate JS protos to gen/proto/js
* Move lib/teleterm Go protobufs to gen/proto/go
* Move lib/prehog Go protobufs to gen/proto/go
* Rename lib/teleterm proto package
* Re-enable linter rules for teleterm & prehog
* Update prehogv1 path in usagereporter_test.go
* Use except instead of ignore_only to allow Google API-style responses
* Add UNARY_RPC to api/proto & proto
* Ignore gen/ when running addlicense
* buf-js.gen.yaml: Remove comment about lack of go_package for JS
* Move prehog protos to proto/prehog/v1alpha
* Adjust prehog's go_package to match proto package
* Add Plugin resource schema, methods
* Improve shebang of genproto.sh
Execute using bash, no matter where it actually lives
* Use Metadata.Expiry()
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Remove field reservations from PluginStatusCode
* Add plugin (un)marshaling
* Snake case fields of Plugin (and children)
* Ensure timestamp fields on Plugin are always UTC
https://github.com/gogo/protobuf/issues/519
* Rename credentials according to proto conventions
* Fold check for nil settings into the type switch
* Remove extraneous field checks
These are set in setStaticFields()
* Add missing godocs
---------
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Add check if protos are up to date.
A new check has been added that will detect if protobufs are up to date. The
script will exit abnormally if protobufs need to be regenerated.
* Alan's feedback.
* Restoring the script.
* Update script comment.
* Add in the set -eu.
* Add a comment for the pull_request/merge_group bit in the new github action.
* Remove helper script.
* Reduce the runner size.
* Conditionally build the UI if there are changes.
`ensure-webassets` and `ensure-webassets-e` will now build the UI if the
hash-of-hashes has changed since the last time the webassets were built.
* Update Makefile
Co-authored-by: Jeff Pihach <hatched@users.noreply.github.com>
* Check for the existence of the make command.
* Choose a shasum exec from a preconfigured list or use an override.
* Shellcheck fix.
* Remove yarn.lock from sha calculation.
* Recalc SHA on successful build.
* Fix linting issues.
* Use Python instead of shasum.
* Remove unnecessary comments from sh file.
* Unconditionally rebuild if Python executable is not found.
* Revert "Unconditionally rebuild if Python executable is not found."
This reverts commit 9d2b0e3409.
* Revert "Remove unnecessary comments from sh file."
This reverts commit 600e02811b.
* Revert use of Python, use simplified shell scripting.
* Linting fixes.
* Apply suggestions from code review
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Remove reference to realpath as apparently the MacOS builder doesn't have it.
* calculate_sha must return 0.
* Use echo in calculate_sha or else set -o pipefail causes failures.
* Suppress SC2005.
---------
Co-authored-by: Jeff Pihach <hatched@users.noreply.github.com>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Remove CLANG_FORMAT from Makefiles
It was used to format protos but we use Buf for that since v10.
* Move installing grpc_node_plugin into Dockerfile
This commit basically takes grpc_node_plugin compilation from
Dockerfile-teleterm and moves it to Dockerfile.
* Replace Dockerfile-teleterm with Dockerfile
After moving grpc_node_plugin compilation to Dockerfile, the only remaining
thing that Dockerfile-teleterm does is installing rpm so that we can make
an RPM package for Connect during tag builds.
Installing this package can be simply moved to Dockerfile.
* Remove grpc-teleterm Make target in favor of grpc
* Add updated protobufs
It looks like they're a result of someone changing protos in lib/prehog
without running `make grpc-teleterm` separately. Which is why we're getting
rid of grpc-teleterm as a separate Make target in the first place. ;)
Rafał Cieślak