Commit Graph
56 Commits
Author SHA1 Message Date
Colin Adler 5de6f86959 feat: trace httpapi.{Read,Write} (#4134) 2022-09-21 17:07:00 -05:00
Colin Adler b340634aaa feat: add rbac tracing (#4093) 2022-09-16 18:32:15 +00:00
Colin Adler 77acf0c340 feat: provisionerd tracing, add public trace ingestion (#4070) 2022-09-16 11:43:22 -05:00
Ammar Bandukwala 30f8fd9b95 Daily Active User Metrics (#3735)
* agent: add StatsReporter

* Stabilize protoc
2022-09-01 14:58:23 -05:00
Kyle Carberry b0fe9bcdd1 chore: Upgrade to Go 1.19 (#3617)
This is required as part of #3505.
2022-08-21 22:32:53 +00:00
Mathias Fredriksson 4730c589fe chore: Use standardized test timeouts and delays (#3291) 2022-08-01 15:45:05 +03:00
Spike Curtis f310aeb4cb Disable skipping job acquire log (#3240)
Signed-off-by: Spike Curtis <spike@coder.com>
2022-07-26 16:36:45 -07:00
Mathias Fredriksson 6916d34458 fix: Fix cleanup in test helpers, prefer defer in tests (#3113)
* fix: Change uses of t.Cleanup -> defer in test bodies

Mixing t.Cleanup and defer can lead to unexpected order of execution.

* fix: Ensure t.Cleanup is not aborted by require

* chore: Add helper annotations
2022-07-25 19:22:02 +03:00
Kyle Carberry 4f1df88529 fix: Always output job failure reason in provisioner daemon tests (#2850)
This flake can be seen here: https://github.com/coder/coder/runs/7186604615?check_suite_focus=true
2022-07-10 14:52:33 -05:00
Spike Curtis 22febc749a provisionerd sends failed or complete last (#2732)
* provisionerd sends failed or complete last

Signed-off-by: Spike Curtis <spike@coder.com>

* Move runner into package

Signed-off-by: Spike Curtis <spike@coder.com>

* Remove jobRunner interface

Signed-off-by: Spike Curtis <spike@coder.com>

* renames and slight reworking from code review

Signed-off-by: Spike Curtis <spike@coder.com>

* Reword comment about okToSend

Signed-off-by: Spike Curtis <spike@coder.com>
2022-07-01 09:55:46 -07:00
Kyle Carberry 024ab6df57 fix: Use in-memory filesystem for echo provisioner tests (#2408)
* fix: Use in-memory filesystem for echo provisioner tests

This should reduce IO in CI to shave some time off tests!

* test: Increase timeouts to reduce flakes

It's difficult to understand what's timing out due to a lock
vs. taking a long time. This should help resolve! 🕵️
2022-06-16 15:09:22 +00:00
Kyle Carberry ccd061652b feat: Add built-in PostgreSQL for simple production setup (#2345)
* feat: Add built-in PostgreSQL for simple production setup

Fixes #2321.

* Use fork of embedded-postgres for cache path
2022-06-15 16:02:18 -05:00
Dean Sheather 45eb1b4980 feat: improve terraform template parsing errors (#2331) 2022-06-16 04:12:17 +10:00
Dean Sheather 6be8a373e0 feat: run a terraform plan before creating workspaces with the given template parameters (#1732) 2022-06-02 00:44:53 +10:00
David Wahler 0c4a65b113 fix: manually fix coderd/database/dump.sql and make style/gen check run more reliably (#1607) 2022-05-19 22:37:22 +00:00
Garrett Delfosse e2ed581708 Add stages to all proto.Logs (#1563) 2022-05-18 17:33:29 +00:00
Colin Adler 98ccd0eb89 feat: add README parsing to template versions (#1500) 2022-05-17 15:00:48 -05:00
Kyle Carberry 914a2f477c fix: Restore terminal on interrupt when connecting (#1312)
Fixes #1292.
2022-05-05 20:11:44 -05:00
Colin Adler 9319c39257 fix: additional provisionerd test double closes (#1267) 2022-05-03 08:02:54 -05:00
Kyle Carberry 603b7da413 test: Wrap provisionerd channel closes in sync.Once (#1181)
This caused a few flakes, so figured I'd tackle all of them:
https://github.com/coder/coder/runs/6167856950?check_suite_focus=true#step:9:246
2022-04-26 09:44:16 -05:00
Kyle Carberry 877854a2f3 fix: Display proper access URL on server start (#1172)
Fixes #1170.
2022-04-25 23:30:22 +00:00
Kyle Carberry 947e8f9d2e fix: Manually format external URL (#1168)
path.Join escaped the double slash!
2022-04-25 22:22:31 +00:00
Kyle Carberry 66d45f391e test: Check if provisionerd is closed before setting run chan (#1145)
This race can be seen here:
https://github.com/coder/coder/runs/6159662393?check_suite_focus=true
2022-04-25 12:00:54 -05:00
Kyle Carberry c8e566fe42 feat: Add links for registering git key (#1125) 2022-04-25 04:27:45 +00:00
Kyle Carberry 68f67c54b6 fix: Add sync.Once to prevent double close in test (#1124)
https://github.com/coder/coder/runs/6151451291?check_suite_focus=true
2022-04-25 04:06:18 +00:00
Kyle Carberry a6ea99541e fix: Return from update loop when job completes (#1121)
Update was running forever, which stopped jobs from timing
out unless a restart occurred. This also fixes complete properly
reporting an error.
2022-04-25 03:41:03 +00:00
Kyle Carberry db7ed4d019 fix: Add resiliency to daemon connections (#1116)
Connections could fail when massive payloads were transmitted.
This fixes an upstream bug in dRPC where the connection would
end with a context canceled if a message was too large.

This adds retransmission of completion and failures too. If
Coder somehow loses connection with a provisioner daemon,
upon the next connection the state will be properly reported.
2022-04-24 20:33:19 -05:00
Steven Masley da3681246e chore: Bump protoc to 3.20.0 (#1104)
* chore: Bump protoc to 3.20.0
* Make gen with 3.20.0 protoc
2022-04-23 01:53:22 +00:00
Kyle CarberryandCian Johnston fb9dc4f346 feat: Improve resource preview and first-time experience (#946)
* Improve CLI documentation

* feat: Allow workspace resources to attach multiple agents

This enables a "kubernetes_pod" to attach multiple agents that
could be for multiple services. Each agent is required to have
a unique name, so SSH syntax is:

`coder ssh <workspace>.<agent>`

A resource can have zero agents too, they aren't required.

* Add tree view

* Improve table UI

* feat: Allow workspace resources to attach multiple agents

This enables a "kubernetes_pod" to attach multiple agents that
could be for multiple services. Each agent is required to have
a unique name, so SSH syntax is:

`coder ssh <workspace>.<agent>`

A resource can have zero agents too, they aren't required.

* Rename `tunnel` to `skip-tunnel`

This command was `true` by default, which causes
a confusing user experience.

* Add disclaimer about editing templates

* Add help to template create

* Improve workspace create flow

* Add end-to-end test for config-ssh

* Improve testing of config-ssh

* Fix workspace list

* Fix config ssh tests

* Update cli/configssh.go

Co-authored-by: Cian Johnston <public@cianjohnston.ie>

* Fix requested changes

* Remove socat requirement

* Fix resources not reading in TTY

Co-authored-by: Cian Johnston <public@cianjohnston.ie>
2022-04-11 18:54:30 -05:00
Kyle Carberry 02ad3f14f5 chore: Rename Projects to Templates (#880)
Customer feedback indicated projects was a confusing name.
After querying the team internally, it seemed unanimous
that it is indeed a confusing name.

Here's for a lil less confusion @ashmeer7 🥂
2022-04-06 12:42:40 -05:00
Kyle Carberry a502a5fa14 feat: Add AWS instance identity authentication (#570)
* feat: Add AWS instance identity authentication

This allows zero-trust authentication for all AWS instances.

Prior to this, AWS instances could be used by passing `CODER_TOKEN`
as an environment variable to the startup script. AWS explicitly
states that secrets should not be passed in startup scripts because
it's user-readable.

* Fix sha256 verbosity

* Fix HTTP client being exposed on auth
2022-03-28 19:31:03 +00:00
Kyle Carberry b33dec9d38 feat: Add stage to build logs (#577)
* feat: Add stage to build logs

This adds a stage property to logs, and refactors the job logs
cliui.

It also adds tests to the cliui for build logs!

* Fix comments
2022-03-28 18:43:22 +00:00
Kyle Carberry 26d24f4508 chore: Improve CI builds by caching Go modules (#528)
* chore: Improve CI builds by caching Go modules

* Skip running with `race` on non-Linux systems

* Fix darwin file descriptor error

* Fix log after close

* Improve PostgreSQL test speeds

* Fix parallel connections with PostgreSQL tests

* Fix CI flake

* Separate test/go into PostgreSQL
2022-03-22 17:09:04 -05:00
Kyle Carberry c451f4e685 feat: Add templates to create working release (#422)
* Add templates

* Move API structs to codersdk

* Back to green tests!

* It all works, but now with tea! 🧋

* It works!

* Add cancellation to provisionerd

* Tests pass!

* Add deletion of workspaces and projects

* Fix agent lock

* Add clog

* Fix linting errors

* Remove unused CLI tests

* Rename daemon to start

* Fix leaking command

* Fix promptui test

* Update agent connection frequency

* Skip login tests on Windows

* Increase tunnel connect timeout

* Fix templater

* Lower test requirements

* Fix embed

* Disable promptui tests for Windows

* Fix write newline

* Fix PTY write newline

* Fix CloseReader

* Fix compilation on Windows

* Fix linting error

* Remove bubbletea

* Cleanup readwriter

* Use embedded templates instead of serving over API

* Move templates to examples

* Improve workspace create flow

* Fix Windows build

* Fix tests

* Fix linting errors

* Fix untar with extracting max size

* Fix newline char
2022-03-22 13:17:50 -06:00
Colin Adler 706e60bb3b chore: pluralize table names (#463) 2022-03-17 18:10:46 -05:00
Kyle Carberry bf0ae8f573 feat: Refactor API routes to use UUIDs instead of friendly names (#401)
* Add client for agent

* Cleanup code

* Fix linting error

* Rename routes to be simpler

* Rename workspace history to workspace build

* Refactor HTTP middlewares to use UUIDs

* Cleanup routes

* Compiles!

* Fix files and organizations

* Fix querying

* Fix agent lock

* Cleanup database abstraction

* Add parameters

* Fix linting errors

* Fix log race

* Lock on close wait

* Fix log cleanup

* Fix e2e tests

* Fix upstream version of opencensus-go

* Update coderdtest.go

* Fix coverpkg

* Fix codecov ignore
2022-03-07 11:40:54 -06:00
Kyle Carberry ea5efbd37f test: Fix flake with context.Cancelled in provisionerd (#386)
This occurred because the context can cancel in the same time a response
is sent. This isn't a bug, because the complete still occurs.
2022-03-01 01:56:54 +00:00
Kyle Carberry fd5eceb0b8 test: Fix test flake panic in provisionerd (#383)
Closes #382.
2022-02-28 13:10:34 -08:00
Kyle Carberry 9d2803e07a feat: Add graceful exits to provisionerd (#372)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* fix: Convert all jobs to use a common resource and agent type

This enables a consistent API for project import and provisioned resources.

* Add "coder_workspace" data source

* feat: Remove magical parameters from being injected

This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.

* feat: Add graceful exits to provisionerd

Terraform (or other provisioners) may need to cleanup state, or
cancel actions before exit. This adds the ability to gracefully
exit provisionerd.

* Fix cancel error check
2022-02-28 18:40:49 +00:00
Kyle Carberry e5c95552cd feat: Remove magical parameters from being injected (#371)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* fix: Convert all jobs to use a common resource and agent type

This enables a consistent API for project import and provisioned resources.

* Add "coder_workspace" data source

* feat: Remove magical parameters from being injected

This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.
2022-02-28 18:26:01 +00:00
Kyle Carberry 65de96c8b4 fix: Leaking yamux session after HTTP handler is closed (#329)
* fix: Leaking yamux session after HTTP handler is closed

Closes #317. The httptest server cancels the context after the connection
is closed, but if a connection takes a long time to close, the request
would never end. This applies a context to the entire listener that cancels
on test cleanup.

After discussion with @bryphe-coder, reducing the parallel limit on
Windows is likely to reduce failures as well.

* Switch to windows-2022 to improve decompression

* Invalidate cache on matrix OS
2022-02-18 22:06:56 -06:00
Kyle Carberry 6bdef0697c test: Fix race condition in provisionerd on cleanup (#322)
These goroutines could be ran after the pipe has already been closed.
I'm not certain this resolves this specific leak:

https://github.com/coder/coder/runs/5249481202?check_suite_focus=true#step:7:186

...but I find it likely.
2022-02-18 10:43:56 -06:00
Kyle Carberry 154b9bce57 feat: Add "coder projects create" command (#246)
* Refactor parameter parsing to return nil values if none computed

* Refactor parameter to allow for hiding redisplay

* Refactor parameters to enable schema matching

* Refactor provisionerd to dynamically update parameter schemas

* Refactor job update for provisionerd

* Handle multiple states correctly when provisioning a project

* Add project import job resource table

* Basic creation flow works!

* Create project fully works!!!

* Only show job status if completed

* Add create workspace support

* Replace Netflix/go-expect with ActiveState

* Fix linting errors

* Use forked chzyer/readline

* Add create workspace CLI

* Add CLI test

* Move jobs to their own APIs

* Remove go-expect

* Fix requested changes

* Skip workspacecreate test on windows
2022-02-12 13:34:04 -06:00
Kyle Carberry 7364933e65 refactor: Allow provisioner jobs to be disconnected from projects (#194)
* Nest jobs under an organization

* Rename project parameter to parameter schema

* Update references when computing project parameters

* Add files endpoint

* Allow one-off project import jobs

* Allow variables to be injected that are not defined by the schema

* Update API to use jobs first

* Fix CLI tests

* Fix linting

* Fix hex length for files table

* Reduce memory allocation for windows
2022-02-08 12:00:44 -06:00
Kyle Carberry 795bba2af4 feat: Add dry run for provisioners (#178)
* refactor: Rename ProjectParameter to ProjectVersionParameter

This was confusing with ParameterValue before. It still is a bit,
but this should help distinguish scope.

* Add project version resources table

* Allow project parameters to optionally have user and workspace

* Add dry run for provisioners

* Add resource detection on project import
2022-02-07 19:35:18 -06:00
Kyle Carberry ed705f6af2 refactor: Generalize log ownership to allow for scratch jobs (#182)
* refactor: Generalize log ownership to allow for scratch jobs

Importing may fail when creating a project. We don't want to lose this output,
but we don't want to allow users to create a failing project.

This generalizes logs to soon enable one-off situations where a user can upload
their archive, create a project, and watch the output parse to completion.

* Improve file table schema by using hash

* Fix racey test by allowing logs before

* Add debug logging for PostgreSQL insert
2022-02-07 15:32:37 -06:00
Kyle CarberryandBryan 1796dc6c2f chore: Add test helpers to improve coverage (#166)
* chore: Rename ProjectHistory to ProjectVersion

Version more accurately represents version storage. This
forks from the WorkspaceHistory name, but I think it's
easier to understand Workspace history.

* Rename files

* Standardize tests a bit more

* Remove Server struct from coderdtest

* Improve test coverage for workspace history

* Fix linting errors

* Fix coderd test leak

* Fix coderd test leak

* Improve workspace history logs

* Standardize test structure for codersdk

* Fix linting errors

* Fix WebSocket compression

* Update coderd/workspaces.go

Co-authored-by: Bryan <bryan@coder.com>

* Add test for listing project parameters

* Cache npm dependencies with setup node

* Remove windows npm cache key

Co-authored-by: Bryan <bryan@coder.com>
2022-02-05 18:24:51 -06:00
Kyle Carberry f19770b2c6 chore: Rename ProjectHistory to ProjectVersion (#165)
* chore: Rename ProjectHistory to ProjectVersion

Version more accurately represents version storage. This
forks from the WorkspaceHistory name, but I think it's
easier to understand Workspace history.

* Rename files
2022-02-04 20:47:58 -06:00
Kyle Carberry 65de6eef9c feat: Add streaming endpoint for workspace history (#157)
* feat: Add parameter querying to the API

* feat: Add streaming endpoint for workspace history

Enables a buildlog-like flow for reading job output.

* Fix empty parameter source and destination

* Add comment for usage of workspace history logs endpoint
2022-02-04 19:36:43 +00:00
Kyle Carberry b6d27adba0 fix: Attempt cleanup multiple times on Windows (#161)
See: https://github.com/golang/go/issues/50510
2022-02-04 11:54:07 -06:00