Commit Graph
72 Commits
Author SHA1 Message Date
Danny KoppingandSas Swart 4c33846f6d chore: add prebuilds system user (#16916)
Pre-requisite for https://github.com/coder/coder/pull/16891

Closes https://github.com/coder/internal/issues/515

This PR introduces a new concept of a "system" user.

Our data model requires that all workspaces have an owner (a `users`
relation), and prebuilds is a feature that will spin up workspaces to be
claimed later by actual users - and thus needs to own the workspaces in
the interim.

Naturally, introducing a change like this touches a few aspects around
the codebase and we've taken the approach _default hidden_ here; in
other words, queries for users will by default _exclude_ all system
users, but there is a flag to ensure they can be displayed. This keeps
the changeset relatively small.

This user has minimal permissions (it's equivalent to a `member` since
it has no roles). It will be associated with the default org in the
initial migration, and thereafter we'll need to somehow ensure its
membership aligns with templates (which are org-scoped) for which it'll
need to provision prebuilds; that's a solution we'll have in a
subsequent PR.

---------

Signed-off-by: Danny Kopping <dannykopping@gmail.com>
Co-authored-by: Sas Swart <sas.swart.cdk@gmail.com>
2025-03-25 12:18:06 +00:00
Danny Kopping b16275b7cd chore: fix regex bug in migration number fixer (#16822)
This fixes a slight regex bug on Bash 5, where `[:/]` would only match
`:` but not both `:/`.

```bash
$ git remote -v | grep "github.com[:/]coder/coder.*(fetch)" | cut -f1

$ git remote -v | grep "github.com[:/]*coder/coder.*(fetch)" | cut -f1
origin
```

The former will actually cause the whole script to bork because of
`pipefail`, since `grep` exits 1.

Signed-off-by: Danny Kopping <dannykopping@gmail.com>
2025-03-06 12:21:14 +02:00
Danny Kopping 546d915d32 chore: install libgbm-dev to allow headless chrome e2e tests to run (#16695)
Without this lib, Chrome can’t set up its offscreen rendering buffers -
apparently.

I've validated this manually in my workspace.

Signed-off-by: Danny Kopping <danny@coder.com>
2025-02-25 12:33:17 +00:00
Danny Kopping 4c438bd4d3 feat(cli): add local and UTC time options to ping cmd (#16648)
It's sometimes useful to see when each pong was received, for
correlating these times with other events.

---------

Signed-off-by: Danny Kopping <danny@coder.com>
2025-02-24 05:38:17 +00:00
Danny Kopping 833ca53e51 chore: document docker-compose development workflow (#16618) 2025-02-19 14:56:24 +02:00
Danny Kopping 2d36b587c0 chore(site): log errors when page fails with not_available in e2e tests (#16219)
https://github.com/coder/internal/issues/279 tracks a flake we've been
seeing of late.

I noticed log lines like these:

```
[onResponse] url=http://localhost:3111/api/v2/workspaces/2fedd3ad-58a9-49be-9e61-177878be7611/watch status=200 body=not_available
[onResponse] url=http://localhost:3111/api/v2/workspaceagents/753ee86d-46cb-4641-97f6-7b4c9c9a9e27/watch-metadata status=200 body=not_available
```

No other debugging info seems to be available for these responses, so
let's add some.

Signed-off-by: Danny Kopping <danny@coder.com>
2025-01-22 10:27:03 +02:00
Danny KoppingandMuhammad Atif Ali 5b72a4376d chore: improve CI reliability (#16169)
We have an effort underway to replace `dbmem` (#15109), and consequently
we've begun running our full test-suite (with Postgres) on all supported
OSs - Windows, MacOS, and Linux, since #15520.

Since this change, we've seen a marked decrease in the success rate of
our builds on `main` (note how the Windows/MacOS failures account for
the vast majority of failed builds):


![image](https://github.com/user-attachments/assets/a02c15b7-037d-428a-a600-2aed60553ac0)

We're still investigating why these OSs are a lot less reliable. It's
likely that the VMs on which the builds are run have different
characteristics from our Ubuntu runners such as disk I/O, network
latency, or something else.

**In the meantime, we need to start trusting CI failures in `main`
again, as the current failures are too noisy / vague for us to
correct.**

We've also considered hosting our own runners where possible so we can
get OS-level observability to rule out some possibilities.

See the [meeting
notes](https://www.notion.so/coderhq/CI-Investigation-Call-Notes-17dd579be59280d8897cc9fe4bb46695?pvs=6&utm_content=17dd579b-e592-80d8-897c-c9fe4bb46695&utm_campaign=T1ZPT2FL0&n=slack&n=slack_link_unfurl)
where we linked into this for more detail.

This PR introduces several changes:

1. Moves the full test-suite with Postgres on Windows/MacOS to the
`nightly-gauntlet` workflow
tradeoff: this means that any regressions may be more difficult to
discover since we merge to main several times a day
2. Run only the CLI test-suite on each PR / merge to `main` on
Windows/MacOS
3. `test-go` is still running the full test-suite against all OSs
(including the CLI ones), but will soon be removed once #15109 is
completed since it uses `dbmem`
4. Changes `nightly-gauntlet` to run at 4AM: we've seen several
instances of the runner being stopped externally, and we're _guessing_
this may have something to do with the midnight UTC execution time, when
other cron jobs may run
5. Removes the existing `nightly-gauntlet` jobs since they haven't
passed in a long time, indicating that nobody cares enough to fix them
and they don't provide diagnostic value; we can restore them later if
necessary

I've manually run both these new workflows successfully:

- `ci`:
https://github.com/coder/coder/actions/runs/12825874176/job/35764724907
- `nightly-gauntlet`:
https://github.com/coder/coder/actions/runs/12825539092

---------

Signed-off-by: Danny Kopping <danny@coder.com>
Co-authored-by: Muhammad Atif Ali <atif@coder.com>
2025-01-20 07:06:33 +00:00
Danny Kopping 22236f2988 chore: only notify about CI failure on main if required job fails (#16114)
This should be the last PR to get this working

Looks like the `nightly-gauntlet` is working as expected, and this is a
clone of that.

---------

Signed-off-by: Danny Kopping <danny@coder.com>
2025-01-13 14:06:43 +00:00
Danny Kopping f9f72de1d6 chore: predicate slack notification job on other jobs (#16106)
`always()` does not seem to work

Extending https://github.com/coder/coder/pull/16105

Signed-off-by: Danny Kopping <danny@coder.com>
2025-01-13 13:42:37 +02:00
Danny Kopping 73d8dde6ed chore: notify #dev of nightly gauntlet failures (#16105)
Expands on https://github.com/coder/coder/pull/16102

This workflow is currently failing every night, so this will not only
raise immediate awareness but will also be easy to validate this job.

Signed-off-by: Danny Kopping <danny@coder.com>
2025-01-13 10:15:15 +00:00
Danny Kopping 859abcde4e chore: send notification to #dev on any CI failure on main (#16102)
We've had a [few failures in
main](https://github.com/coder/coder/actions?query=branch%3Amain+is%3Afailure)
of late, and unless the committer of the change has CI notifications
enabled we may not be aware of the failure.

This PR sends a Slack notification to the #dev channel so everyone has
visibility.

Signed-off-by: Danny Kopping <danny@coder.com>
2025-01-13 11:28:45 +02:00
Danny Kopping 401f8b4fea provide useful debug info when retrieving a version from GitHub fails (#15956)
Closes https://github.com/coder/coder/issues/15851

This fails the installation when the version cannot be retrieved, and
prints useful debug info.

`install.sh` could use with more error-handling in general, but this at
least ameliorates the linked issue.

Signed-off-by: Danny Kopping <danny@coder.com>
2025-01-09 09:47:20 +00:00
Danny Kopping f38f746f5d chore: ensure coder is logged before querying orgs (#15618) 2024-11-21 16:04:21 +01:00
Danny Kopping cb1a006ae4 chore: add link to RBAC usage doc in README (#15510)
Some new joiners had found the README, but not the usage doc

Signed-off-by: Danny Kopping <danny@coder.com>
2024-11-14 10:01:19 +00:00
Danny KoppingandEdwardAngert 111029ec07 chore: clarify template provisioning speed doc (#15383)
I noticed a few inaccuracies in this doc, or aspects which could've been
explained a little more. In other cases, I've added some specificity in
service of clarity.

---------

Signed-off-by: Danny Kopping <danny@coder.com>
Co-authored-by: EdwardAngert <17991901+EdwardAngert@users.noreply.github.com>
2024-11-08 16:06:20 +00:00
Danny Kopping cafb963bf4 chore: clarify section description to indicate that notifications can be disabled (#15381)
The current description of "Configure your notification preferences.
Icons on the right of each notification indicate delivery method, either
SMTP or Webhook." doesn't really say what this page is for.

I've adjusted the wording, and also changed the tooltip of the delivery
method icon to be more clear.

Signed-off-by: Danny Kopping <danny@coder.com>
2024-11-06 18:43:39 +01:00
Danny Kopping b9d441ff5f fix: use concurrency-safe bytes buffer to avoid race (#15142)
Fixes https://github.com/coder/internal/issues/93

`bytes.Buffer` is not concurrency-safe.

`cmd` could write to the buffer concurrently while we're reading the
buffer in

```
require.Eventually(t, func() bool {
	return bytes.Contains(output.Bytes(), []byte("ERROR: Downloaded agent binary returned unexpected version output"))
}, testutil.WaitShort, testutil.IntervalSlow)
```

Not sure about the `os: process already finished` flake, though.

---------

Signed-off-by: Danny Kopping <danny@coder.com>
2024-10-21 17:24:18 +11:00
Danny Kopping 1b5f3418d3 chore: align active version terminology and link to docs (#14639) 2024-09-11 12:25:07 +02:00
Danny Kopping 914f35a3a3 chore: document RBAC usage (#14065) 2024-09-10 15:15:30 +00:00
Danny Kopping 8b1c46fbe0 fix: analyze build times over 30 days not 30 months (#14584) 2024-09-06 17:03:01 +02:00
Danny Kopping f23a05075e feat: support optional SMTP auth (#14533) 2024-09-03 11:51:34 +02:00
Danny Kopping 66c8060605 Minor fixups, added troubleshooting (#14519) 2024-09-02 16:10:54 +02:00
Danny Kopping 043f4f5327 docs: add documentation for notifications feature (#14478) 2024-08-30 17:30:06 +02:00
Danny Kopping 9596f236c1 fix: use negative deadline to ensure timeout in TestWebhook/timeout (#14498) 2024-08-30 16:46:13 +02:00
Danny Kopping 0f414a00d3 fix: restore closing SMTP message on method exit (#14496) 2024-08-30 13:03:25 +02:00
Danny Kopping c90be9b0c1 fix: correctly close SMTP message and await response (#14495) 2024-08-30 11:37:50 +02:00
Danny Kopping f24cb5cc96 fix: prevent test flakiness (#14467)
Signed-off-by: Danny Kopping <danny@coder.com>
2024-08-28 16:33:27 +02:00
Danny Kopping a4d785dec5 chore: use idiomatic test setup in notification tests (#14416) 2024-08-23 16:29:27 +02:00
Danny Kopping cc944209ae fix: include provisioner timing action in hash func (#14388) 2024-08-22 12:31:50 +02:00
Danny Kopping fa733318e0 Add missing content (#14380) 2024-08-21 12:46:11 +00:00
Danny Kopping 6960d194ae feat: add provisioning timings to understand slow build times (#14274) 2024-08-21 14:18:58 +02:00
Danny Kopping 9c8c6a952d feat: add notification deduplication trigger (#14172) 2024-08-21 11:18:03 +02:00
Danny Kopping c90e6d7b47 chore: fix up migration number fixer (#14266) 2024-08-14 11:05:03 +02:00
Danny Kopping d79a7adf99 docs: advise against shared CODER_CACHE_DIRECTORY dir usage in note (#14216) 2024-08-08 12:42:47 +00:00
Danny Kopping c6076d2d0d chore: improve notification template tests' resilience (#14196) 2024-08-07 11:33:26 +02:00
Danny Kopping 58428aafce fix: allow all users to read system notification templates (#14181) 2024-08-06 15:37:49 +02:00
Danny Kopping e164b1e71c feat: add notification preferences database & audit support (#14100) 2024-08-05 16:18:45 +02:00
Danny Kopping c3390993dd chore: update generated files after pnpm upgrade (#14036) 2024-07-31 17:23:55 +02:00
Danny Kopping b817c863ef fix: webhook endpoint YAML attribute (#13983)
Signed-off-by: Danny Kopping <danny@coder.com>
2024-07-23 14:59:27 +00:00
Danny Kopping 695afb80e6 fix: address TestPendingUpdatesMetric flake 2024-07-23 14:39:57 +00:00
Danny Kopping c88e4162d8 fix: TestPendingUpdatesMetric flake (#13944)
Signed-off-by: Danny Kopping <danny@coder.com>
2024-07-19 12:54:15 +02:00
Danny Kopping 492ab1cc7e chore: add webhook tests for notification subsystem (#13942) 2024-07-19 12:03:29 +02:00
Danny Kopping 943ea7c52a feat: add SMTP auth & TLS support (#13902) 2024-07-19 09:22:15 +02:00
Danny Kopping 1691768fb9 chore: use store enqueuer with external provisioners (#13881) 2024-07-12 13:51:13 +02:00
Danny Kopping b2dab3308d feat: implement observability of notifications subsystem (#13799) 2024-07-11 10:57:49 +02:00
Danny Kopping 542fff7df0 chore: improve notifications tests (#13863) 2024-07-10 15:25:23 +02:00
Danny Kopping bdd2caf95d feat: implement thin vertical slice of system-generated notifications (#13537) 2024-07-08 15:38:50 +02:00
Danny Kopping 98c09bf5d2 fix: add policy.go as dependency of Makefile rbac target (#13757) 2024-07-02 14:29:00 +00:00
Danny Kopping 0a221e8d5b feat: create database tables and queries for notifications (#13536) 2024-06-28 09:21:25 +00:00
Danny Kopping d5d8b918d7 feat: lint github actions workflows (#13552) 2024-06-26 10:28:16 +02:00
Danny Kopping 7569cccc51 chore: remove git pinning (#13414)
Alpine 3.20 includes 2.45.1 by default: https://git.alpinelinux.org/aports/tree/main/git/APKBUILD?h=3.20-stable#n56

Follow-up from https://github.com/coder/coder/pull/13411#issuecomment-2139028721

Signed-off-by: Danny Kopping <danny@coder.com>
2024-05-30 15:58:32 +02:00
Danny Kopping 59ab5053b1 fix: return error if agent init script fails to download valid binary (#13280) 2024-05-30 13:33:00 +02:00
Danny Kopping a1671a633c Upgrade to git v2.45.1 to fix alpine 3.20 builds (#13411)
Possibly fixes https://github.com/coder/coder/issues/13407

Signed-off-by: Danny Kopping <danny@coder.com>
2024-05-30 08:36:24 +00:00
Danny Kopping 989575c5b6 chore: prevent commit signing in tests (#13222) 2024-05-10 16:35:59 +02:00
Danny Kopping 4671ebb330 feat: measure pubsub latencies and expose metrics (#13126) 2024-05-10 12:31:49 +00:00
Danny Kopping 319fd5bf1d chore: add e2e test against an external auth provider during workspace creation (#12985) 2024-04-18 19:43:10 +02:00
Danny Kopping b85d5d8491 feat: add warning about use of old/removed/invalid experiments (#12962) 2024-04-17 16:59:31 +02:00
Danny Kopping 06e042acfa chore: apply shellcheck recommendation which was causing "make lint" to fail locally (#12972) 2024-04-16 13:06:08 +02:00
Danny Kopping 79fb8e43c5 feat: expose workspace statuses (with details) as a prometheus metric (#12762)
Implements #12462
2024-04-02 09:57:36 +02:00
Danny Kopping d734f3fb74 chore: reduce azure CA cert validity check period to 2 months (#12788)
Signed-off-by: Danny Kopping <danny@coder.com>
2024-03-28 11:17:02 +02:00
Danny Kopping 6cb1fc8956 chore: add note about options use in numeric parameters (#12770) 2024-03-27 13:46:41 +02:00
Danny Kopping a74ef4096e feat: allow number options with monotonic validation (#12726)
NOTE: terraform-provider-coder was updated to facilitate this change, and your template will require v0.19.0 for this feature to work. You can run terraform init -upgrade in your template directory. If you have a version constraint set, ensure it points to this version.
2024-03-27 08:54:42 +00:00
Danny Kopping 9cfd5baa91 feat(coderd): export metric indicating each experiment's status (#12657) 2024-03-19 14:11:27 +02:00
Danny Kopping ab95ae827d feat(coderd): add enabled experiments to telemetry (#12656) 2024-03-19 11:05:29 +02:00
Danny Kopping 93933d7905 feat(cli): show queue position during workspace builds (#12606) 2024-03-18 12:05:05 +02:00
Danny Kopping 53b58ed74b fix: correct troubleshooting links (#12608) 2024-03-18 08:52:20 +02:00
Danny Kopping 14130deb07 fix: clean template destination path for pull (#12559) 2024-03-14 12:41:23 +00:00
Danny Kopping da54c8a51f fix: fix data race in TestLabelsAggregation tests (#12578) 2024-03-13 13:47:22 +02:00
Danny Kopping 7a7105ad66 feat: make agent stats' cardinality configurable (#12535) 2024-03-13 12:03:36 +02:00
Danny Kopping 90d00190ea chore: remove pr_number param from deploy-pr.sh (#12549)
Field was removed in https://github.com/coder/coder/pull/11259

Signed-off-by: Danny Kopping <danny@coder.com>
2024-03-12 12:01:31 +00:00
Danny Kopping 21d1873d97 feat: make agent stats' cardinality configurable (#12468)
Closes #12221
2024-03-11 16:04:08 +02:00
Danny Kopping 7a92154e67 Install pnpm before calling pnpm exec in make stages (#12471)
Signed-off-by: Danny Kopping <danny@coder.com>
2024-03-08 14:33:28 +02:00