mirror of
https://github.com/coder/coder.git
synced 2026-09-24 15:04:27 +08:00
docs: restructure docs (#14421)
Closes #13434 Supersedes #14182 --------- Co-authored-by: Ethan <39577870+ethanndickson@users.noreply.github.com> Co-authored-by: Ethan Dickson <ethan@coder.com> Co-authored-by: Ben Potter <ben@coder.com> Co-authored-by: Stephen Kirby <58410745+stirby@users.noreply.github.com> Co-authored-by: Stephen Kirby <me@skirby.dev> Co-authored-by: EdwardAngert <17991901+EdwardAngert@users.noreply.github.com> Co-authored-by: Edward Angert <EdwardAngert@users.noreply.github.com>
This commit is contained in:
Muhammad Atif Ali
co-authored by
Ethan
Ethan Dickson
Ben Potter
Stephen Kirby
Stephen Kirby
EdwardAngert
Edward Angert
Ethan
Ethan Dickson
Ben Potter
Stephen Kirby
Stephen Kirby
EdwardAngert
Edward Angert
parent
288df75686
commit
419eba5fb6
@@ -0,0 +1,130 @@
|
||||
# Architecture
|
||||
|
||||
The Coder deployment model is flexible and offers various components that
|
||||
platform administrators can deploy and scale depending on their use case. This
|
||||
page describes possible deployments, challenges, and risks associated with them.
|
||||
|
||||
<div class="tabs">
|
||||
|
||||
## Community Edition
|
||||
|
||||
|
||||
|
||||
## Enterprise
|
||||
|
||||
|
||||
|
||||
## Multi-Region Enterprise
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
## Primary components
|
||||
|
||||
### coderd
|
||||
|
||||
_coderd_ is the service created by running `coder server`. It is a thin API that
|
||||
connects workspaces, provisioners and users. _coderd_ stores its state in
|
||||
Postgres and is the only service that communicates with Postgres.
|
||||
|
||||
It offers:
|
||||
|
||||
- Dashboard (UI)
|
||||
- HTTP API
|
||||
- Dev URLs (HTTP reverse proxy to workspaces)
|
||||
- Workspace Web Applications (e.g for easy access to `code-server`)
|
||||
- Agent registration
|
||||
|
||||
### provisionerd
|
||||
|
||||
_provisionerd_ is the execution context for infrastructure modifying providers.
|
||||
At the moment, the only provider is Terraform (running `terraform`).
|
||||
|
||||
By default, the Coder server runs multiple provisioner daemons.
|
||||
[External provisioners](../provisioners.md) can be added for security or
|
||||
scalability purposes.
|
||||
|
||||
### Workspaces
|
||||
|
||||
At the highest level, a workspace is a set of cloud resources. These resources
|
||||
can be VMs, Kubernetes clusters, storage buckets, or whatever else Terraform
|
||||
lets you dream up.
|
||||
|
||||
The resources that run the agent are described as _computational resources_,
|
||||
while those that don't are called _peripheral resources_.
|
||||
|
||||
Each resource may also be _persistent_ or _ephemeral_ depending on whether
|
||||
they're destroyed on workspace stop.
|
||||
|
||||
### Agents
|
||||
|
||||
An agent is the Coder service that runs within a user's remote workspace. It
|
||||
provides a consistent interface for coderd and clients to communicate with
|
||||
workspaces regardless of operating system, architecture, or cloud.
|
||||
|
||||
It offers the following services along with much more:
|
||||
|
||||
- SSH
|
||||
- Port forwarding
|
||||
- Liveness checks
|
||||
- `startup_script` automation
|
||||
|
||||
Templates are responsible for
|
||||
[creating and running agents](../templates/extending-templates/index.md#workspace-agents)
|
||||
within workspaces.
|
||||
|
||||
## Service Bundling
|
||||
|
||||
While _coderd_ and Postgres can be orchestrated independently, our default
|
||||
installation paths bundle them all together into one system service. It's
|
||||
perfectly fine to run a production deployment this way, but there are certain
|
||||
situations that necessitate decomposition:
|
||||
|
||||
- Reducing global client latency (distribute coderd and centralize database)
|
||||
- Achieving greater availability and efficiency (horizontally scale individual
|
||||
services)
|
||||
|
||||
## Data Layer
|
||||
|
||||
### PostgreSQL (Recommended)
|
||||
|
||||
While `coderd` runs a bundled version of PostgreSQL, we recommend running an
|
||||
external PostgreSQL 13+ database for production deployments.
|
||||
|
||||
A managed PostgreSQL database, with daily backups, is recommended:
|
||||
|
||||
- For AWS: Amazon RDS for PostgreSQL
|
||||
- For Azure: Azure Database for PostgreSQL
|
||||
- Flexible Server For GCP: Cloud SQL for PostgreSQL
|
||||
|
||||
Learn more about database requirements:
|
||||
[Database Health](../monitoring/health-check.md#database)
|
||||
|
||||
### Git Providers (Recommended)
|
||||
|
||||
Users will likely need to pull source code and other artifacts from a git
|
||||
provider. The Coder control plane and workspaces will need network connectivity
|
||||
to the git provider.
|
||||
|
||||
- [GitHub Enterprise](../external-auth.md#github-enterprise)
|
||||
- [GitLab](../external-auth.md#gitlab-self-managed)
|
||||
- [BitBucket](../external-auth.md#bitbucket-server)
|
||||
- [Other Providers](../external-auth.md#self-managed-git-providers)
|
||||
|
||||
### Artifact Manager (Optional)
|
||||
|
||||
Workspaces and templates can pull artifacts from an artifact manager, such as
|
||||
JFrog Artifactory. This can be configured on the infrastructure level, or in
|
||||
some cases within Coder:
|
||||
|
||||
- Tutorial: [JFrog Artifactory and Coder](../integrations/jfrog-artifactory.md)
|
||||
|
||||
### Container Registry (Optional)
|
||||
|
||||
If you prefer not to pull container images for the control plane (`coderd`,
|
||||
`provisionerd`) and workspaces from public container registry (Docker Hub,
|
||||
GitHub Container Registry) you can run your own container registry with Coder.
|
||||
|
||||
To shorten the provisioning time, it is recommended to deploy registry mirrors
|
||||
in the same region as the workspace nodes.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Infrastructure
|
||||
|
||||
Learn how to spin up & manage Coder infrastructure.
|
||||
|
||||
## Architecture
|
||||
|
||||
Coder is a self-hosted platform that runs on your own servers. For large
|
||||
deployments, we recommend running the control plane on Kubernetes. Workspaces
|
||||
can be run as VMs or Kubernetes pods. The control plane (`coderd`) runs in a
|
||||
single region. However, workspace proxies, provisioners, and workspaces can run
|
||||
across regions or even cloud providers for the optimal developer experience.
|
||||
|
||||
Learn more about Coder's
|
||||
[architecture, concepts, and dependencies](./architecture.md).
|
||||
|
||||
## Reference Architectures
|
||||
|
||||
We publish [reference architectures](./validated-architectures/index.md) that
|
||||
include best practices around Coder configuration, infrastructure sizing,
|
||||
autoscaling, and operational readiness for different deployment sizes (e.g.
|
||||
`Up to 2000 users`).
|
||||
|
||||
## Scale Tests
|
||||
|
||||
Use our [scale test utility](./scale-utility.md) that can be run on your Coder
|
||||
deployment to simulate user activity and measure performance.
|
||||
|
||||
## Monitoring
|
||||
|
||||
See our dedicated [Monitoring](../monitoring/index.md) section for details
|
||||
around monitoring your Coder deployment via a bundled Grafana dashboard, health
|
||||
check, and/or within your own observability stack via Prometheus metrics.
|
||||
@@ -0,0 +1,235 @@
|
||||
# Scale Testing
|
||||
|
||||
Scaling Coder involves planning and testing to ensure it can handle more load
|
||||
without compromising service. This process encompasses infrastructure setup,
|
||||
traffic projections, and aggressive testing to identify and mitigate potential
|
||||
bottlenecks.
|
||||
|
||||
A dedicated Kubernetes cluster for Coder is recommended to configure, host and
|
||||
manage Coder workloads. Kubernetes provides container orchestration
|
||||
capabilities, allowing Coder to efficiently deploy, scale, and manage workspaces
|
||||
across a distributed infrastructure. This ensures high availability, fault
|
||||
tolerance, and scalability for Coder deployments. Coder is deployed on this
|
||||
cluster using the
|
||||
[Helm chart](../../install/kubernetes.md#install-coder-with-helm).
|
||||
|
||||
## Methodology
|
||||
|
||||
Our scale tests include the following stages:
|
||||
|
||||
1. Prepare environment: create expected users and provision workspaces.
|
||||
|
||||
2. SSH connections: establish user connections with agents, verifying their
|
||||
ability to echo back received content.
|
||||
|
||||
3. Web Terminal: verify the PTY connection used for communication with Web
|
||||
Terminal.
|
||||
|
||||
4. Workspace application traffic: assess the handling of user connections with
|
||||
specific workspace apps, confirming their capability to echo back received
|
||||
content effectively.
|
||||
|
||||
5. Dashboard evaluation: verify the responsiveness and stability of Coder
|
||||
dashboards under varying load conditions. This is achieved by simulating user
|
||||
interactions using instances of headless Chromium browsers.
|
||||
|
||||
6. Cleanup: delete workspaces and users created in step 1.
|
||||
|
||||
## Infrastructure and setup requirements
|
||||
|
||||
The scale tests runner can distribute the workload to overlap single scenarios
|
||||
based on the workflow configuration:
|
||||
|
||||
| | T0 | T1 | T2 | T3 | T4 | T5 | T6 |
|
||||
| -------------------- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| SSH connections | X | X | X | X | | | |
|
||||
| Web Terminal (PTY) | | X | X | X | X | | |
|
||||
| Workspace apps | | | X | X | X | X | |
|
||||
| Dashboard (headless) | | | | X | X | X | X |
|
||||
|
||||
This pattern closely reflects how our customers naturally use the system. SSH
|
||||
connections are heavily utilized because they're the primary communication
|
||||
channel for IDEs with VS Code and JetBrains plugins.
|
||||
|
||||
The basic setup of scale tests environment involves:
|
||||
|
||||
1. Scale tests runner (32 vCPU, 128 GB RAM)
|
||||
2. Coder: 2 replicas (4 vCPU, 16 GB RAM)
|
||||
3. Database: 1 instance (2 vCPU, 32 GB RAM)
|
||||
4. Provisioner: 50 instances (0.5 vCPU, 512 MB RAM)
|
||||
|
||||
The test is deemed successful if users did not experience interruptions in their
|
||||
workflows, `coderd` did not crash or require restarts, and no other internal
|
||||
errors were observed.
|
||||
|
||||
## Traffic Projections
|
||||
|
||||
In our scale tests, we simulate activity from 2000 users, 2000 workspaces, and
|
||||
2000 agents, with two items of workspace agent metadata being sent every 10
|
||||
seconds. Here are the resulting metrics:
|
||||
|
||||
Coder:
|
||||
|
||||
- Median CPU usage for _coderd_: 3 vCPU, peaking at 3.7 vCPU while all tests are
|
||||
running concurrently.
|
||||
- Median API request rate: 350 RPS during dashboard tests, 250 RPS during Web
|
||||
Terminal and workspace apps tests.
|
||||
- 2000 agent API connections with latency: p90 at 60 ms, p95 at 220 ms.
|
||||
- on average 2400 Web Socket connections during dashboard tests.
|
||||
|
||||
Provisionerd:
|
||||
|
||||
- Median CPU usage is 0.35 vCPU during workspace provisioning.
|
||||
|
||||
Database:
|
||||
|
||||
- Median CPU utilization is 80%, with a significant portion dedicated to writing
|
||||
workspace agent metadata.
|
||||
- Memory utilization averages at 40%.
|
||||
- `write_ops_count` between 6.7 and 8.4 operations per second.
|
||||
|
||||
## Available reference architectures
|
||||
|
||||
[Up to 1,000 users](./validated-architectures/1k-users.md)
|
||||
|
||||
[Up to 2,000 users](./validated-architectures/2k-users.md)
|
||||
|
||||
[Up to 3,000 users](./validated-architectures/3k-users.md)
|
||||
|
||||
## Hardware recommendation
|
||||
|
||||
### Control plane: coderd
|
||||
|
||||
To ensure stability and reliability of the Coder control plane, it's essential
|
||||
to focus on node sizing, resource limits, and the number of replicas. We
|
||||
recommend referencing public cloud providers such as AWS, GCP, and Azure for
|
||||
guidance on optimal configurations. A reasonable approach involves using scaling
|
||||
formulas based on factors like CPU, memory, and the number of users.
|
||||
|
||||
While the minimum requirements specify 1 CPU core and 2 GB of memory per
|
||||
`coderd` replica, it is recommended to allocate additional resources depending
|
||||
on the workload size to ensure deployment stability.
|
||||
|
||||
#### CPU and memory usage
|
||||
|
||||
Enabling
|
||||
[agent stats collection](../../reference/cli/index.md#--prometheus-collect-agent-stats)
|
||||
(optional) may increase memory consumption.
|
||||
|
||||
Enabling direct connections between users and workspace agents (apps or SSH
|
||||
traffic) can help prevent an increase in CPU usage. It is recommended to keep
|
||||
[this option enabled](../../reference/cli/index.md#--disable-direct-connections)
|
||||
unless there are compelling reasons to disable it.
|
||||
|
||||
Inactive users do not consume Coder resources.
|
||||
|
||||
#### Scaling formula
|
||||
|
||||
When determining scaling requirements, consider the following factors:
|
||||
|
||||
- `1 vCPU x 2 GB memory` for every 250 users: A reasonable formula to determine
|
||||
resource allocation based on the number of users and their expected usage
|
||||
patterns.
|
||||
- API latency/response time: Monitor API latency and response times to ensure
|
||||
optimal performance under varying loads.
|
||||
- Average number of HTTP requests: Track the average number of HTTP requests to
|
||||
gauge system usage and identify potential bottlenecks. The number of proxied
|
||||
connections: For a very high number of proxied connections, more memory is
|
||||
required.
|
||||
|
||||
**HTTP API latency**
|
||||
|
||||
For a reliable Coder deployment dealing with medium to high loads, it's
|
||||
important that API calls for workspace/template queries and workspace build
|
||||
operations respond within 300 ms. However, API template insights calls, which
|
||||
involve browsing workspace agent stats and user activity data, may require more
|
||||
time. Moreover, Coder API exposes WebSocket long-lived connections for Web
|
||||
Terminal (bidirectional), and Workspace events/logs (unidirectional).
|
||||
|
||||
If the Coder deployment expects traffic from developers spread across the globe,
|
||||
be aware that customer-facing latency might be higher because of the distance
|
||||
between users and the load balancer. Fortunately, the latency can be improved
|
||||
with a deployment of Coder
|
||||
[workspace proxies](../networking/workspace-proxies.md).
|
||||
|
||||
**Node Autoscaling**
|
||||
|
||||
We recommend disabling the autoscaling for `coderd` nodes. Autoscaling can cause
|
||||
interruptions for user connections, see
|
||||
[Autoscaling](./scale-utility.md#autoscaling) for more details.
|
||||
|
||||
### Control plane: Workspace Proxies
|
||||
|
||||
When scaling [workspace proxies](../networking/workspace-proxies.md), follow the
|
||||
same guidelines as for `coderd` above:
|
||||
|
||||
- `1 vCPU x 2 GB memory` for every 250 users.
|
||||
- Disable autoscaling.
|
||||
|
||||
### Control plane: provisionerd
|
||||
|
||||
Each external provisioner can run a single concurrent workspace build. For
|
||||
example, running 10 provisioner containers will allow 10 users to start
|
||||
workspaces at the same time.
|
||||
|
||||
By default, the Coder server runs 3 built-in provisioner daemons, but the
|
||||
_Enterprise_ Coder release allows for running external provisioners to separate
|
||||
the load caused by workspace provisioning on the `coderd` nodes.
|
||||
|
||||
#### Scaling formula
|
||||
|
||||
When determining scaling requirements, consider the following factors:
|
||||
|
||||
- `1 vCPU x 1 GB memory x 2 concurrent workspace build`: A formula to determine
|
||||
resource allocation based on the number of concurrent workspace builds, and
|
||||
standard complexity of a Terraform template. _Rule of thumb_: the more
|
||||
provisioners are free/available, the more concurrent workspace builds can be
|
||||
performed.
|
||||
|
||||
**Node Autoscaling**
|
||||
|
||||
Autoscaling provisioners is not an easy problem to solve unless it can be
|
||||
predicted when a number of concurrent workspace builds increases.
|
||||
|
||||
We recommend disabling autoscaling and adjusting the number of provisioners to
|
||||
developer needs based on the workspace build queuing time.
|
||||
|
||||
### Data plane: Workspaces
|
||||
|
||||
To determine workspace resource limits and keep the best developer experience
|
||||
for workspace users, administrators must be aware of a few assumptions.
|
||||
|
||||
- Workspace pods run on the same Kubernetes cluster, but possibly in a different
|
||||
namespace or on a separate set of nodes.
|
||||
- Workspace limits (per workspace user):
|
||||
- Evaluate the workspace utilization pattern. For instance, web application
|
||||
development does not require high CPU capacity at all times, but will spike
|
||||
during builds or testing.
|
||||
- Evaluate minimal limits for single workspace. Include in the calculation
|
||||
requirements for Coder agent running in an idle workspace - 0.1 vCPU and 256
|
||||
MB. For instance, developers can choose between 0.5-8 vCPUs, and 1-16 GB
|
||||
memory.
|
||||
|
||||
#### Scaling formula
|
||||
|
||||
When determining scaling requirements, consider the following factors:
|
||||
|
||||
- `1 vCPU x 2 GB memory x 1 workspace`: A formula to determine resource
|
||||
allocation based on the minimal requirements for an idle workspace with a
|
||||
running Coder agent and occasional CPU and memory bursts for building
|
||||
projects.
|
||||
|
||||
**Node Autoscaling**
|
||||
|
||||
Workspace nodes can be set to operate in autoscaling mode to mitigate the risk
|
||||
of prolonged high resource utilization.
|
||||
|
||||
One approach is to scale up workspace nodes when total CPU usage or memory
|
||||
consumption reaches 80%. Another option is to scale based on metrics such as the
|
||||
number of workspaces or active users. It's important to note that as new users
|
||||
onboard, the autoscaling configuration should account for ongoing workspaces.
|
||||
|
||||
Scaling down workspace nodes to zero is not recommended, as it will result in
|
||||
longer wait times for workspace provisioning by users. However, this may be
|
||||
necessary for workspaces with special resource requirements (e.g. GPUs) that
|
||||
incur significant cost overheads.
|
||||
@@ -0,0 +1,255 @@
|
||||
# Scale Tests and Utilities
|
||||
|
||||
We scale-test Coder with [a built-in utility](#scale-testing-utility) that can
|
||||
be used in your environment for insights into how Coder scales with your
|
||||
infrastructure. For scale-testing Kubernetes clusters we recommend to install
|
||||
and use the dedicated Coder template,
|
||||
[scaletest-runner](https://github.com/coder/coder/tree/main/scaletest/templates/scaletest-runner).
|
||||
|
||||
Learn more about [Coder’s architecture](./architecture.md) and our
|
||||
[scale-testing methodology](./scale-testing.md).
|
||||
|
||||
## Recent scale tests
|
||||
|
||||
> Note: the below information is for reference purposes only, and are not
|
||||
> intended to be used as guidelines for infrastructure sizing. Review the
|
||||
> [Reference Architectures](./validated-architectures/index.md#node-sizing) for
|
||||
> hardware sizing recommendations.
|
||||
|
||||
| Environment | Coder CPU | Coder RAM | Coder Replicas | Database | Users | Concurrent builds | Concurrent connections (Terminal/SSH) | Coder Version | Last tested |
|
||||
| ---------------- | --------- | --------- | -------------- | ----------------- | ----- | ----------------- | ------------------------------------- | ------------- | ------------ |
|
||||
| Kubernetes (GKE) | 3 cores | 12 GB | 1 | db-f1-micro | 200 | 3 | 200 simulated | `v0.24.1` | Jun 26, 2023 |
|
||||
| Kubernetes (GKE) | 4 cores | 8 GB | 1 | db-custom-1-3840 | 1500 | 20 | 1,500 simulated | `v0.24.1` | Jun 27, 2023 |
|
||||
| Kubernetes (GKE) | 2 cores | 4 GB | 1 | db-custom-1-3840 | 500 | 20 | 500 simulated | `v0.27.2` | Jul 27, 2023 |
|
||||
| Kubernetes (GKE) | 2 cores | 8 GB | 2 | db-custom-2-7680 | 1000 | 20 | 1000 simulated | `v2.2.1` | Oct 9, 2023 |
|
||||
| Kubernetes (GKE) | 4 cores | 16 GB | 2 | db-custom-8-30720 | 2000 | 50 | 2000 simulated | `v2.8.4` | Feb 28, 2024 |
|
||||
| Kubernetes (GKE) | 2 cores | 4 GB | 2 | db-custom-2-7680 | 1000 | 50 | 1000 simulated | `v2.10.2` | Apr 26, 2024 |
|
||||
|
||||
> Note: a simulated connection reads and writes random data at 40KB/s per
|
||||
> connection.
|
||||
|
||||
## Scale testing utility
|
||||
|
||||
Since Coder's performance is highly dependent on the templates and workflows you
|
||||
support, you may wish to use our internal scale testing utility against your own
|
||||
environments.
|
||||
|
||||
> Note: This utility is experimental. It is not subject to any compatibility
|
||||
> guarantees, and may cause interruptions for your users. To avoid potential
|
||||
> outages and orphaned resources, we recommend running scale tests on a
|
||||
> secondary "staging" environment or a dedicated
|
||||
> [Kubernetes playground cluster](https://github.com/coder/coder/tree/main/scaletest/terraform).
|
||||
> Run it against a production environment at your own risk.
|
||||
|
||||
### Create workspaces
|
||||
|
||||
The following command will provision a number of Coder workspaces using the
|
||||
specified template and extra parameters.
|
||||
|
||||
```shell
|
||||
coder exp scaletest create-workspaces \
|
||||
--retry 5 \
|
||||
--count "${SCALETEST_PARAM_NUM_WORKSPACES}" \
|
||||
--template "${SCALETEST_PARAM_TEMPLATE}" \
|
||||
--concurrency "${SCALETEST_PARAM_CREATE_CONCURRENCY}" \
|
||||
--timeout 5h \
|
||||
--job-timeout 5h \
|
||||
--no-cleanup \
|
||||
--output json:"${SCALETEST_RESULTS_DIR}/create-workspaces.json"
|
||||
|
||||
# Run `coder exp scaletest create-workspaces --help` for all usage
|
||||
```
|
||||
|
||||
The command does the following:
|
||||
|
||||
1. Create `${SCALETEST_PARAM_NUM_WORKSPACES}` workspaces concurrently
|
||||
(concurrency level: `${SCALETEST_PARAM_CREATE_CONCURRENCY}`) using the
|
||||
template `${SCALETEST_PARAM_TEMPLATE}`.
|
||||
1. Leave workspaces running to use in next steps (`--no-cleanup` option).
|
||||
1. Store provisioning results in JSON format.
|
||||
1. If you don't want the creation process to be interrupted by any errors, use
|
||||
the `--retry 5` flag.
|
||||
|
||||
### Traffic Generation
|
||||
|
||||
Given an existing set of workspaces created previously with `create-workspaces`,
|
||||
the following command will generate traffic similar to that of Coder's Web
|
||||
Terminal against those workspaces.
|
||||
|
||||
```shell
|
||||
# Produce load at about 1000MB/s (25MB/40ms).
|
||||
coder exp scaletest workspace-traffic \
|
||||
--template "${SCALETEST_PARAM_GREEDY_AGENT_TEMPLATE}" \
|
||||
--bytes-per-tick $((1024 * 1024 * 25)) \
|
||||
--tick-interval 40ms \
|
||||
--timeout "$((delay))s" \
|
||||
--job-timeout "$((delay))s" \
|
||||
--scaletest-prometheus-address 0.0.0.0:21113 \
|
||||
--target-workspaces "0:100" \
|
||||
--trace=false \
|
||||
--output json:"${SCALETEST_RESULTS_DIR}/traffic-${type}-greedy-agent.json"
|
||||
```
|
||||
|
||||
Traffic generation can be parametrized:
|
||||
|
||||
1. Send `bytes-per-tick` every `tick-interval`.
|
||||
1. Enable tracing for performance debugging.
|
||||
1. Target a range of workspaces with `--target-workspaces 0:100`.
|
||||
1. For dashboard traffic: Target a range of users with `--target-users 0:100`.
|
||||
1. Store provisioning results in JSON format.
|
||||
1. Expose a dedicated Prometheus address (`--scaletest-prometheus-address`) for
|
||||
scaletest-specific metrics.
|
||||
|
||||
The `workspace-traffic` supports also other modes - SSH traffic, workspace app:
|
||||
|
||||
1. For SSH traffic: Use `--ssh` flag to generate SSH traffic instead of Web
|
||||
Terminal.
|
||||
1. For workspace app traffic: Use `--app [wsdi|wsec|wsra]` flag to select app
|
||||
behavior. (modes: _WebSocket discard_, _WebSocket echo_, _WebSocket read_).
|
||||
|
||||
### Cleanup
|
||||
|
||||
The scaletest utility will attempt to clean up all workspaces it creates. If you
|
||||
wish to clean up all workspaces, you can run the following command:
|
||||
|
||||
```shell
|
||||
coder exp scaletest cleanup \
|
||||
--cleanup-job-timeout 2h \
|
||||
--cleanup-timeout 15min
|
||||
```
|
||||
|
||||
This will delete all workspaces and users with the prefix `scaletest-`.
|
||||
|
||||
## Scale testing template
|
||||
|
||||
Consider using a dedicated
|
||||
[scaletest-runner](https://github.com/coder/coder/tree/main/scaletest/templates/scaletest-runner)
|
||||
template alongside the CLI utility for testing large-scale Kubernetes clusters.
|
||||
|
||||
The template deploys a main workspace with scripts used to orchestrate Coder,
|
||||
creating workspaces, generating workspace traffic, or load-testing workspace
|
||||
apps.
|
||||
|
||||
### Parameters
|
||||
|
||||
The _scaletest-runner_ offers the following configuration options:
|
||||
|
||||
- Workspace size selection: minimal/small/medium/large (_default_: minimal,
|
||||
which contains just enough resources for a Coder agent to run without
|
||||
additional workloads)
|
||||
- Number of workspaces
|
||||
- Wait duration between scenarios or staggered approach
|
||||
|
||||
The template exposes parameters to control the traffic dimensions for SSH
|
||||
connections, workspace apps, and dashboard tests:
|
||||
|
||||
- Traffic duration of the load test scenario
|
||||
- Traffic percentage of targeted workspaces
|
||||
- Bytes per tick and tick interval
|
||||
- _For workspace apps_: modes (echo, read random data, or write and discard)
|
||||
|
||||
Scale testing concurrency can be controlled with the following parameters:
|
||||
|
||||
- Enable parallel scenarios - interleave different traffic patterns (SSH,
|
||||
workspace apps, dashboard traffic, etc.)
|
||||
- Workspace creation concurrency level (_default_: 10)
|
||||
- Job concurrency level - generate workspace traffic using multiple jobs
|
||||
(_default_: 0)
|
||||
- Cleanup concurrency level
|
||||
|
||||
### Kubernetes cluster
|
||||
|
||||
It is recommended to learn how to operate the _scaletest-runner_ before running
|
||||
it against the staging cluster (or production at your own risk). Coder provides
|
||||
different
|
||||
[workspace configurations](https://github.com/coder/coder/tree/main/scaletest/templates)
|
||||
that operators can deploy depending on the traffic projections.
|
||||
|
||||
There are a few cluster options available:
|
||||
|
||||
| Workspace size | vCPU | Memory | Persisted storage | Details |
|
||||
| -------------- | ---- | ------ | ----------------- | ----------------------------------------------------- |
|
||||
| minimal | 1 | 2 Gi | None | |
|
||||
| small | 1 | 1 Gi | None | |
|
||||
| medium | 2 | 2 Gi | None | Medium-sized cluster offers the greedy agent variant. |
|
||||
| large | 4 | 4 Gi | None | |
|
||||
|
||||
Note: Review the selected cluster template and edit the node affinity to match
|
||||
your setup.
|
||||
|
||||
#### Greedy agent
|
||||
|
||||
The greedy agent variant is a template modification that makes the Coder agent
|
||||
transmit large metadata (size: 4K) while reporting stats. The transmission of
|
||||
large chunks puts extra overhead on coderd instances and agents when handling
|
||||
and storing the data.
|
||||
|
||||
Use this template variant to verify limits of the cluster performance.
|
||||
|
||||
### Observability
|
||||
|
||||
During scale tests, operators can monitor progress using a Grafana dashboard.
|
||||
Coder offers a comprehensive overview
|
||||
[dashboard](https://github.com/coder/coder/blob/main/scaletest/scaletest_dashboard.json)
|
||||
that can seamlessly integrate into the internal Grafana deployment.
|
||||
|
||||
This dashboard provides insights into various aspects, including:
|
||||
|
||||
- Utilization of resources within the Coder control plane (CPU, memory, pods)
|
||||
- Database performance metrics (CPU, memory, I/O, connections, queries)
|
||||
- Coderd API performance (requests, latency, error rate)
|
||||
- Resource consumption within Coder workspaces (CPU, memory, network usage)
|
||||
- Internal metrics related to provisioner jobs
|
||||
|
||||
Note: Database metrics are disabled by default and can be enabled by setting the
|
||||
environment variable `CODER_PROMETHEUS_COLLECT_DB_METRICS` to `true`.
|
||||
|
||||
It is highly recommended to deploy a solution for centralized log collection and
|
||||
aggregation. The presence of error logs may indicate an underscaled deployment
|
||||
of Coder, necessitating action from operators.
|
||||
|
||||
## Autoscaling
|
||||
|
||||
We generally do not recommend using an autoscaler that modifies the number of
|
||||
coderd replicas. In particular, scale down events can cause interruptions for a
|
||||
large number of users.
|
||||
|
||||
Coderd is different from a simple request-response HTTP service in that it
|
||||
services long-lived connections whenever it proxies HTTP applications like IDEs
|
||||
or terminals that rely on websockets, or when it relays tunneled connections to
|
||||
workspaces. Loss of a coderd replica will drop these long-lived connections and
|
||||
interrupt users. For example, if you have 4 coderd replicas behind a load
|
||||
balancer, and an autoscaler decides to reduce it to 3, roughly 25% of the
|
||||
connections will drop. An even larger proportion of users could be affected if
|
||||
they use applications that use more than one websocket.
|
||||
|
||||
The severity of the interruption varies by application. Coder's web terminal,
|
||||
for example, will reconnect to the same session and continue. So, this should
|
||||
not be interpreted as saying coderd replicas should never be taken down for any
|
||||
reason.
|
||||
|
||||
We recommend you plan to run enough coderd replicas to comfortably meet your
|
||||
weekly high-water-mark load, and monitor coderd peak CPU & memory utilization
|
||||
over the long term, reevaluating periodically. When scaling down (or performing
|
||||
upgrades), schedule these outside normal working hours to minimize user
|
||||
interruptions.
|
||||
|
||||
### A note for Kubernetes users
|
||||
|
||||
When running on Kubernetes on cloud infrastructure (i.e. not bare metal), many
|
||||
operators choose to employ a _cluster_ autoscaler that adds and removes
|
||||
Kubernetes _nodes_ according to load. Coder can coexist with such cluster
|
||||
autoscalers, but we recommend you take steps to prevent the autoscaler from
|
||||
evicting coderd pods, as an eviction will cause the same interruptions as
|
||||
described above. For example, if you are using the
|
||||
[Kubernetes cluster autoscaler](https://kubernetes.io/docs/reference/labels-annotations-taints/#cluster-autoscaler-kubernetes-io-safe-to-evict),
|
||||
you may wish to set `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` as
|
||||
an annotation on the coderd deployment.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If a load test fails or if you are experiencing performance issues during
|
||||
day-to-day use, you can leverage Coder's
|
||||
[Prometheus metrics](../integrations/prometheus.md) to identify bottlenecks
|
||||
during scale tests. Additionally, you can use your existing cloud monitoring
|
||||
stack to measure load, view server logs, etc.
|
||||
@@ -0,0 +1,51 @@
|
||||
# Reference Architecture: up to 1,000 users
|
||||
|
||||
The 1,000 users architecture is designed to cover a wide range of workflows.
|
||||
Examples of subjects that might utilize this architecture include medium-sized
|
||||
tech startups, educational units, or small to mid-sized enterprises.
|
||||
|
||||
**Target load**: API: up to 180 RPS
|
||||
|
||||
**High Availability**: non-essential for small deployments
|
||||
|
||||
## Hardware recommendations
|
||||
|
||||
### Coderd nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | ------------------- | ------------------- | --------------- | ---------- | ----------------- |
|
||||
| Up to 1,000 | 2 vCPU, 8 GB memory | 1-2 / 1 coderd each | `n1-standard-2` | `t3.large` | `Standard_D2s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- For small deployments (ca. 100 users, 10 concurrent workspace builds), it is
|
||||
acceptable to deploy provisioners on `coderd` nodes.
|
||||
|
||||
### Provisioner nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | ------------------------------ | ---------------- | ------------ | ----------------- |
|
||||
| Up to 1,000 | 8 vCPU, 32 GB memory | 2 nodes / 30 provisioners each | `t2d-standard-8` | `t3.2xlarge` | `Standard_D8s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- An external provisioner is deployed as Kubernetes pod.
|
||||
|
||||
### Workspace nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | ----------------------- | ---------------- | ------------ | ----------------- |
|
||||
| Up to 1,000 | 8 vCPU, 32 GB memory | 64 / 16 workspaces each | `t2d-standard-8` | `t3.2xlarge` | `Standard_D8s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- Assumed that a workspace user needs at minimum 2 GB memory to perform. We
|
||||
recommend against over-provisioning memory for developer workloads, as this my
|
||||
lead to OOMKiller invocations.
|
||||
- Maximum number of Kubernetes workspace pods per node: 256
|
||||
|
||||
### Database nodes
|
||||
|
||||
| Users | Node capacity | Replicas | Storage | GCP | AWS | Azure |
|
||||
| ----------- | ------------------- | -------- | ------- | ------------------ | ------------- | ----------------- |
|
||||
| Up to 1,000 | 2 vCPU, 8 GB memory | 1 | 512 GB | `db-custom-2-7680` | `db.t3.large` | `Standard_D2s_v3` |
|
||||
@@ -0,0 +1,59 @@
|
||||
# Reference Architecture: up to 2,000 users
|
||||
|
||||
In the 2,000 users architecture, there is a moderate increase in traffic,
|
||||
suggesting a growing user base or expanding operations. This setup is
|
||||
well-suited for mid-sized companies experiencing growth or for universities
|
||||
seeking to accommodate their expanding user populations.
|
||||
|
||||
Users can be evenly distributed between 2 regions or be attached to different
|
||||
clusters.
|
||||
|
||||
**Target load**: API: up to 300 RPS
|
||||
|
||||
**High Availability**: The mode is _enabled_; multiple replicas provide higher
|
||||
deployment reliability under load.
|
||||
|
||||
## Hardware recommendations
|
||||
|
||||
### Coderd nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | ----------------------- | --------------- | ----------- | ----------------- |
|
||||
| Up to 2,000 | 4 vCPU, 16 GB memory | 2 nodes / 1 coderd each | `n1-standard-4` | `t3.xlarge` | `Standard_D4s_v3` |
|
||||
|
||||
### Provisioner nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | ------------------------------ | ---------------- | ------------ | ----------------- |
|
||||
| Up to 2,000 | 8 vCPU, 32 GB memory | 4 nodes / 30 provisioners each | `t2d-standard-8` | `t3.2xlarge` | `Standard_D8s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- An external provisioner is deployed as Kubernetes pod.
|
||||
- It is not recommended to run provisioner daemons on `coderd` nodes.
|
||||
- Consider separating provisioners into different namespaces in favor of
|
||||
zero-trust or multi-cloud deployments.
|
||||
|
||||
### Workspace nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | ------------------------ | ---------------- | ------------ | ----------------- |
|
||||
| Up to 2,000 | 8 vCPU, 32 GB memory | 128 / 16 workspaces each | `t2d-standard-8` | `t3.2xlarge` | `Standard_D8s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- Assumed that a workspace user needs 2 GB memory to perform
|
||||
- Maximum number of Kubernetes workspace pods per node: 256
|
||||
- Nodes can be distributed in 2 regions, not necessarily evenly split, depending
|
||||
on developer team sizes
|
||||
|
||||
### Database nodes
|
||||
|
||||
| Users | Node capacity | Replicas | Storage | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | -------- | ------- | ------------------- | -------------- | ----------------- |
|
||||
| Up to 2,000 | 4 vCPU, 16 GB memory | 1 | 1 TB | `db-custom-4-15360` | `db.t3.xlarge` | `Standard_D4s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- Consider adding more replicas if the workspace activity is higher than 500
|
||||
workspace builds per day or to achieve higher RPS.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Reference Architecture: up to 3,000 users
|
||||
|
||||
The 3,000 users architecture targets large-scale enterprises, possibly with
|
||||
on-premises network and cloud deployments.
|
||||
|
||||
**Target load**: API: up to 550 RPS
|
||||
|
||||
**High Availability**: Typically, such scale requires a fully-managed HA
|
||||
PostgreSQL service, and all Coder observability features enabled for operational
|
||||
purposes.
|
||||
|
||||
**Observability**: Deploy monitoring solutions to gather Prometheus metrics and
|
||||
visualize them with Grafana to gain detailed insights into infrastructure and
|
||||
application behavior. This allows operators to respond quickly to incidents and
|
||||
continuously improve the reliability and performance of the platform.
|
||||
|
||||
## Hardware recommendations
|
||||
|
||||
### Coderd nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | ----------------- | --------------- | ----------- | ----------------- |
|
||||
| Up to 3,000 | 8 vCPU, 32 GB memory | 4 / 1 coderd each | `n1-standard-4` | `t3.xlarge` | `Standard_D4s_v3` |
|
||||
|
||||
### Provisioner nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | ------------------------ | ---------------- | ------------ | ----------------- |
|
||||
| Up to 3,000 | 8 vCPU, 32 GB memory | 8 / 30 provisioners each | `t2d-standard-8` | `t3.2xlarge` | `Standard_D8s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- An external provisioner is deployed as Kubernetes pod.
|
||||
- It is strongly discouraged to run provisioner daemons on `coderd` nodes at
|
||||
this level of scale.
|
||||
- Separate provisioners into different namespaces in favor of zero-trust or
|
||||
multi-cloud deployments.
|
||||
|
||||
### Workspace nodes
|
||||
|
||||
| Users | Node capacity | Replicas | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | ------------------------------ | ---------------- | ------------ | ----------------- |
|
||||
| Up to 3,000 | 8 vCPU, 32 GB memory | 256 nodes / 12 workspaces each | `t2d-standard-8` | `t3.2xlarge` | `Standard_D8s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- Assumed that a workspace user needs 2 GB memory to perform
|
||||
- Maximum number of Kubernetes workspace pods per node: 256
|
||||
- As workspace nodes can be distributed between regions, on-premises networks
|
||||
and cloud areas, consider different namespaces in favor of zero-trust or
|
||||
multi-cloud deployments.
|
||||
|
||||
### Database nodes
|
||||
|
||||
| Users | Node capacity | Replicas | Storage | GCP | AWS | Azure |
|
||||
| ----------- | -------------------- | -------- | ------- | ------------------- | --------------- | ----------------- |
|
||||
| Up to 3,000 | 8 vCPU, 32 GB memory | 2 | 1.5 TB | `db-custom-8-30720` | `db.t3.2xlarge` | `Standard_D8s_v3` |
|
||||
|
||||
**Footnotes**:
|
||||
|
||||
- Consider adding more replicas if the workspace activity is higher than 1500
|
||||
workspace builds per day or to achieve higher RPS.
|
||||
@@ -0,0 +1,366 @@
|
||||
# Coder Validated Architecture
|
||||
|
||||
Many customers operate Coder in complex organizational environments, consisting
|
||||
of multiple business units, agencies, and/or subsidiaries. This can lead to
|
||||
numerous Coder deployments, due to discrepancies in regulatory compliance, data
|
||||
sovereignty, and level of funding across groups. The Coder Validated
|
||||
Architecture (CVA) prescribes a Kubernetes-based deployment approach, enabling
|
||||
your organization to deploy a stable Coder instance that is easier to maintain
|
||||
and troubleshoot.
|
||||
|
||||
The following sections will detail the components of the Coder Validated
|
||||
Architecture, provide guidance on how to configure and deploy these components,
|
||||
and offer insights into how to maintain and troubleshoot your Coder environment.
|
||||
|
||||
- [General concepts](#general-concepts)
|
||||
- [Kubernetes Infrastructure](#kubernetes-infrastructure)
|
||||
- [PostgreSQL Database](#postgresql-database)
|
||||
- [Operational readiness](#operational-readiness)
|
||||
|
||||
## Who is this document for?
|
||||
|
||||
This guide targets the following personas. It assumes a basic understanding of
|
||||
cloud/on-premise computing, containerization, and the Coder platform.
|
||||
|
||||
| Role | Description |
|
||||
| ------------------------- | ------------------------------------------------------------------------------ |
|
||||
| Platform Engineers | Responsible for deploying, operating the Coder deployment and infrastructure |
|
||||
| Enterprise Architects | Responsible for architecting Coder deployments to meet enterprise requirements |
|
||||
| Managed Service Providers | Entities that deploy and run Coder software as a service for customers |
|
||||
|
||||
## CVA Guidance
|
||||
|
||||
| CVA provides: | CVA does not provide: |
|
||||
| ---------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
||||
| Single and multi-region K8s deployment options | Prescribing OS, or cloud vs. on-premise |
|
||||
| Reference architectures for up to 3,000 users | An approval of your architecture; the CVA solely provides recommendations and guidelines |
|
||||
| Best practices for building a Coder deployment | Recommendations for every possible deployment scenario |
|
||||
|
||||
> For higher level design principles and architectural best practices, see
|
||||
> Coder's
|
||||
> [Well-Architected Framework](https://coder.com/blog/coder-well-architected-framework).
|
||||
|
||||
## General concepts
|
||||
|
||||
This section outlines core concepts and terminology essential for understanding
|
||||
Coder's architecture and deployment strategies.
|
||||
|
||||
### Administrator
|
||||
|
||||
An administrator is a user role within the Coder platform with elevated
|
||||
privileges. Admins have access to administrative functions such as user
|
||||
management, template definitions, insights, and deployment configuration.
|
||||
|
||||
### Coder control plane
|
||||
|
||||
Coder's control plane, also known as _coderd_, is the main service recommended
|
||||
for deployment with multiple replicas to ensure high availability. It provides
|
||||
an API for managing workspaces and templates, and serves the dashboard UI. In
|
||||
addition, each _coderd_ replica hosts 3 Terraform [provisioners](#provisioner)
|
||||
by default.
|
||||
|
||||
### User
|
||||
|
||||
A [user](../../users/index.md) is an individual who utilizes the Coder platform
|
||||
to develop, test, and deploy applications using workspaces. Users can select
|
||||
available templates to provision workspaces. They interact with Coder using the
|
||||
web interface, the CLI tool, or directly calling API methods.
|
||||
|
||||
### Workspace
|
||||
|
||||
A [workspace](../../../user-guides/workspace-management.md) refers to an
|
||||
isolated development environment where users can write, build, and run code.
|
||||
Workspaces are fully configurable and can be tailored to specific project
|
||||
requirements, providing developers with a consistent and efficient development
|
||||
environment. Workspaces can be autostarted and autostopped, enabling efficient
|
||||
resource management.
|
||||
|
||||
Users can connect to workspaces using SSH or via workspace applications like
|
||||
`code-server`, facilitating collaboration and remote access. Additionally,
|
||||
workspaces can be parameterized, allowing users to customize settings and
|
||||
configurations based on their unique needs. Workspaces are instantiated using
|
||||
Coder templates and deployed on resources created by provisioners.
|
||||
|
||||
### Template
|
||||
|
||||
A [template](../../../admin/templates/index.md) in Coder is a predefined
|
||||
configuration for creating workspaces. Templates streamline the process of
|
||||
workspace creation by providing pre-configured settings, tooling, and
|
||||
dependencies. They are built by template administrators on top of Terraform,
|
||||
allowing for efficient management of infrastructure resources. Additionally,
|
||||
templates can utilize Coder modules to leverage existing features shared with
|
||||
other templates, enhancing flexibility and consistency across deployments.
|
||||
Templates describe provisioning rules for infrastructure resources offered by
|
||||
Terraform providers.
|
||||
|
||||
### Workspace Proxy
|
||||
|
||||
A [workspace proxy](../../../admin/networking/workspace-proxies.md) serves as a
|
||||
relay connection option for developers connecting to their workspace over SSH, a
|
||||
workspace app, or through port forwarding. It helps reduce network latency for
|
||||
geo-distributed teams by minimizing the distance network traffic needs to
|
||||
travel. Notably, workspace proxies do not handle dashboard connections or API
|
||||
calls.
|
||||
|
||||
### Provisioner
|
||||
|
||||
Provisioners in Coder execute Terraform during workspace and template builds.
|
||||
While the platform includes built-in provisioner daemons by default, there are
|
||||
advantages to employing external provisioners. These external daemons provide
|
||||
secure build environments and reduce server load, improving performance and
|
||||
scalability. Each provisioner can handle a single concurrent workspace build,
|
||||
allowing for efficient resource allocation and workload management.
|
||||
|
||||
### Registry
|
||||
|
||||
The [Coder Registry](https://registry.coder.com) is a platform where you can
|
||||
find starter templates and _Modules_ for various cloud services and platforms.
|
||||
|
||||
Templates help create self-service development environments using
|
||||
Terraform-defined infrastructure, while _Modules_ simplify template creation by
|
||||
providing common features like workspace applications, third-party integrations,
|
||||
or helper scripts.
|
||||
|
||||
Please note that the Registry is a hosted service and isn't available for
|
||||
offline use.
|
||||
|
||||
## Kubernetes Infrastructure
|
||||
|
||||
Kubernetes is the recommended, and supported platform for deploying Coder in the
|
||||
enterprise. It is the hosting platform of choice for a large majority of Coder's
|
||||
Fortune 500 customers, and it is the platform in which we build and test against
|
||||
here at Coder.
|
||||
|
||||
### General recommendations
|
||||
|
||||
In general, it is recommended to deploy Coder into its own respective cluster,
|
||||
separate from production applications. Keep in mind that Coder runs development
|
||||
workloads, so the cluster should be deployed as such, without production-level
|
||||
configurations.
|
||||
|
||||
### Compute
|
||||
|
||||
Deploy your Kubernetes cluster with two node groups, one for Coder's control
|
||||
plane, and another for user workspaces (if you intend on leveraging K8s for
|
||||
end-user compute).
|
||||
|
||||
#### Control plane nodes
|
||||
|
||||
The Coder control plane node group must be static, to prevent scale down events
|
||||
from dropping pods, and thus dropping user connections to the dashboard UI and
|
||||
their workspaces.
|
||||
|
||||
Coder's Helm Chart supports
|
||||
[defining nodeSelectors, affinities, and tolerations](https://github.com/coder/coder/blob/e96652ebbcdd7554977594286b32015115c3f5b6/helm/coder/values.yaml#L221-L249)
|
||||
to schedule the control plane pods on the appropriate node group.
|
||||
|
||||
#### Workspace nodes
|
||||
|
||||
Coder workspaces can be deployed either as Pods or Deployments in Kubernetes.
|
||||
See our
|
||||
[example Kubernetes workspace template](https://github.com/coder/coder/tree/main/examples/templates/kubernetes).
|
||||
Configure the workspace node group to be auto-scaling, to dynamically allocate
|
||||
compute as users start/stop workspaces at the beginning and end of their day.
|
||||
Set nodeSelectors, affinities, and tolerations in Coder templates to assign
|
||||
workspaces to the given node group:
|
||||
|
||||
```tf
|
||||
resource "kubernetes_deployment" "coder" {
|
||||
spec {
|
||||
template {
|
||||
metadata {
|
||||
labels = {
|
||||
app = "coder-workspace"
|
||||
}
|
||||
}
|
||||
|
||||
spec {
|
||||
affinity {
|
||||
pod_anti_affinity {
|
||||
preferred_during_scheduling_ignored_during_execution {
|
||||
weight = 1
|
||||
pod_affinity_term {
|
||||
label_selector {
|
||||
match_expressions {
|
||||
key = "app.kubernetes.io/instance"
|
||||
operator = "In"
|
||||
values = ["coder-workspace"]
|
||||
}
|
||||
}
|
||||
topology_key = # add your node group label here
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
tolerations {
|
||||
# Add your tolerations here
|
||||
}
|
||||
|
||||
node_selector {
|
||||
# Add your node selectors here
|
||||
}
|
||||
|
||||
container {
|
||||
image = "coder-workspace:latest"
|
||||
name = "dev"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Node sizing
|
||||
|
||||
For sizing recommendations, see the below reference architectures:
|
||||
|
||||
- [Up to 1,000 users](1k-users.md)
|
||||
|
||||
- [Up to 2,000 users](2k-users.md)
|
||||
|
||||
- [Up to 3,000 users](3k-users.md)
|
||||
|
||||
### Networking
|
||||
|
||||
It is likely your enterprise deploys Kubernetes clusters with various networking
|
||||
restrictions. With this in mind, Coder requires the following connectivity:
|
||||
|
||||
- Egress from workspace compute to the Coder control plane pods
|
||||
- Egress from control plane pods to Coder's PostgreSQL database
|
||||
- Egress from control plane pods to git and package repositories
|
||||
- Ingress from user devices to the control plane Load Balancer or Ingress
|
||||
controller
|
||||
|
||||
We recommend configuring your network policies in accordance with the above.
|
||||
Note that Coder workspaces do not require any ports to be open.
|
||||
|
||||
### Storage
|
||||
|
||||
If running Coder workspaces as Kubernetes Pods or Deployments, you will need to
|
||||
assign persistent storage. We recommend leveraging a
|
||||
[supported Container Storage Interface (CSI) driver](https://kubernetes-csi.github.io/docs/drivers.html)
|
||||
in your cluster, with Dynamic Provisioning and read/write, to provide on-demand
|
||||
storage to end-user workspaces.
|
||||
|
||||
The following Kubernetes volume types have been validated by Coder internally,
|
||||
and/or by our customers:
|
||||
|
||||
- [PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim)
|
||||
- [NFS](https://kubernetes.io/docs/concepts/storage/volumes/#nfs)
|
||||
- [subPath](https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath)
|
||||
- [cephfs](https://kubernetes.io/docs/concepts/storage/volumes/#cephfs)
|
||||
|
||||
Our
|
||||
[example Kubernetes workspace template](https://github.com/coder/coder/blob/5b9a65e5c137232351381fc337d9784bc9aeecfc/examples/templates/kubernetes/main.tf#L191-L219)
|
||||
provisions a PersistentVolumeClaim block storage device, attached to the
|
||||
Deployment.
|
||||
|
||||
It is not recommended to mount volumes from the host node(s) into workspaces,
|
||||
for security and reliability purposes. The below volume types are _not_
|
||||
recommended for use with Coder:
|
||||
|
||||
- [Local](https://kubernetes.io/docs/concepts/storage/volumes/#local)
|
||||
- [hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath)
|
||||
|
||||
Not that Coder's control plane filesystem is ephemeral, so no persistent storage
|
||||
is required.
|
||||
|
||||
## PostgreSQL database
|
||||
|
||||
Coder requires access to an external PostgreSQL database to store user data,
|
||||
workspace state, template files, and more. Depending on the scale of the
|
||||
user-base, workspace activity, and High Availability requirements, the amount of
|
||||
CPU and memory resources required by Coder's database may differ.
|
||||
|
||||
### Disaster recovery
|
||||
|
||||
Prepare internal scripts for dumping and restoring your database. We recommend
|
||||
scheduling regular database backups, especially before upgrading Coder to a new
|
||||
release. Coder does not support downgrades without initially restoring the
|
||||
database to the prior version.
|
||||
|
||||
### Performance efficiency
|
||||
|
||||
We highly recommend deploying the PostgreSQL instance in the same region (and if
|
||||
possible, same availability zone) as the Coder server to optimize for low
|
||||
latency connections. We recommend keeping latency under 10ms between the Coder
|
||||
server and database.
|
||||
|
||||
When determining scaling requirements, take into account the following
|
||||
considerations:
|
||||
|
||||
- `2 vCPU x 8 GB RAM x 512 GB storage`: A baseline for database requirements for
|
||||
Coder deployment with less than 1000 users, and low activity level (30% active
|
||||
users). This capacity should be sufficient to support 100 external
|
||||
provisioners.
|
||||
- Storage size depends on user activity, workspace builds, log verbosity,
|
||||
overhead on database encryption, etc.
|
||||
- Allocate two additional CPU core to the database instance for every 1000
|
||||
active users.
|
||||
- Enable High Availability mode for database engine for large scale deployments.
|
||||
|
||||
If you enable
|
||||
[database encryption](../../../admin/security/database-encryption.md) in Coder,
|
||||
consider allocating an additional CPU core to every `coderd` replica.
|
||||
|
||||
#### Resource utilization guidelines
|
||||
|
||||
Below are general recommendations for sizing your PostgreSQL instance:
|
||||
|
||||
- Increase number of vCPU if CPU utilization or database latency is high.
|
||||
- Allocate extra memory if database performance is poor, CPU utilization is low,
|
||||
and memory utilization is high.
|
||||
- Utilize faster disk options (higher IOPS) such as SSDs or NVMe drives for
|
||||
optimal performance enhancement and possibly reduce database load.
|
||||
|
||||
## Operational readiness
|
||||
|
||||
Operational readiness in Coder is about ensuring that everything is set up
|
||||
correctly before launching a platform into production. It involves making sure
|
||||
that the service is reliable, secure, and easily scales accordingly to user-base
|
||||
needs. Operational readiness is crucial because it helps prevent issues that
|
||||
could affect workspace users experience once the platform is live.
|
||||
|
||||
### Helm Chart Configuration
|
||||
|
||||
1. Reference our [Helm chart values file](../../../../helm/coder/values.yaml)
|
||||
and identify the required values for deployment.
|
||||
1. Create a `values.yaml` and add it to your version control system.
|
||||
1. Determine the necessary environment variables. Here is the
|
||||
[full list of supported server environment variables](../../../reference/cli/server.md).
|
||||
1. Follow our documented
|
||||
[steps for installing Coder via Helm](../../../install/kubernetes.md).
|
||||
|
||||
### Template configuration
|
||||
|
||||
1. Establish dedicated accounts for users with the _Template Administrator_
|
||||
role.
|
||||
1. Maintain Coder templates using
|
||||
[version control](../../templates/managing-templates/change-management.md).
|
||||
1. Consider implementing a GitOps workflow to automatically push new template
|
||||
versions into Coder from git. For example, on Github, you can use the
|
||||
[Setup Coder](https://github.com/marketplace/actions/setup-coder) action.
|
||||
1. Evaluate enabling
|
||||
[automatic template updates](../../templates/managing-templates/index.md#template-update-policies-enterprise-premium)
|
||||
upon workspace startup.
|
||||
|
||||
### Observability
|
||||
|
||||
1. Enable the Prometheus endpoint (environment variable:
|
||||
`CODER_PROMETHEUS_ENABLE`).
|
||||
1. Deploy the
|
||||
[Coder Observability bundle](https://github.com/coder/observability) to
|
||||
leverage pre-configured dashboards, alerts, and runbooks for monitoring
|
||||
Coder. This includes integrations between Prometheus, Grafana, Loki, and
|
||||
Alertmanager.
|
||||
1. Review the [Prometheus response](../../integrations/prometheus.md) and set up
|
||||
alarms on selected metrics.
|
||||
|
||||
### User support
|
||||
|
||||
1. Incorporate [support links](../../setup/appearance.md#support-links) into
|
||||
internal documentation accessible from the user context menu. Ensure that
|
||||
hyperlinks are valid and lead to up-to-date materials.
|
||||
1. Encourage the use of `coder support bundle` to allow workspace users to
|
||||
generate and provide network-related diagnostic data.
|
||||
Reference in New Issue
Block a user