mirror of
https://github.com/coder/coder.git
synced 2026-09-21 20:51:01 +08:00
chore: delete scaletest infrastructure (#19603)
We've successfully migrated the latest iteration of our scaletest infrastructure (`scaletest/terraform/action`) to https://github.com/coder/scaletest (private repo). This PR removes the older iterations, and the scriptsfor spinning up & running the load generators against that infrastructure (`scaletest.sh`). The tooling for generating load against a Coder deployment remains untouched, as does the public documentation for that tooling (i.e. `coder exp scaletest`). If we ever need that old scaletest Terraform code, it's always in the git history!
This commit is contained in:
Ethan
@@ -44,7 +44,7 @@ environments.
|
||||
> for your users.
|
||||
> To avoid potential outages and orphaned resources, we recommend that you run
|
||||
> scale tests on a secondary "staging" environment or a dedicated
|
||||
> [Kubernetes playground cluster](https://github.com/coder/coder/tree/main/scaletest/terraform).
|
||||
> Kubernetes playground cluster.
|
||||
>
|
||||
> Run it against a production environment at your own risk.
|
||||
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
# Scale Testing
|
||||
|
||||
This folder contains CLI commands, Terraform code, and scripts to aid in performing load tests of Coder.
|
||||
At a high level, it performs the following steps:
|
||||
|
||||
- Using the Terraform code in `./terraform`, stands up a preconfigured Google Cloud environment
|
||||
consisting of a VPC, GKE Cluster, and CloudSQL instance.
|
||||
> **Note: You must have an existing Google Cloud project available.**
|
||||
- Creates a dedicated namespace for Coder and installs Coder using the Helm chart in this namespace.
|
||||
- Configures the Coder deployment with random credentials and a predefined Kubernetes template.
|
||||
> **Note:** These credentials are stored in `${PROJECT_ROOT}/scaletest/.coderv2/coder.env`.
|
||||
- Creates a number of workspaces and waits for them to all start successfully. These workspaces
|
||||
are ephemeral and do not contain any persistent resources.
|
||||
- Waits for 10 minutes to allow things to settle and establish a baseline.
|
||||
- Generates web terminal traffic to all workspaces for 30 minutes.
|
||||
- Directly after traffic generation, captures goroutine and heap snapshots of the Coder deployment.
|
||||
- Tears down all resources (unless `--skip-cleanup` is specified).
|
||||
|
||||
## Usage
|
||||
|
||||
The main entrypoint is the `scaletest.sh` script.
|
||||
|
||||
```console
|
||||
$ scaletest.sh --help
|
||||
Usage: scaletest.sh --name <name> --project <project> --num-workspaces <num-workspaces> --scenario <scenario> [--dry-run] [--skip-cleanup]
|
||||
```
|
||||
|
||||
### Required arguments
|
||||
|
||||
- `--name`: Name for the loadtest. This is added as a prefix to resources created by Terraform (e.g. `joe-big-loadtest`).
|
||||
- `--project`: Google Cloud project in which to create the resources (example: `my-loadtest-project`).
|
||||
- `--num-workspaces`: Number of workspaces to create (example: `10`).
|
||||
- `--scenario`: Deployment scenario to use (example: `small`). See `terraform/scenario-*.tfvars`.
|
||||
|
||||
> **Note:** In order to capture Prometheus metrics, you must define the environment variables
|
||||
> `SCALETEST_PROMETHEUS_REMOTE_WRITE_USER` and `SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD`.
|
||||
|
||||
### Optional arguments
|
||||
|
||||
- `--dry-run`: Do not perform any action and instead print what would be executed.
|
||||
- `--skip-cleanup`: Do not perform any cleanup. You will be responsible for deleting any resources this creates.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
All of the above arguments may be specified as environment variables. Consult the script for details.
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
To capture Prometheus metrics from the loadtest, two environment variables are required:
|
||||
|
||||
- `SCALETEST_PROMETHEUS_REMOTE_WRITE_USER`
|
||||
- `SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD`
|
||||
|
||||
### Enterprise License
|
||||
|
||||
To add an Enterprise license, set the `SCALETEST_CODER_LICENSE` environment variable to the JWT string
|
||||
|
||||
## Scenarios
|
||||
|
||||
A scenario defines a number of variables that override the default Terraform variables.
|
||||
A number of existing scenarios are provided in `scaletest/terraform/scenario-*.tfvars`.
|
||||
|
||||
For example, `scenario-small.tfvars` includes the following variable definitions:
|
||||
|
||||
```hcl
|
||||
nodepool_machine_type_coder = "t2d-standard-2"
|
||||
nodepool_machine_type_workspaces = "t2d-standard-2"
|
||||
coder_cpu = "1000m" # Leaving 1 CPU for system workloads
|
||||
coder_mem = "4Gi" # Leaving 4GB for system workloads
|
||||
```
|
||||
|
||||
To create your own scenario, simply add a new file `terraform/scenario-$SCENARIO_NAME.tfvars`.
|
||||
In this file, override variables as required, consulting `vars.tf` as needed.
|
||||
You can then use this scenario by specifying `--scenario $SCENARIO_NAME`.
|
||||
For example, if your scenario file were named `scenario-big-whopper2x.tfvars`, you would specify
|
||||
`--scenario=big-whopper2x`.
|
||||
|
||||
## Utility scripts
|
||||
|
||||
A number of utility scripts are provided in `lib`, and are used by `scaletest.sh`:
|
||||
|
||||
- `coder_shim.sh`: a convenience script to run the `coder` binary with a predefined config root.
|
||||
This is intended to allow running Coder CLI commands against the loadtest cluster without
|
||||
modifying a user's existing Coder CLI configuration.
|
||||
- `coder_init.sh`: Performs first-time user setup of an existing Coder instance, generating
|
||||
a random password for the admin user. The admin user is named `admin@coder.com` by default.
|
||||
Credentials are written to `scaletest/.coderv2/coder.env`.
|
||||
- `coder_workspacetraffic.sh`: Runs traffic generation against the loadtest cluster and creates
|
||||
a monitoring manifest for the traffic generation pod. This pod will restart automatically
|
||||
after the traffic generation has completed.
|
||||
|
||||
## Grafana Dashboard
|
||||
|
||||
A sample Grafana dashboard is provided in `scaletest_dashboard.json`. This dashboard is intended
|
||||
to be imported into an existing Grafana instance. It provides a number of useful metrics:
|
||||
|
||||
- **Control Plane Resources**: CPU, memory, and network usage for the Coder deployment, as well as the number of pod restarts.
|
||||
- **Database**: Rows inserted/updated/deleted/returned, active connections, and transactions per second. Fine-grained `sqlQuerier` metrics are provided for Coder's database as well, broken down my query method.
|
||||
- **HTTP requests**: Number of HTTP requests per second, broken down by status code and path.
|
||||
- **Workspace Resources**: CPU, memory, and network usage for all workspaces.
|
||||
- **Workspace Agents**: Workspace agent network usage, connection latency, and number of active connections.
|
||||
- **Workspace Traffic**: Statistics related to workspace traffic generation.
|
||||
- **Internals**: Provisioner job timings, concurrency, workspace builds, and AuthZ duration.
|
||||
|
||||
A subset of these metrics may be useful for a production deployment, but some are only useful
|
||||
for load testing.
|
||||
|
||||
> **Note:** in particular, `sqlQuerier` metrics produce a large number of time series and may cause
|
||||
> increased charges in your metrics provider.
|
||||
@@ -1,240 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
[[ -n ${VERBOSE:-} ]] && set -x
|
||||
set -euo pipefail
|
||||
|
||||
PROJECT_ROOT="$(git rev-parse --show-toplevel)"
|
||||
# shellcheck source=scripts/lib.sh
|
||||
source "${PROJECT_ROOT}/scripts/lib.sh"
|
||||
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
SCALETEST_NAME="${SCALETEST_NAME:-}"
|
||||
SCALETEST_NUM_WORKSPACES="${SCALETEST_NUM_WORKSPACES:-}"
|
||||
SCALETEST_SCENARIO="${SCALETEST_SCENARIO:-}"
|
||||
SCALETEST_PROJECT="${SCALETEST_PROJECT:-}"
|
||||
SCALETEST_PROMETHEUS_REMOTE_WRITE_USER="${SCALETEST_PROMETHEUS_REMOTE_WRITE_USER:-}"
|
||||
SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD="${SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD:-}"
|
||||
SCALETEST_CODER_LICENSE="${SCALETEST_CODER_LICENSE:-}"
|
||||
SCALETEST_SKIP_CLEANUP="${SCALETEST_SKIP_CLEANUP:-0}"
|
||||
SCALETEST_CREATE_CONCURRENCY="${SCALETEST_CREATE_CONCURRENCY:-10}"
|
||||
SCALETEST_TRAFFIC_BYTES_PER_TICK="${SCALETEST_TRAFFIC_BYTES_PER_TICK:-1024}"
|
||||
SCALETEST_TRAFFIC_TICK_INTERVAL="${SCALETEST_TRAFFIC_TICK_INTERVAL:-10s}"
|
||||
SCALETEST_DESTROY="${SCALETEST_DESTROY:-0}"
|
||||
|
||||
script_name=$(basename "$0")
|
||||
args="$(getopt -o "" -l create-concurrency:,destroy,dry-run,help,name:,num-workspaces:,project:,scenario:,skip-cleanup,traffic-bytes-per-tick:,traffic-tick-interval:, -- "$@")"
|
||||
eval set -- "$args"
|
||||
while true; do
|
||||
case "$1" in
|
||||
--create-concurrency)
|
||||
SCALETEST_CREATE_CONCURRENCY="$2"
|
||||
shift 2
|
||||
;;
|
||||
--destroy)
|
||||
SCALETEST_DESTROY=1
|
||||
shift
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=1
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
echo "Usage: $script_name --name <name> --project <project> --num-workspaces <num-workspaces> --scenario <scenario> [--create-concurrency <create-concurrency>] [--destroy] [--dry-run] [--skip-cleanup] [--traffic-bytes-per-tick <number>] [--traffic-tick-interval <duration>]"
|
||||
exit 1
|
||||
;;
|
||||
--name)
|
||||
SCALETEST_NAME="$2"
|
||||
shift 2
|
||||
;;
|
||||
--num-workspaces)
|
||||
SCALETEST_NUM_WORKSPACES="$2"
|
||||
shift 2
|
||||
;;
|
||||
--project)
|
||||
SCALETEST_PROJECT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--scenario)
|
||||
SCALETEST_SCENARIO="$2"
|
||||
shift 2
|
||||
;;
|
||||
--skip-cleanup)
|
||||
SCALETEST_SKIP_CLEANUP=1
|
||||
shift
|
||||
;;
|
||||
--traffic-bytes-per-tick)
|
||||
SCALETEST_TRAFFIC_BYTES_PER_TICK="$2"
|
||||
shift 2
|
||||
;;
|
||||
--traffic-tick-interval)
|
||||
SCALETEST_TRAFFIC_TICK_INTERVAL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--)
|
||||
shift
|
||||
break
|
||||
;;
|
||||
*)
|
||||
error "Unrecognized option: $1"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
dependencies gcloud kubectl terraform
|
||||
|
||||
if [[ -z "${SCALETEST_NAME}" ]]; then
|
||||
echo "Must specify --name"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${SCALETEST_PROJECT}" ]]; then
|
||||
echo "Must specify --project"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${SCALETEST_NUM_WORKSPACES}" ]]; then
|
||||
echo "Must specify --num-workspaces"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${SCALETEST_SCENARIO}" ]]; then
|
||||
echo "Must specify --scenario"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${SCALETEST_PROMETHEUS_REMOTE_WRITE_USER}" ]] || [[ -z "${SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD}" ]]; then
|
||||
echo "SCALETEST_PROMETHEUS_REMOTE_WRITE_USER or SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD not specified."
|
||||
echo "No prometheus metrics will be collected!"
|
||||
read -p "Continue (y/N)? " -n1 -r
|
||||
if [[ "${REPLY}" != [yY] ]]; then
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
SCALETEST_SCENARIO_VARS="${PROJECT_ROOT}/scaletest/terraform/scenario-${SCALETEST_SCENARIO}.tfvars"
|
||||
if [[ ! -f "${SCALETEST_SCENARIO_VARS}" ]]; then
|
||||
echo "Scenario ${SCALETEST_SCENARIO_VARS} not found."
|
||||
echo "Please create it or choose another scenario:"
|
||||
find "${PROJECT_ROOT}/scaletest/terraform" -type f -name 'scenario-*.tfvars'
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${SCALETEST_SKIP_CLEANUP}" == 1 ]]; then
|
||||
log "WARNING: you told me to not clean up after myself, so this is now your job!"
|
||||
fi
|
||||
|
||||
CONFIG_DIR="${PROJECT_ROOT}/scaletest/.coderv2"
|
||||
if [[ -d "${CONFIG_DIR}" ]] && files=$(ls -qAH -- "${CONFIG_DIR}") && [[ -z "$files" ]]; then
|
||||
echo "Cleaning previous configuration"
|
||||
maybedryrun "$DRY_RUN" rm -fv "${CONFIG_DIR}/*"
|
||||
fi
|
||||
maybedryrun "$DRY_RUN" mkdir -p "${CONFIG_DIR}"
|
||||
|
||||
SCALETEST_SCENARIO_VARS="${PROJECT_ROOT}/scaletest/terraform/scenario-${SCALETEST_SCENARIO}.tfvars"
|
||||
SCALETEST_SECRETS="${PROJECT_ROOT}/scaletest/terraform/secrets.tfvars"
|
||||
SCALETEST_SECRETS_TEMPLATE="${PROJECT_ROOT}/scaletest/terraform/secrets.tfvars.tpl"
|
||||
|
||||
log "Writing scaletest secrets to file."
|
||||
SCALETEST_NAME="${SCALETEST_NAME}" \
|
||||
SCALETEST_PROJECT="${SCALETEST_PROJECT}" \
|
||||
SCALETEST_PROMETHEUS_REMOTE_WRITE_USER="${SCALETEST_PROMETHEUS_REMOTE_WRITE_USER}" \
|
||||
SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD="${SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD}" \
|
||||
envsubst <"${SCALETEST_SECRETS_TEMPLATE}" >"${SCALETEST_SECRETS}"
|
||||
|
||||
pushd "${PROJECT_ROOT}/scaletest/terraform"
|
||||
|
||||
echo "Initializing terraform."
|
||||
maybedryrun "$DRY_RUN" terraform init
|
||||
|
||||
echo "Setting up infrastructure."
|
||||
maybedryrun "$DRY_RUN" terraform apply --var-file="${SCALETEST_SCENARIO_VARS}" --var-file="${SCALETEST_SECRETS}" --var state=started --auto-approve
|
||||
|
||||
if [[ "${DRY_RUN}" != 1 ]]; then
|
||||
SCALETEST_CODER_URL=$(<"${CONFIG_DIR}/url")
|
||||
else
|
||||
SCALETEST_CODER_URL="http://coder.dryrun.local:3000"
|
||||
fi
|
||||
KUBECONFIG="${PROJECT_ROOT}/scaletest/.coderv2/${SCALETEST_NAME}-cluster.kubeconfig"
|
||||
echo "Waiting for Coder deployment at ${SCALETEST_CODER_URL} to become ready"
|
||||
max_attempts=10
|
||||
for attempt in $(seq 1 $max_attempts); do
|
||||
maybedryrun "$DRY_RUN" curl --silent --fail --output /dev/null "${SCALETEST_CODER_URL}/api/v2/buildinfo"
|
||||
curl_status=$?
|
||||
if [[ $curl_status -eq 0 ]]; then
|
||||
break
|
||||
fi
|
||||
if attempt -eq $max_attempts; then
|
||||
echo
|
||||
echo "Coder deployment failed to become ready in time!"
|
||||
exit 1
|
||||
fi
|
||||
echo "Coder deployment not ready yet (${attempt}/${max_attempts}), sleeping 3 seconds"
|
||||
maybedryrun "$DRY_RUN" sleep 3
|
||||
done
|
||||
|
||||
echo "Initializing Coder deployment."
|
||||
DRY_RUN="$DRY_RUN" "${PROJECT_ROOT}/scaletest/lib/coder_init.sh" "${SCALETEST_CODER_URL}"
|
||||
|
||||
if [[ -n "${SCALETEST_CODER_LICENSE}" ]]; then
|
||||
echo "Applying Coder Enterprise License"
|
||||
DRY_RUN="$DRY_RUN" "${PROJECT_ROOT}/scaletest/lib/coder_shim.sh" license add -l "${SCALETEST_CODER_LICENSE}"
|
||||
fi
|
||||
|
||||
echo "Creating ${SCALETEST_NUM_WORKSPACES} workspaces."
|
||||
DRY_RUN="$DRY_RUN" "${PROJECT_ROOT}/scaletest/lib/coder_shim.sh" exp scaletest create-workspaces \
|
||||
--count "${SCALETEST_NUM_WORKSPACES}" \
|
||||
--template=kubernetes \
|
||||
--concurrency "${SCALETEST_CREATE_CONCURRENCY}" \
|
||||
--no-cleanup
|
||||
|
||||
echo "Sleeping 10 minutes to establish a baseline measurement."
|
||||
maybedryrun "$DRY_RUN" sleep 600
|
||||
|
||||
echo "Sending traffic to workspaces"
|
||||
maybedryrun "$DRY_RUN" "${PROJECT_ROOT}/scaletest/lib/coder_workspacetraffic.sh" \
|
||||
--name "${SCALETEST_NAME}" \
|
||||
--traffic-bytes-per-tick "${SCALETEST_TRAFFIC_BYTES_PER_TICK}" \
|
||||
--traffic-tick-interval "${SCALETEST_TRAFFIC_TICK_INTERVAL}"
|
||||
maybedryrun "$DRY_RUN" kubectl --kubeconfig="${KUBECONFIG}" -n "coder-${SCALETEST_NAME}" wait pods coder-scaletest-workspace-traffic --for condition=Ready
|
||||
|
||||
echo "Sleeping 15 minutes for traffic generation"
|
||||
maybedryrun "$DRY_RUN" sleep 900
|
||||
|
||||
echo "Starting pprof"
|
||||
maybedryrun "$DRY_RUN" kubectl -n "coder-${SCALETEST_NAME}" port-forward deployment/coder 6061:6060 &
|
||||
pfpid=$!
|
||||
maybedryrun "$DRY_RUN" trap "kill $pfpid" EXIT
|
||||
|
||||
echo "Waiting for pprof endpoint to become available"
|
||||
pprof_attempt_counter=0
|
||||
while ! maybedryrun "$DRY_RUN" timeout 1 bash -c "echo > /dev/tcp/localhost/6061"; do
|
||||
if [[ $pprof_attempt_counter -eq 10 ]]; then
|
||||
echo
|
||||
echo "pprof failed to become ready in time!"
|
||||
exit 1
|
||||
fi
|
||||
((pprof_attempt_counter += 1))
|
||||
maybedryrun "$DRY_RUN" sleep 3
|
||||
done
|
||||
|
||||
echo "Taking pprof snapshots"
|
||||
maybedryrun "$DRY_RUN" curl --silent --fail --output "${SCALETEST_NAME}-heap.pprof.gz" http://localhost:6061/debug/pprof/heap
|
||||
maybedryrun "$DRY_RUN" curl --silent --fail --output "${SCALETEST_NAME}-goroutine.pprof.gz" http://localhost:6061/debug/pprof/goroutine
|
||||
# No longer need to port-forward
|
||||
maybedryrun "$DRY_RUN" kill "$pfpid"
|
||||
maybedryrun "$DRY_RUN" trap - EXIT
|
||||
|
||||
if [[ "${SCALETEST_SKIP_CLEANUP}" == 1 ]]; then
|
||||
echo "Leaving resources up for you to inspect."
|
||||
echo "Please don't forget to clean up afterwards:"
|
||||
echo "cd terraform && terraform destroy --var-file=${SCALETEST_SCENARIO_VARS} --var-file=${SCALETEST_SECRETS} --auto-approve"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "${SCALETEST_DESTROY}" == 1 ]]; then
|
||||
echo "Destroying infrastructure"
|
||||
maybedryrun "$DRY_RUN" terraform destroy --var-file="${SCALETEST_SCENARIO_VARS}" --var-file="${SCALETEST_SECRETS}" --auto-approve
|
||||
else
|
||||
echo "Scaling down infrastructure"
|
||||
maybedryrun "$DRY_RUN" terraform apply --var-file="${SCALETEST_SCENARIO_VARS}" --var-file="${SCALETEST_SECRETS}" --var state=stopped --auto-approve
|
||||
fi
|
||||
@@ -1 +0,0 @@
|
||||
*.tfvars
|
||||
@@ -1,21 +0,0 @@
|
||||
data "cloudflare_zone" "domain" {
|
||||
name = var.cloudflare_domain
|
||||
}
|
||||
|
||||
resource "cloudflare_record" "coder" {
|
||||
for_each = local.deployments
|
||||
zone_id = data.cloudflare_zone.domain.zone_id
|
||||
name = "${each.value.subdomain}.${var.cloudflare_domain}"
|
||||
content = google_compute_address.coder[each.key].address
|
||||
type = "A"
|
||||
ttl = 3600
|
||||
}
|
||||
|
||||
resource "cloudflare_record" "coder_wildcard" {
|
||||
for_each = local.deployments
|
||||
zone_id = data.cloudflare_zone.domain.id
|
||||
name = each.value.wildcard_subdomain
|
||||
content = cloudflare_record.coder[each.key].name
|
||||
type = "CNAME"
|
||||
ttl = 3600
|
||||
}
|
||||
@@ -1,120 +0,0 @@
|
||||
coder:
|
||||
workspaceProxy: ${workspace_proxy}
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${node_pool}"]
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 1
|
||||
podAffinityTerm:
|
||||
topologyKey: "kubernetes.io/hostname"
|
||||
labelSelector:
|
||||
matchExpressions:
|
||||
- key: "app.kubernetes.io/instance"
|
||||
operator: "In"
|
||||
values: ["${release_name}"]
|
||||
env:
|
||||
%{~ if workspace_proxy ~}
|
||||
- name: "CODER_ACCESS_URL"
|
||||
value: "${access_url}"
|
||||
- name: "CODER_WILDCARD_ACCESS_URL"
|
||||
value: "${wildcard_access_url}"
|
||||
- name: CODER_PRIMARY_ACCESS_URL
|
||||
value: "${primary_url}"
|
||||
- name: CODER_PROXY_SESSION_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: token
|
||||
name: "${proxy_token}"
|
||||
%{~ endif ~}
|
||||
%{~ if provisionerd ~}
|
||||
- name: "CODER_URL"
|
||||
value: "${access_url}"
|
||||
- name: "CODER_PROVISIONERD_TAGS"
|
||||
value: "scope=organization,deployment=${deployment}"
|
||||
- name: "CODER_PROVISIONER_DAEMON_NAME"
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: "CODER_CONFIG_DIR"
|
||||
value: "/tmp/config"
|
||||
%{~ endif ~}
|
||||
%{~ if !workspace_proxy && !provisionerd ~}
|
||||
- name: "CODER_ACCESS_URL"
|
||||
value: "${access_url}"
|
||||
- name: "CODER_WILDCARD_ACCESS_URL"
|
||||
value: "${wildcard_access_url}"
|
||||
- name: "CODER_PG_CONNECTION_URL"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: "${db_secret}"
|
||||
key: url
|
||||
- name: "CODER_PROVISIONER_DAEMONS"
|
||||
value: "0"
|
||||
- name: CODER_PROVISIONER_DAEMON_PSK
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: psk
|
||||
name: "${provisionerd_psk}"
|
||||
- name: "CODER_PROMETHEUS_COLLECT_AGENT_STATS"
|
||||
value: "true"
|
||||
- name: "CODER_PROMETHEUS_COLLECT_DB_METRICS"
|
||||
value: "true"
|
||||
- name: "CODER_PPROF_ENABLE"
|
||||
value: "true"
|
||||
%{~ endif ~}
|
||||
- name: "CODER_CACHE_DIRECTORY"
|
||||
value: "/tmp/coder"
|
||||
- name: "CODER_TELEMETRY_ENABLE"
|
||||
value: "false"
|
||||
- name: "CODER_LOGGING_HUMAN"
|
||||
value: "/dev/null"
|
||||
- name: "CODER_LOGGING_STACKDRIVER"
|
||||
value: "/dev/stderr"
|
||||
- name: "CODER_PROMETHEUS_ENABLE"
|
||||
value: "true"
|
||||
- name: "CODER_VERBOSE"
|
||||
value: "true"
|
||||
- name: "CODER_EXPERIMENTS"
|
||||
value: "${experiments}"
|
||||
- name: "CODER_DANGEROUS_DISABLE_RATE_LIMITS"
|
||||
value: "true"
|
||||
- name: "CODER_DANGEROUS_ALLOW_PATH_APP_SITE_OWNER_ACCESS"
|
||||
value: "true"
|
||||
image:
|
||||
repo: ${image_repo}
|
||||
tag: ${image_tag}
|
||||
replicaCount: "${replicas}"
|
||||
resources:
|
||||
requests:
|
||||
cpu: "${cpu_request}"
|
||||
memory: "${mem_request}"
|
||||
limits:
|
||||
cpu: "${cpu_limit}"
|
||||
memory: "${mem_limit}"
|
||||
securityContext:
|
||||
readOnlyRootFilesystem: true
|
||||
%{~ if !provisionerd ~}
|
||||
service:
|
||||
enable: true
|
||||
sessionAffinity: None
|
||||
loadBalancerIP: "${ip_address}"
|
||||
%{~ endif ~}
|
||||
volumeMounts:
|
||||
- mountPath: "/tmp"
|
||||
name: cache
|
||||
readOnly: false
|
||||
volumes:
|
||||
- emptyDir:
|
||||
sizeLimit: 1024Mi
|
||||
name: cache
|
||||
%{~ if !provisionerd ~}
|
||||
tls:
|
||||
secretNames:
|
||||
- "${tls_secret_name}"
|
||||
%{~ endif ~}
|
||||
@@ -1,102 +0,0 @@
|
||||
data "http" "coder_healthy" {
|
||||
url = local.deployments.primary.url
|
||||
// Wait up to 5 minutes for DNS to propagate
|
||||
retry {
|
||||
attempts = 30
|
||||
min_delay_ms = 10000
|
||||
}
|
||||
|
||||
lifecycle {
|
||||
postcondition {
|
||||
condition = self.status_code == 200
|
||||
error_message = "${self.url} returned an unhealthy status code"
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [helm_release.coder_primary, cloudflare_record.coder["primary"]]
|
||||
}
|
||||
|
||||
resource "null_resource" "api_key" {
|
||||
provisioner "local-exec" {
|
||||
interpreter = ["/bin/bash", "-c"]
|
||||
command = <<EOF
|
||||
set -e
|
||||
|
||||
curl '${local.deployments.primary.url}/api/v2/users/first' \
|
||||
--data-raw $'{"email":"${local.coder_admin_email}","password":"${local.coder_admin_password}","username":"${local.coder_admin_user}","name":"${local.coder_admin_full_name}","trial":false}' \
|
||||
--insecure --silent --output /dev/null
|
||||
|
||||
session_token=$(curl '${local.deployments.primary.url}/api/v2/users/login' \
|
||||
--data-raw $'{"email":"${local.coder_admin_email}","password":"${local.coder_admin_password}"}' \
|
||||
--insecure --silent | jq -r .session_token)
|
||||
|
||||
echo -n $${session_token} > ${path.module}/.coderv2/session_token
|
||||
|
||||
api_key=$(curl '${local.deployments.primary.url}/api/v2/users/me/keys/tokens' \
|
||||
-H "Coder-Session-Token: $${session_token}" \
|
||||
--data-raw '{"token_name":"terraform","scope":"all"}' \
|
||||
--insecure --silent | jq -r .key)
|
||||
|
||||
echo -n $${api_key} > ${path.module}/.coderv2/api_key
|
||||
EOF
|
||||
}
|
||||
|
||||
depends_on = [data.http.coder_healthy]
|
||||
}
|
||||
|
||||
data "local_file" "api_key" {
|
||||
filename = "${path.module}/.coderv2/api_key"
|
||||
depends_on = [null_resource.api_key]
|
||||
}
|
||||
|
||||
resource "null_resource" "license" {
|
||||
provisioner "local-exec" {
|
||||
interpreter = ["/bin/bash", "-c"]
|
||||
command = <<EOF
|
||||
curl '${local.deployments.primary.url}/api/v2/licenses' \
|
||||
-H "Coder-Session-Token: ${trimspace(data.local_file.api_key.content)}" \
|
||||
--data-raw '{"license":"${var.coder_license}"}' \
|
||||
--insecure --silent --output /dev/null
|
||||
EOF
|
||||
}
|
||||
}
|
||||
|
||||
resource "null_resource" "europe_proxy_token" {
|
||||
provisioner "local-exec" {
|
||||
interpreter = ["/bin/bash", "-c"]
|
||||
command = <<EOF
|
||||
curl '${local.deployments.primary.url}/api/v2/workspaceproxies' \
|
||||
-H "Coder-Session-Token: ${trimspace(data.local_file.api_key.content)}" \
|
||||
--data-raw '{"name":"europe","display_name":"Europe","icon":"/emojis/1f950.png"}' \
|
||||
--insecure --silent \
|
||||
| jq -r .proxy_token > ${path.module}/.coderv2/europe_proxy_token
|
||||
EOF
|
||||
}
|
||||
|
||||
depends_on = [null_resource.license]
|
||||
}
|
||||
|
||||
data "local_file" "europe_proxy_token" {
|
||||
filename = "${path.module}/.coderv2/europe_proxy_token"
|
||||
depends_on = [null_resource.europe_proxy_token]
|
||||
}
|
||||
|
||||
resource "null_resource" "asia_proxy_token" {
|
||||
provisioner "local-exec" {
|
||||
interpreter = ["/bin/bash", "-c"]
|
||||
command = <<EOF
|
||||
curl '${local.deployments.primary.url}/api/v2/workspaceproxies' \
|
||||
-H "Coder-Session-Token: ${trimspace(data.local_file.api_key.content)}" \
|
||||
--data-raw '{"name":"asia","display_name":"Asia","icon":"/emojis/1f35b.png"}' \
|
||||
--insecure --silent \
|
||||
| jq -r .proxy_token > ${path.module}/.coderv2/asia_proxy_token
|
||||
EOF
|
||||
}
|
||||
|
||||
depends_on = [null_resource.license]
|
||||
}
|
||||
|
||||
data "local_file" "asia_proxy_token" {
|
||||
filename = "${path.module}/.coderv2/asia_proxy_token"
|
||||
depends_on = [null_resource.asia_proxy_token]
|
||||
}
|
||||
@@ -1,340 +0,0 @@
|
||||
resource "local_file" "kubernetes_template" {
|
||||
filename = "${path.module}/.coderv2/templates/kubernetes/main.tf"
|
||||
content = <<EOF
|
||||
terraform {
|
||||
required_providers {
|
||||
coder = {
|
||||
source = "coder/coder"
|
||||
version = "~> 2.1.0"
|
||||
}
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = "~> 2.30"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "coder" {}
|
||||
|
||||
provider "kubernetes" {
|
||||
config_path = null # always use host
|
||||
}
|
||||
|
||||
data "coder_workspace" "me" {}
|
||||
data "coder_workspace_owner" "me" {}
|
||||
|
||||
resource "coder_agent" "main" {
|
||||
os = "linux"
|
||||
arch = "amd64"
|
||||
}
|
||||
|
||||
resource "coder_script" "websocat" {
|
||||
agent_id = coder_agent.main.id
|
||||
display_name = "websocat"
|
||||
script = <<EOF2
|
||||
curl -sSL -o /tmp/websocat https://github.com/vi/websocat/releases/download/v1.12.0/websocat.x86_64-unknown-linux-musl
|
||||
chmod +x /tmp/websocat
|
||||
|
||||
/tmp/websocat --exit-on-eof --binary ws-l:127.0.0.1:1234 mirror:
|
||||
EOF2
|
||||
run_on_start = true
|
||||
}
|
||||
|
||||
resource "coder_app" "wsecho" {
|
||||
agent_id = coder_agent.main.id
|
||||
slug = "wsec"
|
||||
display_name = "WebSocket Echo"
|
||||
url = "http://localhost:1234"
|
||||
share = "authenticated"
|
||||
}
|
||||
|
||||
resource "kubernetes_pod" "main" {
|
||||
count = data.coder_workspace.me.start_count
|
||||
metadata {
|
||||
name = "coder-$${lower(data.coder_workspace_owner.me.name)}-$${lower(data.coder_workspace.me.name)}"
|
||||
namespace = "${local.coder_namespace}"
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "coder-workspace"
|
||||
"app.kubernetes.io/instance" = "coder-workspace-$${lower(data.coder_workspace_owner.me.name)}-$${lower(data.coder_workspace.me.name)}"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
security_context {
|
||||
run_as_user = "1000"
|
||||
fs_group = "1000"
|
||||
}
|
||||
container {
|
||||
name = "dev"
|
||||
image = "${var.workspace_image}"
|
||||
image_pull_policy = "Always"
|
||||
command = ["sh", "-c", coder_agent.main.init_script]
|
||||
security_context {
|
||||
run_as_user = "1000"
|
||||
}
|
||||
env {
|
||||
name = "CODER_AGENT_TOKEN"
|
||||
value = coder_agent.main.token
|
||||
}
|
||||
resources {
|
||||
requests = {
|
||||
"cpu" = "${local.scenarios[var.scenario].workspaces.cpu_request}"
|
||||
"memory" = "${local.scenarios[var.scenario].workspaces.mem_request}"
|
||||
}
|
||||
limits = {
|
||||
"cpu" = "${local.scenarios[var.scenario].workspaces.cpu_limit}"
|
||||
"memory" = "${local.scenarios[var.scenario].workspaces.mem_limit}"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["primary_workspaces"].name}","${google_container_node_pool.node_pool["europe_workspaces"].name}","${google_container_node_pool.node_pool["asia_workspaces"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
}
|
||||
|
||||
resource "kubernetes_config_map" "template_primary" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
metadata {
|
||||
name = "coder-template"
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
}
|
||||
|
||||
data = {
|
||||
"main.tf" = local_file.kubernetes_template.content
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "push_template_primary" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
metadata {
|
||||
name = "${var.name}-push-template"
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-push-template"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["primary_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = [
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"templates",
|
||||
"push",
|
||||
"--directory=/home/coder/template",
|
||||
"--provisioner-tag=scope=organization",
|
||||
"--provisioner-tag=deployment=primary",
|
||||
"--yes",
|
||||
"kubernetes-primary"
|
||||
]
|
||||
volume_mount {
|
||||
name = "coder-template"
|
||||
mount_path = "/home/coder/template/main.tf"
|
||||
sub_path = "main.tf"
|
||||
}
|
||||
}
|
||||
volume {
|
||||
name = "coder-template"
|
||||
config_map {
|
||||
name = kubernetes_config_map.template_primary.metadata.0.name
|
||||
}
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
depends_on = [helm_release.provisionerd_primary]
|
||||
}
|
||||
|
||||
resource "kubernetes_config_map" "template_europe" {
|
||||
provider = kubernetes.europe
|
||||
|
||||
metadata {
|
||||
name = "coder-template"
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
}
|
||||
|
||||
data = {
|
||||
"main.tf" = local_file.kubernetes_template.content
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "push_template_europe" {
|
||||
provider = kubernetes.europe
|
||||
|
||||
metadata {
|
||||
name = "${var.name}-push-template"
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-push-template"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["europe_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = [
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"templates",
|
||||
"push",
|
||||
"--directory=/home/coder/template",
|
||||
"--provisioner-tag=scope=organization",
|
||||
"--provisioner-tag=deployment=europe",
|
||||
"--yes",
|
||||
"kubernetes-europe"
|
||||
]
|
||||
volume_mount {
|
||||
name = "coder-template"
|
||||
mount_path = "/home/coder/template/main.tf"
|
||||
sub_path = "main.tf"
|
||||
}
|
||||
}
|
||||
volume {
|
||||
name = "coder-template"
|
||||
config_map {
|
||||
name = kubernetes_config_map.template_europe.metadata.0.name
|
||||
}
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
depends_on = [helm_release.provisionerd_europe]
|
||||
}
|
||||
|
||||
resource "kubernetes_config_map" "template_asia" {
|
||||
provider = kubernetes.asia
|
||||
|
||||
metadata {
|
||||
name = "coder-template"
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
}
|
||||
|
||||
data = {
|
||||
"main.tf" = local_file.kubernetes_template.content
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "push_template_asia" {
|
||||
provider = kubernetes.asia
|
||||
|
||||
metadata {
|
||||
name = "${var.name}-push-template"
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-push-template"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["asia_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = [
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"templates",
|
||||
"push",
|
||||
"--directory=/home/coder/template",
|
||||
"--provisioner-tag=scope=organization",
|
||||
"--provisioner-tag=deployment=asia",
|
||||
"--yes",
|
||||
"kubernetes-asia"
|
||||
]
|
||||
volume_mount {
|
||||
name = "coder-template"
|
||||
mount_path = "/home/coder/template/main.tf"
|
||||
sub_path = "main.tf"
|
||||
}
|
||||
}
|
||||
volume {
|
||||
name = "coder-template"
|
||||
config_map {
|
||||
name = kubernetes_config_map.template_asia.metadata.0.name
|
||||
}
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
depends_on = [helm_release.provisionerd_asia]
|
||||
}
|
||||
@@ -1,228 +0,0 @@
|
||||
locals {
|
||||
wait_baseline_duration = "5m"
|
||||
bytes_per_tick = 1024
|
||||
tick_interval = "100ms"
|
||||
|
||||
traffic_types = {
|
||||
ssh = {
|
||||
duration = "30m"
|
||||
job_timeout = "35m"
|
||||
flags = [
|
||||
"--ssh",
|
||||
]
|
||||
}
|
||||
webterminal = {
|
||||
duration = "25m"
|
||||
job_timeout = "30m"
|
||||
flags = []
|
||||
}
|
||||
app = {
|
||||
duration = "20m"
|
||||
job_timeout = "25m"
|
||||
flags = [
|
||||
"--app=wsec",
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "time_sleep" "wait_baseline" {
|
||||
depends_on = [
|
||||
kubernetes_job.create_workspaces_primary,
|
||||
kubernetes_job.create_workspaces_europe,
|
||||
kubernetes_job.create_workspaces_asia,
|
||||
helm_release.prometheus_chart_primary,
|
||||
helm_release.prometheus_chart_europe,
|
||||
helm_release.prometheus_chart_asia,
|
||||
]
|
||||
|
||||
create_duration = local.wait_baseline_duration
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "workspace_traffic_primary" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
for_each = local.traffic_types
|
||||
metadata {
|
||||
name = "${var.name}-workspace-traffic-${each.key}"
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-workspace-traffic-${each.key}"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
backoff_limit = 0
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["primary_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = concat([
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"exp",
|
||||
"scaletest",
|
||||
"workspace-traffic",
|
||||
"--template=kubernetes-primary",
|
||||
"--concurrency=0",
|
||||
"--bytes-per-tick=${local.bytes_per_tick}",
|
||||
"--tick-interval=${local.tick_interval}",
|
||||
"--scaletest-prometheus-wait=30s",
|
||||
"--job-timeout=${local.traffic_types[each.key].duration}",
|
||||
], local.traffic_types[each.key].flags)
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
timeouts {
|
||||
create = local.traffic_types[each.key].job_timeout
|
||||
}
|
||||
|
||||
depends_on = [time_sleep.wait_baseline]
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "workspace_traffic_europe" {
|
||||
provider = kubernetes.europe
|
||||
|
||||
for_each = local.traffic_types
|
||||
metadata {
|
||||
name = "${var.name}-workspace-traffic-${each.key}"
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-workspace-traffic-${each.key}"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
backoff_limit = 0
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["europe_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = concat([
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"exp",
|
||||
"scaletest",
|
||||
"workspace-traffic",
|
||||
"--template=kubernetes-europe",
|
||||
"--concurrency=0",
|
||||
"--bytes-per-tick=${local.bytes_per_tick}",
|
||||
"--tick-interval=${local.tick_interval}",
|
||||
"--scaletest-prometheus-wait=30s",
|
||||
"--job-timeout=${local.traffic_types[each.key].duration}",
|
||||
"--workspace-proxy-url=${local.deployments.europe.url}",
|
||||
], local.traffic_types[each.key].flags)
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
timeouts {
|
||||
create = local.traffic_types[each.key].job_timeout
|
||||
}
|
||||
|
||||
depends_on = [time_sleep.wait_baseline]
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "workspace_traffic_asia" {
|
||||
provider = kubernetes.asia
|
||||
|
||||
for_each = local.traffic_types
|
||||
metadata {
|
||||
name = "${var.name}-workspace-traffic-${each.key}"
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-workspace-traffic-${each.key}"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
backoff_limit = 0
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["asia_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = concat([
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"exp",
|
||||
"scaletest",
|
||||
"workspace-traffic",
|
||||
"--template=kubernetes-asia",
|
||||
"--concurrency=0",
|
||||
"--bytes-per-tick=${local.bytes_per_tick}",
|
||||
"--tick-interval=${local.tick_interval}",
|
||||
"--scaletest-prometheus-wait=30s",
|
||||
"--job-timeout=${local.traffic_types[each.key].duration}",
|
||||
"--workspace-proxy-url=${local.deployments.asia.url}",
|
||||
], local.traffic_types[each.key].flags)
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
timeouts {
|
||||
create = local.traffic_types[each.key].job_timeout
|
||||
}
|
||||
|
||||
depends_on = [time_sleep.wait_baseline]
|
||||
}
|
||||
@@ -1,180 +0,0 @@
|
||||
locals {
|
||||
create_workspace_timeout = "30m"
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "create_workspaces_primary" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
metadata {
|
||||
name = "${var.name}-create-workspaces"
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-create-workspaces"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
backoff_limit = 0
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["primary_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = [
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"exp",
|
||||
"scaletest",
|
||||
"create-workspaces",
|
||||
"--count=${local.scenarios[var.scenario].workspaces.count_per_deployment}",
|
||||
"--template=kubernetes-primary",
|
||||
"--concurrency=${local.scenarios[var.scenario].provisionerd.replicas}",
|
||||
"--no-cleanup"
|
||||
]
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
timeouts {
|
||||
create = local.create_workspace_timeout
|
||||
}
|
||||
|
||||
depends_on = [kubernetes_job.push_template_primary]
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "create_workspaces_europe" {
|
||||
provider = kubernetes.europe
|
||||
|
||||
metadata {
|
||||
name = "${var.name}-create-workspaces"
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-create-workspaces"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
backoff_limit = 0
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["europe_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = [
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"exp",
|
||||
"scaletest",
|
||||
"create-workspaces",
|
||||
"--count=${local.scenarios[var.scenario].workspaces.count_per_deployment}",
|
||||
"--template=kubernetes-europe",
|
||||
"--concurrency=${local.scenarios[var.scenario].provisionerd.replicas}",
|
||||
"--no-cleanup"
|
||||
]
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
timeouts {
|
||||
create = local.create_workspace_timeout
|
||||
}
|
||||
|
||||
depends_on = [kubernetes_job.push_template_europe]
|
||||
}
|
||||
|
||||
resource "kubernetes_job" "create_workspaces_asia" {
|
||||
provider = kubernetes.asia
|
||||
|
||||
metadata {
|
||||
name = "${var.name}-create-workspaces"
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "${var.name}-create-workspaces"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
completions = 1
|
||||
backoff_limit = 0
|
||||
template {
|
||||
metadata {}
|
||||
spec {
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${google_container_node_pool.node_pool["asia_misc"].name}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
container {
|
||||
name = "cli"
|
||||
image = "${var.coder_image_repo}:${var.coder_image_tag}"
|
||||
command = [
|
||||
"/opt/coder",
|
||||
"--verbose",
|
||||
"--url=${local.deployments.primary.url}",
|
||||
"--token=${trimspace(data.local_file.api_key.content)}",
|
||||
"exp",
|
||||
"scaletest",
|
||||
"create-workspaces",
|
||||
"--count=${local.scenarios[var.scenario].workspaces.count_per_deployment}",
|
||||
"--template=kubernetes-asia",
|
||||
"--concurrency=${local.scenarios[var.scenario].provisionerd.replicas}",
|
||||
"--no-cleanup"
|
||||
]
|
||||
}
|
||||
restart_policy = "Never"
|
||||
}
|
||||
}
|
||||
}
|
||||
wait_for_completion = true
|
||||
|
||||
timeouts {
|
||||
create = local.create_workspace_timeout
|
||||
}
|
||||
|
||||
depends_on = [kubernetes_job.push_template_asia]
|
||||
}
|
||||
@@ -1,162 +0,0 @@
|
||||
data "google_compute_default_service_account" "default" {
|
||||
project = var.project_id
|
||||
depends_on = [google_project_service.api["compute.googleapis.com"]]
|
||||
}
|
||||
|
||||
locals {
|
||||
deployments = {
|
||||
primary = {
|
||||
subdomain = "primary.${var.name}"
|
||||
wildcard_subdomain = "*.primary.${var.name}"
|
||||
url = "https://primary.${var.name}.${var.cloudflare_domain}"
|
||||
wildcard_access_url = "*.primary.${var.name}.${var.cloudflare_domain}"
|
||||
region = "us-east1"
|
||||
zone = "us-east1-c"
|
||||
subnet = "scaletest"
|
||||
}
|
||||
europe = {
|
||||
subdomain = "europe.${var.name}"
|
||||
wildcard_subdomain = "*.europe.${var.name}"
|
||||
url = "https://europe.${var.name}.${var.cloudflare_domain}"
|
||||
wildcard_access_url = "*.europe.${var.name}.${var.cloudflare_domain}"
|
||||
region = "europe-west1"
|
||||
zone = "europe-west1-b"
|
||||
subnet = "scaletest"
|
||||
}
|
||||
asia = {
|
||||
subdomain = "asia.${var.name}"
|
||||
wildcard_subdomain = "*.asia.${var.name}"
|
||||
url = "https://asia.${var.name}.${var.cloudflare_domain}"
|
||||
wildcard_access_url = "*.asia.${var.name}.${var.cloudflare_domain}"
|
||||
region = "asia-southeast1"
|
||||
zone = "asia-southeast1-a"
|
||||
subnet = "scaletest"
|
||||
}
|
||||
}
|
||||
node_pools = {
|
||||
primary_coder = {
|
||||
name = "coder"
|
||||
cluster = "primary"
|
||||
}
|
||||
primary_workspaces = {
|
||||
name = "workspaces"
|
||||
cluster = "primary"
|
||||
}
|
||||
primary_misc = {
|
||||
name = "misc"
|
||||
cluster = "primary"
|
||||
}
|
||||
europe_coder = {
|
||||
name = "coder"
|
||||
cluster = "europe"
|
||||
}
|
||||
europe_workspaces = {
|
||||
name = "workspaces"
|
||||
cluster = "europe"
|
||||
}
|
||||
europe_misc = {
|
||||
name = "misc"
|
||||
cluster = "europe"
|
||||
}
|
||||
asia_coder = {
|
||||
name = "coder"
|
||||
cluster = "asia"
|
||||
}
|
||||
asia_workspaces = {
|
||||
name = "workspaces"
|
||||
cluster = "asia"
|
||||
}
|
||||
asia_misc = {
|
||||
name = "misc"
|
||||
cluster = "asia"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_container_cluster" "cluster" {
|
||||
for_each = local.deployments
|
||||
name = "${var.name}-${each.key}"
|
||||
location = each.value.zone
|
||||
project = var.project_id
|
||||
network = google_compute_network.network.name
|
||||
subnetwork = google_compute_subnetwork.subnetwork[each.key].name
|
||||
networking_mode = "VPC_NATIVE"
|
||||
default_max_pods_per_node = 256
|
||||
ip_allocation_policy { # Required with networking_mode=VPC_NATIVE
|
||||
cluster_secondary_range_name = local.secondary_ip_range_k8s_pods
|
||||
services_secondary_range_name = local.secondary_ip_range_k8s_services
|
||||
}
|
||||
release_channel {
|
||||
# Setting release channel as STABLE can cause unexpected cluster upgrades.
|
||||
channel = "UNSPECIFIED"
|
||||
}
|
||||
initial_node_count = 1
|
||||
remove_default_node_pool = true
|
||||
|
||||
network_policy {
|
||||
enabled = true
|
||||
}
|
||||
depends_on = [
|
||||
google_project_service.api["container.googleapis.com"]
|
||||
]
|
||||
monitoring_config {
|
||||
enable_components = ["SYSTEM_COMPONENTS"]
|
||||
managed_prometheus {
|
||||
enabled = false
|
||||
}
|
||||
}
|
||||
workload_identity_config {
|
||||
workload_pool = "${data.google_project.project.project_id}.svc.id.goog"
|
||||
}
|
||||
|
||||
lifecycle {
|
||||
ignore_changes = [
|
||||
maintenance_policy,
|
||||
release_channel,
|
||||
remove_default_node_pool
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_container_node_pool" "node_pool" {
|
||||
for_each = local.node_pools
|
||||
name = each.value.name
|
||||
location = local.deployments[each.value.cluster].zone
|
||||
project = var.project_id
|
||||
cluster = google_container_cluster.cluster[each.value.cluster].name
|
||||
node_count = local.scenarios[var.scenario][each.value.name].nodepool_size
|
||||
node_config {
|
||||
oauth_scopes = [
|
||||
"https://www.googleapis.com/auth/logging.write",
|
||||
"https://www.googleapis.com/auth/monitoring",
|
||||
"https://www.googleapis.com/auth/trace.append",
|
||||
"https://www.googleapis.com/auth/devstorage.read_only",
|
||||
"https://www.googleapis.com/auth/service.management.readonly",
|
||||
"https://www.googleapis.com/auth/servicecontrol",
|
||||
]
|
||||
disk_size_gb = 100
|
||||
machine_type = local.scenarios[var.scenario][each.value.name].machine_type
|
||||
image_type = "cos_containerd"
|
||||
service_account = data.google_compute_default_service_account.default.email
|
||||
tags = ["gke-node", "${var.project_id}-gke"]
|
||||
labels = {
|
||||
env = var.project_id
|
||||
}
|
||||
metadata = {
|
||||
disable-legacy-endpoints = "true"
|
||||
}
|
||||
kubelet_config {
|
||||
cpu_manager_policy = ""
|
||||
cpu_cfs_quota = false
|
||||
pod_pids_limit = 0
|
||||
}
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [
|
||||
management[0].auto_repair,
|
||||
management[0].auto_upgrade,
|
||||
timeouts,
|
||||
node_config[0].resource_labels
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,89 +0,0 @@
|
||||
resource "google_sql_database_instance" "db" {
|
||||
name = "${var.name}-coder"
|
||||
project = var.project_id
|
||||
region = local.deployments.primary.region
|
||||
database_version = "POSTGRES_14"
|
||||
deletion_protection = false
|
||||
|
||||
depends_on = [google_service_networking_connection.private_vpc_connection]
|
||||
|
||||
settings {
|
||||
tier = local.scenarios[var.scenario].cloudsql.tier
|
||||
activation_policy = "ALWAYS"
|
||||
availability_type = "ZONAL"
|
||||
|
||||
location_preference {
|
||||
zone = local.deployments.primary.zone
|
||||
}
|
||||
|
||||
database_flags {
|
||||
name = "max_connections"
|
||||
value = local.scenarios[var.scenario].cloudsql.max_connections
|
||||
}
|
||||
|
||||
ip_configuration {
|
||||
ipv4_enabled = false
|
||||
private_network = google_compute_network.network.id
|
||||
}
|
||||
|
||||
insights_config {
|
||||
query_insights_enabled = true
|
||||
query_string_length = 1024
|
||||
record_application_tags = false
|
||||
record_client_address = false
|
||||
}
|
||||
}
|
||||
|
||||
lifecycle {
|
||||
ignore_changes = [deletion_protection, timeouts]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_sql_database" "coder" {
|
||||
project = var.project_id
|
||||
instance = google_sql_database_instance.db.id
|
||||
name = "${var.name}-coder"
|
||||
# required for postgres, otherwise db fails to delete
|
||||
deletion_policy = "ABANDON"
|
||||
lifecycle {
|
||||
ignore_changes = [deletion_policy]
|
||||
}
|
||||
}
|
||||
|
||||
resource "random_password" "coder_postgres_password" {
|
||||
length = 12
|
||||
}
|
||||
|
||||
resource "random_password" "prometheus_postgres_password" {
|
||||
length = 12
|
||||
}
|
||||
|
||||
resource "google_sql_user" "coder" {
|
||||
project = var.project_id
|
||||
instance = google_sql_database_instance.db.id
|
||||
name = "${var.name}-coder"
|
||||
type = "BUILT_IN"
|
||||
password = random_password.coder_postgres_password.result
|
||||
# required for postgres, otherwise user fails to delete
|
||||
deletion_policy = "ABANDON"
|
||||
lifecycle {
|
||||
ignore_changes = [deletion_policy, password]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_sql_user" "prometheus" {
|
||||
project = var.project_id
|
||||
instance = google_sql_database_instance.db.id
|
||||
name = "${var.name}-prometheus"
|
||||
type = "BUILT_IN"
|
||||
password = random_password.prometheus_postgres_password.result
|
||||
# required for postgres, otherwise user fails to delete
|
||||
deletion_policy = "ABANDON"
|
||||
lifecycle {
|
||||
ignore_changes = [deletion_policy, password]
|
||||
}
|
||||
}
|
||||
|
||||
locals {
|
||||
coder_db_url = "postgres://${google_sql_user.coder.name}:${urlencode(random_password.coder_postgres_password.result)}@${google_sql_database_instance.db.private_ip_address}/${google_sql_database.coder.name}?sslmode=disable"
|
||||
}
|
||||
@@ -1,27 +0,0 @@
|
||||
locals {
|
||||
project_apis = [
|
||||
"cloudtrace",
|
||||
"compute",
|
||||
"container",
|
||||
"logging",
|
||||
"monitoring",
|
||||
"servicemanagement",
|
||||
"servicenetworking",
|
||||
"sqladmin",
|
||||
"stackdriver",
|
||||
"storage-api",
|
||||
]
|
||||
}
|
||||
|
||||
data "google_project" "project" {
|
||||
project_id = var.project_id
|
||||
}
|
||||
|
||||
resource "google_project_service" "api" {
|
||||
for_each = toset(local.project_apis)
|
||||
project = data.google_project.project.project_id
|
||||
service = "${each.value}.googleapis.com"
|
||||
|
||||
disable_dependent_services = false
|
||||
disable_on_destroy = false
|
||||
}
|
||||
@@ -1,154 +0,0 @@
|
||||
locals {
|
||||
# Generate a /14 for each deployment.
|
||||
cidr_networks = cidrsubnets(
|
||||
"172.16.0.0/12",
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
)
|
||||
|
||||
networks = {
|
||||
alpha = local.cidr_networks[0]
|
||||
bravo = local.cidr_networks[1]
|
||||
charlie = local.cidr_networks[2]
|
||||
}
|
||||
|
||||
# Generate a bunch of /18s within the subnet we're using from the above map.
|
||||
cidr_subnetworks = cidrsubnets(
|
||||
local.networks[var.name],
|
||||
4, # PSA
|
||||
4, # primary subnetwork
|
||||
4, # primary k8s pod network
|
||||
4, # primary k8s services network
|
||||
4, # europe subnetwork
|
||||
4, # europe k8s pod network
|
||||
4, # europe k8s services network
|
||||
4, # asia subnetwork
|
||||
4, # asia k8s pod network
|
||||
4, # asia k8s services network
|
||||
)
|
||||
|
||||
psa_range_address = split("/", local.cidr_subnetworks[0])[0]
|
||||
psa_range_prefix_length = tonumber(split("/", local.cidr_subnetworks[0])[1])
|
||||
|
||||
subnetworks = {
|
||||
primary = local.cidr_subnetworks[1]
|
||||
europe = local.cidr_subnetworks[4]
|
||||
asia = local.cidr_subnetworks[7]
|
||||
}
|
||||
cluster_ranges = {
|
||||
primary = {
|
||||
pods = local.cidr_subnetworks[2]
|
||||
services = local.cidr_subnetworks[3]
|
||||
}
|
||||
europe = {
|
||||
pods = local.cidr_subnetworks[5]
|
||||
services = local.cidr_subnetworks[6]
|
||||
}
|
||||
asia = {
|
||||
pods = local.cidr_subnetworks[8]
|
||||
services = local.cidr_subnetworks[9]
|
||||
}
|
||||
}
|
||||
|
||||
secondary_ip_range_k8s_pods = "k8s-pods"
|
||||
secondary_ip_range_k8s_services = "k8s-services"
|
||||
}
|
||||
|
||||
# Create a VPC for the deployment
|
||||
resource "google_compute_network" "network" {
|
||||
project = var.project_id
|
||||
name = "${var.name}-scaletest"
|
||||
description = "scaletest network for ${var.name}"
|
||||
auto_create_subnetworks = false
|
||||
}
|
||||
|
||||
# Create a subnetwork with a unique range for each region
|
||||
resource "google_compute_subnetwork" "subnetwork" {
|
||||
for_each = local.subnetworks
|
||||
name = "${var.name}-${each.key}"
|
||||
# Use the deployment region
|
||||
region = local.deployments[each.key].region
|
||||
network = google_compute_network.network.id
|
||||
project = var.project_id
|
||||
ip_cidr_range = each.value
|
||||
private_ip_google_access = true
|
||||
|
||||
secondary_ip_range {
|
||||
range_name = local.secondary_ip_range_k8s_pods
|
||||
ip_cidr_range = local.cluster_ranges[each.key].pods
|
||||
}
|
||||
|
||||
secondary_ip_range {
|
||||
range_name = local.secondary_ip_range_k8s_services
|
||||
ip_cidr_range = local.cluster_ranges[each.key].services
|
||||
}
|
||||
}
|
||||
|
||||
# Create a public IP for each region
|
||||
resource "google_compute_address" "coder" {
|
||||
for_each = local.deployments
|
||||
project = var.project_id
|
||||
region = each.value.region
|
||||
name = "${var.name}-${each.key}-coder"
|
||||
address_type = "EXTERNAL"
|
||||
network_tier = "PREMIUM"
|
||||
}
|
||||
|
||||
# Reserve an internal range for Google-managed services (PSA), used for Cloud
|
||||
# SQL
|
||||
resource "google_compute_global_address" "psa_peering" {
|
||||
project = var.project_id
|
||||
name = "${var.name}-sql-peering"
|
||||
purpose = "VPC_PEERING"
|
||||
address_type = "INTERNAL"
|
||||
address = local.psa_range_address
|
||||
prefix_length = local.psa_range_prefix_length
|
||||
network = google_compute_network.network.self_link
|
||||
}
|
||||
|
||||
resource "google_service_networking_connection" "private_vpc_connection" {
|
||||
network = google_compute_network.network.id
|
||||
service = "servicenetworking.googleapis.com"
|
||||
reserved_peering_ranges = [google_compute_global_address.psa_peering.name]
|
||||
}
|
||||
|
||||
# Join the new network to the observability network so we can talk to the
|
||||
# Prometheus instance
|
||||
data "google_compute_network" "observability" {
|
||||
project = var.project_id
|
||||
name = var.observability_cluster_vpc
|
||||
}
|
||||
|
||||
resource "google_compute_network_peering" "scaletest_to_observability" {
|
||||
name = "peer-${google_compute_network.network.name}-to-${data.google_compute_network.observability.name}"
|
||||
network = google_compute_network.network.self_link
|
||||
peer_network = data.google_compute_network.observability.self_link
|
||||
import_custom_routes = true
|
||||
export_custom_routes = true
|
||||
}
|
||||
|
||||
resource "google_compute_network_peering" "observability_to_scaletest" {
|
||||
name = "peer-${data.google_compute_network.observability.name}-to-${google_compute_network.network.name}"
|
||||
network = data.google_compute_network.observability.self_link
|
||||
peer_network = google_compute_network.network.self_link
|
||||
import_custom_routes = true
|
||||
export_custom_routes = true
|
||||
}
|
||||
|
||||
# Allow traffic from the scaletest network into the observability network so we
|
||||
# can connect to Prometheus
|
||||
resource "google_compute_firewall" "observability_allow_from_scaletest" {
|
||||
project = var.project_id
|
||||
name = "allow-from-scaletest-${var.name}"
|
||||
network = data.google_compute_network.observability.self_link
|
||||
direction = "INGRESS"
|
||||
source_ranges = [local.networks[var.name]]
|
||||
allow {
|
||||
protocol = "icmp"
|
||||
}
|
||||
allow {
|
||||
protocol = "tcp"
|
||||
ports = ["0-65535"]
|
||||
}
|
||||
}
|
||||
@@ -1,131 +0,0 @@
|
||||
resource "kubernetes_namespace" "coder_asia" {
|
||||
provider = kubernetes.asia
|
||||
|
||||
metadata {
|
||||
name = local.coder_namespace
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_default_service_account]
|
||||
}
|
||||
|
||||
depends_on = [google_container_node_pool.node_pool["asia_misc"]]
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "provisionerd_psk_asia" {
|
||||
provider = kubernetes.asia
|
||||
|
||||
type = "Opaque"
|
||||
metadata {
|
||||
name = "coder-provisioner-psk"
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
psk = random_password.provisionerd_psk.result
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "proxy_token_asia" {
|
||||
provider = kubernetes.asia
|
||||
|
||||
type = "Opaque"
|
||||
metadata {
|
||||
name = "coder-proxy-token"
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
token = trimspace(data.local_file.asia_proxy_token.content)
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "coder_tls_asia" {
|
||||
provider = kubernetes.asia
|
||||
|
||||
type = "kubernetes.io/tls"
|
||||
metadata {
|
||||
name = "coder-tls"
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
"tls.crt" = data.kubernetes_secret.coder_tls["asia"].data["tls.crt"]
|
||||
"tls.key" = data.kubernetes_secret.coder_tls["asia"].data["tls.key"]
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "helm_release" "coder_asia" {
|
||||
provider = helm.asia
|
||||
|
||||
repository = local.coder_helm_repo
|
||||
chart = local.coder_helm_chart
|
||||
name = local.coder_release_name
|
||||
version = var.coder_chart_version
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
values = [templatefile("${path.module}/coder_helm_values.tftpl", {
|
||||
workspace_proxy = true,
|
||||
provisionerd = false,
|
||||
primary_url = local.deployments.primary.url,
|
||||
proxy_token = kubernetes_secret.proxy_token_asia.metadata.0.name,
|
||||
db_secret = null,
|
||||
ip_address = google_compute_address.coder["asia"].address,
|
||||
provisionerd_psk = null,
|
||||
access_url = local.deployments.asia.url,
|
||||
wildcard_access_url = local.deployments.asia.wildcard_access_url,
|
||||
node_pool = google_container_node_pool.node_pool["asia_coder"].name,
|
||||
release_name = local.coder_release_name,
|
||||
experiments = var.coder_experiments,
|
||||
image_repo = var.coder_image_repo,
|
||||
image_tag = var.coder_image_tag,
|
||||
replicas = local.scenarios[var.scenario].coder.replicas,
|
||||
cpu_request = local.scenarios[var.scenario].coder.cpu_request,
|
||||
mem_request = local.scenarios[var.scenario].coder.mem_request,
|
||||
cpu_limit = local.scenarios[var.scenario].coder.cpu_limit,
|
||||
mem_limit = local.scenarios[var.scenario].coder.mem_limit,
|
||||
deployment = "asia",
|
||||
tls_secret_name = kubernetes_secret.coder_tls_asia.metadata.0.name,
|
||||
})]
|
||||
|
||||
depends_on = [null_resource.license]
|
||||
}
|
||||
|
||||
resource "helm_release" "provisionerd_asia" {
|
||||
provider = helm.asia
|
||||
|
||||
repository = local.coder_helm_repo
|
||||
chart = local.provisionerd_helm_chart
|
||||
name = local.provisionerd_release_name
|
||||
version = var.provisionerd_chart_version
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
values = [templatefile("${path.module}/coder_helm_values.tftpl", {
|
||||
workspace_proxy = false,
|
||||
provisionerd = true,
|
||||
primary_url = null,
|
||||
proxy_token = null,
|
||||
db_secret = null,
|
||||
ip_address = null,
|
||||
provisionerd_psk = kubernetes_secret.provisionerd_psk_asia.metadata.0.name,
|
||||
access_url = local.deployments.primary.url,
|
||||
wildcard_access_url = null,
|
||||
node_pool = google_container_node_pool.node_pool["asia_coder"].name,
|
||||
release_name = local.coder_release_name,
|
||||
experiments = var.coder_experiments,
|
||||
image_repo = var.coder_image_repo,
|
||||
image_tag = var.coder_image_tag,
|
||||
replicas = local.scenarios[var.scenario].provisionerd.replicas,
|
||||
cpu_request = local.scenarios[var.scenario].provisionerd.cpu_request,
|
||||
mem_request = local.scenarios[var.scenario].provisionerd.mem_request,
|
||||
cpu_limit = local.scenarios[var.scenario].provisionerd.cpu_limit,
|
||||
mem_limit = local.scenarios[var.scenario].provisionerd.mem_limit,
|
||||
deployment = "asia",
|
||||
tls_secret_name = null,
|
||||
})]
|
||||
|
||||
depends_on = [null_resource.license]
|
||||
}
|
||||
@@ -1,131 +0,0 @@
|
||||
resource "kubernetes_namespace" "coder_europe" {
|
||||
provider = kubernetes.europe
|
||||
|
||||
metadata {
|
||||
name = local.coder_namespace
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_default_service_account]
|
||||
}
|
||||
|
||||
depends_on = [google_container_node_pool.node_pool["europe_misc"]]
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "provisionerd_psk_europe" {
|
||||
provider = kubernetes.europe
|
||||
|
||||
type = "Opaque"
|
||||
metadata {
|
||||
name = "coder-provisioner-psk"
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
psk = random_password.provisionerd_psk.result
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "proxy_token_europe" {
|
||||
provider = kubernetes.europe
|
||||
|
||||
type = "Opaque"
|
||||
metadata {
|
||||
name = "coder-proxy-token"
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
token = trimspace(data.local_file.europe_proxy_token.content)
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "coder_tls_europe" {
|
||||
provider = kubernetes.europe
|
||||
|
||||
type = "kubernetes.io/tls"
|
||||
metadata {
|
||||
name = "coder-tls"
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
"tls.crt" = data.kubernetes_secret.coder_tls["europe"].data["tls.crt"]
|
||||
"tls.key" = data.kubernetes_secret.coder_tls["europe"].data["tls.key"]
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "helm_release" "coder_europe" {
|
||||
provider = helm.europe
|
||||
|
||||
repository = local.coder_helm_repo
|
||||
chart = local.coder_helm_chart
|
||||
name = local.coder_release_name
|
||||
version = var.coder_chart_version
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
values = [templatefile("${path.module}/coder_helm_values.tftpl", {
|
||||
workspace_proxy = true,
|
||||
provisionerd = false,
|
||||
primary_url = local.deployments.primary.url,
|
||||
proxy_token = kubernetes_secret.proxy_token_europe.metadata.0.name,
|
||||
db_secret = null,
|
||||
ip_address = google_compute_address.coder["europe"].address,
|
||||
provisionerd_psk = null,
|
||||
access_url = local.deployments.europe.url,
|
||||
wildcard_access_url = local.deployments.europe.wildcard_access_url,
|
||||
node_pool = google_container_node_pool.node_pool["europe_coder"].name,
|
||||
release_name = local.coder_release_name,
|
||||
experiments = var.coder_experiments,
|
||||
image_repo = var.coder_image_repo,
|
||||
image_tag = var.coder_image_tag,
|
||||
replicas = local.scenarios[var.scenario].coder.replicas,
|
||||
cpu_request = local.scenarios[var.scenario].coder.cpu_request,
|
||||
mem_request = local.scenarios[var.scenario].coder.mem_request,
|
||||
cpu_limit = local.scenarios[var.scenario].coder.cpu_limit,
|
||||
mem_limit = local.scenarios[var.scenario].coder.mem_limit,
|
||||
deployment = "europe",
|
||||
tls_secret_name = kubernetes_secret.coder_tls_europe.metadata.0.name,
|
||||
})]
|
||||
|
||||
depends_on = [null_resource.license]
|
||||
}
|
||||
|
||||
resource "helm_release" "provisionerd_europe" {
|
||||
provider = helm.europe
|
||||
|
||||
repository = local.coder_helm_repo
|
||||
chart = local.provisionerd_helm_chart
|
||||
name = local.provisionerd_release_name
|
||||
version = var.provisionerd_chart_version
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
values = [templatefile("${path.module}/coder_helm_values.tftpl", {
|
||||
workspace_proxy = false,
|
||||
provisionerd = true,
|
||||
primary_url = null,
|
||||
proxy_token = null,
|
||||
db_secret = null,
|
||||
ip_address = null,
|
||||
provisionerd_psk = kubernetes_secret.provisionerd_psk_europe.metadata.0.name,
|
||||
access_url = local.deployments.primary.url,
|
||||
wildcard_access_url = null,
|
||||
node_pool = google_container_node_pool.node_pool["europe_coder"].name,
|
||||
release_name = local.coder_release_name,
|
||||
experiments = var.coder_experiments,
|
||||
image_repo = var.coder_image_repo,
|
||||
image_tag = var.coder_image_tag,
|
||||
replicas = local.scenarios[var.scenario].provisionerd.replicas,
|
||||
cpu_request = local.scenarios[var.scenario].provisionerd.cpu_request,
|
||||
mem_request = local.scenarios[var.scenario].provisionerd.mem_request,
|
||||
cpu_limit = local.scenarios[var.scenario].provisionerd.cpu_limit,
|
||||
mem_limit = local.scenarios[var.scenario].provisionerd.mem_limit,
|
||||
deployment = "europe",
|
||||
tls_secret_name = null,
|
||||
})]
|
||||
|
||||
depends_on = [null_resource.license]
|
||||
}
|
||||
@@ -1,160 +0,0 @@
|
||||
data "google_client_config" "default" {}
|
||||
|
||||
locals {
|
||||
coder_admin_email = "admin@coder.com"
|
||||
coder_admin_full_name = "Coder Admin"
|
||||
coder_admin_user = "coder"
|
||||
coder_admin_password = random_password.coder_admin_password.result
|
||||
coder_helm_repo = "https://helm.coder.com/v2"
|
||||
coder_helm_chart = "coder"
|
||||
coder_namespace = "coder"
|
||||
coder_release_name = "${var.name}-coder"
|
||||
provisionerd_helm_chart = "coder-provisioner"
|
||||
provisionerd_release_name = "${var.name}-provisionerd"
|
||||
|
||||
}
|
||||
|
||||
resource "random_password" "provisionerd_psk" {
|
||||
length = 26
|
||||
}
|
||||
|
||||
resource "random_password" "coder_admin_password" {
|
||||
length = 16
|
||||
special = true
|
||||
}
|
||||
|
||||
resource "kubernetes_namespace" "coder_primary" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
metadata {
|
||||
name = local.coder_namespace
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_default_service_account]
|
||||
}
|
||||
|
||||
depends_on = [google_container_node_pool.node_pool["primary_misc"]]
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "coder_db" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
type = "Opaque"
|
||||
metadata {
|
||||
name = "coder-db-url"
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
url = local.coder_db_url
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "provisionerd_psk_primary" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
type = "Opaque"
|
||||
metadata {
|
||||
name = "coder-provisioner-psk"
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
psk = random_password.provisionerd_psk.result
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "coder_tls_primary" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
type = "kubernetes.io/tls"
|
||||
metadata {
|
||||
name = "coder-tls"
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
"tls.crt" = data.kubernetes_secret.coder_tls["primary"].data["tls.crt"]
|
||||
"tls.key" = data.kubernetes_secret.coder_tls["primary"].data["tls.key"]
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "helm_release" "coder_primary" {
|
||||
provider = helm.primary
|
||||
|
||||
repository = local.coder_helm_repo
|
||||
chart = local.coder_helm_chart
|
||||
name = local.coder_release_name
|
||||
version = var.coder_chart_version
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
values = [templatefile("${path.module}/coder_helm_values.tftpl", {
|
||||
workspace_proxy = false,
|
||||
provisionerd = false,
|
||||
primary_url = null,
|
||||
proxy_token = null,
|
||||
db_secret = kubernetes_secret.coder_db.metadata.0.name,
|
||||
ip_address = google_compute_address.coder["primary"].address,
|
||||
provisionerd_psk = kubernetes_secret.provisionerd_psk_primary.metadata.0.name,
|
||||
access_url = local.deployments.primary.url,
|
||||
wildcard_access_url = local.deployments.primary.wildcard_access_url,
|
||||
node_pool = google_container_node_pool.node_pool["primary_coder"].name,
|
||||
release_name = local.coder_release_name,
|
||||
experiments = var.coder_experiments,
|
||||
image_repo = var.coder_image_repo,
|
||||
image_tag = var.coder_image_tag,
|
||||
replicas = local.scenarios[var.scenario].coder.replicas,
|
||||
cpu_request = local.scenarios[var.scenario].coder.cpu_request,
|
||||
mem_request = local.scenarios[var.scenario].coder.mem_request,
|
||||
cpu_limit = local.scenarios[var.scenario].coder.cpu_limit,
|
||||
mem_limit = local.scenarios[var.scenario].coder.mem_limit,
|
||||
deployment = "primary",
|
||||
tls_secret_name = kubernetes_secret.coder_tls_primary.metadata.0.name,
|
||||
})]
|
||||
}
|
||||
|
||||
resource "helm_release" "provisionerd_primary" {
|
||||
provider = helm.primary
|
||||
|
||||
repository = local.coder_helm_repo
|
||||
chart = local.provisionerd_helm_chart
|
||||
name = local.provisionerd_release_name
|
||||
version = var.provisionerd_chart_version
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
values = [templatefile("${path.module}/coder_helm_values.tftpl", {
|
||||
workspace_proxy = false,
|
||||
provisionerd = true,
|
||||
primary_url = null,
|
||||
proxy_token = null,
|
||||
db_secret = null,
|
||||
ip_address = null,
|
||||
provisionerd_psk = kubernetes_secret.provisionerd_psk_primary.metadata.0.name,
|
||||
access_url = local.deployments.primary.url,
|
||||
wildcard_access_url = null,
|
||||
node_pool = google_container_node_pool.node_pool["primary_coder"].name,
|
||||
release_name = local.coder_release_name,
|
||||
experiments = var.coder_experiments,
|
||||
image_repo = var.coder_image_repo,
|
||||
image_tag = var.coder_image_tag,
|
||||
replicas = local.scenarios[var.scenario].provisionerd.replicas,
|
||||
cpu_request = local.scenarios[var.scenario].provisionerd.cpu_request,
|
||||
mem_request = local.scenarios[var.scenario].provisionerd.mem_request,
|
||||
cpu_limit = local.scenarios[var.scenario].provisionerd.cpu_limit,
|
||||
mem_limit = local.scenarios[var.scenario].provisionerd.mem_limit,
|
||||
deployment = "primary",
|
||||
tls_secret_name = null,
|
||||
})]
|
||||
|
||||
depends_on = [null_resource.license]
|
||||
}
|
||||
|
||||
output "coder_admin_password" {
|
||||
description = "Randomly generated Coder admin password"
|
||||
value = random_password.coder_admin_password.result
|
||||
# Deliberately not sensitive, so it appears in terraform apply logs
|
||||
}
|
||||
@@ -1,17 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Config
|
||||
current-context: ${name}
|
||||
clusters:
|
||||
- name: ${name}
|
||||
cluster:
|
||||
certificate-authority-data: ${cluster_ca_certificate}
|
||||
server: ${endpoint}
|
||||
contexts:
|
||||
- context:
|
||||
cluster: ${name}
|
||||
user: ${name}
|
||||
name: ${name}
|
||||
users:
|
||||
- name: ${name}
|
||||
user:
|
||||
token: ${access_token}
|
||||
@@ -1,141 +0,0 @@
|
||||
terraform {
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 4.36"
|
||||
}
|
||||
|
||||
random = {
|
||||
source = "hashicorp/random"
|
||||
version = "~> 3.5"
|
||||
}
|
||||
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = "~> 2.20"
|
||||
}
|
||||
|
||||
// We use the kubectl provider to apply Custom Resources.
|
||||
// The kubernetes provider requires the CRD is already present
|
||||
// and would require a separate apply step beforehand.
|
||||
// https://github.com/hashicorp/terraform-provider-kubernetes/issues/1367
|
||||
kubectl = {
|
||||
source = "alekc/kubectl"
|
||||
version = ">= 2.0.0"
|
||||
}
|
||||
|
||||
helm = {
|
||||
source = "hashicorp/helm"
|
||||
version = "~> 2.9"
|
||||
}
|
||||
|
||||
tls = {
|
||||
source = "hashicorp/tls"
|
||||
version = "~> 4.0"
|
||||
}
|
||||
|
||||
cloudflare = {
|
||||
source = "cloudflare/cloudflare"
|
||||
version = "~> 4.0"
|
||||
}
|
||||
}
|
||||
|
||||
required_version = ">= 1.9.0"
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
}
|
||||
|
||||
data "google_secret_manager_secret_version_access" "cloudflare_api_token_dns" {
|
||||
secret = "cloudflare-api-token-dns"
|
||||
project = var.project_id
|
||||
}
|
||||
|
||||
provider "cloudflare" {
|
||||
api_token = coalesce(var.cloudflare_api_token, data.google_secret_manager_secret_version_access.cloudflare_api_token_dns.secret_data)
|
||||
}
|
||||
|
||||
data "google_container_cluster" "observability" {
|
||||
name = var.observability_cluster_name
|
||||
location = var.observability_cluster_location
|
||||
project = var.project_id
|
||||
}
|
||||
|
||||
provider "kubernetes" {
|
||||
alias = "primary"
|
||||
host = "https://${google_container_cluster.cluster["primary"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["primary"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
}
|
||||
|
||||
provider "kubernetes" {
|
||||
alias = "europe"
|
||||
host = "https://${google_container_cluster.cluster["europe"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["europe"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
}
|
||||
|
||||
provider "kubernetes" {
|
||||
alias = "asia"
|
||||
host = "https://${google_container_cluster.cluster["asia"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["asia"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
}
|
||||
|
||||
provider "kubernetes" {
|
||||
alias = "observability"
|
||||
host = "https://${data.google_container_cluster.observability.endpoint}"
|
||||
cluster_ca_certificate = base64decode(data.google_container_cluster.observability.master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
}
|
||||
|
||||
provider "kubectl" {
|
||||
alias = "primary"
|
||||
host = "https://${google_container_cluster.cluster["primary"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["primary"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
load_config_file = false
|
||||
}
|
||||
|
||||
provider "kubectl" {
|
||||
alias = "europe"
|
||||
host = "https://${google_container_cluster.cluster["europe"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["europe"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
load_config_file = false
|
||||
}
|
||||
|
||||
provider "kubectl" {
|
||||
alias = "asia"
|
||||
host = "https://${google_container_cluster.cluster["asia"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["asia"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
load_config_file = false
|
||||
}
|
||||
|
||||
provider "helm" {
|
||||
alias = "primary"
|
||||
kubernetes {
|
||||
host = "https://${google_container_cluster.cluster["primary"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["primary"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
}
|
||||
}
|
||||
|
||||
provider "helm" {
|
||||
alias = "europe"
|
||||
kubernetes {
|
||||
host = "https://${google_container_cluster.cluster["europe"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["europe"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
}
|
||||
}
|
||||
|
||||
provider "helm" {
|
||||
alias = "asia"
|
||||
kubernetes {
|
||||
host = "https://${google_container_cluster.cluster["asia"].endpoint}"
|
||||
cluster_ca_certificate = base64decode(google_container_cluster.cluster["asia"].master_auth.0.cluster_ca_certificate)
|
||||
token = data.google_client_config.default.access_token
|
||||
}
|
||||
}
|
||||
@@ -1,174 +0,0 @@
|
||||
locals {
|
||||
prometheus_helm_repo = "https://prometheus-community.github.io/helm-charts"
|
||||
prometheus_helm_chart = "kube-prometheus-stack"
|
||||
prometheus_release_name = "prometheus"
|
||||
prometheus_remote_write_send_interval = "15s"
|
||||
prometheus_remote_write_metrics_regex = ".*"
|
||||
prometheus_postgres_exporter_helm_repo = "https://prometheus-community.github.io/helm-charts"
|
||||
prometheus_postgres_exporter_helm_chart = "prometheus-postgres-exporter"
|
||||
prometheus_postgres_exporter_release_name = "prometheus-postgres-exporter"
|
||||
}
|
||||
|
||||
resource "helm_release" "prometheus_chart_primary" {
|
||||
provider = helm.primary
|
||||
|
||||
repository = local.prometheus_helm_repo
|
||||
chart = local.prometheus_helm_chart
|
||||
name = local.prometheus_release_name
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
values = [templatefile("${path.module}/prometheus_helm_values.tftpl", {
|
||||
deployment_name = var.name,
|
||||
nodepool = google_container_node_pool.node_pool["primary_misc"].name,
|
||||
cluster = "primary",
|
||||
prometheus_remote_write_url = var.prometheus_remote_write_url,
|
||||
prometheus_remote_write_metrics_regex = local.prometheus_remote_write_metrics_regex,
|
||||
prometheus_remote_write_send_interval = local.prometheus_remote_write_send_interval,
|
||||
})]
|
||||
}
|
||||
|
||||
resource "kubectl_manifest" "pod_monitor_primary" {
|
||||
provider = kubectl.primary
|
||||
|
||||
yaml_body = <<YAML
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
namespace: ${kubernetes_namespace.coder_primary.metadata.0.name}
|
||||
name: coder-monitoring
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
"app.kubernetes.io/name": coder
|
||||
podMetricsEndpoints:
|
||||
- port: prometheus-http
|
||||
interval: 30s
|
||||
YAML
|
||||
|
||||
depends_on = [helm_release.prometheus_chart_primary]
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "prometheus_postgres_password" {
|
||||
provider = kubernetes.primary
|
||||
|
||||
type = "kubernetes.io/basic-auth"
|
||||
metadata {
|
||||
name = "prometheus-postgres"
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
username = "${var.name}-prometheus"
|
||||
password = random_password.prometheus_postgres_password.result
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "helm_release" "prometheus_postgres_exporter" {
|
||||
provider = helm.primary
|
||||
|
||||
repository = local.prometheus_postgres_exporter_helm_repo
|
||||
chart = local.prometheus_postgres_exporter_helm_chart
|
||||
name = local.prometheus_postgres_exporter_release_name
|
||||
namespace = kubernetes_namespace.coder_primary.metadata.0.name
|
||||
values = [<<EOF
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${google_container_node_pool.node_pool["primary_misc"].name}"]
|
||||
config:
|
||||
datasource:
|
||||
host: "${google_sql_database_instance.db.private_ip_address}"
|
||||
user: "${var.name}-prometheus"
|
||||
database: "${var.name}-coder"
|
||||
passwordSecret:
|
||||
name: "${kubernetes_secret.prometheus_postgres_password.metadata.0.name}"
|
||||
key: password
|
||||
autoDiscoverDatabases: true
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
EOF
|
||||
]
|
||||
|
||||
depends_on = [helm_release.prometheus_chart_primary]
|
||||
}
|
||||
|
||||
resource "helm_release" "prometheus_chart_europe" {
|
||||
provider = helm.europe
|
||||
|
||||
repository = local.prometheus_helm_repo
|
||||
chart = local.prometheus_helm_chart
|
||||
name = local.prometheus_release_name
|
||||
namespace = kubernetes_namespace.coder_europe.metadata.0.name
|
||||
values = [templatefile("${path.module}/prometheus_helm_values.tftpl", {
|
||||
deployment_name = var.name,
|
||||
nodepool = google_container_node_pool.node_pool["europe_misc"].name,
|
||||
cluster = "europe",
|
||||
prometheus_remote_write_url = var.prometheus_remote_write_url,
|
||||
prometheus_remote_write_metrics_regex = local.prometheus_remote_write_metrics_regex,
|
||||
prometheus_remote_write_send_interval = local.prometheus_remote_write_send_interval,
|
||||
})]
|
||||
}
|
||||
|
||||
resource "kubectl_manifest" "pod_monitor_europe" {
|
||||
provider = kubectl.europe
|
||||
|
||||
yaml_body = <<YAML
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
namespace: ${kubernetes_namespace.coder_europe.metadata.0.name}
|
||||
name: coder-monitoring
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
"app.kubernetes.io/name": coder
|
||||
podMetricsEndpoints:
|
||||
- port: prometheus-http
|
||||
interval: 30s
|
||||
YAML
|
||||
|
||||
depends_on = [helm_release.prometheus_chart_europe]
|
||||
}
|
||||
|
||||
resource "helm_release" "prometheus_chart_asia" {
|
||||
provider = helm.asia
|
||||
|
||||
repository = local.prometheus_helm_repo
|
||||
chart = local.prometheus_helm_chart
|
||||
name = local.prometheus_release_name
|
||||
namespace = kubernetes_namespace.coder_asia.metadata.0.name
|
||||
values = [templatefile("${path.module}/prometheus_helm_values.tftpl", {
|
||||
deployment_name = var.name,
|
||||
nodepool = google_container_node_pool.node_pool["asia_misc"].name,
|
||||
cluster = "asia",
|
||||
prometheus_remote_write_url = var.prometheus_remote_write_url,
|
||||
prometheus_remote_write_metrics_regex = local.prometheus_remote_write_metrics_regex,
|
||||
prometheus_remote_write_send_interval = local.prometheus_remote_write_send_interval,
|
||||
})]
|
||||
}
|
||||
|
||||
resource "kubectl_manifest" "pod_monitor_asia" {
|
||||
provider = kubectl.asia
|
||||
|
||||
yaml_body = <<YAML
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
namespace: ${kubernetes_namespace.coder_asia.metadata.0.name}
|
||||
name: coder-monitoring
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
"app.kubernetes.io/name": coder
|
||||
podMetricsEndpoints:
|
||||
- port: prometheus-http
|
||||
interval: 30s
|
||||
YAML
|
||||
|
||||
depends_on = [helm_release.prometheus_chart_asia]
|
||||
}
|
||||
@@ -1,38 +0,0 @@
|
||||
alertmanager:
|
||||
enabled: false
|
||||
grafana:
|
||||
enabled: false
|
||||
prometheusOperator:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${nodepool}"]
|
||||
prometheus:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${nodepool}"]
|
||||
prometheusSpec:
|
||||
externalLabels:
|
||||
deployment_name: "${deployment_name}"
|
||||
cluster: "${cluster}"
|
||||
podMonitorSelectorNilUsesHelmValues: false
|
||||
serviceMonitorSelectorNilUsesHelmValues: false
|
||||
remoteWrite:
|
||||
- url: "${prometheus_remote_write_url}"
|
||||
tlsConfig:
|
||||
insecureSkipVerify: true
|
||||
writeRelabelConfigs:
|
||||
- sourceLabels: [__name__]
|
||||
regex: "${prometheus_remote_write_metrics_regex}"
|
||||
action: keep
|
||||
metadataConfig:
|
||||
sendInterval: "${prometheus_remote_write_send_interval}"
|
||||
@@ -1,74 +0,0 @@
|
||||
locals {
|
||||
scenarios = {
|
||||
large = {
|
||||
coder = {
|
||||
nodepool_size = 3
|
||||
machine_type = "c2d-standard-8"
|
||||
replicas = 3
|
||||
cpu_request = "4000m"
|
||||
mem_request = "12Gi"
|
||||
cpu_limit = "4000m"
|
||||
mem_limit = "12Gi"
|
||||
}
|
||||
provisionerd = {
|
||||
replicas = 30
|
||||
cpu_request = "100m"
|
||||
mem_request = "256Mi"
|
||||
cpu_limit = "1000m"
|
||||
mem_limit = "1Gi"
|
||||
}
|
||||
workspaces = {
|
||||
count_per_deployment = 100
|
||||
nodepool_size = 3
|
||||
machine_type = "c2d-standard-32"
|
||||
cpu_request = "100m"
|
||||
mem_request = "128Mi"
|
||||
cpu_limit = "100m"
|
||||
mem_limit = "128Mi"
|
||||
}
|
||||
misc = {
|
||||
nodepool_size = 1
|
||||
machine_type = "c2d-standard-32"
|
||||
}
|
||||
cloudsql = {
|
||||
tier = "db-custom-2-7680"
|
||||
max_connections = 500
|
||||
}
|
||||
}
|
||||
small = {
|
||||
coder = {
|
||||
nodepool_size = 3
|
||||
machine_type = "c2d-standard-8"
|
||||
replicas = 3
|
||||
cpu_request = "4000m"
|
||||
mem_request = "12Gi"
|
||||
cpu_limit = "4000m"
|
||||
mem_limit = "12Gi"
|
||||
}
|
||||
provisionerd = {
|
||||
replicas = 5
|
||||
cpu_request = "100m"
|
||||
mem_request = "256Mi"
|
||||
cpu_limit = "1000m"
|
||||
mem_limit = "1Gi"
|
||||
}
|
||||
workspaces = {
|
||||
count_per_deployment = 10
|
||||
nodepool_size = 3
|
||||
machine_type = "c2d-standard-8"
|
||||
cpu_request = "100m"
|
||||
mem_request = "128Mi"
|
||||
cpu_limit = "100m"
|
||||
mem_limit = "128Mi"
|
||||
}
|
||||
misc = {
|
||||
nodepool_size = 1
|
||||
machine_type = "c2d-standard-8"
|
||||
}
|
||||
cloudsql = {
|
||||
tier = "db-custom-2-7680"
|
||||
max_connections = 100
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,13 +0,0 @@
|
||||
locals {
|
||||
coder_certs_namespace = "coder-certs"
|
||||
}
|
||||
|
||||
# These certificates are managed by flux and cert-manager.
|
||||
data "kubernetes_secret" "coder_tls" {
|
||||
for_each = local.deployments
|
||||
provider = kubernetes.observability
|
||||
metadata {
|
||||
name = "coder-${var.name}-${each.key}-tls"
|
||||
namespace = local.coder_certs_namespace
|
||||
}
|
||||
}
|
||||
@@ -1,112 +0,0 @@
|
||||
variable "name" {
|
||||
description = "The name all resources will be prefixed with. Must be one of alpha, bravo, or charlie."
|
||||
validation {
|
||||
condition = contains(["alpha", "bravo", "charlie"], var.name)
|
||||
error_message = "Name must be one of alpha, bravo, or charlie."
|
||||
}
|
||||
}
|
||||
|
||||
variable "scenario" {
|
||||
description = "The scenario to deploy"
|
||||
validation {
|
||||
condition = contains(["small", "medium", "large"], var.scenario)
|
||||
error_message = "Scenario must be one of small, medium, or large"
|
||||
}
|
||||
}
|
||||
|
||||
// GCP
|
||||
variable "project_id" {
|
||||
description = "The project in which to provision resources"
|
||||
default = "coder-scaletest"
|
||||
}
|
||||
|
||||
variable "k8s_version" {
|
||||
description = "Kubernetes version to provision."
|
||||
default = "1.24"
|
||||
}
|
||||
|
||||
// Cloudflare
|
||||
variable "cloudflare_api_token" {
|
||||
description = "Cloudflare API token."
|
||||
sensitive = true
|
||||
# only override if you want to change the cloudflare_domain; pulls the token for scaletest.dev from Google Secrets
|
||||
# Manager if null.
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "cloudflare_domain" {
|
||||
description = "Cloudflare coder domain."
|
||||
default = "scaletest.dev"
|
||||
}
|
||||
|
||||
// Coder
|
||||
variable "coder_license" {
|
||||
description = "Coder license key."
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
variable "coder_chart_version" {
|
||||
description = "Version of the Coder Helm chart to install. Defaults to latest."
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "coder_image_tag" {
|
||||
description = "Tag to use for Coder image."
|
||||
default = "latest"
|
||||
}
|
||||
|
||||
variable "coder_image_repo" {
|
||||
description = "Repository to use for Coder image."
|
||||
default = "ghcr.io/coder/coder"
|
||||
}
|
||||
|
||||
variable "coder_experiments" {
|
||||
description = "Coder Experiments to enable."
|
||||
default = ""
|
||||
}
|
||||
|
||||
// Workspaces
|
||||
variable "workspace_image" {
|
||||
description = "Image and tag to use for workspaces."
|
||||
default = "docker.io/codercom/enterprise-minimal:ubuntu"
|
||||
}
|
||||
|
||||
variable "provisionerd_chart_version" {
|
||||
description = "Version of the Provisionerd Helm chart to install. Defaults to latest."
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "provisionerd_image_repo" {
|
||||
description = "Repository to use for Provisionerd image."
|
||||
default = "ghcr.io/coder/coder"
|
||||
}
|
||||
|
||||
variable "provisionerd_image_tag" {
|
||||
description = "Tag to use for Provisionerd image."
|
||||
default = "latest"
|
||||
}
|
||||
|
||||
variable "observability_cluster_name" {
|
||||
description = "Name of the observability GKE cluster."
|
||||
default = "observability"
|
||||
}
|
||||
|
||||
variable "observability_cluster_location" {
|
||||
description = "Location of the observability GKE cluster."
|
||||
default = "us-east1-b"
|
||||
}
|
||||
|
||||
variable "observability_cluster_vpc" {
|
||||
description = "Name of the observability cluster VPC network to peer with."
|
||||
default = "default"
|
||||
}
|
||||
|
||||
variable "cloudflare_api_token_secret" {
|
||||
description = "Name of the Google Secret Manager secret containing the Cloudflare API token."
|
||||
default = "cloudflare-api-token-dns"
|
||||
}
|
||||
|
||||
// Prometheus
|
||||
variable "prometheus_remote_write_url" {
|
||||
description = "URL to push prometheus metrics to."
|
||||
}
|
||||
@@ -1,186 +0,0 @@
|
||||
data "google_compute_default_service_account" "default" {
|
||||
project = var.project_id
|
||||
}
|
||||
|
||||
locals {
|
||||
abs_module_path = abspath(path.module)
|
||||
rel_kubeconfig_path = "../../.coderv2/${var.name}-cluster.kubeconfig"
|
||||
cluster_kubeconfig_path = abspath("${local.abs_module_path}/${local.rel_kubeconfig_path}")
|
||||
}
|
||||
|
||||
resource "google_container_cluster" "primary" {
|
||||
name = var.name
|
||||
location = var.zone
|
||||
project = var.project_id
|
||||
network = google_compute_network.vpc.name
|
||||
subnetwork = google_compute_subnetwork.subnet.name
|
||||
networking_mode = "VPC_NATIVE"
|
||||
default_max_pods_per_node = 256
|
||||
ip_allocation_policy { # Required with networking_mode=VPC_NATIVE
|
||||
|
||||
}
|
||||
release_channel {
|
||||
# Setting release channel as STABLE can cause unexpected cluster upgrades.
|
||||
channel = "UNSPECIFIED"
|
||||
}
|
||||
initial_node_count = 1
|
||||
remove_default_node_pool = true
|
||||
|
||||
network_policy {
|
||||
enabled = true
|
||||
}
|
||||
depends_on = [
|
||||
google_project_service.api["container.googleapis.com"]
|
||||
]
|
||||
monitoring_config {
|
||||
enable_components = ["SYSTEM_COMPONENTS"]
|
||||
managed_prometheus {
|
||||
enabled = false
|
||||
}
|
||||
}
|
||||
workload_identity_config {
|
||||
workload_pool = "${data.google_project.project.project_id}.svc.id.goog"
|
||||
}
|
||||
|
||||
|
||||
lifecycle {
|
||||
ignore_changes = [
|
||||
maintenance_policy,
|
||||
release_channel,
|
||||
remove_default_node_pool
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_container_node_pool" "coder" {
|
||||
name = "${var.name}-coder"
|
||||
location = var.zone
|
||||
project = var.project_id
|
||||
cluster = google_container_cluster.primary.name
|
||||
autoscaling {
|
||||
min_node_count = 1
|
||||
max_node_count = var.nodepool_size_coder
|
||||
}
|
||||
node_config {
|
||||
oauth_scopes = [
|
||||
"https://www.googleapis.com/auth/logging.write",
|
||||
"https://www.googleapis.com/auth/monitoring",
|
||||
"https://www.googleapis.com/auth/trace.append",
|
||||
"https://www.googleapis.com/auth/devstorage.read_only",
|
||||
"https://www.googleapis.com/auth/service.management.readonly",
|
||||
"https://www.googleapis.com/auth/servicecontrol",
|
||||
]
|
||||
disk_size_gb = var.node_disk_size_gb
|
||||
machine_type = var.nodepool_machine_type_coder
|
||||
image_type = var.node_image_type
|
||||
preemptible = var.node_preemptible
|
||||
service_account = data.google_compute_default_service_account.default.email
|
||||
tags = ["gke-node", "${var.project_id}-gke"]
|
||||
labels = {
|
||||
env = var.project_id
|
||||
}
|
||||
metadata = {
|
||||
disable-legacy-endpoints = "true"
|
||||
}
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [management[0].auto_repair, management[0].auto_upgrade, timeouts]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_container_node_pool" "workspaces" {
|
||||
name = "${var.name}-workspaces"
|
||||
location = var.zone
|
||||
project = var.project_id
|
||||
cluster = google_container_cluster.primary.name
|
||||
autoscaling {
|
||||
min_node_count = 0
|
||||
total_max_node_count = var.nodepool_size_workspaces
|
||||
}
|
||||
management {
|
||||
auto_upgrade = false
|
||||
}
|
||||
node_config {
|
||||
oauth_scopes = [
|
||||
"https://www.googleapis.com/auth/logging.write",
|
||||
"https://www.googleapis.com/auth/monitoring",
|
||||
"https://www.googleapis.com/auth/trace.append",
|
||||
"https://www.googleapis.com/auth/devstorage.read_only",
|
||||
"https://www.googleapis.com/auth/service.management.readonly",
|
||||
"https://www.googleapis.com/auth/servicecontrol",
|
||||
]
|
||||
disk_size_gb = var.node_disk_size_gb
|
||||
machine_type = var.nodepool_machine_type_workspaces
|
||||
image_type = var.node_image_type
|
||||
preemptible = var.node_preemptible
|
||||
service_account = data.google_compute_default_service_account.default.email
|
||||
tags = ["gke-node", "${var.project_id}-gke"]
|
||||
labels = {
|
||||
env = var.project_id
|
||||
}
|
||||
metadata = {
|
||||
disable-legacy-endpoints = "true"
|
||||
}
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [management[0].auto_repair, management[0].auto_upgrade, timeouts]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_container_node_pool" "misc" {
|
||||
name = "${var.name}-misc"
|
||||
location = var.zone
|
||||
project = var.project_id
|
||||
cluster = google_container_cluster.primary.name
|
||||
node_count = var.state == "stopped" ? 0 : var.nodepool_size_misc
|
||||
management {
|
||||
auto_upgrade = false
|
||||
}
|
||||
node_config {
|
||||
oauth_scopes = [
|
||||
"https://www.googleapis.com/auth/logging.write",
|
||||
"https://www.googleapis.com/auth/monitoring",
|
||||
"https://www.googleapis.com/auth/trace.append",
|
||||
"https://www.googleapis.com/auth/devstorage.read_only",
|
||||
"https://www.googleapis.com/auth/service.management.readonly",
|
||||
"https://www.googleapis.com/auth/servicecontrol",
|
||||
]
|
||||
disk_size_gb = var.node_disk_size_gb
|
||||
machine_type = var.nodepool_machine_type_misc
|
||||
image_type = var.node_image_type
|
||||
preemptible = var.node_preemptible
|
||||
service_account = data.google_compute_default_service_account.default.email
|
||||
tags = ["gke-node", "${var.project_id}-gke"]
|
||||
labels = {
|
||||
env = var.project_id
|
||||
}
|
||||
metadata = {
|
||||
disable-legacy-endpoints = "true"
|
||||
}
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [management[0].auto_repair, management[0].auto_upgrade, timeouts]
|
||||
}
|
||||
}
|
||||
|
||||
resource "null_resource" "cluster_kubeconfig" {
|
||||
depends_on = [google_container_cluster.primary]
|
||||
triggers = {
|
||||
path = local.cluster_kubeconfig_path
|
||||
name = google_container_cluster.primary.name
|
||||
project_id = var.project_id
|
||||
zone = var.zone
|
||||
}
|
||||
provisioner "local-exec" {
|
||||
command = <<EOF
|
||||
KUBECONFIG=${self.triggers.path} gcloud container clusters get-credentials ${self.triggers.name} --project=${self.triggers.project_id} --zone=${self.triggers.zone}
|
||||
EOF
|
||||
}
|
||||
|
||||
provisioner "local-exec" {
|
||||
when = destroy
|
||||
command = <<EOF
|
||||
rm -f ${self.triggers.path}
|
||||
EOF
|
||||
}
|
||||
}
|
||||
@@ -1,88 +0,0 @@
|
||||
resource "google_sql_database_instance" "db" {
|
||||
name = var.name
|
||||
region = var.region
|
||||
database_version = var.cloudsql_version
|
||||
deletion_protection = false
|
||||
|
||||
depends_on = [google_service_networking_connection.private_vpc_connection]
|
||||
|
||||
settings {
|
||||
tier = var.cloudsql_tier
|
||||
activation_policy = "ALWAYS"
|
||||
availability_type = "ZONAL"
|
||||
|
||||
location_preference {
|
||||
zone = var.zone
|
||||
}
|
||||
|
||||
database_flags {
|
||||
name = "max_connections"
|
||||
value = var.cloudsql_max_connections
|
||||
}
|
||||
|
||||
ip_configuration {
|
||||
ipv4_enabled = false
|
||||
private_network = google_compute_network.vpc.id
|
||||
}
|
||||
|
||||
insights_config {
|
||||
query_insights_enabled = true
|
||||
query_string_length = 1024
|
||||
record_application_tags = false
|
||||
record_client_address = false
|
||||
}
|
||||
}
|
||||
|
||||
lifecycle {
|
||||
ignore_changes = [deletion_protection, timeouts]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_sql_database" "coder" {
|
||||
project = var.project_id
|
||||
instance = google_sql_database_instance.db.id
|
||||
name = "${var.name}-coder"
|
||||
# required for postgres, otherwise db fails to delete
|
||||
deletion_policy = "ABANDON"
|
||||
lifecycle {
|
||||
ignore_changes = [deletion_policy]
|
||||
}
|
||||
}
|
||||
|
||||
resource "random_password" "coder-postgres-password" {
|
||||
length = 12
|
||||
}
|
||||
|
||||
resource "random_password" "prometheus-postgres-password" {
|
||||
length = 12
|
||||
}
|
||||
|
||||
resource "google_sql_user" "coder" {
|
||||
project = var.project_id
|
||||
instance = google_sql_database_instance.db.id
|
||||
name = "${var.name}-coder"
|
||||
type = "BUILT_IN"
|
||||
password = random_password.coder-postgres-password.result
|
||||
# required for postgres, otherwise user fails to delete
|
||||
deletion_policy = "ABANDON"
|
||||
lifecycle {
|
||||
ignore_changes = [deletion_policy, password]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_sql_user" "prometheus" {
|
||||
project = var.project_id
|
||||
instance = google_sql_database_instance.db.id
|
||||
name = "${var.name}-prometheus"
|
||||
type = "BUILT_IN"
|
||||
password = random_password.prometheus-postgres-password.result
|
||||
# required for postgres, otherwise user fails to delete
|
||||
deletion_policy = "ABANDON"
|
||||
lifecycle {
|
||||
ignore_changes = [deletion_policy, password]
|
||||
}
|
||||
}
|
||||
|
||||
locals {
|
||||
coder_db_url = "postgres://${google_sql_user.coder.name}:${urlencode(random_password.coder-postgres-password.result)}@${google_sql_database_instance.db.private_ip_address}/${google_sql_database.coder.name}?sslmode=disable"
|
||||
}
|
||||
@@ -1,27 +0,0 @@
|
||||
locals {
|
||||
project_apis = [
|
||||
"cloudtrace",
|
||||
"compute",
|
||||
"container",
|
||||
"logging",
|
||||
"monitoring",
|
||||
"servicemanagement",
|
||||
"servicenetworking",
|
||||
"sqladmin",
|
||||
"stackdriver",
|
||||
"storage-api",
|
||||
]
|
||||
}
|
||||
|
||||
data "google_project" "project" {
|
||||
project_id = var.project_id
|
||||
}
|
||||
|
||||
resource "google_project_service" "api" {
|
||||
for_each = toset(local.project_apis)
|
||||
project = data.google_project.project.project_id
|
||||
service = "${each.value}.googleapis.com"
|
||||
|
||||
disable_dependent_services = false
|
||||
disable_on_destroy = false
|
||||
}
|
||||
@@ -1,39 +0,0 @@
|
||||
resource "google_compute_network" "vpc" {
|
||||
project = var.project_id
|
||||
name = var.name
|
||||
auto_create_subnetworks = "false"
|
||||
depends_on = [
|
||||
google_project_service.api["compute.googleapis.com"]
|
||||
]
|
||||
}
|
||||
|
||||
resource "google_compute_subnetwork" "subnet" {
|
||||
name = var.name
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
network = google_compute_network.vpc.name
|
||||
ip_cidr_range = var.subnet_cidr
|
||||
}
|
||||
|
||||
resource "google_compute_global_address" "sql_peering" {
|
||||
project = var.project_id
|
||||
name = "${var.name}-sql-peering"
|
||||
purpose = "VPC_PEERING"
|
||||
address_type = "INTERNAL"
|
||||
prefix_length = 16
|
||||
network = google_compute_network.vpc.id
|
||||
}
|
||||
|
||||
resource "google_compute_address" "coder" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
name = "${var.name}-coder"
|
||||
address_type = "EXTERNAL"
|
||||
network_tier = "PREMIUM"
|
||||
}
|
||||
|
||||
resource "google_service_networking_connection" "private_vpc_connection" {
|
||||
network = google_compute_network.vpc.id
|
||||
service = "servicenetworking.googleapis.com"
|
||||
reserved_peering_ranges = [google_compute_global_address.sql_peering.name]
|
||||
}
|
||||
@@ -1,20 +0,0 @@
|
||||
terraform {
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 4.36"
|
||||
}
|
||||
|
||||
random = {
|
||||
source = "hashicorp/random"
|
||||
version = "~> 3.5"
|
||||
}
|
||||
}
|
||||
|
||||
required_version = "~> 1.5.0"
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
region = var.region
|
||||
project = var.project_id
|
||||
}
|
||||
@@ -1,73 +0,0 @@
|
||||
output "coder_db_url" {
|
||||
description = "URL of the database for Coder."
|
||||
value = local.coder_db_url
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
output "coder_address" {
|
||||
description = "IP address to use for the Coder service."
|
||||
value = google_compute_address.coder.address
|
||||
}
|
||||
|
||||
output "kubernetes_kubeconfig_path" {
|
||||
description = "Kubeconfig path."
|
||||
value = local.cluster_kubeconfig_path
|
||||
}
|
||||
|
||||
output "kubernetes_nodepool_coder" {
|
||||
description = "Name of the nodepool on which to run Coder."
|
||||
value = google_container_node_pool.coder.name
|
||||
}
|
||||
|
||||
output "kubernetes_nodepool_misc" {
|
||||
description = "Name of the nodepool on which to run everything else."
|
||||
value = google_container_node_pool.misc.name
|
||||
}
|
||||
|
||||
output "kubernetes_nodepool_workspaces" {
|
||||
description = "Name of the nodepool on which to run workspaces."
|
||||
value = google_container_node_pool.workspaces.name
|
||||
}
|
||||
|
||||
output "prometheus_external_label_cluster" {
|
||||
description = "Value for the Prometheus external label named cluster."
|
||||
value = google_container_cluster.primary.name
|
||||
}
|
||||
|
||||
output "prometheus_postgres_dbname" {
|
||||
description = "Name of the database for Prometheus to monitor."
|
||||
value = google_sql_database.coder.name
|
||||
}
|
||||
|
||||
output "prometheus_postgres_host" {
|
||||
description = "Hostname of the database for Prometheus to connect to."
|
||||
value = google_sql_database_instance.db.private_ip_address
|
||||
}
|
||||
|
||||
output "prometheus_postgres_password" {
|
||||
description = "Postgres password for Prometheus."
|
||||
value = random_password.prometheus-postgres-password.result
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
output "prometheus_postgres_user" {
|
||||
description = "Postgres username for Prometheus."
|
||||
value = google_sql_user.prometheus.name
|
||||
}
|
||||
|
||||
resource "local_file" "outputs" {
|
||||
filename = "${path.module}/../../.coderv2/infra_outputs.tfvars"
|
||||
content = <<EOF
|
||||
coder_db_url = "${local.coder_db_url}"
|
||||
coder_address = "${google_compute_address.coder.address}"
|
||||
kubernetes_kubeconfig_path = "${local.cluster_kubeconfig_path}"
|
||||
kubernetes_nodepool_coder = "${google_container_node_pool.coder.name}"
|
||||
kubernetes_nodepool_misc = "${google_container_node_pool.misc.name}"
|
||||
kubernetes_nodepool_workspaces = "${google_container_node_pool.workspaces.name}"
|
||||
prometheus_external_label_cluster = "${google_container_cluster.primary.name}"
|
||||
prometheus_postgres_dbname = "${google_sql_database.coder.name}"
|
||||
prometheus_postgres_host = "${google_sql_database_instance.db.private_ip_address}"
|
||||
prometheus_postgres_password = "${random_password.prometheus-postgres-password.result}"
|
||||
prometheus_postgres_user = "${google_sql_user.prometheus.name}"
|
||||
EOF
|
||||
}
|
||||
@@ -1,107 +0,0 @@
|
||||
variable "state" {
|
||||
description = "The state of the cluster. Valid values are 'started', and 'stopped'."
|
||||
validation {
|
||||
condition = contains(["started", "stopped"], var.state)
|
||||
error_message = "value must be one of 'started' or 'stopped'"
|
||||
}
|
||||
default = "started"
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "The project in which to provision resources"
|
||||
}
|
||||
|
||||
variable "name" {
|
||||
description = "Adds a prefix to resources."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
description = "GCP region in which to provision resources."
|
||||
default = "us-east1"
|
||||
}
|
||||
|
||||
variable "zone" {
|
||||
description = "GCP zone in which to provision resources."
|
||||
default = "us-east1-c"
|
||||
}
|
||||
|
||||
variable "subnet_cidr" {
|
||||
description = "CIDR range for the subnet."
|
||||
default = "10.200.0.0/24"
|
||||
}
|
||||
|
||||
variable "k8s_version" {
|
||||
description = "Kubernetes version to provision."
|
||||
default = "1.24"
|
||||
}
|
||||
|
||||
variable "node_disk_size_gb" {
|
||||
description = "Size of the root disk for cluster nodes."
|
||||
default = 100
|
||||
}
|
||||
|
||||
variable "node_image_type" {
|
||||
description = "Image type to use for cluster nodes."
|
||||
default = "cos_containerd"
|
||||
}
|
||||
|
||||
// Preemptible nodes are way cheaper, but can be pulled out
|
||||
// from under you at any time. Caveat emptor.
|
||||
variable "node_preemptible" {
|
||||
description = "Use preemptible nodes."
|
||||
default = false
|
||||
}
|
||||
|
||||
// We create three nodepools:
|
||||
// - One for the Coder control plane
|
||||
// - One for workspaces
|
||||
// - One for everything else (for example, load generation)
|
||||
|
||||
// These variables control the node pool dedicated to Coder.
|
||||
variable "nodepool_machine_type_coder" {
|
||||
description = "Machine type to use for Coder control plane nodepool."
|
||||
default = "t2d-standard-4"
|
||||
}
|
||||
|
||||
variable "nodepool_size_coder" {
|
||||
description = "Number of cluster nodes for the Coder control plane nodepool."
|
||||
default = 1
|
||||
}
|
||||
|
||||
// These variables control the node pool dedicated to workspaces.
|
||||
variable "nodepool_machine_type_workspaces" {
|
||||
description = "Machine type to use for the workspaces nodepool."
|
||||
default = "t2d-standard-4"
|
||||
}
|
||||
|
||||
variable "nodepool_size_workspaces" {
|
||||
description = "Number of cluster nodes for the workspaces nodepool."
|
||||
default = 1
|
||||
}
|
||||
|
||||
// These variables control the node pool for everything else.
|
||||
variable "nodepool_machine_type_misc" {
|
||||
description = "Machine type to use for the misc nodepool."
|
||||
default = "t2d-standard-4"
|
||||
}
|
||||
|
||||
variable "nodepool_size_misc" {
|
||||
description = "Number of cluster nodes for the misc nodepool."
|
||||
default = 1
|
||||
}
|
||||
|
||||
// These variables control the size of the database to be used by Coder.
|
||||
variable "cloudsql_version" {
|
||||
description = "CloudSQL version to provision"
|
||||
default = "POSTGRES_14"
|
||||
}
|
||||
|
||||
variable "cloudsql_tier" {
|
||||
description = "CloudSQL database tier."
|
||||
default = "db-f1-micro"
|
||||
}
|
||||
|
||||
variable "cloudsql_max_connections" {
|
||||
description = "CloudSQL database max_connections"
|
||||
default = 500
|
||||
}
|
||||
@@ -1,67 +0,0 @@
|
||||
# Terraform configuration for cert-manaer
|
||||
|
||||
locals {
|
||||
cert_manager_namespace = "cert-manager"
|
||||
cert_manager_helm_repo = "https://charts.jetstack.io"
|
||||
cert_manager_helm_chart = "cert-manager"
|
||||
cert_manager_release_name = "cert-manager"
|
||||
cert_manager_chart_version = "1.12.2"
|
||||
cloudflare_issuer_private_key_secret_name = "cloudflare-issuer-private-key"
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "cloudflare-api-key" {
|
||||
metadata {
|
||||
name = "cloudflare-api-key-secret"
|
||||
namespace = local.cert_manager_namespace
|
||||
}
|
||||
data = {
|
||||
api-token = var.cloudflare_api_token
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_namespace" "cert-manager-namespace" {
|
||||
metadata {
|
||||
name = local.cert_manager_namespace
|
||||
}
|
||||
}
|
||||
|
||||
resource "helm_release" "cert-manager" {
|
||||
repository = local.cert_manager_helm_repo
|
||||
chart = local.cert_manager_helm_chart
|
||||
name = local.cert_manager_release_name
|
||||
namespace = kubernetes_namespace.cert-manager-namespace.metadata.0.name
|
||||
values = [<<EOF
|
||||
installCRDs: true
|
||||
EOF
|
||||
]
|
||||
}
|
||||
|
||||
resource "kubernetes_manifest" "cloudflare-cluster-issuer" {
|
||||
manifest = {
|
||||
apiVersion = "cert-manager.io/v1"
|
||||
kind = "ClusterIssuer"
|
||||
metadata = {
|
||||
name = "cloudflare-issuer"
|
||||
}
|
||||
spec = {
|
||||
acme = {
|
||||
email = var.cloudflare_email
|
||||
privateKeySecretRef = {
|
||||
name = local.cloudflare_issuer_private_key_secret_name
|
||||
}
|
||||
solvers = [
|
||||
{
|
||||
dns01 = {
|
||||
cloudflare = {
|
||||
apiTokenSecretRef = {
|
||||
name = kubernetes_secret.cloudflare-api-key.metadata.0.name
|
||||
key = "api-token"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,375 +0,0 @@
|
||||
data "google_client_config" "default" {}
|
||||
|
||||
locals {
|
||||
coder_url = var.coder_access_url
|
||||
coder_admin_email = "admin@coder.com"
|
||||
coder_admin_user = "coder"
|
||||
coder_helm_repo = "https://helm.coder.com/v2"
|
||||
coder_helm_chart = "coder"
|
||||
coder_namespace = "coder-${var.name}"
|
||||
coder_release_name = var.name
|
||||
provisionerd_helm_chart = "coder-provisioner"
|
||||
provisionerd_release_name = "${var.name}-provisionerd"
|
||||
}
|
||||
|
||||
resource "kubernetes_namespace" "coder_namespace" {
|
||||
metadata {
|
||||
name = local.coder_namespace
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_default_service_account]
|
||||
}
|
||||
}
|
||||
|
||||
resource "random_password" "provisionerd_psk" {
|
||||
length = 26
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "coder-db" {
|
||||
type = "Opaque"
|
||||
metadata {
|
||||
name = "coder-db-url"
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
url = var.coder_db_url
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "provisionerd_psk" {
|
||||
type = "Opaque"
|
||||
metadata {
|
||||
name = "coder-provisioner-psk"
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
psk = random_password.provisionerd_psk.result
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
# OIDC secret needs to be manually provisioned for now.
|
||||
data "kubernetes_secret" "coder_oidc" {
|
||||
metadata {
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
name = "coder-oidc"
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_manifest" "coder_certificate" {
|
||||
manifest = {
|
||||
apiVersion = "cert-manager.io/v1"
|
||||
kind = "Certificate"
|
||||
metadata = {
|
||||
name = "${var.name}"
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
}
|
||||
spec = {
|
||||
secretName = "${var.name}-tls"
|
||||
dnsNames = regex("https?://([^/]+)", local.coder_url)
|
||||
issuerRef = {
|
||||
name = kubernetes_manifest.cloudflare-cluster-issuer.manifest.metadata.name
|
||||
kind = "ClusterIssuer"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
data "kubernetes_secret" "coder_tls" {
|
||||
metadata {
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
name = "${var.name}-tls"
|
||||
}
|
||||
depends_on = [kubernetes_manifest.coder_certificate]
|
||||
}
|
||||
|
||||
resource "helm_release" "coder-chart" {
|
||||
repository = local.coder_helm_repo
|
||||
chart = local.coder_helm_chart
|
||||
name = local.coder_release_name
|
||||
version = var.coder_chart_version
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
values = [<<EOF
|
||||
coder:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${var.kubernetes_nodepool_coder}"]
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 1
|
||||
podAffinityTerm:
|
||||
topologyKey: "kubernetes.io/hostname"
|
||||
labelSelector:
|
||||
matchExpressions:
|
||||
- key: "app.kubernetes.io/instance"
|
||||
operator: "In"
|
||||
values: ["${local.coder_release_name}"]
|
||||
env:
|
||||
- name: "CODER_ACCESS_URL"
|
||||
value: "${local.coder_url}"
|
||||
- name: "CODER_CACHE_DIRECTORY"
|
||||
value: "/tmp/coder"
|
||||
- name: "CODER_TELEMETRY_ENABLE"
|
||||
value: "false"
|
||||
- name: "CODER_LOGGING_HUMAN"
|
||||
value: "/dev/null"
|
||||
- name: "CODER_LOGGING_STACKDRIVER"
|
||||
value: "/dev/stderr"
|
||||
- name: "CODER_PG_CONNECTION_URL"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: "${kubernetes_secret.coder-db.metadata.0.name}"
|
||||
key: url
|
||||
- name: "CODER_PPROF_ENABLE"
|
||||
value: "true"
|
||||
- name: "CODER_PROMETHEUS_ENABLE"
|
||||
value: "true"
|
||||
- name: "CODER_PROMETHEUS_COLLECT_AGENT_STATS"
|
||||
value: "true"
|
||||
- name: "CODER_PROMETHEUS_COLLECT_DB_METRICS"
|
||||
value: "true"
|
||||
- name: "CODER_VERBOSE"
|
||||
value: "true"
|
||||
- name: "CODER_EXPERIMENTS"
|
||||
value: "${var.coder_experiments}"
|
||||
- name: "CODER_DANGEROUS_DISABLE_RATE_LIMITS"
|
||||
value: "true"
|
||||
# Disabling built-in provisioner daemons
|
||||
- name: "CODER_PROVISIONER_DAEMONS"
|
||||
value: "0"
|
||||
- name: CODER_PROVISIONER_DAEMON_PSK
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: psk
|
||||
name: "${kubernetes_secret.provisionerd_psk.metadata.0.name}"
|
||||
# Enable OIDC
|
||||
- name: "CODER_OIDC_ISSUER_URL"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: issuer-url
|
||||
name: "${data.kubernetes_secret.coder_oidc.metadata.0.name}"
|
||||
- name: "CODER_OIDC_EMAIL_DOMAIN"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: email-domain
|
||||
name: "${data.kubernetes_secret.coder_oidc.metadata.0.name}"
|
||||
- name: "CODER_OIDC_CLIENT_ID"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: client-id
|
||||
name: "${data.kubernetes_secret.coder_oidc.metadata.0.name}"
|
||||
- name: "CODER_OIDC_CLIENT_SECRET"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: client-secret
|
||||
name: "${data.kubernetes_secret.coder_oidc.metadata.0.name}"
|
||||
# Send OTEL traces to the cluster-local collector to sample 10%
|
||||
- name: "OTEL_EXPORTER_OTLP_ENDPOINT"
|
||||
value: "http://${kubernetes_manifest.otel-collector.manifest.metadata.name}-collector.${kubernetes_namespace.coder_namespace.metadata.0.name}.svc.cluster.local:4317"
|
||||
- name: "OTEL_TRACES_SAMPLER"
|
||||
value: parentbased_traceidratio
|
||||
- name: "OTEL_TRACES_SAMPLER_ARG"
|
||||
value: "0.1"
|
||||
image:
|
||||
repo: ${var.coder_image_repo}
|
||||
tag: ${var.coder_image_tag}
|
||||
replicaCount: "${var.coder_replicas}"
|
||||
resources:
|
||||
requests:
|
||||
cpu: "${var.coder_cpu_request}"
|
||||
memory: "${var.coder_mem_request}"
|
||||
limits:
|
||||
cpu: "${var.coder_cpu_limit}"
|
||||
memory: "${var.coder_mem_limit}"
|
||||
securityContext:
|
||||
readOnlyRootFilesystem: true
|
||||
service:
|
||||
enable: true
|
||||
sessionAffinity: None
|
||||
loadBalancerIP: "${var.coder_address}"
|
||||
volumeMounts:
|
||||
- mountPath: "/tmp"
|
||||
name: cache
|
||||
readOnly: false
|
||||
volumes:
|
||||
- emptyDir:
|
||||
sizeLimit: 1024Mi
|
||||
name: cache
|
||||
EOF
|
||||
]
|
||||
}
|
||||
|
||||
resource "helm_release" "provisionerd-chart" {
|
||||
repository = local.coder_helm_repo
|
||||
chart = local.provisionerd_helm_chart
|
||||
name = local.provisionerd_release_name
|
||||
version = var.provisionerd_chart_version
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
values = [<<EOF
|
||||
coder:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${var.kubernetes_nodepool_coder}"]
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 1
|
||||
podAffinityTerm:
|
||||
topologyKey: "kubernetes.io/hostname"
|
||||
labelSelector:
|
||||
matchExpressions:
|
||||
- key: "app.kubernetes.io/instance"
|
||||
operator: "In"
|
||||
values: ["${local.coder_release_name}"]
|
||||
env:
|
||||
- name: "CODER_URL"
|
||||
value: "${local.coder_url}"
|
||||
- name: "CODER_VERBOSE"
|
||||
value: "true"
|
||||
- name: "CODER_CACHE_DIRECTORY"
|
||||
value: "/tmp/coder"
|
||||
- name: "CODER_TELEMETRY_ENABLE"
|
||||
value: "false"
|
||||
- name: "CODER_LOGGING_HUMAN"
|
||||
value: "/dev/null"
|
||||
- name: "CODER_LOGGING_STACKDRIVER"
|
||||
value: "/dev/stderr"
|
||||
- name: "CODER_PROMETHEUS_ENABLE"
|
||||
value: "true"
|
||||
- name: "CODER_PROVISIONERD_TAGS"
|
||||
value = "socpe=organization"
|
||||
image:
|
||||
repo: ${var.provisionerd_image_repo}
|
||||
tag: ${var.provisionerd_image_tag}
|
||||
replicaCount: "${var.provisionerd_replicas}"
|
||||
resources:
|
||||
requests:
|
||||
cpu: "${var.provisionerd_cpu_request}"
|
||||
memory: "${var.provisionerd_mem_request}"
|
||||
limits:
|
||||
cpu: "${var.provisionerd_cpu_limit}"
|
||||
memory: "${var.provisionerd_mem_limit}"
|
||||
securityContext:
|
||||
readOnlyRootFilesystem: true
|
||||
volumeMounts:
|
||||
- mountPath: "/tmp"
|
||||
name: cache
|
||||
readOnly: false
|
||||
volumes:
|
||||
- emptyDir:
|
||||
sizeLimit: 1024Mi
|
||||
name: cache
|
||||
EOF
|
||||
]
|
||||
}
|
||||
|
||||
resource "local_file" "kubernetes_template" {
|
||||
filename = "${path.module}/../.coderv2/templates/kubernetes/main.tf"
|
||||
content = <<EOF
|
||||
terraform {
|
||||
required_providers {
|
||||
coder = {
|
||||
source = "coder/coder"
|
||||
version = "~> 0.23.0"
|
||||
}
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = "~> 2.30"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "coder" {}
|
||||
|
||||
provider "kubernetes" {
|
||||
config_path = null # always use host
|
||||
}
|
||||
|
||||
data "coder_workspace" "me" {}
|
||||
data "coder_workspace_owner" "me" {}
|
||||
|
||||
resource "coder_agent" "main" {
|
||||
os = "linux"
|
||||
arch = "amd64"
|
||||
}
|
||||
|
||||
resource "kubernetes_pod" "main" {
|
||||
count = data.coder_workspace.me.start_count
|
||||
metadata {
|
||||
name = "coder-$${lower(data.coder_workspace_owner.me.name)}-$${lower(data.coder_workspace.me.name)}"
|
||||
namespace = "${local.coder_namespace}"
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "coder-workspace"
|
||||
"app.kubernetes.io/instance" = "coder-workspace-$${lower(data.coder_workspace_owner.me.name)}-$${lower(data.coder_workspace.me.name)}"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
security_context {
|
||||
run_as_user = "1000"
|
||||
fs_group = "1000"
|
||||
}
|
||||
container {
|
||||
name = "dev"
|
||||
image = "${var.workspace_image}"
|
||||
image_pull_policy = "Always"
|
||||
command = ["sh", "-c", coder_agent.main.init_script]
|
||||
security_context {
|
||||
run_as_user = "1000"
|
||||
}
|
||||
env {
|
||||
name = "CODER_AGENT_TOKEN"
|
||||
value = coder_agent.main.token
|
||||
}
|
||||
resources {
|
||||
requests = {
|
||||
"cpu" = "${var.workspace_cpu_request}"
|
||||
"memory" = "${var.workspace_mem_request}"
|
||||
}
|
||||
limits = {
|
||||
"cpu" = "${var.workspace_cpu_limit}"
|
||||
"memory" = "${var.workspace_mem_limit}"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
affinity {
|
||||
node_affinity {
|
||||
required_during_scheduling_ignored_during_execution {
|
||||
node_selector_term {
|
||||
match_expressions {
|
||||
key = "cloud.google.com/gke-nodepool"
|
||||
operator = "In"
|
||||
values = ["${var.kubernetes_nodepool_workspaces}"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
}
|
||||
|
||||
resource "local_file" "output_vars" {
|
||||
filename = "${path.module}/../../.coderv2/url"
|
||||
content = local.coder_url
|
||||
}
|
||||
|
||||
output "coder_url" {
|
||||
description = "URL of the Coder deployment"
|
||||
value = local.coder_url
|
||||
}
|
||||
@@ -1,35 +0,0 @@
|
||||
terraform {
|
||||
required_providers {
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = "~> 2.20"
|
||||
}
|
||||
|
||||
helm = {
|
||||
source = "hashicorp/helm"
|
||||
version = "~> 2.9"
|
||||
}
|
||||
|
||||
random = {
|
||||
source = "hashicorp/random"
|
||||
version = "~> 3.5"
|
||||
}
|
||||
|
||||
tls = {
|
||||
source = "hashicorp/tls"
|
||||
version = "~> 4.0"
|
||||
}
|
||||
}
|
||||
|
||||
required_version = "~> 1.5.0"
|
||||
}
|
||||
|
||||
provider "kubernetes" {
|
||||
config_path = var.kubernetes_kubeconfig_path
|
||||
}
|
||||
|
||||
provider "helm" {
|
||||
kubernetes {
|
||||
config_path = var.kubernetes_kubeconfig_path
|
||||
}
|
||||
}
|
||||
@@ -1,69 +0,0 @@
|
||||
# Terraform configuration for OpenTelemetry Operator
|
||||
|
||||
locals {
|
||||
otel_namespace = "opentelemetry-operator-system"
|
||||
otel_operator_helm_repo = "https://open-telemetry.github.io/opentelemetry-helm-charts"
|
||||
otel_operator_helm_chart = "opentelemtry-operator"
|
||||
otel_operator_release_name = "opentelemetry-operator"
|
||||
otel_operator_chart_version = "0.34.1"
|
||||
}
|
||||
|
||||
resource "kubernetes_namespace" "otel-namespace" {
|
||||
metadata {
|
||||
name = local.otel_namespace
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_default_service_account]
|
||||
}
|
||||
}
|
||||
|
||||
resource "helm_release" "otel-operator" {
|
||||
repository = local.otel_operator_helm_repo
|
||||
chart = local.otel_operator_helm_chart
|
||||
name = local.otel_operator_release_name
|
||||
namespace = kubernetes_namespace.otel-namespace.metadata.0.name
|
||||
# Default values
|
||||
values = []
|
||||
}
|
||||
|
||||
resource "kubernetes_manifest" "otel-collector" {
|
||||
manifest = {
|
||||
apiVersion = "opentelemetry.io/v1alpha1"
|
||||
kind = "OpenTelemetryCollector"
|
||||
metadata = {
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
name = "otel"
|
||||
}
|
||||
spec = {
|
||||
config = jsonencode({
|
||||
receivers = {
|
||||
otlp = {
|
||||
protocols : {
|
||||
grpc : {}
|
||||
http : {}
|
||||
}
|
||||
}
|
||||
}
|
||||
exporters = {
|
||||
googlecloud = {
|
||||
logging = {
|
||||
loglevel = "debug"
|
||||
}
|
||||
}
|
||||
}
|
||||
service = {
|
||||
pipelines = {
|
||||
traces = {
|
||||
receivers = ["otlp"]
|
||||
processors = []
|
||||
exporters = ["logging", "googlecloud"]
|
||||
}
|
||||
}
|
||||
}
|
||||
image = "otel/open-telemetry-collector-contrib:latest"
|
||||
mode = "deployment"
|
||||
replicas = 1
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,173 +0,0 @@
|
||||
locals {
|
||||
prometheus_helm_repo = "https://charts.bitnami.com/bitnami"
|
||||
prometheus_helm_chart = "kube-prometheus"
|
||||
prometheus_exporter_helm_repo = "https://prometheus-community.github.io/helm-charts"
|
||||
prometheus_exporter_helm_chart = "prometheus-postgres-exporter"
|
||||
prometheus_release_name = "prometheus"
|
||||
prometheus_exporter_release_name = "prometheus-postgres-exporter"
|
||||
prometheus_namespace = "prometheus"
|
||||
prometheus_remote_write_enabled = var.prometheus_remote_write_password != ""
|
||||
}
|
||||
|
||||
# Create a namespace to hold our Prometheus deployment.
|
||||
resource "kubernetes_namespace" "prometheus_namespace" {
|
||||
metadata {
|
||||
name = local.prometheus_namespace
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_default_service_account]
|
||||
}
|
||||
}
|
||||
|
||||
# Create a secret to store the remote write key
|
||||
resource "kubernetes_secret" "prometheus-credentials" {
|
||||
count = local.prometheus_remote_write_enabled ? 1 : 0
|
||||
type = "kubernetes.io/basic-auth"
|
||||
metadata {
|
||||
name = "prometheus-credentials"
|
||||
namespace = kubernetes_namespace.prometheus_namespace.metadata.0.name
|
||||
}
|
||||
|
||||
data = {
|
||||
username = var.prometheus_remote_write_user
|
||||
password = var.prometheus_remote_write_password
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
# Install Prometheus using the Bitnami Prometheus helm chart.
|
||||
resource "helm_release" "prometheus-chart" {
|
||||
repository = local.prometheus_helm_repo
|
||||
chart = local.prometheus_helm_chart
|
||||
name = local.prometheus_release_name
|
||||
namespace = kubernetes_namespace.prometheus_namespace.metadata.0.name
|
||||
values = [<<EOF
|
||||
alertmanager:
|
||||
enabled: false
|
||||
blackboxExporter:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${var.kubernetes_nodepool_misc}"]
|
||||
operator:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${var.kubernetes_nodepool_misc}"]
|
||||
prometheus:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${var.kubernetes_nodepool_misc}"]
|
||||
externalLabels:
|
||||
cluster: "${var.prometheus_external_label_cluster}"
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClass: standard
|
||||
%{if local.prometheus_remote_write_enabled~}
|
||||
remoteWrite:
|
||||
- url: "${var.prometheus_remote_write_url}"
|
||||
basicAuth:
|
||||
username:
|
||||
name: "${kubernetes_secret.prometheus-credentials[0].metadata[0].name}"
|
||||
key: username
|
||||
password:
|
||||
name: "${kubernetes_secret.prometheus-credentials[0].metadata[0].name}"
|
||||
key: password
|
||||
tlsConfig:
|
||||
insecureSkipVerify: ${var.prometheus_remote_write_insecure_skip_verify}
|
||||
writeRelabelConfigs:
|
||||
- sourceLabels: [__name__]
|
||||
regex: "${var.prometheus_remote_write_metrics_regex}"
|
||||
action: keep
|
||||
metadataConfig:
|
||||
sendInterval: "${var.prometheus_remote_write_send_interval}"
|
||||
%{endif~}
|
||||
EOF
|
||||
]
|
||||
}
|
||||
|
||||
resource "kubernetes_secret" "prometheus-postgres-password" {
|
||||
type = "kubernetes.io/basic-auth"
|
||||
metadata {
|
||||
name = "prometheus-postgres"
|
||||
namespace = kubernetes_namespace.prometheus_namespace.metadata.0.name
|
||||
}
|
||||
data = {
|
||||
username = var.prometheus_postgres_user
|
||||
password = var.prometheus_postgres_password
|
||||
}
|
||||
lifecycle {
|
||||
ignore_changes = [timeouts, wait_for_service_account_token]
|
||||
}
|
||||
}
|
||||
|
||||
# Install Prometheus Postgres exporter helm chart
|
||||
resource "helm_release" "prometheus-exporter-chart" {
|
||||
depends_on = [helm_release.prometheus-chart]
|
||||
repository = local.prometheus_exporter_helm_repo
|
||||
chart = local.prometheus_exporter_helm_chart
|
||||
name = local.prometheus_exporter_release_name
|
||||
namespace = local.prometheus_namespace
|
||||
values = [<<EOF
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: "cloud.google.com/gke-nodepool"
|
||||
operator: "In"
|
||||
values: ["${var.kubernetes_nodepool_misc}"]
|
||||
config:
|
||||
datasource:
|
||||
host: "${var.prometheus_postgres_host}"
|
||||
user: "${var.prometheus_postgres_user}"
|
||||
database: "${var.prometheus_postgres_dbname}"
|
||||
passwordSecret:
|
||||
name: "${kubernetes_secret.prometheus-postgres-password.metadata.0.name}"
|
||||
key: password
|
||||
autoDiscoverDatabases: true
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
EOF
|
||||
]
|
||||
}
|
||||
|
||||
resource "kubernetes_manifest" "coder_monitoring" {
|
||||
depends_on = [helm_release.prometheus-chart]
|
||||
manifest = {
|
||||
apiVersion = "monitoring.coreos.com/v1"
|
||||
kind = "PodMonitor"
|
||||
metadata = {
|
||||
namespace = kubernetes_namespace.coder_namespace.metadata.0.name
|
||||
name = "coder-monitoring"
|
||||
}
|
||||
spec = {
|
||||
selector = {
|
||||
matchLabels = {
|
||||
"app.kubernetes.io/name" : "coder"
|
||||
}
|
||||
}
|
||||
podMetricsEndpoints = [
|
||||
{
|
||||
port = "prometheus-http"
|
||||
interval = "30s"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,219 +0,0 @@
|
||||
variable "state" {
|
||||
description = "The state of the cluster. Valid values are 'started', and 'stopped'."
|
||||
validation {
|
||||
condition = contains(["started", "stopped"], var.state)
|
||||
error_message = "value must be one of 'started' or 'stopped'"
|
||||
}
|
||||
default = "started"
|
||||
}
|
||||
|
||||
variable "name" {
|
||||
description = "Adds a prefix to resources."
|
||||
}
|
||||
|
||||
variable "kubernetes_kubeconfig_path" {
|
||||
description = "Path to kubeconfig to use to provision resources."
|
||||
}
|
||||
|
||||
variable "kubernetes_nodepool_coder" {
|
||||
description = "Name of the nodepool on which to run Coder."
|
||||
}
|
||||
|
||||
variable "kubernetes_nodepool_workspaces" {
|
||||
description = "Name of the nodepool on which to run workspaces."
|
||||
}
|
||||
|
||||
variable "kubernetes_nodepool_misc" {
|
||||
description = "Name of the nodepool on which to run everything else."
|
||||
}
|
||||
|
||||
// These variables control the Coder deployment.
|
||||
variable "coder_access_url" {
|
||||
description = "Access URL for the Coder deployment."
|
||||
}
|
||||
variable "coder_replicas" {
|
||||
description = "Number of Coder replicas to provision."
|
||||
default = 1
|
||||
}
|
||||
|
||||
variable "coder_address" {
|
||||
description = "IP address to use for Coder service."
|
||||
}
|
||||
|
||||
variable "coder_db_url" {
|
||||
description = "URL of the database for Coder to use."
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
// Ensure that requests allow for at least two replicas to be scheduled
|
||||
// on a single node temporarily, otherwise deployments may fail due to
|
||||
// lack of resources.
|
||||
variable "coder_cpu_request" {
|
||||
description = "CPU request to allocate to Coder."
|
||||
default = "500m"
|
||||
}
|
||||
|
||||
variable "coder_mem_request" {
|
||||
description = "Memory request to allocate to Coder."
|
||||
default = "512Mi"
|
||||
}
|
||||
|
||||
variable "coder_cpu_limit" {
|
||||
description = "CPU limit to allocate to Coder."
|
||||
default = "1000m"
|
||||
}
|
||||
|
||||
variable "coder_mem_limit" {
|
||||
description = "Memory limit to allocate to Coder."
|
||||
default = "1024Mi"
|
||||
}
|
||||
|
||||
// Allow independently scaling provisionerd resources
|
||||
variable "provisionerd_cpu_request" {
|
||||
description = "CPU request to allocate to provisionerd."
|
||||
default = "100m"
|
||||
}
|
||||
|
||||
variable "provisionerd_mem_request" {
|
||||
description = "Memory request to allocate to provisionerd."
|
||||
default = "1Gi"
|
||||
}
|
||||
|
||||
variable "provisionerd_cpu_limit" {
|
||||
description = "CPU limit to allocate to provisionerd."
|
||||
default = "1000m"
|
||||
}
|
||||
|
||||
variable "provisionerd_mem_limit" {
|
||||
description = "Memory limit to allocate to provisionerd."
|
||||
default = "1Gi"
|
||||
}
|
||||
|
||||
variable "provisionerd_replicas" {
|
||||
description = "Number of Provisionerd replicas."
|
||||
default = 1
|
||||
}
|
||||
|
||||
variable "provisionerd_chart_version" {
|
||||
description = "Version of the Provisionerd Helm chart to install. Defaults to latest."
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "provisionerd_image_repo" {
|
||||
description = "Repository to use for Provisionerd image."
|
||||
default = "ghcr.io/coder/coder"
|
||||
}
|
||||
|
||||
variable "provisionerd_image_tag" {
|
||||
description = "Tag to use for Provisionerd image."
|
||||
default = "latest"
|
||||
}
|
||||
|
||||
variable "coder_chart_version" {
|
||||
description = "Version of the Coder Helm chart to install. Defaults to latest."
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "coder_image_repo" {
|
||||
description = "Repository to use for Coder image."
|
||||
default = "ghcr.io/coder/coder"
|
||||
}
|
||||
|
||||
variable "coder_image_tag" {
|
||||
description = "Tag to use for Coder image."
|
||||
default = "latest"
|
||||
}
|
||||
|
||||
variable "coder_experiments" {
|
||||
description = "Coder Experiments to enable."
|
||||
default = ""
|
||||
}
|
||||
|
||||
// These variables control the default workspace template.
|
||||
variable "workspace_image" {
|
||||
description = "Image and tag to use for workspaces."
|
||||
default = "docker.io/codercom/enterprise-minimal:ubuntu"
|
||||
}
|
||||
|
||||
variable "workspace_cpu_request" {
|
||||
description = "CPU request to allocate to workspaces."
|
||||
default = "100m"
|
||||
}
|
||||
|
||||
variable "workspace_cpu_limit" {
|
||||
description = "CPU limit to allocate to workspaces."
|
||||
default = "100m"
|
||||
}
|
||||
|
||||
variable "workspace_mem_request" {
|
||||
description = "Memory request to allocate to workspaces."
|
||||
default = "128Mi"
|
||||
}
|
||||
|
||||
variable "workspace_mem_limit" {
|
||||
description = "Memory limit to allocate to workspaces."
|
||||
default = "128Mi"
|
||||
}
|
||||
|
||||
// These variables control the Prometheus deployment.
|
||||
variable "prometheus_external_label_cluster" {
|
||||
description = "Value for the Prometheus external label named cluster."
|
||||
}
|
||||
|
||||
variable "prometheus_postgres_dbname" {
|
||||
description = "Database for Postgres to monitor."
|
||||
}
|
||||
|
||||
variable "prometheus_postgres_host" {
|
||||
description = "Database hostname for Prometheus."
|
||||
}
|
||||
|
||||
variable "prometheus_postgres_password" {
|
||||
description = "Postgres password for Prometheus."
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
variable "prometheus_postgres_user" {
|
||||
description = "Postgres username for Prometheus."
|
||||
}
|
||||
|
||||
variable "prometheus_remote_write_user" {
|
||||
description = "Username for Prometheus remote write."
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "prometheus_remote_write_password" {
|
||||
description = "Password for Prometheus remote write."
|
||||
default = ""
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
variable "prometheus_remote_write_url" {
|
||||
description = "URL for Prometheus remote write. Defaults to stats.dev.c8s.io."
|
||||
default = "https://stats.dev.c8s.io:9443/api/v1/write"
|
||||
}
|
||||
|
||||
variable "prometheus_remote_write_insecure_skip_verify" {
|
||||
description = "Skip TLS verification for Prometheus remote write."
|
||||
default = true
|
||||
}
|
||||
|
||||
variable "prometheus_remote_write_metrics_regex" {
|
||||
description = "Allowlist regex of metrics for Prometheus remote write."
|
||||
default = ".*"
|
||||
}
|
||||
|
||||
variable "prometheus_remote_write_send_interval" {
|
||||
description = "Prometheus remote write interval."
|
||||
default = "15s"
|
||||
}
|
||||
|
||||
variable "cloudflare_api_token" {
|
||||
description = "Cloudflare API token."
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
variable "cloudflare_email" {
|
||||
description = "Cloudflare email address."
|
||||
sensitive = true
|
||||
}
|
||||
@@ -1,9 +0,0 @@
|
||||
nodepool_machine_type_coder = "t2d-standard-8"
|
||||
nodepool_size_coder = 3
|
||||
nodepool_machine_type_workspaces = "t2d-standard-8"
|
||||
cloudsql_tier = "db-custom-2-7680"
|
||||
coder_cpu_request = "3000m"
|
||||
coder_mem_request = "12Gi"
|
||||
coder_cpu_limit = "6000m" # Leaving 2 CPUs for system workloads
|
||||
coder_mem_limit = "24Gi" # Leaving 8 GB for system workloads
|
||||
coder_replicas = 3
|
||||
@@ -1,7 +0,0 @@
|
||||
nodepool_machine_type_coder = "t2d-standard-8"
|
||||
nodepool_machine_type_workspaces = "t2d-standard-8"
|
||||
cloudsql_tier = "db-custom-1-3840"
|
||||
coder_cpu_request = "3000m"
|
||||
coder_mem_request = "12Gi"
|
||||
coder_cpu_limit = "6000m" # Leaving 2 CPUs for system workloads
|
||||
coder_mem_limit = "24Gi" # Leaving 8 GB for system workloads
|
||||
@@ -1,6 +0,0 @@
|
||||
nodepool_machine_type_coder = "t2d-standard-4"
|
||||
nodepool_machine_type_workspaces = "t2d-standard-4"
|
||||
coder_cpu_request = "1000m"
|
||||
coder_mem_request = "6Gi"
|
||||
coder_cpu_limit = "2000m" # Leaving 2 CPUs for system workloads
|
||||
coder_mem_limit = "12Gi" # Leaving 4GB for system workloads
|
||||
@@ -1,4 +0,0 @@
|
||||
name = "${SCALETEST_NAME}"
|
||||
project_id = "${SCALETEST_PROJECT}"
|
||||
prometheus_remote_write_user = "${SCALETEST_PROMETHEUS_REMOTE_WRITE_USER}"
|
||||
prometheus_remote_write_password = "${SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD}"
|
||||
Reference in New Issue
Block a user