From 6acf32701e93df71ae749a8358f4fddb3b38c2d2 Mon Sep 17 00:00:00 2001 From: Nick Vigilante Date: Tue, 23 Jun 2026 12:55:15 -0400 Subject: [PATCH] fix: preserve Vale severity in CI annotations and add three-severity demo (#26587) Closes DOCS-426. Follow-up to [#26586](https://github.com/coder/coder/pull/26586) (DOCS-425, strip), which merged first. ## Problem The Vale problem matcher at `.github/vale-problem-matcher.json` hard-codes `"severity": "warning"`. Every Vale finding renders as a GitHub `warning` annotation, regardless of Vale's actual severity. Nick observed this on PR [#25501](https://github.com/coder/coder/pull/25501): error-level findings from `Coder.BrandNames` appear as warnings. This collapsed the doctrine's three-severity ladder (`error` / `warning` / `suggestion`) into a single advisory channel for the reader of a PR diff. This PR restores the ladder visually so contributors and reviewers see each rule's intended severity. ## Root cause GitHub Actions problem matchers expect either a regex capture group for severity or a hard-coded severity. Vale's `--output=line` format produces `path:line:col:rule:message` with severity stripped, so the matcher had no severity to capture and fell back on the hard-coded value. ## Fix ### Commit 1: severity rendering Switch the Vale prose lint step to `vale --output=JSON` and pipe through `jq` to emit GitHub workflow commands directly. Drop the problem matcher file. | Vale severity | GitHub workflow command | |---|---| | `suggestion` | `::notice::` | | `warning` | `::warning::` | | `error` | `::error::` | Message bodies are URL-encoded for `%`, `\r`, and `\n` per the GitHub Actions workflow command spec. The Vale step stays advisory (`continue-on-error: true`, `vale --no-exit`); rendering becomes correct but the step never fails the job. ### Commit 2: three-severity demo Three throwaway `Coder.Demo*` rules at `level: suggestion`, `level: warning`, and `level: error`, plus a `docs/.style/_vale-annotation-demo.md` file that triggers each rule exactly once. Together with the rendering fix above, this PR's CI surfaces three GitHub annotations in three distinct severities (notice, warning, error). Use the Files Changed view to inspect rendering. The demo files live permanently in `docs/.style/`, which is excluded from coder.com. They re-trigger annotations only on PRs that touch the demo file itself, so they don't pollute CI on day-to-day PRs. ## Sample output image image ## Out of scope Blocking merge on `error`-level findings is the natural next step but is sequenced as the **final** step of the prose-style rollout. It was prototyped in this PR (commit 3, since backed out) and verified end-to-end against the demo doc. The work moved to [DOCS-433](https://linear.app/codercom/issue/DOCS-433/block-merge-on-vale-error-level-findings-final-step-of-prose-style) so the corpus of enabled rules is broad enough by the time the gate lands that it catches real violations rather than novelty failures from a single rule. ## Expected CI state on this PR `lint-docs` passes. The three demo annotations render at lines 17 / 19 / 21 of `docs/.style/_vale-annotation-demo.md` as `::notice::`, `::warning::`, and `::error::` respectively. The `::error::` annotation does not fail the job because the Vale step is still advisory under this PR. Local verification of the rendering pipeline: ``` $ printf '%s\n' 'docs/.style/_vale-annotation-demo.md' \ | xargs -d '\n' vale --no-exit --output=JSON \ | jq -r '...' ::notice file=docs/.style/_vale-annotation-demo.md,line=17,col=3,title=Coder.DemoSuggestion::[Demo] Suggestion-level Vale annotation. ::warning file=docs/.style/_vale-annotation-demo.md,line=19,col=3,title=Coder.DemoWarning::[Demo] Warning-level Vale annotation. ::error file=docs/.style/_vale-annotation-demo.md,line=21,col=3,title=Coder.DemoError::[Demo] Error-level Vale annotation. ```
Decision log - **Workflow commands vs custom Vale template + updated matcher**: chose workflow commands because the transform is a 10-line jq pipeline with no extra files to maintain, and it bypasses GitHub Actions problem-matcher limitations entirely. The custom-template option would have kept the matcher infrastructure but required an additional Go template file under `.github/`. - **Throwaway demo rules vs reusing existing rules**: chose throwaway because we wanted each severity to fire deterministically from a single unambiguous marker. Reusing existing rules would couple the demo to corpus content and obscure the signal. - **Demo persists vs drops before merge**: persists. The merge-gate constraint that originally forced the demo to drop is gone (deferred to DOCS-433). The four demo files live in `docs/.style/`, excluded from coder.com, and only annotate PRs that touch them. They double as a permanent canary so a future regression in severity rendering surfaces immediately on whichever PR introduces it, and as the verification artifact DOCS-433 uses when re-installing the merge gate. - **`docs/.style/_vale-annotation-demo.md` filename**: underscore prefix follows Coder convention for files that exist outside the normal docs taxonomy. Not surfaced on coder.com/docs because `docs/.style/` is excluded from the manifest, deploy workflow, and docs preview. - **Merge-block deferred to DOCS-433**: the rendering fix and the merge gate are independent changes. Shipping the rendering first lets contributors see the three-severity ladder while the rule catalogue is still small and the false-positive policy hasn't been stress-tested yet. The gate lands as the final step of the rollout, after the catalogue is broad enough that the gate covers real prose-style policy rather than one rule's enforcement.
--- *Filed via [Coder Agents](https://coder.com/docs/ai-coder/agents) on Nick's behalf.* --- .github/vale-problem-matcher.json | 18 --------------- .github/workflows/ci.yaml | 22 +++++++++++++++--- docs/.style/_vale-annotation-demo.md | 25 +++++++++++++++++++++ docs/.style/styles/Coder/DemoError.yml | 11 +++++++++ docs/.style/styles/Coder/DemoSuggestion.yml | 11 +++++++++ docs/.style/styles/Coder/DemoWarning.yml | 11 +++++++++ 6 files changed, 77 insertions(+), 21 deletions(-) delete mode 100644 .github/vale-problem-matcher.json create mode 100644 docs/.style/_vale-annotation-demo.md create mode 100644 docs/.style/styles/Coder/DemoError.yml create mode 100644 docs/.style/styles/Coder/DemoSuggestion.yml create mode 100644 docs/.style/styles/Coder/DemoWarning.yml diff --git a/.github/vale-problem-matcher.json b/.github/vale-problem-matcher.json deleted file mode 100644 index bedf0e0b3e..0000000000 --- a/.github/vale-problem-matcher.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "problemMatcher": [ - { - "owner": "vale", - "severity": "warning", - "pattern": [ - { - "regexp": "^(.+):(\\d+):(\\d+):([^:]+):(.+)$", - "file": 1, - "line": 2, - "column": 3, - "code": 4, - "message": 5 - } - ] - } - ] -} diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 321c647bdf..39ac276db1 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -234,9 +234,25 @@ jobs: echo "No changed Markdown files under docs/ on disk; skipping Vale." exit 0 fi - echo "::add-matcher::.github/vale-problem-matcher.json" - printf '%s\n' "$files" | xargs -d '\n' mise exec "aqua:errata-ai/vale" -- vale --no-exit --output=line - echo "::remove-matcher owner=vale::" + # Vale's --output=line strips per-finding severity, so the + # previous problem-matcher approach collapsed every finding to + # a single hard-coded severity. Use --output=JSON instead and + # emit GitHub workflow commands directly so error/warning/ + # suggestion render with their actual Vale severities. URL- + # encode message bodies for `%`, `\r`, and `\n` per the + # GitHub Actions workflow command spec. See DOCS-426. + printf '%s\n' "$files" \ + | xargs -d '\n' mise exec "aqua:errata-ai/vale" -- vale --no-exit --output=JSON \ + | jq -r ' + to_entries[] + | .key as $file + | .value[] + | (if .Severity == "suggestion" then "notice" + elif .Severity == "warning" then "warning" + else "error" end) as $level + | (.Message | gsub("%"; "%25") | gsub("\r"; "%0D") | gsub("\n"; "%0A")) as $msg + | "::\($level) file=\($file),line=\(.Line),col=\(.Span[0]),title=\(.Check)::\($msg)" + ' - name: Save Vale styles # Only the default branch is trusted to write the cache, so PR diff --git a/docs/.style/_vale-annotation-demo.md b/docs/.style/_vale-annotation-demo.md new file mode 100644 index 0000000000..facbf2487e --- /dev/null +++ b/docs/.style/_vale-annotation-demo.md @@ -0,0 +1,25 @@ +# Vale annotation rendering demo + +> [!NOTE] +> This page exists only to verify how GitHub renders Vale annotations +> at each severity level. +> The three `Coder.Demo*` rules under +> [`docs/.style/styles/Coder/`](styles/Coder/) fire on the marker +> strings below. +> This page and its rules disappear in a follow-up commit on the same +> PR once the rendering check completes; see DOCS-426. + +## Markers + +Each marker fires exactly one Vale annotation when this page lints: + +- Suggestion (rendered as GitHub `notice`): + vale-demo-suggestion-marker. +- Warning (rendered as GitHub `warning`): + vale-demo-warning-marker. +- Error (rendered as GitHub `error`): + vale-demo-error-marker. + +The rules use Vale's `existence` extension type and target a single +literal token each, so each marker produces a single annotation at its +exact location. diff --git a/docs/.style/styles/Coder/DemoError.yml b/docs/.style/styles/Coder/DemoError.yml new file mode 100644 index 0000000000..366736701a --- /dev/null +++ b/docs/.style/styles/Coder/DemoError.yml @@ -0,0 +1,11 @@ +# DemoError - canary rule used to verify how GitHub renders error-level +# Vale annotations in PR diffs. Fires on the literal phrase +# `vale-demo-error-marker`, which appears only in +# docs/.style/_vale-annotation-demo.md. Demo rules drop in a follow-up +# commit before the parent PR merges; see DOCS-426. +extends: existence +message: "[Demo] Error-level Vale annotation." +link: https://github.com/coder/coder/blob/main/docs/.style/_vale-annotation-demo.md +level: error +tokens: + - vale-demo-error-marker diff --git a/docs/.style/styles/Coder/DemoSuggestion.yml b/docs/.style/styles/Coder/DemoSuggestion.yml new file mode 100644 index 0000000000..dd6dda6a34 --- /dev/null +++ b/docs/.style/styles/Coder/DemoSuggestion.yml @@ -0,0 +1,11 @@ +# DemoSuggestion - canary rule used to verify how GitHub renders +# suggestion-level Vale annotations in PR diffs. Fires on the literal +# phrase `vale-demo-suggestion-marker`, which appears only in +# docs/.style/_vale-annotation-demo.md. Demo rules drop in a follow-up +# commit before the parent PR merges; see DOCS-426. +extends: existence +message: "[Demo] Suggestion-level Vale annotation." +link: https://github.com/coder/coder/blob/main/docs/.style/_vale-annotation-demo.md +level: suggestion +tokens: + - vale-demo-suggestion-marker diff --git a/docs/.style/styles/Coder/DemoWarning.yml b/docs/.style/styles/Coder/DemoWarning.yml new file mode 100644 index 0000000000..bd03e1c8db --- /dev/null +++ b/docs/.style/styles/Coder/DemoWarning.yml @@ -0,0 +1,11 @@ +# DemoWarning - canary rule used to verify how GitHub renders +# warning-level Vale annotations in PR diffs. Fires on the literal phrase +# `vale-demo-warning-marker`, which appears only in +# docs/.style/_vale-annotation-demo.md. Demo rules drop in a follow-up +# commit before the parent PR merges; see DOCS-426. +extends: existence +message: "[Demo] Warning-level Vale annotation." +link: https://github.com/coder/coder/blob/main/docs/.style/_vale-annotation-demo.md +level: warning +tokens: + - vale-demo-warning-marker