feat: make CLI/API doc generators emit front-matter metadata (Phase 2) (#27246)

## Summary Phase 2 of the H1 → front-matter migration (`DOCS-483`; parent `DOCS-477`). Makes the two reference-doc generators emit per-page metadata as YAML front matter instead of a leading `# H1`, so generated pages are self-describing and `make gen` stops reverting migrated pages (Phase 3). Phase 1 (`DOCS-482`) made the coder.com renderers prefer a front-matter `title` (manifest fallback). > [!NOTE] > Rebased onto `main` and fully regenerated, and updated across two rounds of Coder Agents Review — see **Review follow-ups** below. ## Changes - **`scripts/clidocgen/command.tpl` + `gen.go` + `main.go`** — front matter now carries `title` (from `fullName`) and `description` (from the command's `Short`), and the leading `# H1` is dropped. The CLI index page's front matter is taken from the manifest `Command Line` route (title/description/icon_path). - **`scripts/apidocgen/postprocess/main.go`** — reads the manifest and, at write time, injects front matter carrying each section's `title` plus any curated `description`, `state`, and `icon_path`. The API index page's front matter is taken from the manifest `REST API` route. - **`scripts/docgenenv`** (new shared code) — one `YAMLScalar` front-matter escaper, one `Route`/`Manifest` schema + `LoadManifest`/`FindRoute`, and one `FrontMatter(Route)` emitter, all imported by both generators (no duplicated helpers, types, or emitters). - Regenerated all **166 CLI + 31 API** reference pages. ### Metadata → front matter, and what stays in the manifest Every *per-page* manifest field is mirrored into the page's front matter: `title`, `description`, `state`, `icon_path`. The **structural** fields stay in `manifest.json`: - `children` — the nav tree (explicitly out of scope). - `path` — the manifest's pointer to the file; a page carrying its own path is redundant/error-prone, so it's treated like `children`. The fields are **duplicated** into front matter and **`manifest.json` is left unchanged**, so this is a **no-op for rendering today** (coder.com strips front matter for `llms`, and Algolia + the renderer read only `title`). Removing the fields from the manifest is the natural follow-up, gated on the renderer reading them from front matter first. ### Why the API side changes the postprocessor, not the `.dot` templates The issue text suggested editing `scripts/apidocgen/markdown-template/*`. I deliberately did **not**, because the postprocessor derives each page's **filename, section title, and manifest route** from the leading `# {name}` line (`extractSectionName`). Emitting front matter from the template would break that extraction. Instead the widdershins templates still emit `# {name}`, the postprocessor reads it (and now verifies it), and then swaps the heading for a front-matter block as each section is written. ## Review follow-ups (Coder Agents Review) ### Round 1 — addressed in `e53d5e03` (all threads resolved) - **CRF-1 / CRF-4** — de-duplicated the escaper and the `route`/`manifest` schema + traversal into `scripts/docgenenv` (shared by both generators). - **CRF-2** — `YAMLScalar` now quotes YAML-reserved scalars (`true/false/null/…`, numbers); no current value is affected. - **CRF-3** — added unit tests: a `YAMLScalar` round-trip, `FindRoute`, and `prependFrontMatter`. - **CRF-5** — the CLI and API **index** pages now mirror their manifest route's title/description/icon_path instead of a hardcoded `coder`/`API`, fixing a rendered-heading regression (`REST API`/`Command Line` were being overwritten). - **CRF-6** — dropped the dead `#login` anchor in `docs/support/support-bundle.md` (the migrated `login.md` no longer mints that heading anchor). - **CRF-7 / CRF-8 / CRF-11** — renamed to `prependFrontMatter`, switched to `bytes.Cut`, and it now strips the first line only when it is the `# {name}` heading (`extractSectionName` errors otherwise). - **CRF-9** — removed the orphan `docs/reference/api/chat.md` (not in the manifest, not linked; the real page is `chats.md`). - **CRF-10** — the metadata read and the manifest rewrite now share one `FindRoute` traversal. - **CRF-13** — moot under squash-merge; this branch is a single scopeless commit. - **CRF-15** — the pre-existing `sort.Slice`/`slices.IsSorted` comparator is left as-is per the review (out of scope; safe today because section names are unique). ### Round 2 — addressed in `ee796e7107` (all threads resolved) - **CRF-16** (P1) — removed three em-dashes from new doc comments (the only `make lint` failure on the prior head); the emdash gate is green. - **CRF-17 / CRF-18** — unified front-matter emission into one shared `docgenenv.FrontMatter(Route)`, used by the API postprocessor directly and by `command.tpl` via a `frontMatter` template func. This retires the hand-written template YAML and the `indexTitle`/`indexDescription`/`indexIconPath` closures, so a new front-matter field is wired in one place, and it gives the CLI index the `state` arm it previously lacked. Verified byte-identical: a full CLI + API regen produces zero page changes. - **CRF-19** — CLI child sort switched to `slices.SortFunc` + `cmp.Compare` (typed comparator). - **CRF-20** — reworded the `prependFrontMatter` comment: `extractSectionName`'s fail-fast is the load-bearing guard; the prefix check is a defensive backstop. - **CRF-21** — added `icon_path`/`state` coverage in `docgenenv`'s `TestFrontMatter/AllFields` (the branch the index page relies on, previously at 0%). - **CRF-22** — `YAMLScalar` no longer emits a trailing-space value as a bare scalar (YAML strips it on read, so it would not round-trip); added test coverage. - **CRF-24** — the shared emitter removed the duplicated `cliIndexRoute` doc comment; the rationale now lives in one place. - **CRF-23** (Phase 3, out of scope here) — noted: the API generator wipes and regenerates `reference/api/` from the manifest, so removing curated metadata from the manifest in Phase 3 needs another source first (a generator that preserves existing front matter, or metadata carried alongside the swagger annotations). - **Process (Mafu-san)** — the verification set below now leads with `make lint`, the mandatory CI gate that the earlier list omitted. ## Cross-repo dependency **Resolved — this PR no longer has a hard merge-ordering gate** (CRF-14 was right; the earlier "must merge after #968" note was stale). The coder.com surfaces that would otherwise leak raw front matter from `coder/coder` `main` are already front-matter-aware on merged PRs: - **coder.com#964** (`DOCS-554`, llms-full.txt corpus + Algolia) — **merged**. - **coder.com#974** (`DOCS-574`, the `.md` proxy twin + `llms.txt` index titles) — **merged**. coder.com#968 (`DOCS-577`) was re-scoped to only the renderer route-metadata generalization; it's a no-op on today's corpus and its own description confirms the "deploy before the generators" constraint no longer applies (that was driven by the llms corpus, now in #964). Worth a final confirmation that #964/#974 are **deployed** before merge, but there's no branch/PR ordering blocker left. ## Verification & evidence AI was the primary author of this PR (see disclosure below); per the [AI Contribution Guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING) here is manual verification. - `make lint` (golangci-lint + the emdash gate) passes; `go build` / `go vet` / `go test` are clean for the generators + `scripts/docgenenv`; `pnpm check-docs` passes. - `swagger.json`, `docs.go`, and `manifest.json` are **unchanged** — metadata is duplicated into front matter; command/section names and routes did not move. - The diff is purely additive front matter (`title`/`description`/`state`/`icon_path`) + the leading H1 removal; no body reflow. A full CLI + API regen produces **zero** page changes beyond the two index pages. <details> <summary>Terminal evidence</summary> CLI `description` from the command's `Short` (`YAMLScalar` quotes when needed, e.g. a `Short` with a colon): ```md --- title: server description: Start a Coder server --- ``` API pages inherit curated manifest metadata (only Agents/Chats have any today): ```md --- title: Chats description: "REST endpoints for Coder Agents Chats API (programmatic agent sessions)." state: - early access --- ``` Diff scope + "no body changes" proof (uses an explicit `base..HEAD` range, so it actually tests the claim): ``` $ git diff --shortstat origin/main 210 files changed, 1447 insertions(+), 344 deletions(-) # = 166 CLI + 31 API reference pages + generators + scripts/docgenenv # swagger.json / docs.go / manifest.json: NOT modified # Every removed line under docs/reference is a leading "# H1"; nothing else: $ git diff origin/main..HEAD -- docs/reference/ | grep '^-' | grep -v '^---' | grep -v '^-# ' (empty) $ pnpm check-docs Summary: 0 error(s) ``` </details> Linear: DOCS-483 > This PR was created with AI assistance (Coder Agents).
2026-09-24 15:04:27 +08:00 · 2026-08-11 15:22:55 +00:00
parent 5febde0ebb
commit 0a7bb80a1e
218 changed files with 1941 additions and 352 deletions
@@ -0,0 +1,65 @@
+package docgenenv
+
+import "strings"
+
+// generatedMarker is the YAML comment placed as the first line inside every
+// generated reference page's front matter. It follows the canonical
+// "Code generated ... DO NOT EDIT." convention so the signal is recognizable at
+// the top of the file. Because it lives inside the front matter fence, every
+// coder.com surface (rendered HTML, the .md proxy twin, and the llms corpus)
+// strips it with the rest of the block, so it never reaches readers.
+const generatedMarker = "# Code generated by make gen. DO NOT EDIT."
+
+// GeneratedContentBanner is the HTML comment placed in the Markdown body of
+// every generated reference page, directly below the front matter. The in-fence
+// generatedMarker flags the page metadata as generated; this body banner flags
+// the whole page as generated, so an editor who reads past the front matter is
+// still warned. Both documentation generators reference this one constant so
+// the wording cannot drift between them.
+const GeneratedContentBanner = "<!-- DO NOT EDIT | GENERATED CONTENT -->"
+
+// FrontMatter renders r's metadata (title plus any description, icon_path, and
+// state) as a YAML front matter block. The block opens with generatedMarker,
+// carries the fences, and ends with the blank line that separates it from the
+// Markdown body. Optional fields are omitted when empty.
+//
+// Both documentation generators emit their front matter through this single
+// function so a new per-page metadata field is wired in one place instead of
+// drifting between the CLI template and the API generator. Structural manifest
+// fields (path, children) are intentionally not mirrored here.
+func FrontMatter(r Route) string {
+	// strings.Builder and bytes.Buffer expose only error-returning Write
+	// methods, which the revive unhandled-error linter flags, so assemble the
+	// block as a []string and join it.
+	lines := []string{
+		"---",
+		generatedMarker,
+		"title: " + YAMLScalar(r.Title),
+	}
+	if r.Description != "" {
+		lines = append(lines, "description: "+YAMLScalar(r.Description))
+	}
+	if r.IconPath != "" {
+		lines = append(lines, "icon_path: "+YAMLScalar(r.IconPath))
+	}
+	if len(r.State) > 0 {
+		lines = append(lines, "state:")
+		for _, s := range r.State {
+			lines = append(lines, "  - "+YAMLScalar(s))
+		}
+	}
+	// The trailing empty strings produce the closing fence followed by the
+	// blank line that must separate front matter from the body.
+	lines = append(lines, "---", "", "")
+	return strings.Join(lines, "\n")
+}
+
+// GeneratedHeader returns the full generated-page preamble: the front matter
+// block (from FrontMatter) followed by GeneratedContentBanner and the blank
+// line before the Markdown body. The API generator prepends this to every page
+// so CLI and API reference pages share the same two markers; the CLI template
+// composes the equivalent preamble inline from the frontMatter and
+// generatedContentBanner template helpers.
+func GeneratedHeader(r Route) string {
+	return FrontMatter(r) + GeneratedContentBanner + "\n\n"
+}
@@ -0,0 +1,58 @@
+package docgenenv_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+
+	"github.com/coder/coder/v2/scripts/docgenenv"
+)
+
+func TestFrontMatter(t *testing.T) {
+	t.Parallel()
+
+	t.Run("TitleOnly", func(t *testing.T) {
+		t.Parallel()
+
+		got := docgenenv.FrontMatter(docgenenv.Route{Title: "General"})
+		require.Equal(t, "---\n# Code generated by make gen. DO NOT EDIT.\ntitle: General\n---\n\n", got)
+	})
+
+	// AllFields exercises every optional branch, including icon_path (the
+	// curated field the index pages rely on, previously uncovered) and the
+	// state sequence.
+	t.Run("AllFields", func(t *testing.T) {
+		t.Parallel()
+
+		got := docgenenv.FrontMatter(docgenenv.Route{
+			Title:       "REST API",
+			Description: "Learn how to use Coderd API",
+			IconPath:    "./images/icons/api.svg",
+			State:       []string{"early access"},
+		})
+		want := "---\n" +
+			"# Code generated by make gen. DO NOT EDIT.\n" +
+			"title: REST API\n" +
+			"description: Learn how to use Coderd API\n" +
+			`icon_path: "./images/icons/api.svg"` + "\n" +
+			"state:\n" +
+			"  - early access\n" +
+			"---\n\n"
+		require.Equal(t, want, got)
+	})
+}
+
+// TestGeneratedHeader pins the full generated-page preamble: the front matter
+// block (with its in-fence marker) followed by the body banner and the blank
+// line before the body.
+func TestGeneratedHeader(t *testing.T) {
+	t.Parallel()
+
+	got := docgenenv.GeneratedHeader(docgenenv.Route{Title: "General"})
+	want := "---\n" +
+		"# Code generated by make gen. DO NOT EDIT.\n" +
+		"title: General\n" +
+		"---\n\n" +
+		"<!-- DO NOT EDIT | GENERATED CONTENT -->\n\n"
+	require.Equal(t, want, got)
+}
@@ -0,0 +1,72 @@
+package docgenenv
+
+import (
+	"encoding/json"
+	"os"
+
+	"golang.org/x/xerrors"
+)
+
+// Route is an individual page object in the docs manifest.json. Per-page
+// metadata (title, description, icon_path, state) is mirrored into page front
+// matter by the doc generators; the structural fields (path, children) stay in
+// the manifest.
+type Route struct {
+	Title       string   `json:"title,omitempty"`
+	Description string   `json:"description,omitempty"`
+	Path        string   `json:"path,omitempty"`
+	IconPath    string   `json:"icon_path,omitempty"`
+	State       []string `json:"state,omitempty"`
+	Children    []Route  `json:"children,omitempty"`
+}
+
+// Manifest describes the entire documentation index (docs/manifest.json).
+type Manifest struct {
+	Versions []string `json:"versions,omitempty"`
+	Routes   []Route  `json:"routes,omitempty"`
+}
+
+// LoadManifest reads and unmarshals the manifest.json at path. Its errors wrap
+// the path and cause, so callers should return the error as-is rather than
+// wrapping it again.
+func LoadManifest(path string) (*Manifest, error) {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return nil, xerrors.Errorf("read manifest %q: %w", path, err)
+	}
+	var m Manifest
+	if err := json.Unmarshal(b, &m); err != nil {
+		return nil, xerrors.Errorf("unmarshal manifest %q: %w", path, err)
+	}
+	return &m, nil
+}
+
+// FindRoute walks the manifest, following titles as a breadcrumb from the
+// top-level routes, and returns a pointer to the matching route (or nil if any
+// title in the path has no match). The returned pointer aliases the manifest,
+// so mutating it (for example, replacing Children) updates the manifest in place.
+//
+// Both documentation generators resolve their target route through this single
+// traversal so the route they read metadata from and the route they rewrite
+// cannot drift apart.
+func (m *Manifest) FindRoute(titles ...string) *Route {
+	if m == nil || len(titles) == 0 {
+		return nil
+	}
+	routes := m.Routes
+	var match *Route
+	for _, title := range titles {
+		match = nil
+		for i := range routes {
+			if routes[i].Title == title {
+				match = &routes[i]
+				break
+			}
+		}
+		if match == nil {
+			return nil
+		}
+		routes = match.Children
+	}
+	return match
+}
@@ -0,0 +1,39 @@
+package docgenenv_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+
+	"github.com/coder/coder/v2/scripts/docgenenv"
+)
+
+func TestManifestFindRoute(t *testing.T) {
+	t.Parallel()
+
+	m := &docgenenv.Manifest{
+		Routes: []docgenenv.Route{
+			{Title: "Reference", Children: []docgenenv.Route{
+				{Title: "Command Line", Description: "Learn how to use Coder CLI"},
+				{Title: "REST API", Description: "Learn how to use Coderd API", Children: []docgenenv.Route{
+					{Title: "General"},
+				}},
+			}},
+		},
+	}
+
+	rest := m.FindRoute("Reference", "REST API")
+	require.NotNil(t, rest)
+	require.Equal(t, "Learn how to use Coderd API", rest.Description)
+
+	// The returned pointer aliases the manifest, so mutations persist.
+	rest.Children = nil
+	require.Nil(t, m.FindRoute("Reference", "REST API").Children)
+
+	require.NotNil(t, m.FindRoute("Reference", "Command Line"))
+
+	// Misses return nil rather than panicking.
+	require.Nil(t, m.FindRoute("Reference", "Nope"))
+	require.Nil(t, m.FindRoute())
+	require.Nil(t, m.FindRoute("REST API")) // Not a top-level route.
+}
@@ -0,0 +1,52 @@
+package docgenenv
+
+import (
+	"encoding/json"
+	"regexp"
+	"strings"
+)
+
+// safeScalarRegex matches values that can be emitted as a bare (unquoted) YAML
+// scalar: they start with an alphanumeric and contain only alphanumerics,
+// spaces, and a small set of punctuation YAML never treats specially. A
+// trailing space is disallowed because YAML strips it on read, so a bare scalar
+// ending in a space would not round-trip back to the original string.
+var safeScalarRegex = regexp.MustCompile(`^[A-Za-z0-9]([A-Za-z0-9 ._/-]*[A-Za-z0-9._/-])?$`)
+
+// numberScalarRegex matches values YAML would resolve to an integer or float.
+var numberScalarRegex = regexp.MustCompile(`^[+-]?(\d+\.?\d*|\.\d+)([eE][+-]?\d+)?$`)
+
+// YAMLScalar renders s as a YAML scalar suitable for a front matter value.
+// Simple values are emitted verbatim. Anything YAML could misparse, such as
+// special characters or a bare word/number YAML would otherwise resolve to a
+// bool, null, or number, is JSON-encoded, which is valid YAML that quotes and
+// escapes the value so it round-trips back to the original string.
+//
+// Both the CLI and API documentation generators share this helper so their
+// front-matter escaping cannot silently diverge.
+func YAMLScalar(s string) string {
+	if isBareScalar(s) {
+		return s
+	}
+	b, err := json.Marshal(s)
+	if err != nil {
+		return `""`
+	}
+	return string(b)
+}
+
+// isBareScalar reports whether s can be emitted unquoted without YAML
+// reinterpreting it as a non-string type.
+func isBareScalar(s string) bool {
+	if !safeScalarRegex.MatchString(s) {
+		return false
+	}
+	// Even when every character is safe, quote values YAML would resolve to a
+	// bool or null (e.g. a title of "true" or "null") so they stay strings.
+	switch strings.ToLower(s) {
+	case "true", "false", "yes", "no", "on", "off", "y", "n", "null", "none", "~":
+		return false
+	}
+	// Likewise quote anything that parses as a number (e.g. "123", "1.5").
+	return !numberScalarRegex.MatchString(s)
+}
@@ -0,0 +1,99 @@
+package docgenenv_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	"gopkg.in/yaml.v3"
+
+	"github.com/coder/coder/v2/scripts/docgenenv"
+)
+
+// TestYAMLScalarRoundTrip asserts the emitted scalar parses back to the exact
+// input string. Unmarshaling into a Go string returns the scalar's text
+// regardless of the type YAML would resolve it to, so this catches the values
+// that break parsing or resolve away (quotes, colons, newlines, trailing space,
+// null and ~) but not the reserved-word or number quoting intent. That intent
+// is pinned directly in TestYAMLScalarBareWhenSafe.
+func TestYAMLScalarRoundTrip(t *testing.T) {
+	t.Parallel()
+
+	cases := []string{
+		// Values seen in practice today.
+		"server",
+		"templates create",
+		"Start a Coder server",
+		"early access",
+		"DEPRECATED: Create a template from the current directory or as specified by flag",
+		"./images/icons/api.svg",
+		// Special characters that must be escaped.
+		`has "quotes" and: a colon`,
+		"trailing backtick `code`",
+		"line one\nline two",
+		// Trailing space: YAML strips it on read, so a bare scalar would not
+		// round-trip.
+		"trailing space ",
+		// Values YAML would otherwise resolve to a non-string type.
+		"true",
+		"False",
+		"NULL",
+		"no",
+		"on",
+		"123",
+		"1.5",
+		"-42",
+		"~",
+	}
+	for _, in := range cases {
+		t.Run(in, func(t *testing.T) {
+			t.Parallel()
+
+			doc := "value: " + docgenenv.YAMLScalar(in) + "\n"
+			var got struct {
+				Value string `yaml:"value"`
+			}
+			require.NoErrorf(t, yaml.Unmarshal([]byte(doc), &got), "emitted YAML must parse: %q", doc)
+			require.Equalf(t, in, got.Value, "scalar must round-trip as a string, doc=%q", doc)
+		})
+	}
+}
+
+// TestYAMLScalarBareWhenSafe pins the bare-vs-quoted decision directly against
+// the emitted text, which the round-trip test cannot do for the reserved-word
+// and number class (unmarshaling into a Go string hands back the text whether
+// or not YAMLScalar quoted it). Common values stay bare so regenerated pages
+// don't churn; anything a YAML reader would resolve to a bool, null, or number
+// is quoted. Trimming the isBareScalar switch or the number check fails here.
+func TestYAMLScalarBareWhenSafe(t *testing.T) {
+	t.Parallel()
+
+	// Safe, stringy values stay bare.
+	for _, in := range []string{
+		"server",
+		"templates create",
+		"Start a Coder server",
+		"early access",
+	} {
+		require.Equalf(t, in, docgenenv.YAMLScalar(in), "safe value %q must stay bare", in)
+	}
+
+	// Values that look bare but a YAML reader would resolve to a non-string
+	// type must be quoted. json.Marshal wraps each of these verbatim, so the
+	// expected form is simply the input in double quotes.
+	for _, in := range []string{
+		// YAML 1.1 bool aliases, in the casings isBareScalar folds.
+		"true", "false", "yes", "no", "on", "off", "y", "n", "none",
+		"True", "False", "Yes", "No", "On", "Off", "None", "NULL",
+		// Null forms.
+		"null", "~",
+		// Numbers (integer, float, signed, scientific).
+		"123", "1.5", "-42", "+7", "1e3",
+	} {
+		require.Equalf(t, `"`+in+`"`, docgenenv.YAMLScalar(in), "ambiguous value %q must be quoted", in)
+	}
+
+	// Values with characters YAML would misparse are quoted too.
+	require.Equal(t, `"./images/icons/api.svg"`, docgenenv.YAMLScalar("./images/icons/api.svg"))
+	// A trailing space forces quoting so the value round-trips.
+	require.Equal(t, `"foo "`, docgenenv.YAMLScalar("foo "))
+}