fix(forks): stop copying connector-managed knowledge base documents (#6818)

* fix(forks): stop copying connector-managed knowledge base documents

A fork copies a KB's documents but never its connectors, so a
connector-sourced document arrives with `connector_id` nulled and its
`external_id` intact. The sync engine keys every existing/tombstone/
exclusion lookup off `connector_id`, so that copy is invisible to it -
never updated, reconciled, or purged - and `doc_connector_external_id_idx`
does not constrain it either, since its `connector_id` is NULL.

Attaching a connector in the child then re-ingests every page as a NEW
row on top of the snapshot. Each fork hop re-copies the previous hop's
orphans and adds one more generation, so a prod -> UAT -> staging chain
leaves three rows per page and a knowledge search returns the same page
three times, one of them serving content frozen at the fork date.

Exclude connector-managed documents from all four doors a document can
enter a fork through: the whole-KB content copy, the in-transaction
placeholder pre-creation, the sync-only copy into an already-mapped KB,
and the content fill (guarded for payloads planned by a pre-change
worker mid-rollout). The placeholder path matters as much as the copy
loop - filtering only the content phase would leave a permanently
archived row behind a persisted `knowledge_document` mapping. Skipped on
both sides, the reference clears like any other uncopied document's.

A document whose connector was deleted already has a null `connector_id`
(the FK is ON DELETE SET NULL) and is static in the source too, so it
still copies. One count(*) per copied KB logs what was left behind, since
a fully connector-synced KB now forks to zero documents.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(forks): keep the skipped-document count from failing a copied KB

The connector-managed count feeds a log line, but it sat inside the KB's
try block, so a transient failure on a COUNT(*) would roll back a copy
that had otherwise succeeded and clear every reference to it.

Move it into a helper that swallows its own error. Counting is not
copying: only the copy itself may fail a resource. Test proven red by
removing the catch - the mutation reports a knowledge-base failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(forks): clean up full-KB placeholders planned before the exclusion

The mapped-KB fill guarded a pre-change plan, but the full-KB path did
not: a placeholder planned by an old worker for a connector-managed
document is simply no longer returned by the page query, so nothing fills
it and it stays archived behind a live mapping that a remapped
document-selector still resolves to.

Report those child ids as failed documents so the shared cleanup clears
their references and drops the rows, and delete their persisted identity
so a later sync does not resolve to a row cleanup removes. Keyed on the
SOURCE being connector-managed, which can never become copyable, so it
cannot race a concurrent attempt mid-fill the way a "source is gone"
check could.

The mapping drop is now one helper shared with the mapped-KB catch.
Test proven red by removing the reconciliation block.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(forks): make the stale-plan probe best-effort

The probe ran inside the KB try, so a transient SELECT would reach the
catch, roll back a complete copy, delete the child base, and clear every
reference to it. Weighing it as "load-bearing, so fail closed" was wrong:
the probe runs on EVERY copied KB that has referenced documents, while
the state it repairs exists only inside a rollout window. Failing closed
traded a common-path outage against a rare-squared one.

It now swallows its own failure with a loud error log, leaving that
pre-existing state in place rather than destroying a good copy. Test
proven red by removing the catch - the mutation reports the KB failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vikhyath Mondreti
2026-08-18 14:23:19 -07:00
committed by GitHub
co-authored by Claude Opus 5
parent d1e3eeea9e
commit 3a03774e42
3 changed files with 356 additions and 24 deletions
@@ -174,7 +174,7 @@ How each resource behaves at **fork** time vs **sync** time. Use this when you a
| [Excluded workflows](#excluded-workflows) | Never | Never — not sent, not overwritten, not archived |
| Files | Optional copy (default on) | Map or copy |
| Tables | Optional copy (default on) | Map or copy |
| Knowledge bases (+ documents) | Optional copy; referenced docs come with the KB | Map or copy; documents follow the KB |
| Knowledge bases (+ documents) | Optional copy; uploaded documents come with the KB, [connector-synced ones do not](#connector-synced-documents-are-not-copied) | Map or copy; documents follow the KB |
| Custom tools | Optional copy (default on) | Map or copy |
| Skills | Optional copy (default on) | Map or copy |
| External MCP servers | Optional copy (config only; sign-in cleared) | Map or copy (config only; sign-in cleared) |
@@ -227,10 +227,27 @@ Only **deployed** workflows move. Deploy is the commit; sync is the force push/p
| | Behavior |
|---|----------|
| **Fork** | Optional copy (default on). Tag definitions come with the knowledge base. Documents that the forked workflows actually reference are included. Deselect → knowledge base / document fields clear. |
| **Fork** | Optional copy (default on). Tag definitions come with the knowledge base, along with every **uploaded** document in it. Deselect → knowledge base / document fields clear. |
| **Sync** | Map or copy the knowledge base. Documents are not mapped by themselves — they follow the knowledge base (copied with it, or re-picked when you map to an existing one). |
**Example:** An agent searches knowledge base “Product docs.” Fork with that knowledge base selected → the child gets the base, tags, and the documents the agent used. On sync, mapping to the child’s existing “Product docs” means re-picking which document the tool should use.
**Example:** An agent searches knowledge base “Product docs.” Fork with that knowledge base selected → the child gets the base, tags, and the uploaded documents. On sync, mapping to the child’s existing “Product docs” means re-picking which document the tool should use.
#### Connector-synced documents are not copied
Connectors themselves never cross a fork edge — the child gets no Confluence, Notion, Google Drive, or other sync running against it. Documents that a **connector** put in the knowledge base are therefore not copied either. Only documents you **uploaded** come across.
<Callout type="warn">
Fork a knowledge base whose content is entirely connector-synced and the child gets the base, its tags, and its settings — but **no documents**. Add the connector in the child to fill it.
</Callout>
This is deliberate. A copied connector document would arrive detached from any connector, so nothing would ever update, re-sync, or remove it — and when you added the connector in the child it would ingest every page again *alongside* the stale copy. Chain a few forks (prod → UAT → staging) and each hop leaves another dead generation behind, so one page comes back several times in a single knowledge search. Skipping them keeps the child’s own connector the single owner of that content.
| To get connector content into the child | Do this |
|---|---|
| Keep it live | Add the same connector in the child and let it sync. It re-ingests everything, so nothing is lost. |
| Keep a frozen snapshot | Download the documents from the source and upload them to the child’s knowledge base — uploaded documents copy on every later fork. |
A document whose connector was **deleted** in the source is no longer connector-managed, so it copies like any other uploaded document.
---