The search instructions advertised firecrawl_research_* as searching public
code repositories, which is the same corpus firecrawl_developer_search now
covers, leaving an agent no rule for choosing between them. Scope the research
clause to its GitHub slice and route general programming questions to the
developer tool, and name skills: "only" as the capability the developer
category filter cannot reach.
Cover skills forwarding in the search-profile test: it was the only parameter
on the tool with no assertion in either test file.
Co-authored-by: Cursor <cursoragent@cursor.com>
Point keyless recovery at an API key instead of OAuth, scope the keyless profile to the tools that session actually has, and fix developer-search copy to match the live data.web shape.
The non-cloud session builder labeled header-sourced API keys as environment credentials. Preserve their resolved API-key source so Core 401 responses receive CREDENTIAL_INVALID recovery, while true environment credentials retain their existing self-hosted error behavior.
Replace the fixed stderr delay with a failing issuer that counts requests. The OAuth request proves the fixture records a transport failure, and the API-key request proves no second introspection call occurs.
Forward raw fc- credentials unchanged and let Core authenticate them. Keep strict, route-scoped introspection for fco_ access tokens.
Translate Core 401 responses into CREDENTIAL_INVALID for API-key sessions across SDK and fetch-based tool paths, including feedback, while preserving other session and status behavior. Cover every MCP route and credential transport with zero-introspection regression tests.
Three defects in the recovery mapping, all confirmed against the code.
The matcher recognised only FirecrawlSdkError and axios, so the tools
that reach Core through plain fetch never mapped. firecrawl_parse and the
monitor tools both threw a bare Error carrying no status, and an invalid
key there produced a generic failure instead of the recovery. Give those
paths a CoreHttpError that carries the status, then read the status
rather than sniffing the error's type. That also drops a match on a
runtime name string, which a JobTimeoutError would not have satisfied.
The wrapper applied to every session, so an OAuth session whose key Core
rejects was told to replace a key on this server when the right
instruction is to reconnect the account. Gate it on the only session type
that forwards a credential nothing resolved first.
Restore the four assertions the outage test lost when its loop became
linear, and scope the fixture's invalid-credential guard to fc- so an
fco_ token containing "invalid" cannot silently alter unrelated tests.
The feedback tools are deliberately left alone: they return a structured
4xx payload rather than throwing, on purpose, so agents do not retry-loop
on a terminal rejection. A 401 there is already legible as
{success:false, status:401}.
The carve-out kept introspect_unusable_credential failing closed, which
denied a key Core can validate because the authorization server sent a
payload we could not read. That is a statement about the answer, not
about the key, and being stricter there than when no answer arrives at
all was arbitrary.
A verdict still arrives as a value rather than a throw: an inactive
credential and a non-general purpose are both handled on the returned
payload and both still fail closed.
Removing the call outright also removed the account attribution that
gates the action-log POST, so API-key traffic would have stopped being
recorded. Keep the call and demote it instead.
Core authenticates an API key on every request, so introspecting one
resolves it to itself. What the round trip supplies is the
credential_purpose check and the account attribution, not a decision Core
does not already make. An authorization server that cannot answer now
costs an API-key session that enrichment and nothing else; an answer we
do get is still honoured, so behaviour is unchanged whenever
introspection works.
An answered-but-unusable payload still fails closed. Only the classes
that mean no usable answer are treated as absent enrichment.
OAuth access tokens are untouched and still fail closed, because Core
cannot resolve one on its own and there is nothing to fall back to.
Keep the Core 401 mapping: when enrichment was unavailable, an invalid
key is discovered at tool time, and it must still reach the agent as the
CREDENTIAL_INVALID recovery rather than a raw upstream error.
Note this preserves request volume against the introspection endpoint. It
removes the failure, not the load.
Nothing is rejected at connect time now that a well formed API key is
admitted and judged by Core, so the old name described behaviour the
assertions no longer check.
An API key is already the credential Core authenticates on every call.
Resolving it through the authorization server returned the same key back,
so the round trip added no authorization decision Core does not already
make, while making every request depend on a third service. OAuth access
tokens are unchanged: Core cannot resolve one on its own, so those still
introspect and still fail closed.
Forward the key and let Core be the authority. A rejection is no longer
discovered at connect time, so translate a 401 from Core into the same
CREDENTIAL_INVALID recovery the agent already knows how to act on. Tools
reach Core two ways, the SDK helpers and the raw HTTP layer that search
uses to keep its response envelope, and both carry the upstream status.
Only 401 is matched. A 403 can mean an entitlement the key legitimately
lacks, which is not a verdict on the credential.
This gives up the credential_purpose check, which was defence in depth
rather than a boundary since Core validates the key itself, and team
metadata on API-key sessions, which fed action telemetry only.
Constructing the class directly compiles and silently skips the record,
which is the regression this PR exists to close. Say so where someone
adding a throw site will read it.
That path was the one record emitted with no context beyond its reason.
Pass the resource the token was validated against, using expectedAudience
rather than the profile URL so the legacy audience fallback is reflected.
The parameter is optional because outbound signing has no resource in
scope and should not carry a misleading one.
Logging from the authenticate catch covered only one of the two call
stacks that raise this error. The outbound client checks in getClient run
during tool execution, so their reasons were labels that could never
appear in a log, and any throw site added later would have been dropped
the same way.
Build the error through a factory that emits the record and returns it.
Every site is covered regardless of where it is caught or whether it is
caught at all.
Replace the profile field with the resource being validated. Profile
describes the request rather than the error, it is not available where
the error is raised, and the resource URL is both more precise and
already carried into introspection.
Note that delegated_credential_unavailable is unreachable: the
interceptor is installed only for a session holding a managed key, and
signing such a session always yields a non-empty credential or throws.
It stays because the signature is string | undefined, so dropping it
would mean asserting rather than checking.
An introspection body that failed to parse, or that parsed to null,
threw past every tagged error in introspectToken. It reached the client
as a 401 with WWW-Authenticate and the raw parser message in
error_description, so an upstream fault pushed a working session into
reauthorization and no reason was logged.
Parse defensively and treat a missing boolean `active` as
introspect_malformed_body, matching the guard this file already uses on
other upstream JSON reads. Also scope the telemetry doc comment: it
covers rejections raised while authenticating, not the outbound client
tags raised during tool execution.
Every failed credential check returned the same 503 body with no server
log, so an introspection outage, a missing secret, and a credential that
introspected cleanly but is unusable here were indistinguishable after
the fact.
Tag each throw site with a reason and emit one low cardinality server log
line carrying the reason, the introspection HTTP status, elapsed ms, and
whether the request was aborted by its own budget. Never log the token,
the resolved API key, or the upstream body.
Keep the 503 and the existing sentence. Add Retry-After so the client has
a concrete wait, and drop the OAuth error code, which RFC 6749 reserves
for 400 and 401 responses and which clients surface as an authentication
verdict. No WWW-Authenticate: this is not a verdict on the credential.
The full-profile MCP instructions omitted the developer index entirely,
while the search-only profile already described it. Add one capability
sentence naming firecrawl_developer_search (and the categories
fallback). The keyless profile is left unchanged: its instruction text
stays scoped to the keyless tool surface (Search, Scrape, Parse).
Also extend the metadata-policy test gates to the full, keyless, and
account profile instructions, so future rewrites of these constants
into routing playbooks fail in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Quota, blocked-tool, and ineligible-access errors now say how to attach an API key or use the OAuth URL. Invalid-key and broken-OAuth recovery use a short replace/reauth Fix instead of a human-reconnect essay, and tell the agent to start a new session after the config change.
* fix(mcp): disambiguate research paper tools from the research category
Why: measured over 40 agent traces, two features answer to "research" —
the paper index reached via firecrawl_research_*, and firecrawl_search
with categories: ["research"], which is only a website filter. Agents
picked the website filter on biomedical tasks and never called the paper
tools.
Summary:
- categories enum describe now states that `research` filters ordinary
web results to research-affiliated websites, separate from the paper
tools.
- FULL, KEYLESS, and SEARCH profile instructions each gained a neutral
boundary sentence naming the two surfaces.
- search_papers description restores biomedical coverage (PubMed,
bioRxiv, medRxiv) alongside arXiv, plus a multi-framing line.
- related_papers title is no longer arXiv-specific.
- Escape-free rewrite of the boundary sentence in FULL/KEYLESS: the
backticks around `research` terminated the enclosing template literal
and broke the TypeScript parse. Uses the categories: ["research"] form
already used by SEARCH_PROFILE_INSTRUCTIONS.
* fix(mcp): put the research disambiguation where agents can read it
Why: the categories enum `.describe()` never reaches an agent. Every
parameter description is stripped during schema serialization — a dump
of tools/list shows 0 of 156 property descriptions across all 26 tools
survive. The disambiguation therefore had no agent-visible carrier on
the firecrawl_search side.
Summary:
- Add a `categories: ["research"]` paragraph to the firecrawl_search
tool description, mirroring the existing `["developer"]` paragraph.
It states what the website filter does and that firecrawl_research_*
is a separate surface over PubMed/bioRxiv/medRxiv/arXiv.
- Retarget the smoke-test assertion for search_papers onto the new
biomedical coverage text, and pin both the tool description and the
initialize instructions so the boundary cannot silently regress.
The categories `.describe()` edit is kept: it is correct, it documents
the field for readers of the source, and it costs nothing.
* docs(readme): separate paper research from the research website filter
Why: the README taught the exact confusion the tool metadata was fixing.
The worked firecrawl_search example was a paper-search query ("latest AI
research papers 2023") shown served by plain web search, the "How to
Choose a Tool" list had a bullet for every tool except research, and
Section 12 was the thinnest tool section with no statement of what the
paper index actually covers.
Summary:
- Replace the search worked example and its prompt example with a
genuinely web-shaped query, and cross-reference the research tools.
- Add a research bullet to "How to Choose a Tool" that names the two
surfaces.
- Section 12 gains a "Covers:" line (PubMed, bioRxiv, medRxiv, arXiv),
per-tool bullets that say what each tool does, and a disambiguation
sentence mirroring Section 14's developer-index pattern.
* chore: 3.24.0 — add research vocabulary to package metadata
Why: the npm and MCP-registry listings described only web search and
scraping, so the paper-search tools were invisible to anyone searching
for "pubmed", "arxiv", or "literature search". Registry copy is the
first discovery surface, before any tool description is ever loaded.
Summary:
- package.json description now names research paper search over
biomedical and arXiv literature, citation-graph expansion, and
full-text reading.
- Add research, paper-search, literature-search, biomedical, pubmed,
and arxiv keywords.
- Minor bump to 3.24.0: agent-visible behaviour changes (tool
descriptions and server instructions), no API change.
- server.json was stale at 3.23.6 in both the top-level and package
version fields; both now track package.json at 3.24.0, and its
description picks up the same vocabulary (83 chars, within the
registry limit).
* docs(readme): describe search_github by what it actually indexes
Why: the bullet said "public code repositories", but the tool searches
indexed GitHub issue, pull-request, and README content. Matching the
tool description keeps the README from setting the wrong expectation.
* fix(mcp): keep keyless instructions accurate to the keyless tool surface
The boundary sentence added in 913341e told every reader of
KEYLESS_PROFILE_INSTRUCTIONS to reach for the firecrawl_research_* tools
on biomedical questions. That profile also serves keyless sessions, where
tools/list is exactly KEYLESS_TOOL_NAMES (firecrawl_scrape,
firecrawl_search, firecrawl_parse) and guardHostedTool rejects every
research call with KEYLESS_TOOL_NOT_AVAILABLE. The instructions were
recommending tools the client can neither see nor call.
Rewrite the keyless variant so it leads with the surface that is actually
reachable -- firecrawl_search with categories: ["research"] as a filter
over ordinary web results -- and presents the paper index as something an
OAuth connection or Authorization bearer API key makes available. The
condition is phrased on the credential, not on absence, so the same string
still reads correctly for the API-key sessions this profile also serves,
where the research tools do list and do run.
FULL_PROFILE_INSTRUCTIONS keeps its sentence unchanged: the account
surface exposes those tools.
The pinned smoke assertion is replaced by two, one per half of the
corrected boundary, so neither the reachable-now surface nor the
authentication gate can silently regress.
Verified: findAgentMetadataPolicyViolations returns [] for the new
sentence and for the whole string; npx tsc --noEmit clean; npm run lint
clean; npm test 71/71.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(mcp): make invalid API key recovery reach the agent
An invalid or revoked API key on /v2/mcp returned HTTP 401 at initialize and
tools/list. MCP clients treat a 401 at connect as "server unavailable" and
never surface the response body to the model, so the CREDENTIAL_INVALID
recovery payload — the whole point of #363 for this case — was unreachable in
a real agent session. The agent saw "no Firecrawl tools connected" and fell
back to generic, often outdated setup advice.
Fix: on the keyless+API-key endpoint, admit a supplied-but-invalid credential
as a session flagged credentialError instead of throwing a 401. The connection
succeeds, tools list, and every tool call returns the CREDENTIAL_INVALID
recovery payload as a 200 isError result — the same agent-legible path keyless
quota recovery already uses. No credential is forwarded and no tool executes,
so this grants zero functional access (strictly less than the previously
rejected "downgrade to keyless" option).
Scope: only profiles that accept API keys as a valid auth mode (allowKeyless).
OAuth-only surfaces such as /v2/mcp-search keep their hard 401 credential
rejection unchanged.
Verified end to end: a live claude session wired to an invalid-key server now
receives the payload and relays both recovery options to the human without
asking for a key in chat. Smoke suite updated to the 200/isError contract;
no net-new failures vs main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(mcp): bump to 3.23.8 for the invalid-key recovery fix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): stop CREDENTIAL_INVALID payload from advertising unusable keyless tools
The credentialError guard in guardHostedTool's execute() runs before the
keyless branch, so no tool (including keyless ones) is actually callable in a
CREDENTIAL_INVALID session. recoveryPayload() was still including
available_tools: KEYLESS_TOOL_NAMES for this code, which could send the agent
into a retry loop against tools that just return the same recovery error.
Omit available_tools when code is CREDENTIAL_INVALID, matching the existing
isKeylessAccessUnavailable carve-out.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): scope CREDENTIAL_INVALID tools/list to the keyless surface; pin next_actions shape
canList previously returned true for every registered tool in a
credentialError session, so tools/list disclosed the full authenticated tool
schema (including non-keyless tools) to any request carrying an unrecognized
or invalid credential -- more schema disclosure than a real keyless session
gets. Scope it to the same keyless-tool surface a keyless session already
lists; recovery guidance still surfaces on any listed tool call since
execute() gates on credentialError before the keyless branch either way.
Also tighten the invalid-key next_actions assertion in mcp-smoke.test.mjs
from a loose non-empty-array check to pinning the human_reconnect_account /
operator_configure_api_key kinds and their consent flags, so a regression
that drops or reorders the recovery actions is caught.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): make keyless recovery consent-first
* fix(mcp): complete safe keyless recovery
* fix(mcp): guide invalid credential recovery
* fix(mcp): make account-only recovery fallback-first
* refactor(mcp): dedupe security-boundary guidance strings and recovery payload shape
Addresses cubic P2 (src/index.ts:1007, confidence 6): the "never put an API
key in chat/URL" and "retry after a new session/run" sentences were
duplicated verbatim across HUMAN_CONNECTION_GUIDANCE,
invalidApiKeyRecoveryPayload, and both invalidOAuthRecoveryPayload branches;
the two invalidApiKeyRecoveryPayload/invalidOAuthRecoveryPayload builders
also repeated the same code/auth_mode/docs_url/next_actions shape.
Extracts NEVER_SHARE_API_KEY_SENTENCE and RETRY_AFTER_CONFIRM_SUFFIX as the
single source for that phrasing, and a small connectionRecoveryPayload()
base builder for the two recovery payloads. Pure dedup: every emitted
message/payload byte is unchanged (verified by diffing the built dist
output against the pre-refactor build).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): gate reconnect-through-client guidance on OAuth being enabled
With CLOUD_SERVICE=false the invalid-OAuth recovery told the user to sign in
through the client's account-connection flow even though this server cannot
start one. The advertise branch now also requires isMcpOAuthEnabled(),
matching the existing composition used for tool listing; disabled-OAuth
deployments fall through to the guidance for servers that do not start
account sign-in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(mcp): co-locate guidance URL constants with resource defaults
Pure relocation addressing review feedback: the guidance URL constants
lived ~830 lines from the DEFAULT_* resource constants. They now share one
section, with a comment explaining why MCP_OAUTH_SERVER_URL repeats the
oauth resource value without aliasing it (protocol identity is
env-overridable per deployment; the human-facing copy is fixed).
No behavior change: bundle content is line-identical modulo ordering.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(keyless): put recovery URLs in the message text agents actually read
Agent hosts surface only the error message string to the model;
structuredContent (next_actions, docs_url, client_commands) is not
passed through. Verified empirically: an agent asked to repeat the
error verbatim reported 'no URLs present in the response'.
Add the OAuth endpoint and the free-API-key signup URL directly to the
four recovery messages (quota exhausted, keyless unavailable, invalid
credential, account-tool required), matching the API-side precedent in
apps/api keyless.ts. With the change, agents relay both URLs and the
Authorization header format to users unprompted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(keyless): align CREDENTIAL_INVALID message with sibling recovery guidance
Add the Authorization: Bearer wiring instruction so a user replacing a
broken key sees the same how-to as the quota and access messages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: 3.23.5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs(scrape): disclose maxAge freshness/speed tradeoff and liveness limits
The firecrawl_scrape description sold cache speed ("500% faster") without
disclosing the freshness tradeoff or how to bypass the index, and the maxAge
param had no describe(). This adds a Freshness-vs-speed line, a Liveness line
(a successful render does not prove a listed item is still active), describe()
text on maxAge in both scrapeParamsSchema and parseOptionParamsSchema, and one
reinforcing sentence in the server instructions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(scrape): make liveness guidance domain-neutral
* docs(scrape,search): trim freshness copy and correct maxAge claims
Addresses review on #321.
- Drop the maxAge describe() from scrapeParamsSchema. That schema is
embedded in the scrape, search, crawl and interact tool schemas, so a
single describe shipped four times.
- Mark parse's maxAge as ignored instead of documenting it. /v2/parse
force-sets maxAge: 0 and storeInCache: false after spreading the
client body (apps/api/src/controllers/v2/parse.ts:501-503), so a
client-supplied value is inert.
- Keep one freshness surface: a single line in the firecrawl_scrape
description, plus one short liveness sentence. The server-instructions
sentence is dropped.
- Note on firecrawl_search that scrapeOptions pages reuse indexed
content up to three days old and do not honor maxAge. The search
scrape path hard-sets maxAge to 3 days after spreading scrapeOptions
(apps/api/src/search/scrape.ts:77-80), so maxAge: 0 does not bypass
reuse there.
Net effect on shipped tool-description text: ~1585 chars (~396 tok)
down to ~481 chars (~120 tok).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(scrape,search): describe reuse behavior without index internals
The concrete reuse windows are index internals that can drift, so state
the user-visible behavior only: scrape's window varies by domain and is
bounded by maxAge, and search uses a fixed window that ignores maxAge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The developer tool called /v2/developer/search, the path that may be
withdrawn. Call /v2/search/developer, the public path.
The API returns 10 results by default, not 20. Correct the parameter
description.
Developer search leaves beta, so document the tool in the README beside
the other tools and drop the experimental note.
Bump to 3.23.3 in package.json and both server.json fields.
Erik Engervall