Commit Graph
6548 Commits
Author SHA1 Message Date
Abimael MartellandClaude Opus 5.5 4c33951663 fix(branding): use black as the primary color for monochrome brands (#5098)
Brands with a black logo and black calls to action, with color only in links
or artwork, had their primary color reported as whatever link or artwork color
the model saw, because the merge step rejects near-black on light pages to keep
dark nav bars out. When the chosen primary button is a neutral near-black on a
light page, and the chosen color doesn't also fill a button, the primary color
is now that black and the colored pick moves to accent. Navy buttons, dark
pages, and brands that pair a black button with colored ones are unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v2.11.508
2026-10-12 09:40:46 -07:00
tomandtomsideguide 94ba4a3eb1 fix(keyless): point suspicious IPs to Google/GitHub signup (#5097)
* fix(keyless): point suspicious IPs to Google/GitHub signup

* fix(keyless): say VPN or proxy instead of suspicious

---------

Co-authored-by: tomsideguide <tom@sideguide.dev>
v2.11.507
2026-10-12 13:25:42 +01:00
Rakshith Ramprakashandcapy-ai[bot] c22afa500a feat(api): expose Alexandria domain providers as scrape agent hint signals (#5091)
* feat(api): expose Alexandria domain providers as scrape agent hint signals

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): keep failed domain provider lookups distinct from no match

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.506
2026-10-11 22:17:47 +00:00
Rakshith Ramprakashandcapy-ai[bot] f84845c864 fix(api): make bounded progress on monitor finalization (#5046)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.505
2026-10-10 23:02:40 +00:00
Abimael MartellandClaude Opus 5.5 6f892bc502 feat(branding): find more of the page's real call-to-action buttons (#5049)
* feat(branding): find more of the page's real call-to-action buttons

The in-page scan missed many pages' main call to action before any model saw
it:

- Sampling took the first 100 button matches and first 100 links in document
  order, so menus and footers used up the budget before the hero. It now
  considers every button and button-like link and keeps the visible ones
  nearest the top of the page.
- Buttons whose fill lives on an inner element or a ::before/::after layer
  read as transparent and were dropped. The fill is now taken from there.
- Filled or outlined buttons inside the header nav were discarded as
  navigation. They stay buttons; plain menu links still don't count.
- Hidden copies (closed menus, dialogs) are left out, snapshots carry their
  page position and visibility, and buttons on the first screen rank higher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(branding): take the text color from the page's text

The text color was the first dark color in a vote over every sampled color,
weighted toward large backgrounds and button fills. Sampling more buttons let
dark button fills outvote the paragraphs, and on dark pages it often returned
the background. It is now the most common text color of non-button elements
that stands out from the background, with the old pick as a fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(branding): tighten visibility and position in the button scan

- An element inside a faded-out ancestor (opacity 0) no longer counts as
  visible: opacity doesn't inherit, so ancestors are checked too.
- A ::before/::after layer at opacity 0 (hover-only) is no longer a fill.
- Fixed and sticky elements keep their on-screen position instead of adding
  the scroll offset, so a header button isn't pushed below the first screen.
- Hidden snapshots don't vote on the text color.
- The hidden-copy test gives the hidden button a real box, so it exercises
  display: none, and covers a faded-out parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v2.11.504
2026-10-09 22:23:46 -07:00
Abimael Martellandcapy-ai[bot] ebd43f734d feat(api): label FirePDF async total duration by origin (#5050)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.503
2026-10-09 22:02:52 -07:00
claude[bot]andClaude 3f5f906ca2 feat(browser): add +2 credits/min ZDR surcharge to Interact sessions (#5048)
ZDR browser/Interact sessions now bill 240/hr code-only (4/min) and
540/hr with prompts (9/min). The surcharge applies to the create-time
estimate, the first-prompt reservation, and settlement.


Claude-Session: https://claude.ai/code/session_013R47uwW8MefsRuLhpzaiq2

Co-authored-by: Claude <noreply@anthropic.com>
v2.11.502
2026-10-09 16:22:21 -04:00
Abimael MartellandClaude Opus 5.5 35f64ae2a0 fix(branding): keep fallback and duplicate fonts out of Jev's font list (#5045)
* fix(branding): keep fallback and duplicate fonts out of Jev's font list

Jev's font answer kept every family a page lists, so system fallbacks
(Arial, Segoe UI, Roboto, Times New Roman, ...) and unplaced extras showed up
next to the real typefaces.

- Fallback fonts are dropped unless the page's typography uses them for text
  or nothing else is left.
- Once headings or body are covered, fonts with no known role are dropped.
- A font stack reports its first family, "Variable"/"VF" suffixes are
  stripped, and names that differ only in spacing merge into one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(branding): match font roles on the same key families merge on

Merging "CormorantGaramond" into "Cormorant Garamond" kept the spaced name, so
the page's typography no longer matched it and the font lost its heading or
body role. Roles and merging now share one key (lowercase, whitespace removed),
and hyphenated names are no longer merged with spaced ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(branding): keep typographyRole's doc comment on typographyRole

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v2.11.501
2026-10-09 11:43:13 -07:00
Abimael MartellandClaude Opus 5.5 ff02eeb993 fix(branding): send Jev only well-formed Unicode text (#5043)
Page text can contain lone UTF-16 surrogates: the branding script truncates
text by code unit, which can split a character outside the BMP (emoji,
mathematical bold letters), and clip() could do the same. TypeSafe rejects
such requests with "400 Request contains invalid Unicode text", so the call
fell back to the LLM.

clip() no longer ends on the first half of a surrogate pair, button labels
and file names go through it, and every string in the Jev state has any
remaining lone surrogate replaced with U+FFFD before sending.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v2.11.500
2026-10-09 10:54:39 -07:00
897354b915 feat(api): the job log is Pub/Sub only, no PostgreSQL copy (#5010)
* feat(api): the job log is Pub/Sub only, no PostgreSQL copy

log_job publishes each row and stops there: the PostgreSQL insert, its
retry flag, and the Drizzle definitions of the seventeen job tables go.
E2E helpers and tests that read the job tables read the ClickHouse job
log instead; the feedback tests seed and back-date feedback jobs in
Bigtable rather than searches rows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(api): share job log polling and skip job log snips without ClickHouse

- jobLogJson returns null for empty or malformed JSON text.
- getCostTrackingCalls reads through jobLogRows/jobLogJson and keeps polling
  until the scrape row carries parseable cost tracking.
- zdr-helpers polls through a shared waitForJobLogRows instead of its own
  query loop; waitForJobLogRow builds on it.
- HAS_JOB_LOG gates every snip that reads the job log (developer, research,
  parse, zdr, lockdown, branding and X cost tracking).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.499
2026-10-09 15:12:12 +02:00
Rakshith Ramprakashandcapy-ai[bot] e5df790e83 fix(api): preserve monitor credit reservation ownership (#5031)
* fix(api): preserve monitor credit reservation ownership

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): guard monitor reservation admission after awaits

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(api): keep unresolved monitor read failures non-destructive

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ci(api): run monitor reservation concurrency regressions

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.498
2026-10-09 06:24:59 +00:00
Abimael MartellandClaude Opus 5.5 dccff26012 feat(branding): stop calling Jev for a minute after repeated failures (#5032)
* feat(branding): stop calling Jev for a minute after repeated failures

After 5 Jev calls in a row fail (timeouts or upstream errors), branding skips
Jev for 60 seconds and answers with the LLM straight away instead of waiting
out the timeout on every request. Calls go through again after the cooldown,
and one more failure reopens it. Unusable answers don't count, and the shadow
pauses while it is open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(branding): keep the Jev breaker armed when a late call succeeds

A call that started before the breaker opened and succeeded afterwards reset
the failure count, so the first failure after the cooldown no longer reopened
it. Successes only reset the count while the breaker is closed. The breaker
tests now fake Date with timers restored in afterEach, so a failing assertion
can't leave the clock frozen for later tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v2.11.497
2026-10-08 19:54:22 -07:00
Gergő Móriczandcapy-ai[bot] 0c3819648e fix(api): move monitor change judge to gemini-3.5-flash (#5030)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.496
2026-10-09 03:38:37 +02:00
Gergő Móriczandcapy-ai[bot] 8d62750016 feat(api): run browser and interact sessions from a chosen country (#5029)
* fix(api): run interact sessions from the scrape's location country

Carry the scrape's location.country and proxy type on BrowserOptions and
map them to Hangar's proxy selection, so an interact session egresses from
the country (and pool) the scrape used.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(api): accept location on browser session create

/v2/browser and /v2/interact take location.country (ISO 3166-1 alpha-2)
and pass it to Hangar's proxy selection. Add the option to the JS, Python,
Go, Java, PHP and Elixir SDKs.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(sdks): cover omitted Elixir browser location; add profile and location to the PHP facade annotation

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(php-sdk): type the facade browser() array parameters

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.495
2026-10-09 02:39:22 +02:00
Gergő Móriczandcapy-ai[bot] e55325ca27 feat(sdks): top-level checkPromptInjection in the .NET, Elixir, Go, Java, PHP, Ruby and Rust SDKs (#5028)
* feat(sdks): expose top-level checkPromptInjection in the .NET, Elixir, Go, Java, PHP, Ruby and Rust SDKs

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(sdks): checkPromptInjection on parse options; address review

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(sdks): cover unset checkPromptInjection on Rust parse and the Elixir parse_file body

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-08 23:24:58 +02:00
Developers Digest f0dd0592a5 Honor Exchange terms overrides while preserving universal scrape gates (#5026)
* Honor Exchange terms overrides without changing universal scrape

* Preserve revocation recovery and acceptance compatibility

* Run Exchange terms compatibility regressions in CI
v2.11.494
2026-10-08 16:55:02 -04:00
Gergő Móriczandcapy-ai[bot] fa7ec5f0a9 feat(api): make checkPromptInjection a top-level scrape option (#5024)
* feat(api): make checkPromptInjection a top-level scrape option

The json format's checkPromptInjection is lifted into the new top-level
flag at parse time. The guard runs once per scrape on the page markdown,
before any LLM-backed format, and bills +4 regardless of format.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): project the prompt injection guard fee for keyless scrapes; cover SmartScrape page scans

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(sdks): say checkPromptInjection works with any format

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.493
2026-10-08 22:35:13 +02:00
Gaurav Chadhaandcapy-ai[bot] ec849f3d60 fix(api): send Retry-After on per-minute rate limit 429s (#5025)
* fix(api): send Retry-After on per-minute rate limit 429s

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): round keyless Retry-After up from the key's PTTL

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.492
2026-10-08 14:10:23 -04:00
Gergő Móriczandcapy-ai[bot] 5a75dca210 chore(js-sdk): move tests from Jest to Vitest (#5021)
* chore(js-sdk): move tests from Jest to Vitest

Replace jest.config.js with a minimal vitest.config.ts, port the unit and e2e suites to vitest imports and vi, and drop jest, ts-jest, @jest/globals, @types/jest and @types/mocha along with the pnpm overrides and build allowances that only Jest's tree needed. Also drop the remaining Jest mentions from repo docs, API test comments and the v1 client's test-env check.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: run vitest through pnpm exec so the harness waits for the API

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.491
2026-10-08 17:51:12 +02:00
Gaurav Chadhaandcapy-ai[bot] 8a0e91b000 fix(api): reject deeply nested sitemaps instead of crashing the process (#5016)
* fix(api): reject deeply nested sitemaps instead of crashing the process

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): ignore '>' inside quoted attributes in the sitemap depth scan

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): reject entity declarations in sitemaps before parsing

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.490
2026-10-08 11:08:36 -04:00
Gergő Móriczandcapy-ai[bot] 244f83bdd6 fix(security): remove braces, sprintf-js, csv-parse and postcss-selector-parser from every lockfile (#5012)
* fix(security): drop braces, sprintf-js and csv-parse from api, js-sdk and test-suite

knip 6 and lint-staged 16.4 no longer pull micromatch/braces in apps/api.
jest 30.5 drops micromatch in the JS SDK, and load-nyc-config's js-yaml is
moved to 4.x so argparse@1 and sprintf-js leave the tree. The artillery
load test in apps/test-suite targeted the retired v0 staging host and
nothing ran it, so it is removed with its npm project and CI wiring.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore(ui): remove the unmaintained ingestion-ui template

Nothing builds or deploys apps/ui/ingestion-ui; it has had no functional
change since 2024 and only receives dependency patches. Removing it takes
tailwindcss 3 and postcss-selector-parser 6 out of the repo.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.489
2026-10-08 14:26:17 +02:00
David TippettandClaude Opus 5.5 b00172b23b feat(api): point scrapes of search results pages to /search (#5003)
* feat(api): point scrapes of search results pages to /search

Scrapes and batch scrapes of a search engine results page
(google.*/search?q=, bing.com/search?q=, DuckDuckGo, Brave, Yahoo,
Baidu, Yandex, Ecosia, Startpage) get a warning that /search returns
ranked results for the query directly. The warning is off by default:
SERP_SCRAPE_WARNING turns it on for every team,
SERP_SCRAPE_WARNING_TEAM_IDS for listed teams, and
SERP_SCRAPE_WARNING_ROLLOUT_PERCENT for a stable share of teams. Every
results-page scrape is logged with whether the warning was shown, so
warned and unwarned teams can be compared.

Hint-enabled /v2/scrape responses for a results page get a next-step
hint with a ready firecrawl_search call, and provider rules can match
the new page_serp_engine and page_serp_query signals.

Also drops export from four agent-hint constants and types that only
their own module uses, which knip reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(api): tighten DuckDuckGo and Yahoo results-page matching

DuckDuckGo now matches only its result paths (/, /html, /lite) and Yahoo
only /search, /search/... and /search;... so pages such as
duckduckgo.com/about?q= and /searchfoo are not treated as results pages.
Tests now pin that the requested URL decides when the final URL differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v2.11.488
2026-10-08 08:24:15 -04:00
Rakshith Ramprakashandcapy-ai[bot] 7cca3edf96 refactor(api): remove built-in agent hints (#4992)
Agent hint content now comes only from provider-supplied response
guidance rules. Without a provider, responses carry no agent_hints.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.487
2026-10-08 05:50:03 +00:00
Gergő Móriczandcapy-ai[bot] f9f2e3dd54 fix(api): keep blocklist keyword exemptions off URLs the Exchange claims (#5011)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.486
2026-10-08 03:00:30 +02:00
Gergő Móriczandcapy-ai[bot] 8d737f12ed fix(api): check robots.txt before fetching a crawl's start URL (#5005)
The crawl start URL was scraped first and only checked against robots.txt
afterwards, so a disallowed start URL was fetched and then discarded. Check
it against the crawl's robots.txt before the scrape starts, honoring
ignoreRobotsTxt.

Also record robots.txt blocks the crawl already found but dropped:
- sitemap and index URLs that filterLinks denies for robots.txt
- redirect targets robots.txt disallows, recorded as the target URL, with
  nothing crawled from a disallowed start URL target

CrawlDenialError carries the blocked URL as robotsBlockedUrl instead of a
robots flag, and filterLinks returns robotsBlocked.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.485
2026-10-08 02:29:40 +02:00
Gergő Móriczandcapy-ai[bot] e3986ebd0a feat(api): log every auth denial as one canonical auth/denied line (#5008)
* feat(api): log every auth denial as one canonical auth/denied line

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): redact the admin secret from auth/denied routes

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.484
2026-10-08 01:19:19 +02:00
Gergő Móriczandcapy-ai[bot] 27220e882f fix(security): override geoip-country's ip-address to 10.7.3 in apps/api (#5007)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.483
2026-10-08 01:18:50 +02:00
Gergő Móriczandcapy-ai[bot] 2a48715def fix(security): patch open Dependabot alerts in examples, Rust SDK and Go html-to-md (#5006)
* fix(security): patch open Dependabot alerts in examples, Rust SDK and Go html-to-md

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(examples): require Python 3.10+ where pins now need it

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.482
2026-10-08 00:34:08 +02:00
23b68aefd2 feat(js-sdk): abortable crawl() that cancels its job; cancelCrawl returns false on 409 (#4898)
* feat(js-sdk): abortable crawl() that cancels its job; cancelCrawl returns false on 409

cancelCrawl now resolves false when the API answers 409 because the crawl
is already completed. Other errors still throw.

crawl() accepts an optional AbortSignal. When it aborts, polling stops,
the SDK sends a best-effort cancel request for the job, and the promise
rejects with signal.reason. A signal that is already aborted rejects
before any request. waitForCrawlCompletion also accepts the signal, but
only stops polling and does not cancel the job.

Bumps the JS SDK to 4.43.0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H5YqtAzmu8fHaWAUDQLZa4

* chore(js-sdk): bump to 4.45.0 for the crawl() abort signal

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(js-sdk): reject an aborted crawl() without waiting for the cancel request

The best-effort cancel now runs in the background, so a slow or hanging
DELETE no longer delays the abort rejection.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(js-sdk): read the abort reason after the abort in the hanging-cancel test

The test passed `controller.signal.reason` to `rejects.toBe()` before the
abort happened, so the expected value was `undefined`. Catch the error
first, then compare it with the reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H5YqtAzmu8fHaWAUDQLZa4

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mogery <mogery@sideguide.dev>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-07 23:22:17 +02:00
a73af1d5df fix(python-sdk): cancel the crawl when async crawl() is cancelled; cancel_crawl returns False on 409 (#4897)
* fix(python-sdk): cancel the crawl when async crawl() is cancelled; cancel_crawl returns False on 409

cancel_crawl now returns False when the API answers 409, which it does
when the crawl already completed. Before, it raised FirecrawlError. Other
error codes still raise. This applies to the sync and async clients.

The async crawl() starts a crawl and waits for it. If the task that
awaits it is cancelled (asyncio.timeout, asyncio.wait_for, Task.cancel),
crawl() now sends a best-effort cancel for the started job, then
re-raises the CancelledError unchanged. The cancel runs in a shielded
task with a 10 second limit, and its errors are ignored. start_crawl and
wait_crawl do not change, and the SDK timeout parameter still raises
CrawlJobTimeoutError without a cancel.

Bump python-sdk to 4.46.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H5YqtAzmu8fHaWAUDQLZa4

* fix(python-sdk): address review on crawl cancellation

- README and the async crawl() docstring: the best-effort cancel runs
  only after start_crawl returns the job id. Task.cancel() gives
  CancelledError, while asyncio.timeout and asyncio.wait_for give
  TimeoutError.
- cancel_crawl docstrings: also list the transport error that the HTTP
  client can raise (httpx.HTTPError async, requests.RequestException
  sync).
- _cancel_abandoned_crawl: add a done callback that reads the cancel
  task's outcome. A second cancellation of the caller no longer leaves
  a "Task exception was never retrieved" log.
- Tests: time the slow-cancel test (under 1 s), and add a test that
  cancels twice and checks that the loop reports no error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H5YqtAzmu8fHaWAUDQLZa4

* chore(python-sdk): bump to 4.49.3

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mogery <mogery@sideguide.dev>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-07 23:22:02 +02:00
9788805295 fix(api): record robots.txt denials on crawls so the status warning and robotsBlocked work (#4900)
* fix(api): record robots.txt denials on crawls

The crawl status robots warning and the robotsBlocked list on
/crawl/:id/errors matched one exact message text. Only the flag-gated
scrape robots check used that text, so default crawls recorded nothing.
Discovered links never reached the robots check either, because link
extraction dropped them first.

- Add isRobotsDenialReason() for the crawler robots denial.
- Give CrawlDenialError a robots marker that survives serialization.
  The scrape robots check and the start-URL robots check set it.
- Skip robots in link extraction. filterLinks applies robots after it
  and reports the denial.
- Reword the warning to point at robotsBlocked on the errors endpoint.
- Add a test-site robots fixture, snips tests and unit tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H5YqtAzmu8fHaWAUDQLZa4

* fix(api): batch robots.txt block records and count them with SCARD

Record every robots.txt-blocked link from a page with one SADD instead of
one SADD and EXPIRE per link, and read the warning count with SCARD so
status polls do not load the whole set. Reuse the snips crawlErrors helper
in the crawl robots tests.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): chunk robots.txt block records under the Redis argument limit

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mogery <mogery@sideguide.dev>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.481
2026-10-07 23:21:26 +02:00
claude[bot]andClaude Opus 5.5 924298df58 fix(api): report crawl completedAt correctly when no page succeeds (#4899)
* fix(api): report crawl completedAt correctly when no page succeeds

The crawl status took completedAt from the newest entry in
jobs_donez_ordered. That set only holds successful jobs. When no page
succeeded, completedAt fell back to createdAt and duration was 0.

addCrawlJobDone now also stores the finish time of the latest job of any
outcome in crawl:<id>:last_job_done_at. getLastDoneJobTimestamp returns
the later of that value and the newest successful job. The semantics of
jobs_donez_ordered do not change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H5YqtAzmu8fHaWAUDQLZa4

* fix(api): keep the latest crawl job finish time with an atomic max

- Write crawl:<id>:last_job_done_at with a small Lua script. The script
  sets the value only if it is greater than the stored value, and keeps
  the 24h TTL. Jobs can finish out of order, so a plain SET could replace
  a newer time with an older one.
- Refresh the TTL of last_job_done_at in getLastDoneJobTimestamp, next to
  the refresh of jobs_donez_ordered.
- In the snips test, replace the comparison with the /errors timestamp
  with a wall-clock bound taken after the crawl returns.
- Remove 4 unused exports that knip reports on main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H5YqtAzmu8fHaWAUDQLZa4

---------

Co-authored-by: Claude <noreply@anthropic.com>
v2.11.480
2026-10-07 23:21:12 +02:00
Abimael MartellandClaude Opus 5.5 0010c5f217 feat(api): record token usage and team on TypeSafe spans (#5004)
* chore(api): drop unused exports flagged by knip

MAX_RULE_HINTS, AGENT_HINTS_PROVIDER_CONTRACT_VERSION, AgentHintRuleScalar and
AgentHintRuleOp are only used in their own modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(api): record token usage and team on TypeSafe spans

Every typesafe.systemone span now carries the call's input and output tokens
and the team, so LLM spend tracking can price Jev calls and attribute them:

- Safe search spans gain typesafe.usage.input_tokens / output_tokens and
  teamId (removeExplicitResults takes the team).
- Branding spans already had input tokens; they gain output tokens and teamId.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v2.11.479
2026-10-07 11:32:51 -07:00
Gergő Móricz 859aec996b revert(api): return undici to 7.30.0 (#4976) v2.11.478 2026-10-07 19:53:24 +02:00
Gergő Móriczandcapy-ai[bot] 6c673c968c fix(api): name the option that keeps a provider out of a blocklisted URL (#5002)
* fix(api): name the option that keeps a provider out of a blocklisted URL

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): raise the unsupported-option refusal from engine selection too

Policy-forced zero data retention is named in plain words and the message no longer tells the caller to drop it.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.477
2026-10-07 19:27:13 +02:00
Gergő Móriczandcapy-ai[bot] bd654c5071 fix(api): admit Exchange-served blocklisted URLs in v2 batch scrape (#4999)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.476
2026-10-07 18:10:16 +02:00
Gergő Móriczandcapy-ai[bot] 7599de41e7 feat(rust-sdk): add DocumentMetadata::provider for Exchange scrapes (#5001)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-07 18:10:05 +02:00
Gergő Móriczandcapy-ai[bot] a2833245be feat(api): report the serving provider as metadata.provider on Exchange scrapes (#5000)
* feat(api): report the serving provider as metadata.provider on Exchange scrapes

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): fall back to the direct provider for empty waterfalls or invalid step costs

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.475
2026-10-07 17:42:20 +02:00
Gergő Móriczandcapy-ai[bot] ecd7934e04 fix(api): return 4xx for Exchange not_found and invalid_exchange_url refusals (#4998)
* fix(api): return 4xx for Exchange not_found and invalid_exchange_url refusals

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): keep the unsupported-URL refusal provider-neutral

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.474
2026-10-07 16:30:57 +02:00
Gaurav Chadhaandmogery 88569f4222 fix(sdks): surface success:false scrape errors in Go, Java, .NET, Ruby and PHP (#4971)
* fix(sdks): surface success:false scrape errors in Go, Java, .NET, Ruby and PHP

* fix(dotnet-sdk): branch on success flag and keep details; assert scrape request in Java test

---------

Co-authored-by: mogery <mogery@sideguide.dev>
2026-10-07 07:21:04 -04:00
claude[bot]andClaude Opus 5.5 9ae2451b39 fix(sdks): match the Alexandria client timeout to the API's 120 s cap (#4995)
* fix(sdks): match the Alexandria client timeout to the API's 120 s cap

The JS and Python SDKs capped the Alexandria execution timeout at 50 s.
With the 30 s response margin, the client gave up at 80 s. The API now
defaults and caps the timeout at 120 s, so the server continued and
billed a late success that the caller never got.

Default and cap the client timeout at 120 s with named constants, and
keep the 30 s margin (150 s transport timeout). Bump JS to 4.44.1 and
Python to 4.49.2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PHiJVRAaJ4rUWemp1cQBqq

* fix(sdks): apply the 120 s Alexandria timeout cap to Go and Rust

The Go SDK waited for the caller's timeout plus 5 s, or its 5 min
client default. The Rust SDK waited for the caller's timeout plus 5 s,
or used no timeout. Use the same rule as the JS and Python SDKs: cap
the execution timeout at the API's 120 s default and add a 30 s
margin for response delivery (150 s when no timeout is given).

Bump Go to 1.16.4 and Rust to 2.21.4.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PHiJVRAaJ4rUWemp1cQBqq

---------

Co-authored-by: Claude <noreply@anthropic.com>
apps/go-sdk/v1.16.4
2026-10-07 03:20:29 -04:00
Rakshith Ramprakashandcapy-ai[bot] 0b25e17057 feat(api): evaluate provider-supplied response guidance rules (#4991)
* feat(api): evaluate provider-supplied response guidance rules

Compute a fixed set of response signals locally, evaluate provider rules
with a small generic matcher, and fill placeholders in the API. The
built-in hints remain the fallback when no provider rules are available.
Bumps the provider contract to version 2; version 1 hints keep working.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): tighten rule placeholder limits and document rule caps

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore(api): remove AGENT_HINTS.md

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(api): never let agent hint computation change a response

Any error while computing hints now sends the original body unchanged,
logs a warning and counts an eval_error. Provider rule sets are
validated as a whole: a set with any invalid rule is treated as a failed
lookup and negative-cached.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
v2.11.473
2026-10-07 03:17:11 +00:00
Gergő Móricz 1bf1215ca3 feat(elixir-sdk): agent thread_id, mode and exchange (#4990)
* feat(elixir-sdk): agent thread_id, mode and exchange from the docs spec

Regenerated from firecrawl-docs fire-98/docs-agent-exchange. start_agent
gains thread_id, mode and exchange; also picks up docs main drift
(gov_search, gov_search_post, create_browser_session block_ads).
Bumps 1.11.3 -> 1.12.0.

* fix(elixir-sdk): validate closed objects' keys and send an empty one as {}

Objects with additionalProperties: false (auditMetadata, exchange) now get
nested NimbleOptions keys and a nested key mapping, replacing the
auditMetadata special case. exchange: [] sends {} instead of [].
2026-10-07 00:44:24 +02:00
Gergő Móricz 36f31d9055 feat(php-sdk): add agent exchange options, threadId and mode (#4989)
AgentOptions gains exchange (AgentExchangeOptions with nested approve and
decline), threadId and mode. AgentResponse exposes threadId/threadTurn and
AgentStatusResponse exposes thread fields, the chat message, pendingApproval
and the exchange run summary. Bumps the SDK to 1.16.2.
2026-10-07 00:44:13 +02:00
Gergő Móricz 776a87f060 feat(rust-sdk): add agent exchange, threadId and mode options (#4988) 2026-10-07 00:44:05 +02:00
Gergő Móricz 13fcb07bd8 feat(go-sdk): add agent exchange, threadId and mode options (#4987)
* feat(go-sdk): add agent exchange, threadId and mode options

* fix(go-sdk): send explicit empty toolkits and callIds lists
apps/go-sdk/v1.16.3
2026-10-07 00:43:54 +02:00
Gergő Móricz 9da7ff609c feat(ruby-sdk): add agent exchange options, threadId and mode (#4985)
* feat(ruby-sdk): add agent exchange options, threadId and mode

* fix(ruby-sdk): split terms and calls approval fixtures, release as 1.17.3

* test(ruby-sdk): keep the terms approval unresolved in the fixture
2026-10-07 00:43:38 +02:00
Gergő Móricz 3a5f89c5d2 feat(java): agent exchange, threadId and mode (#4984)
* feat(java): add exchange, threadId and mode to agent options

Adds AgentExchangeOptions (with Approve/Decline) plus threadId and mode to
AgentOptions, threadId/threadTurn to AgentResponse, and threadId, threadTurn,
mode, message, pendingApproval and exchange to AgentStatusResponse. Bumps the
SDK to 1.18.3.

* fix(java): require approvalId on Approve/Decline, null-guard README exchange example
2026-10-07 00:43:30 +02:00
Gergő Móricz 4b00a298c7 fix(python-sdk): expose get_agent_thread on Firecrawl and AsyncFirecrawl (#4983)
* fix(python-sdk): expose get_agent_thread on Firecrawl and AsyncFirecrawl

* test(python-sdk): assert the async thread read path
2026-10-07 00:43:02 +02:00
NicolasandCursor 204456d040 feat(api): 20-minute Alexandria feedback window (#4986)
* feat(api): 20-minute Alexandria feedback window

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(api): stub non-default windows in the Alexandria activity test

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
v2.11.472
2026-10-06 14:55:52 -07:00