opensource/sim
3
0
Fork 0
mirror of https://github.com/simstudioai/sim.git synced 2026-09-24 15:45:35 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(connectors): harden 10 KB connectors after audit (#4410)

Browse Source 
* fix(connectors): harden 10 KB connectors after audit

Validated each issue against provider docs before fixing.

- jira: migrate from deprecated /rest/api/3/search (Atlassian sunset
  May 2025) to /rest/api/3/search/jql with nextPageToken pagination
- confluence: unify stub hash across v1 CQL (`when`) and v2
  (`createdAt`) paths via shared pageToStub helper using version.number
- salesforce: replace hardcoded login.salesforce.com userinfo with
  host fallback so sandbox-issued tokens (test.salesforce.com) work
- servicenow: validate sys_id against /^[a-f0-9]{32}$/ and switch
  getDocument to path-based /api/now/table/{table}/{sys_id} to close
  encoded-query injection; reject `^` in kbCategory filter
- zendesk: URL-encode Search API query via URLSearchParams; whitelist
  ticket statuses; encode locale path segment
- github: add 10MB cap and /git/blobs/{sha} fallback for files >1MB
  that /contents/ returns with encoding:"none"
- slack: replace SHA-256 over formatted-message window with metadata
  hash slack:{channelId}:{latestTs}:{count} so list and getDocument
  agree; cache auth.test team_id on syncContext
- obsidian: drop syncRunId from stub hash (Local REST API has no
  HEAD/Last-Modified per OpenAPI spec); fall back to path-only stub
  so engine two-stage check short-circuits unchanged notes
- evernote: title fallback for attachments-only notes — breaks
  infinite hydration loop where empty plaintext returned null
- google-docs: drop residual `error instanceof Error` pattern

Confluence and Slack hash format changes self-heal with a one-time
re-sync; no data loss.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(connectors/slack): include oldestTs in hash to catch window-shift edits

Audit caught that the metadata hash slack:{channel}:{latestTs}:{count}
misses edits to the oldest message in the rolling window — count and
latestTs both stay constant. Adding oldestTs catches window-shift
when a new message arrives and pushes the oldest out without changing
the newest.

Also drop residual error instanceof Error pattern in validateConfig.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(connectors): address remaining audit findings

- jira: thread VALIDATE_RETRY_OPTIONS through getJiraCloudId so the
  validate path uses the tighter retry budget
- confluence: thread VALIDATE_RETRY_OPTIONS through getConfluenceCloudId;
  drop residual error instanceof Error pattern
- zendesk: log a warning when statusFilter is set and limit exceeds
  the Search API 1000-result cap; add priority to tagDefinitions and
  mapTags so the metadata isn't orphaned

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(connectors): address PR review comments

- github: skip blob fetch for empty 0-byte files
- salesforce: add fallback message for empty error
- zendesk: warn on invalid statusFilter instead of silent fallthrough
- evernote: remove dead content.trim() guard

* fix(zendesk): add fallback message for empty validateConfig error

* docs(obsidian): clarify hash strategy and re-hydration behavior

* fix(connectors): address parallel audit findings

- servicenow: case-insensitive sys_id pattern; validate workflowState/incidentState/incidentPriority as numeric, kbCategory via allowlist
- salesforce: include 400 in userinfo host fallthrough for sandbox tokens
- zendesk: add missing 'new' and 'hold' options to ticketStatus dropdown

* fix(servicenow): allow string workflowState values (published/draft/review/retired)

* fix(obsidian): validateConfig must check authenticated field, not just status

GET / is the only public endpoint per the Local REST API spec — it returns
200 regardless of auth and exposes the real signal via response.authenticated.
The prior status-based check let invalid API keys pass validation.

* fix(jira): encode collected count in cursor so maxIssues cap works without syncContext

* fix(connectors): address remaining audit findings

- slack: detect message edits and new threaded replies via edited.ts/latest_reply in contentHash
- github: skip binary files via NUL-byte sniff and filter oversized files at tree-list time
- servicenow: add 'outdated' workflow state to allowlist and dropdown; add Canceled (8) to incident state dropdown
- salesforce: drop 400 from userinfo host fallthrough (Salesforce returns 401/403 for cross-env tokens)

* fix(github): apply binary check to blob fallback path

fetchBlobContent now runs isBinaryBuffer on base64-decoded data and
returns null for binaries; getDocument propagates the null so 1-10MB
binary files are skipped instead of being decoded into garbled UTF-8.

* fix(confluence): reuse pageToStub in getDocument to keep hashes in sync

Replace duplicated versionKey/contentHash logic with a single call to
pageToStub so listDocuments and getDocument can never drift.

* fix(connectors): address audit findings across 7 connectors

- github: branch with slashes encoded per-segment in sourceUrl; treat 403
  Contents API responses (>100MB or restricted) as null instead of
  throwing; fetchBlobContent throws on unexpected encoding rather than
  returning empty content
- jira: ADF extractor handles hardBreak, mention, and emoji nodes
- google-docs: paragraph join uses newline so heading/body boundaries
  are preserved
- servicenow: drop legacy 'wiki' field from kb_knowledge requests; only
  the 'text' field is reliably present on modern instances
- salesforce: KnowledgeArticleVersion query adds IsLatestVersion=true to
  avoid duplicate historical versions; encode externalId in sObject GET
- slack: subtype filter switches to a denylist (SLACK_NOISE_SUBTYPES) so
  thread_broadcast / me_message / reminder_add are kept; reply_count
  folded into contentHash to detect threaded reply deletes
- confluence: drop bare-string cursor fallback; unparseable cursors now
  restart the listing instead of silently re-listing blogposts from 0

* fix(servicenow): allow non-ASCII KB category names

Switch from a \w-based allowlist (ASCII only) to a denylist that
rejects only encoded-query-meaningful characters (^, control chars,
quotes). International category names like 'Général' or 'Ação' are
now accepted.

* fix(servicenow): restore wiki field fallback for kb_knowledge

Wiki-template KB articles populate the wiki field and leave text empty;
HTML-template articles do the opposite. Removing wiki caused wiki-format
articles to resolve to empty content and be silently skipped.

* fix(connectors): jira pagination signal and google-docs paragraph join

- jira: rely solely on absence of nextPageToken for end-of-results;
  data.isLast on /rest/api/3/search/jql is unreliable (JRACLOUD-95477)
  and the OR-with-isLast logic could truncate pagination early
- google-docs: strip trailing newline from each paragraph before
  joining so heading->body produces a single newline, not two

* fix(jira): rely solely on nextPageToken for end-of-results

data.isLast on /rest/api/3/search/jql is unreliable (JRACLOUD-95477)
and the previous OR-with-isLast logic could truncate pagination early
if Jira ever returned isLast=true alongside a valid nextPageToken.

* fix(slack): exclude DM channel IDs from direct lookup

DM (D...) channel IDs require im:*/mpim:* OAuth scopes, which the
connector does not request. The regex now matches only public (C) and
private (G) channel IDs to fail fast with a clearer error instead of
hitting missing_scope from Slack.

* fix(slack): align validateConfig channel ID regex with resolveChannel

The previous fix narrowed the regex in resolveChannel to [CG] but missed
the duplicate in validateConfig, which still admitted DM IDs and would
fall through to a name-based search returning 'Channel not found.'

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Waleed
2026-05-02 17:53:57 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 31cfb74dc2
commit 64642d4f65
11 changed files with 631 additions and 204 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/sim/connectors/confluence/confluence.ts
+57 -57
View File
@@ -79,34 +79,56 @@ async function fetchLabelsForPages(
return labelsByPageId
}
/**
* Produces a canonical metadata stub with a deterministic contentHash that
* does not depend on which API surface (v1 CQL or v2) returned the page.
*/
function pageToStub(
page: Record<string, unknown>,
options: {
spaceId?: unknown
labels?: string[]
sourceUrl?: string
} = {}
): ExternalDocument {
const version = page.version as Record<string, unknown> | undefined
const versionNumber = version?.number as number | undefined
const lastModified = (version?.createdAt ?? version?.when ?? '') as string
const versionKey = versionNumber ?? lastModified
return {
externalId: String(page.id),
title: (page.title as string) || 'Untitled',
content: '',
contentDeferred: true,
mimeType: 'text/plain',
sourceUrl: options.sourceUrl,
contentHash: `confluence:${page.id}:${versionKey}`,
metadata: {
spaceId: options.spaceId,
status: page.status,
version: versionNumber,
labels: options.labels ?? [],
lastModified,
},
}
}
/**
* Converts a v1 CQL search result item to a lightweight metadata stub.
*/
function cqlResultToStub(item: Record<string, unknown>, domain: string): ExternalDocument {
const version = item.version as Record<string, unknown> | undefined
const links = item._links as Record<string, string> | undefined
const metadata = item.metadata as Record<string, unknown> | undefined
const labelsWrapper = metadata?.labels as Record<string, unknown> | undefined
const labelResults = (labelsWrapper?.results || []) as Record<string, unknown>[]
const labels = labelResults.map((l) => l.name as string)
const versionNumber = version?.number
return {
externalId: String(item.id),
title: (item.title as string) || 'Untitled',
content: '',
contentDeferred: true,
mimeType: 'text/plain',
return pageToStub(item, {
spaceId: (item.space as Record<string, unknown>)?.key,
labels,
sourceUrl: links?.webui ? `https://${domain}/wiki${links.webui}` : undefined,
contentHash: `confluence:${item.id}:${versionNumber ?? ''}`,
metadata: {
spaceId: (item.space as Record<string, unknown>)?.key,
status: item.status,
version: versionNumber,
labels,
lastModified: version?.when,
},
}
})
}
export const confluenceConnector: ConnectorConfig = {
@@ -285,24 +307,16 @@ export const confluenceConnector: ConnectorConfig = {
const labels = labelMap.get(String(page.id)) ?? []
const links = page._links as Record<string, unknown> | undefined
const version = page.version as Record<string, unknown> | undefined
const versionNumber = version?.number
const stub = pageToStub(page, {
spaceId: page.spaceId,
labels,
sourceUrl: links?.webui ? `https://${domain}/wiki${links.webui}` : undefined,
})
return {
externalId: String(page.id),
title: (page.title as string) || 'Untitled',
...stub,
content: plainText,
contentDeferred: false,
mimeType: 'text/plain',
sourceUrl: links?.webui ? `https://${domain}/wiki${links.webui}` : undefined,
contentHash: `confluence:${page.id}:${versionNumber ?? ''}`,
metadata: {
spaceId: page.spaceId,
status: page.status,
version: versionNumber,
labels,
lastModified: version?.createdAt,
},
}
},
@@ -323,7 +337,7 @@ export const confluenceConnector: ConnectorConfig = {
}
try {
const cloudId = await getConfluenceCloudId(domain, accessToken)
const cloudId = await getConfluenceCloudId(domain, accessToken, VALIDATE_RETRY_OPTIONS)
const spaceUrl = `https://api.atlassian.com/ex/confluence/${cloudId}/wiki/api/v2/spaces?keys=${encodeURIComponent(spaceKey)}&limit=1`
const response = await fetchWithRetry(
spaceUrl,
@@ -345,8 +359,7 @@ export const confluenceConnector: ConnectorConfig = {
}
return { valid: true }
} catch (error) {
const message = error instanceof Error ? error.message : 'Failed to validate configuration'
return { valid: false, error: message }
return { valid: false, error: toError(error).message || 'Failed to validate configuration' }
}
},
@@ -420,28 +433,11 @@ async function listDocumentsV2(
const results = data.results || []
const documents: ExternalDocument[] = results.map((page: Record<string, unknown>) => {
const pageId = String(page.id)
const version = page.version as Record<string, unknown> | undefined
const versionNumber = version?.number
return {
externalId: pageId,
title: (page.title as string) || 'Untitled',
content: '',
contentDeferred: true,
mimeType: 'text/plain',
sourceUrl: (page._links as Record<string, string>)?.webui
? `https://${domain}/wiki${(page._links as Record<string, string>).webui}`
: undefined,
contentHash: `confluence:${pageId}:${versionNumber ?? ''}`,
metadata: {
spaceId: page.spaceId,
status: page.status,
version: versionNumber,
labels: [],
lastModified: version?.createdAt,
},
}
const links = page._links as Record<string, string> | undefined
return pageToStub(page, {
spaceId: page.spaceId,
sourceUrl: links?.webui ? `https://${domain}/wiki${links.webui}` : undefined,
})
})
let nextCursor: string | undefined
@@ -493,7 +489,11 @@ async function listAllContentTypes(
pagesDone = parsed.pagesDone === true
blogsDone = parsed.blogsDone === true
} catch {
pageCursor = cursor
/**
* Older bare-string cursors are no longer emitted; fall through and
* restart instead of silently re-listing blogposts from page 0.
*/
logger.warn('Ignoring unparseable Confluence cursor; restarting listing')
}
}
apps/sim/connectors/evernote/evernote.ts
+5 -4
View File
@@ -462,7 +462,8 @@ export const evernoteConnector: ConnectorConfig = {
const retryOptions = { maxRetries: 3, initialDelayMs: 500 }
const note = await apiGetNote(accessToken, externalId, retryOptions)
const plainText = htmlToPlainText(note.content)
if (!plainText.trim()) return null
const title = note.title || 'Untitled'
const content = plainText.trim() ? plainText : title
const shardId = extractShardId(accessToken)
const userId = extractUserId(accessToken)
@@ -494,8 +495,8 @@ export const evernoteConnector: ConnectorConfig = {
return {
externalId,
title: note.title || 'Untitled',
content: plainText,
title,
content,
contentDeferred: false,
mimeType: 'text/plain',
sourceUrl: `https://${host}/shard/${shardId}/nl/${userId}/${externalId}/`,
@@ -539,7 +540,7 @@ export const evernoteConnector: ConnectorConfig = {
return { valid: true }
} catch (error) {
const message = error instanceof Error ? error.message : 'Failed to connect to Evernote'
const message = toError(error).message || 'Failed to connect to Evernote'
return { valid: false, error: message }
}
},
apps/sim/connectors/github/github.ts
+98 -7
View File
@@ -10,6 +10,20 @@ const logger = createLogger('GitHubConnector')
const GITHUB_API_URL = 'https://api.github.com'
const BATCH_SIZE = 30
const GIT_SHA_PREFIX = 'git-sha:'
const MAX_FILE_SIZE = 10 * 1024 * 1024 // 10 MB
const BINARY_SNIFF_BYTES = 8000
/**
* Heuristic binary detection: Git treats files containing a NUL byte in the
* first 8000 bytes as binary. Matches `git diff` / `git grep` semantics.
*/
function isBinaryBuffer(buf: Buffer): boolean {
const len = Math.min(buf.length, BINARY_SNIFF_BYTES)
for (let i = 0; i < len; i++) {
if (buf[i] === 0) return true
}
return false
}
/**
* Parses the repository string into owner and repo.
@@ -90,6 +104,48 @@ async function fetchTree(
return (data.tree || []).filter((item: TreeItem) => item.type === 'blob')
}
/**
* Fetches blob content via the Git Blobs API. Used as a fallback when the
* `/contents/` endpoint cannot return the file body (files larger than 1 MB
* return `content: ""` and `encoding: "none"`). Supports blobs up to 100 MB.
*/
async function fetchBlobContent(
accessToken: string,
owner: string,
repo: string,
sha: string
): Promise<string | null> {
const url = `${GITHUB_API_URL}/repos/${owner}/${repo}/git/blobs/${encodeURIComponent(sha)}`
const response = await fetchWithRetry(url, {
method: 'GET',
headers: {
Accept: 'application/vnd.github+json',
Authorization: `Bearer ${accessToken}`,
'X-GitHub-Api-Version': '2022-11-28',
},
})
if (!response.ok) {
throw new Error(`Failed to fetch git blob ${sha}: ${response.status}`)
}
const data = await response.json()
const content = (data.content as string) || ''
const encoding = data.encoding as string | undefined
if (encoding === 'base64') {
const buf = Buffer.from(content, 'base64')
if (isBinaryBuffer(buf)) return null
return buf.toString('utf8')
}
/**
* Per https://docs.github.com/en/rest/git/blobs the Blobs API only ever
* returns base64. Refuse to silently persist empty content for an
* unexpected encoding so a sync surfaces the error instead.
*/
throw new Error(`Unexpected git blob encoding for ${sha}: ${encoding ?? 'undefined'}`)
}
/**
* Creates a lightweight stub ExternalDocument from a tree item.
* Uses the Git blob SHA as contentHash for change detection, avoiding
@@ -108,7 +164,7 @@ function treeItemToStub(
content: '',
contentDeferred: true,
mimeType: 'text/plain',
sourceUrl: `https://github.com/${owner}/${repo}/blob/${encodeURIComponent(branch)}/${item.path.split('/').map(encodeURIComponent).join('/')}`,
sourceUrl: `https://github.com/${owner}/${repo}/blob/${branch.split('/').map(encodeURIComponent).join('/')}/${item.path.split('/').map(encodeURIComponent).join('/')}`,
contentHash: `${GIT_SHA_PREFIX}${item.sha}`,
metadata: {
path: item.path,
@@ -189,10 +245,11 @@ export const githubConnector: ConnectorConfig = {
} else {
const tree = await fetchTree(accessToken, owner, repo, branch)
// Filter by path prefix and extensions
// Filter by path prefix, extensions, and size
const filtered = tree.filter((item) => {
if (pathPrefix && !item.path.startsWith(pathPrefix)) return false
if (!matchesExtension(item.path, extSet)) return false
if (typeof item.size === 'number' && item.size > MAX_FILE_SIZE) return false
return true
})
@@ -252,15 +309,49 @@ export const githubConnector: ConnectorConfig = {
if (!response.ok) {
if (response.status === 404) return null
if (response.status === 403) {
logger.info('Skipping GitHub file rejected by Contents API', {
path,
status: response.status,
})
return null
}
throw new Error(`Failed to fetch file ${path}: ${response.status}`)
}
const lastModifiedHeader = response.headers.get('last-modified') || undefined
const data = await response.json()
const content =
data.encoding === 'base64'
? Buffer.from(data.content as string, 'base64').toString('utf-8')
: (data.content as string) || ''
const size = typeof data.size === 'number' ? data.size : 0
if (size > MAX_FILE_SIZE) {
logger.info('Skipping GitHub file exceeding size limit', {
path,
size,
limit: MAX_FILE_SIZE,
})
return null
}
const rawContent = (data.content as string) || ''
const encoding = data.encoding as string | undefined
let content: string
if (encoding === 'base64' && rawContent.length > 0) {
const buf = Buffer.from(rawContent, 'base64')
if (isBinaryBuffer(buf)) {
logger.info('Skipping binary GitHub file', { path, size })
return null
}
content = buf.toString('utf8')
} else if (encoding === 'none' && data.sha && size > 0) {
const blobContent = await fetchBlobContent(accessToken, owner, repo, data.sha as string)
if (blobContent === null) {
logger.info('Skipping binary GitHub file', { path, size })
return null
}
content = blobContent
} else {
content = ''
}
return {
externalId,
@@ -268,7 +359,7 @@ export const githubConnector: ConnectorConfig = {
content,
contentDeferred: false,
mimeType: 'text/plain',
sourceUrl: `https://github.com/${owner}/${repo}/blob/${encodeURIComponent(branch)}/${path.split('/').map(encodeURIComponent).join('/')}`,
sourceUrl: `https://github.com/${owner}/${repo}/blob/${branch.split('/').map(encodeURIComponent).join('/')}/${path.split('/').map(encodeURIComponent).join('/')}`,
contentHash: `${GIT_SHA_PREFIX}${data.sha as string}`,
metadata: {
path,
apps/sim/connectors/google-docs/google-docs.ts
+11 -4
View File
@@ -84,14 +84,22 @@ function extractTextFromDocsBody(doc: DocsDocument): string {
if (!paragraph?.elements) continue
const prefix = headingPrefix(paragraph.paragraphStyle?.namedStyleType)
const text = paragraph.elements.map((el) => el.textRun?.content ?? '').join('')
/**
* Each paragraph's final `textRun.content` already ends with `\n`. Strip
* it before joining with `\n` so a heading followed by a body paragraph
* is separated by a single newline, not two.
*/
const text = paragraph.elements
.map((el) => el.textRun?.content ?? '')
.join('')
.replace(/\n+$/, '')
if (text.trim()) {
parts.push(`${prefix}${text}`)
}
}
return parts.join('').trim()
return parts.join('\n').trim()
}
/**
@@ -349,8 +357,7 @@ export const googleDocsConnector: ConnectorConfig = {
return { valid: true }
} catch (error) {
const message = error instanceof Error ? error.message : 'Failed to validate configuration'
return { valid: false, error: message }
return { valid: false, error: toError(error).message || 'Failed to validate configuration' }
}
},
apps/sim/connectors/jira/jira.ts
+52 -20
View File
@@ -1,4 +1,5 @@
import { createLogger } from '@sim/logger'
import { toError } from '@sim/utils/errors'
import { JiraIcon } from '@/components/icons'
import { fetchWithRetry, VALIDATE_RETRY_OPTIONS } from '@/lib/knowledge/documents/utils'
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
@@ -164,24 +165,42 @@ export const jiraConnector: ConnectorConfig = {
jql = `project = "${safeKey}" AND (${jqlFilter.trim()}) ORDER BY updated DESC`
}
const startAt = cursor ? Number(cursor) : 0
/**
* Collected-count is encoded in the cursor as `${pageToken}|${count}` so
* the maxIssues cap works correctly even when the caller doesn't pass
* syncContext. Falls back to syncContext.collectedCount for backwards
* compatibility with cursors emitted before this format existed.
*/
let pageToken: string | undefined
let collectedSoFar = (syncContext?.collectedCount as number | undefined) ?? 0
if (cursor) {
const sep = cursor.lastIndexOf('|')
if (sep > 0) {
pageToken = cursor.slice(0, sep)
const parsed = Number(cursor.slice(sep + 1))
if (Number.isFinite(parsed) && parsed >= 0) collectedSoFar = parsed
} else {
pageToken = cursor
}
}
const remaining = maxIssues > 0 ? Math.max(0, maxIssues - collectedSoFar) : PAGE_SIZE
if (maxIssues > 0 && remaining === 0) {
return { documents: [], hasMore: false }
}
const params = new URLSearchParams()
params.append('jql', jql)
params.append('startAt', String(startAt))
const remaining = maxIssues > 0 ? Math.max(0, maxIssues - startAt) : PAGE_SIZE
if (remaining === 0) {
return { documents: [], hasMore: false }
}
params.append('maxResults', String(Math.min(PAGE_SIZE, remaining)))
params.append(
'fields',
'summary,issuetype,status,priority,assignee,reporter,project,labels,created,updated'
)
if (pageToken) params.append('nextPageToken', pageToken)
const url = `https://api.atlassian.com/ex/jira/${cloudId}/rest/api/3/search?${params.toString()}`
const url = `https://api.atlassian.com/ex/jira/${cloudId}/rest/api/3/search/jql?${params.toString()}`
logger.info(`Listing Jira issues for project ${projectKey}`, { startAt })
logger.info(`Listing Jira issues for project ${projectKey}`, { hasCursor: Boolean(cursor) })
const response = await fetchWithRetry(url, {
method: 'GET',
@@ -201,17 +220,31 @@ export const jiraConnector: ConnectorConfig = {
}
const data = await response.json()
const issues = (data.issues || []) as Record<string, unknown>[]
const total = (data.total as number) ?? 0
let issues = (data.issues || []) as Record<string, unknown>[]
/**
* `/rest/api/3/search/jql` signals end-of-results purely by the absence
* of `nextPageToken`. `data.isLast` is unreliable on this endpoint and
* has been observed returning `true` alongside a valid token
* (JRACLOUD-95477), so we ignore it.
*/
const nextPageToken = data.nextPageToken as string | undefined
const isLast = !nextPageToken
if (maxIssues > 0 && issues.length > remaining) {
issues = issues.slice(0, remaining)
}
const documents: ExternalDocument[] = issues.map((issue) => issueToStub(issue, domain))
const nextStart = startAt + issues.length
const hasMore = nextStart < total && (maxIssues <= 0 || nextStart < maxIssues)
const newCollected = collectedSoFar + issues.length
if (syncContext) syncContext.collectedCount = newCollected
const reachedCap = maxIssues > 0 && newCollected >= maxIssues
const hasMore = !isLast && !reachedCap
return {
documents,
nextCursor: hasMore ? String(nextStart) : undefined,
nextCursor: hasMore && nextPageToken ? `${nextPageToken}|${newCollected}` : undefined,
hasMore,
}
},
@@ -273,14 +306,14 @@ export const jiraConnector: ConnectorConfig = {
const jqlFilter = (sourceConfig.jql as string | undefined)?.trim() || ''
try {
const cloudId = await getJiraCloudId(domain, accessToken)
const cloudId = await getJiraCloudId(domain, accessToken, VALIDATE_RETRY_OPTIONS)
const params = new URLSearchParams()
const safeKey = projectKey.replace(/\\/g, '\\\\').replace(/"/g, '\\"')
params.append('jql', `project = "${safeKey}"`)
params.append('maxResults', '0')
params.append('maxResults', '1')
const url = `https://api.atlassian.com/ex/jira/${cloudId}/rest/api/3/search?${params.toString()}`
const url = `https://api.atlassian.com/ex/jira/${cloudId}/rest/api/3/search/jql?${params.toString()}`
const response = await fetchWithRetry(
url,
{
@@ -304,9 +337,9 @@ export const jiraConnector: ConnectorConfig = {
if (jqlFilter) {
const filterParams = new URLSearchParams()
filterParams.append('jql', `project = "${safeKey}" AND (${jqlFilter})`)
filterParams.append('maxResults', '0')
filterParams.append('maxResults', '1')
const filterUrl = `https://api.atlassian.com/ex/jira/${cloudId}/rest/api/3/search?${filterParams.toString()}`
const filterUrl = `https://api.atlassian.com/ex/jira/${cloudId}/rest/api/3/search/jql?${filterParams.toString()}`
const filterResponse = await fetchWithRetry(
filterUrl,
{
@@ -326,8 +359,7 @@ export const jiraConnector: ConnectorConfig = {
return { valid: true }
} catch (error) {
const message = error instanceof Error ? error.message : 'Failed to validate configuration'
return { valid: false, error: message }
return { valid: false, error: toError(error).message || 'Failed to validate configuration' }
}
},
apps/sim/connectors/obsidian/obsidian.ts
+21 -8
View File
@@ -215,8 +215,15 @@ export const obsidianConnector: ConnectorConfig = {
const offset = cursor ? Number(cursor) : 0
const pageFiles = allFiles.slice(offset, offset + DOCS_PER_PAGE)
const syncRunId = (syncContext?.syncRunId as string) ?? ''
/**
* The Obsidian Local REST API directory listing returns just
* `{ files: string[] }` — no `stat`/`mtime` and no `HEAD` support to read
* `Last-Modified`, so the stub cannot encode change-detection state. Every
* file is therefore re-hydrated via `getDocument` on every sync. The
* post-hydration hash compare in the sync engine
* (`existing.contentHash === hydratedHash`) prevents redundant DB writes
* when `mtime` is unchanged.
*/
const documents: ExternalDocument[] = pageFiles.map((filePath) => ({
externalId: filePath,
title: titleFromPath(filePath),
@@ -224,7 +231,7 @@ export const obsidianConnector: ConnectorConfig = {
contentDeferred: true,
mimeType: 'text/plain' as const,
sourceUrl: `${baseUrl}/vault/${filePath.split('/').map(encodeURIComponent).join('/')}`,
contentHash: `obsidian:stub:${filePath}:${syncRunId}`,
contentHash: `obsidian:${filePath}`,
metadata: {
folder: filePath.includes('/') ? filePath.substring(0, filePath.lastIndexOf('/')) : '',
},
@@ -306,17 +313,23 @@ export const obsidianConnector: ConnectorConfig = {
VALIDATE_RETRY_OPTIONS
)
if (response.status === 401 || response.status === 403) {
if (!response.ok) {
return { valid: false, error: `Obsidian API returned status ${response.status}` }
}
/**
* `GET /` is the only public endpoint and returns 200 regardless of auth;
* the response body's `authenticated` field is the actual auth signal.
* See https://coddingtonbear.github.io/obsidian-local-rest-api/.
*/
const data = (await response.json()) as { authenticated?: boolean }
if (!data?.authenticated) {
return {
valid: false,
error: 'Invalid API key — check your Obsidian Local REST API settings',
}
}
if (!response.ok) {
return { valid: false, error: `Obsidian API returned status ${response.status}` }
}
const folderPath = (sourceConfig.folderPath as string) || ''
if (folderPath.trim()) {
const entries = await listDirectory(
apps/sim/connectors/salesforce/salesforce.ts
+88 -36
View File
@@ -1,4 +1,5 @@
import { createLogger } from '@sim/logger'
import { toError } from '@sim/utils/errors'
import { SalesforceIcon } from '@/components/icons'
import { fetchWithRetry, VALIDATE_RETRY_OPTIONS } from '@/lib/knowledge/documents/utils'
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
@@ -6,7 +7,13 @@ import { htmlToPlainText, parseTagDate } from '@/connectors/utils'
const logger = createLogger('SalesforceConnector')
const USERINFO_URL = 'https://login.salesforce.com/services/oauth2/userinfo'
/**
* Salesforce serves the userinfo endpoint at the org's authentication host.
* Tokens issued at test.salesforce.com (sandbox) are rejected at login.salesforce.com,
* so we try each host in order and cache the working one in syncContext.
*/
const USERINFO_HOSTS = ['https://login.salesforce.com', 'https://test.salesforce.com'] as const
const USERINFO_PATH = '/services/oauth2/userinfo'
const API_VERSION = 'v62.0'
const PAGE_SIZE = 200
@@ -35,9 +42,75 @@ const OBJECT_FIELDS: Record<string, string[]> = {
/** SOQL WHERE clause additions per object type. */
const OBJECT_WHERE: Record<string, string> = {
KnowledgeArticleVersion: " WHERE PublishStatus='Online' AND Language='en_US'",
KnowledgeArticleVersion:
" WHERE PublishStatus='Online' AND IsLatestVersion=true AND Language='en_US'",
} as const
/**
* Result of a userinfo lookup: either the parsed payload + the auth host that
* served it, or a structured failure describing the last response we saw.
*/
type UserinfoResult =
| { ok: true; data: Record<string, unknown>; host: string }
| { ok: false; status: number | undefined; errorText: string }
/**
* Fetches the Salesforce userinfo payload, trying each candidate auth host in
* order. Sandbox-issued tokens are rejected at login.salesforce.com with 401/403,
* so on those statuses we fall through to test.salesforce.com. The working host
* is cached in syncContext under `_salesforceInstanceUrl` so subsequent calls in
* the same sync run skip the fallback dance.
*/
async function fetchUserinfo(
accessToken: string,
retryOptions?: Parameters<typeof fetchWithRetry>[2],
syncContext?: Record<string, unknown>
): Promise<UserinfoResult> {
const cachedHost =
typeof syncContext?._salesforceInstanceUrl === 'string'
? (syncContext._salesforceInstanceUrl as string)
: undefined
const orderedHosts = cachedHost
? [cachedHost, ...USERINFO_HOSTS.filter((h) => h !== cachedHost)]
: [...USERINFO_HOSTS]
let lastStatus: number | undefined
let lastErrorText = ''
for (const host of orderedHosts) {
const response = await fetchWithRetry(
`${host}${USERINFO_PATH}`,
{
method: 'GET',
headers: {
Accept: 'application/json',
Authorization: `Bearer ${accessToken}`,
},
},
retryOptions
)
if (response.ok) {
const data = (await response.json()) as Record<string, unknown>
if (syncContext) {
syncContext._salesforceInstanceUrl = host
}
return { ok: true, data, host }
}
lastStatus = response.status
lastErrorText = await response.text()
// Only fall through to the next host on auth-shaped failures; surface
// other errors (e.g. 5xx) immediately so we don't mask real problems.
if (response.status !== 401 && response.status !== 403) {
break
}
}
return { ok: false, status: lastStatus, errorText: lastErrorText }
}
/**
* Resolves the Salesforce instance REST URL from the userinfo endpoint.
* Caches the result in syncContext to avoid repeated calls.
@@ -50,21 +123,14 @@ async function resolveInstanceUrl(
return syncContext.instanceUrl as string
}
const response = await fetchWithRetry(USERINFO_URL, {
method: 'GET',
headers: {
Accept: 'application/json',
Authorization: `Bearer ${accessToken}`,
},
})
if (!response.ok) {
const errorText = await response.text()
throw new Error(`Failed to resolve Salesforce instance URL: ${response.status} - ${errorText}`)
const result = await fetchUserinfo(accessToken, undefined, syncContext)
if (!result.ok) {
throw new Error(
`Failed to resolve Salesforce instance URL: ${result.status ?? 'unknown'} - ${result.errorText}`
)
}
const data = await response.json()
const urls = data.urls as Record<string, string> | undefined
const urls = result.data.urls as Record<string, string> | undefined
let restUrl = urls?.rest
if (!restUrl) {
@@ -321,11 +387,10 @@ export const salesforceConnector: ConnectorConfig = {
let instanceUrl = syncContext?.instanceUrl as string | undefined
if (!instanceUrl) {
instanceUrl = await resolveInstanceUrl(accessToken)
if (syncContext) syncContext.instanceUrl = instanceUrl
instanceUrl = await resolveInstanceUrl(accessToken, syncContext)
}
const url = `${instanceUrl}sobjects/${objectType}/${externalId}?fields=${fields.join(',')}`
const url = `${instanceUrl}sobjects/${objectType}/${encodeURIComponent(externalId)}?fields=${fields.join(',')}`
const response = await fetchWithRetry(url, {
method: 'GET',
@@ -372,28 +437,16 @@ export const salesforceConnector: ConnectorConfig = {
}
try {
const userinfoResponse = await fetchWithRetry(
USERINFO_URL,
{
method: 'GET',
headers: {
Accept: 'application/json',
Authorization: `Bearer ${accessToken}`,
},
},
VALIDATE_RETRY_OPTIONS
)
const userinfoResult = await fetchUserinfo(accessToken, VALIDATE_RETRY_OPTIONS)
if (!userinfoResponse.ok) {
const errorText = await userinfoResponse.text()
if (!userinfoResult.ok) {
return {
valid: false,
error: `Failed to authenticate with Salesforce: ${userinfoResponse.status} - ${errorText}`,
error: `Failed to authenticate with Salesforce: ${userinfoResult.status ?? 'unknown'} - ${userinfoResult.errorText}`,
}
}
const userinfo = await userinfoResponse.json()
const urls = userinfo.urls as Record<string, string> | undefined
const urls = userinfoResult.data.urls as Record<string, string> | undefined
let restUrl = urls?.rest
if (!restUrl) {
@@ -427,8 +480,7 @@ export const salesforceConnector: ConnectorConfig = {
return { valid: true }
} catch (error) {
const message = error instanceof Error ? error.message : 'Failed to validate configuration'
return { valid: false, error: message }
return { valid: false, error: toError(error).message || 'Failed to validate configuration' }
}
},
apps/sim/connectors/servicenow/servicenow.ts
+108 -14
View File
@@ -11,6 +11,26 @@ const logger = createLogger('ServiceNowConnector')
const DEFAULT_MAX_ITEMS = 500
const PAGE_SIZE = 100
/**
* ServiceNow sys_id whitelist: 32-character lowercase hex strings.
*
* The encoded query language uses `^` as the AND separator and `^OR` as the
* OR separator with no escape syntax, so any user-supplied value interpolated
* into a `sysparm_query` clause must be validated up front. Path-based
* fetches (`/api/now/table/{table}/{sys_id}`) likewise treat the sys_id as a
* URL path segment and must be constrained to safe characters.
*/
const SYS_ID_PATTERN = /^[a-f0-9]{32}$/i
const NUMERIC_ID_PATTERN = /^\d+$/
/**
* Reject characters that have meaning in a ServiceNow encoded query
* (`^` is the operator separator; control chars and quotes can break the
* URL). All other Unicode characters — including accented letters used in
* categories like "Général" or "Ação" — are allowed.
*/
const KB_CATEGORY_DISALLOWED = /[\^"'`\u0000-\u001f\u007f]/
const VALID_WORKFLOW_STATES = new Set(['published', 'draft', 'review', 'retired', 'outdated'])
interface ServiceNowRecord {
sys_id: string
sys_updated_on?: string
@@ -123,6 +143,50 @@ async function serviceNowApiGet(
}
}
/**
* Fetches a single ServiceNow record by sys_id via the path-based Table API
* endpoint (`GET /api/now/table/{tableName}/{sys_id}`), which returns a
* `{ result: <record> }` object rather than the array shape returned by the
* list endpoint. Returns `null` when the record is not found (404).
*/
async function serviceNowApiGetById(
instanceUrl: string,
tableName: string,
sysId: string,
authHeader: string,
params: Record<string, string>,
retryOptions?: Parameters<typeof fetchWithRetry>[2]
): Promise<Record<string, unknown> | null> {
const queryParams = new URLSearchParams(params)
const queryString = queryParams.toString()
const url = `${instanceUrl}/api/now/table/${tableName}/${sysId}${queryString ? `?${queryString}` : ''}`
const response = await fetchWithRetry(
url,
{
method: 'GET',
headers: {
Authorization: authHeader,
Accept: 'application/json',
'Content-Type': 'application/json',
},
},
retryOptions
)
if (response.status === 404) {
return null
}
if (!response.ok) {
const errorText = await response.text().catch(() => 'Unknown error')
throw new Error(`ServiceNow API error (${response.status}): ${errorText}`)
}
const data = (await response.json()) as { result?: Record<string, unknown> }
return data.result ?? null
}
function isServiceNowRecord(record: unknown): record is ServiceNowRecord & Record<string, unknown> {
return (
typeof record === 'object' &&
@@ -206,6 +270,11 @@ function priorityLabel(priority: string | undefined): string {
*/
function kbArticleToDocument(article: KBArticle, instanceUrl: string): ExternalDocument {
const title = rawValue(article.short_description) || rawValue(article.number) || article.sys_id
/**
* Wiki-template KB articles populate `wiki` with the body and leave
* `text` empty; HTML-template articles do the opposite. Falling back
* to `wiki` keeps both layouts indexable.
*/
const articleText = rawValue(article.text) || rawValue(article.wiki) || ''
const content = htmlToPlainText(articleText)
const sysId = rawValue(article.sys_id) || article.sys_id
@@ -308,12 +377,25 @@ function buildKBQuery(sourceConfig: Record<string, unknown>): string {
const workflowState = sourceConfig.workflowState as string | undefined
if (workflowState && workflowState !== 'all') {
parts.push(`workflow_state=${workflowState}`)
if (VALID_WORKFLOW_STATES.has(workflowState)) {
parts.push(`workflow_state=${workflowState}`)
} else {
logger.warn('Skipping workflowState filter: value is not in the allowed set', {
workflowState,
})
}
}
const kbCategory = sourceConfig.kbCategory as string | undefined
if (kbCategory?.trim()) {
parts.push(`kb_category.label=${kbCategory.trim()}`)
const trimmedCategory = kbCategory?.trim()
if (trimmedCategory) {
if (!KB_CATEGORY_DISALLOWED.test(trimmedCategory)) {
parts.push(`kb_category.label=${trimmedCategory}`)
} else {
logger.warn('Skipping kbCategory filter: value contains disallowed characters', {
kbCategory: trimmedCategory,
})
}
}
parts.push('ORDERBYDESCsys_updated_on')
@@ -328,12 +410,22 @@ function buildIncidentQuery(sourceConfig: Record<string, unknown>): string {
const incidentState = sourceConfig.incidentState as string | undefined
if (incidentState && incidentState !== 'all') {
parts.push(`state=${incidentState}`)
if (NUMERIC_ID_PATTERN.test(incidentState)) {
parts.push(`state=${incidentState}`)
} else {
logger.warn('Skipping incidentState filter: value is not a numeric ID', { incidentState })
}
}
const incidentPriority = sourceConfig.incidentPriority as string | undefined
if (incidentPriority && incidentPriority !== 'all') {
parts.push(`priority=${incidentPriority}`)
if (NUMERIC_ID_PATTERN.test(incidentPriority)) {
parts.push(`priority=${incidentPriority}`)
} else {
logger.warn('Skipping incidentPriority filter: value is not a numeric ID', {
incidentPriority,
})
}
}
parts.push('ORDERBYDESCsys_updated_on')
@@ -393,6 +485,7 @@ export const servicenowConnector: ConnectorConfig = {
{ label: 'Draft', id: 'draft' },
{ label: 'Review', id: 'review' },
{ label: 'Retired', id: 'retired' },
{ label: 'Outdated', id: 'outdated' },
],
},
{
@@ -416,6 +509,7 @@ export const servicenowConnector: ConnectorConfig = {
{ label: 'On Hold', id: '3' },
{ label: 'Resolved', id: '6' },
{ label: 'Closed', id: '7' },
{ label: 'Canceled', id: '8' },
],
},
{
@@ -533,6 +627,14 @@ export const servicenowConnector: ConnectorConfig = {
const isKB = contentType === 'kb_knowledge'
const tableName = isKB ? 'kb_knowledge' : 'incident'
if (!SYS_ID_PATTERN.test(externalId)) {
logger.warn('Rejecting ServiceNow getDocument with invalid sys_id', {
externalId,
table: tableName,
})
return null
}
const fields = isKB
? 'sys_id,short_description,text,wiki,workflow_state,kb_category,kb_knowledge_base,number,author,sys_created_by,sys_updated_by,sys_updated_on,sys_created_on'
: 'sys_id,number,short_description,description,state,priority,category,assigned_to,opened_by,close_notes,resolution_notes,sys_created_by,sys_updated_by,sys_updated_on,sys_created_on'
@@ -540,19 +642,11 @@ export const servicenowConnector: ConnectorConfig = {
const instanceUrl = resolveServiceNowInstanceUrl(sourceConfig.instanceUrl as string)
try {
const { result } = await serviceNowApiGet(instanceUrl, tableName, authHeader, {
sysparm_query: `sys_id=${externalId}`,
sysparm_limit: '1',
sysparm_offset: '0',
const record = await serviceNowApiGetById(instanceUrl, tableName, externalId, authHeader, {
sysparm_fields: fields,
sysparm_display_value: 'all',
})
if (!result || result.length === 0) {
return null
}
const record = result[0]
if (!record || !isServiceNowRecord(record)) {
return null
}
apps/sim/connectors/slack/slack.ts
+152 -48
View File
@@ -3,7 +3,7 @@ import { toError } from '@sim/utils/errors'
import { SlackIcon } from '@/components/icons'
import { fetchWithRetry, VALIDATE_RETRY_OPTIONS } from '@/lib/knowledge/documents/utils'
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
import { computeContentHash, parseTagDate } from '@/connectors/utils'
import { parseTagDate } from '@/connectors/utils'
const logger = createLogger('SlackConnector')
@@ -11,12 +11,42 @@ const SLACK_API_BASE = 'https://slack.com/api'
const DEFAULT_MAX_MESSAGES = 1000
const MESSAGES_PER_PAGE = 200
/**
* Message subtypes that carry no user-authored text (channel events, bot
* lifecycle, etc.). Per https://api.slack.com/events/message every other
* subtype — `bot_message`, `file_share`, `me_message`, `thread_broadcast`,
* `reminder_add`, `file_comment`, etc. — can carry meaningful content.
*/
const SLACK_NOISE_SUBTYPES = new Set([
'channel_join',
'channel_leave',
'channel_topic',
'channel_purpose',
'channel_name',
'channel_archive',
'channel_unarchive',
'group_join',
'group_leave',
'group_topic',
'group_purpose',
'group_name',
'group_archive',
'group_unarchive',
'pinned_item',
'unpinned_item',
'bot_add',
'bot_remove',
])
interface SlackMessage {
type: string
user?: string
text?: string
ts: string
subtype?: string
edited?: { ts: string; user?: string }
latest_reply?: string
reply_count?: number
}
interface SlackChannel {
@@ -130,7 +160,7 @@ async function fetchChannelMessages(
accessToken: string,
channelId: string,
maxMessages: number
): Promise<{ messages: SlackMessage[]; lastActivityTs?: string }> {
): Promise<{ messages: SlackMessage[]; lastActivityTs?: string; oldestTs?: string }> {
const allMessages: SlackMessage[] = []
let cursor: string | undefined
let lastActivityTs: string | undefined
@@ -162,7 +192,9 @@ async function fetchChannelMessages(
cursor = nextCursor
}
return { messages: allMessages.slice(0, maxMessages), lastActivityTs }
const trimmed = allMessages.slice(0, maxMessages)
const oldestTs = trimmed.length > 0 ? trimmed[trimmed.length - 1].ts : undefined
return { messages: trimmed, lastActivityTs, oldestTs }
}
/**
@@ -182,7 +214,13 @@ async function formatMessages(
for (const msg of chronological) {
// Skip non-user messages (join/leave, bot messages without text, etc.)
if (!msg.text) continue
if (msg.subtype && msg.subtype !== 'bot_message' && msg.subtype !== 'file_share') continue
/**
* Drop only known noise subtypes (channel join/leave/topic events,
* bot add/remove, etc.). Per https://api.slack.com/events/message any
* subtype with user-authored text — `thread_broadcast`, `me_message`,
* `bot_message`, `file_share`, `reminder_add`, etc. — should be kept.
*/
if (msg.subtype && SLACK_NOISE_SUBTYPES.has(msg.subtype)) continue
const timestamp = formatSlackTimestamp(msg.ts)
const userName = msg.user
@@ -204,8 +242,9 @@ async function resolveChannel(
): Promise<SlackChannel | null> {
const trimmed = channelInput.trim().replace(/^#/, '')
// If it looks like a channel ID (starts with C, D, or G), try direct lookup
if (/^[CDG][A-Z0-9]+$/.test(trimmed)) {
// If it looks like a channel ID (public C / private G), try direct lookup.
// DMs (D...) and MPIMs require im:*/mpim:* scopes, which we do not request.
if (/^[CG][A-Z0-9]+$/.test(trimmed)) {
try {
const data = await slackApiGet('conversations.info', accessToken, { channel: trimmed })
return data.channel as SlackChannel
@@ -239,6 +278,93 @@ async function resolveChannel(
return null
}
/**
* Resolves the Slack team ID for the current token, caching the result on
* `syncContext._slackTeamId` to avoid repeated `auth.test` calls. The team ID
* is stable per token, so caching for the lifetime of a sync is safe.
*/
async function resolveTeamId(
accessToken: string,
syncContext?: Record<string, unknown>
): Promise<string | undefined> {
const cacheKey = '_slackTeamId'
if (syncContext && typeof syncContext[cacheKey] === 'string') {
return syncContext[cacheKey] as string
}
try {
const authData = await slackApiGet('auth.test', accessToken, {})
const teamId = authData.team_id as string | undefined
if (teamId && syncContext) {
syncContext[cacheKey] = teamId
}
return teamId
} catch (error) {
logger.warn('Failed to resolve Slack team ID', {
error: toError(error).message,
})
return undefined
}
}
/**
* Builds a channel document payload shared by `listDocuments` and `getDocument`.
*
* The `contentHash` is derived from stable Slack metadata — channel ID, the
* newest message `ts`, and the message count — rather than the formatted text.
* This keeps the hash deterministic across calls even though the formatted
* content depends on the user-name cache state and the sliding message window.
*
* Each Slack message has a unique, stable `ts` per channel
* (https://api.slack.com/methods/conversations.history), so `lastActivityTs`
* uniquely identifies the newest message included in the document.
*/
async function buildSlackChannelDocument(
accessToken: string,
channel: SlackChannel,
maxMessages: number,
syncContext?: Record<string, unknown>
): Promise<{
content: string
contentHash: string
messageCount: number
lastActivityTs?: string
}> {
const { messages, lastActivityTs, oldestTs } = await fetchChannelMessages(
accessToken,
channel.id,
maxMessages
)
const content = await formatMessages(accessToken, messages, syncContext)
const messageCount = messages.length
/**
* Edit/thread fingerprint: max(edited.ts) and max(latest_reply) across the
* window. `ts` is immutable for messages, so without these signals an
* in-place edit (chat.update) or a new threaded reply would not change the
* channel hash. Slack returns `edited.ts` only when a message was edited
* and `latest_reply` only when threaded replies exist.
*/
let maxEditTs = ''
let maxReplyTs = ''
let totalReplies = 0
for (const m of messages) {
if (m.edited?.ts && m.edited.ts > maxEditTs) maxEditTs = m.edited.ts
if (m.latest_reply && m.latest_reply > maxReplyTs) maxReplyTs = m.latest_reply
if (typeof m.reply_count === 'number') totalReplies += m.reply_count
}
/**
* `latest_reply` alone misses reply edits and deletes. Folding `reply_count`
* in catches deletes (count drops) but still cannot detect reply edits
* without fetching `conversations.replies` for each parent.
*/
const contentHash = `slack:${channel.id}:${oldestTs ?? 'empty'}:${lastActivityTs ?? 'empty'}:${messageCount}:${maxEditTs || 'noedit'}:${maxReplyTs || 'noreply'}:${totalReplies}`
return { content, contentHash, messageCount, lastActivityTs }
}
export const slackConnector: ConnectorConfig = {
id: 'slack',
name: 'Slack',
@@ -311,31 +437,21 @@ export const slackConnector: ConnectorConfig = {
throw new Error(`Channel not found: ${channelInput}`)
}
const { messages, lastActivityTs } = await fetchChannelMessages(
const { content, contentHash, messageCount, lastActivityTs } = await buildSlackChannelDocument(
accessToken,
channel.id,
maxMessages
channel,
maxMessages,
syncContext
)
const content = await formatMessages(accessToken, messages, syncContext)
if (!content.trim()) {
logger.info(`No messages found in channel: #${channel.name}`)
return { documents: [], hasMore: false }
}
const contentHash = await computeContentHash(content)
// Attempt to get team ID for the source URL
let sourceUrl = `https://app.slack.com/client/${channel.id}`
try {
const authData = await slackApiGet('auth.test', accessToken, {})
const teamId = authData.team_id as string | undefined
if (teamId) {
sourceUrl = `https://app.slack.com/client/${teamId}/${channel.id}`
}
} catch {
// Fall back to URL without team ID
}
const teamId = await resolveTeamId(accessToken, syncContext)
const sourceUrl = teamId
? `https://app.slack.com/client/${teamId}/${channel.id}`
: `https://app.slack.com/client/${channel.id}`
const document: ExternalDocument = {
externalId: channel.id,
@@ -346,7 +462,7 @@ export const slackConnector: ConnectorConfig = {
contentHash,
metadata: {
channelName: channel.name,
messageCount: messages.length,
messageCount,
lastActivity: lastActivityTs ? formatSlackTimestamp(lastActivityTs) : undefined,
topic: channel.topic?.value,
purpose: channel.purpose?.value,
@@ -374,27 +490,14 @@ export const slackConnector: ConnectorConfig = {
const data = await slackApiGet('conversations.info', accessToken, { channel: externalId })
const channel = data.channel as SlackChannel
const { messages, lastActivityTs } = await fetchChannelMessages(
accessToken,
externalId,
maxMessages
)
const content = await formatMessages(accessToken, messages, syncContext)
const { content, contentHash, messageCount, lastActivityTs } =
await buildSlackChannelDocument(accessToken, channel, maxMessages, syncContext)
if (!content.trim()) return null
const contentHash = await computeContentHash(content)
let sourceUrl = `https://app.slack.com/client/${channel.id}`
try {
const authData = await slackApiGet('auth.test', accessToken, {})
const teamId = authData.team_id as string | undefined
if (teamId) {
sourceUrl = `https://app.slack.com/client/${teamId}/${channel.id}`
}
} catch {
// Fall back to URL without team ID
}
const teamId = await resolveTeamId(accessToken, syncContext)
const sourceUrl = teamId
? `https://app.slack.com/client/${teamId}/${channel.id}`
: `https://app.slack.com/client/${channel.id}`
return {
externalId: channel.id,
@@ -405,7 +508,7 @@ export const slackConnector: ConnectorConfig = {
contentHash,
metadata: {
channelName: channel.name,
messageCount: messages.length,
messageCount,
lastActivity: lastActivityTs ? formatSlackTimestamp(lastActivityTs) : undefined,
topic: channel.topic?.value,
purpose: channel.purpose?.value,
@@ -438,8 +541,9 @@ export const slackConnector: ConnectorConfig = {
try {
const trimmed = channelInput.trim().replace(/^#/, '')
// If it looks like a channel ID, verify directly
if (/^[CDG][A-Z0-9]+$/.test(trimmed)) {
// If it looks like a channel ID, verify directly. DMs (D...) are excluded
// because we don't request im:*/mpim:* scopes — see resolveChannel.
if (/^[CG][A-Z0-9]+$/.test(trimmed)) {
await slackApiGet(
'conversations.info',
accessToken,
@@ -478,7 +582,7 @@ export const slackConnector: ConnectorConfig = {
return { valid: false, error: `Channel not found: ${channelInput}` }
} catch (error) {
const message = error instanceof Error ? error.message : 'Failed to validate configuration'
const message = toError(error).message || 'Failed to validate configuration'
return { valid: false, error: message }
}
},
apps/sim/connectors/zendesk/zendesk.ts
+28 -4
View File
@@ -10,6 +10,9 @@ const logger = createLogger('ZendeskConnector')
const ARTICLES_PER_PAGE = 30
const TICKETS_PER_PAGE = 100
const DEFAULT_MAX_TICKETS = 500
const SEARCH_API_RESULT_CAP = 1000
const VALID_TICKET_STATUSES = new Set(['new', 'open', 'pending', 'hold', 'solved', 'closed'])
interface ZendeskArticle {
id: number
@@ -97,7 +100,7 @@ async function fetchArticles(
): Promise<ZendeskArticle[]> {
const allArticles: ZendeskArticle[] = []
const baseUrl = buildBaseUrl(subdomain)
const localePath = locale ? `/${locale}` : ''
const localePath = locale ? `/${encodeURIComponent(locale)}` : ''
let page = 1
while (true) {
@@ -132,7 +135,22 @@ async function fetchTickets(
let url: string | null = `${baseUrl}/api/v2/tickets.json?per_page=${TICKETS_PER_PAGE}`
if (statusFilter && statusFilter !== 'all') {
url = `${baseUrl}/api/v2/search.json?query=type:ticket status:${statusFilter}&per_page=${TICKETS_PER_PAGE}`
if (VALID_TICKET_STATUSES.has(statusFilter)) {
if (limit > SEARCH_API_RESULT_CAP) {
logger.warn(
`Zendesk Search API caps at ${SEARCH_API_RESULT_CAP} results; requested limit ${limit} will be truncated. Remove status filter to use the unbounded tickets endpoint.`
)
}
const params = new URLSearchParams({
query: `type:ticket status:${statusFilter}`,
per_page: String(TICKETS_PER_PAGE),
})
url = `${baseUrl}/api/v2/search.json?${params.toString()}`
} else {
logger.warn(
`Invalid Zendesk statusFilter "${statusFilter}"; falling back to all tickets. Valid values: ${[...VALID_TICKET_STATUSES].join(', ')}.`
)
}
}
while (url && allTickets.length < limit) {
@@ -344,8 +362,10 @@ export const zendeskConnector: ConnectorConfig = {
description: 'Filter tickets by status (applies only when syncing tickets)',
options: [
{ label: 'All Statuses', id: 'all' },
{ label: 'New', id: 'new' },
{ label: 'Open', id: 'open' },
{ label: 'Pending', id: 'pending' },
{ label: 'On Hold', id: 'hold' },
{ label: 'Solved', id: 'solved' },
{ label: 'Closed', id: 'closed' },
],
@@ -496,14 +516,14 @@ export const zendeskConnector: ConnectorConfig = {
await zendeskApiGet(url, accessToken, sourceConfig, VALIDATE_RETRY_OPTIONS)
return { valid: true }
} catch (error) {
const message = error instanceof Error ? error.message : 'Failed to validate configuration'
return { valid: false, error: message }
return { valid: false, error: toError(error).message || 'Failed to validate configuration' }
}
},
tagDefinitions: [
{ id: 'contentType', displayName: 'Content Type', fieldType: 'text' },
{ id: 'status', displayName: 'Status', fieldType: 'text' },
{ id: 'priority', displayName: 'Priority', fieldType: 'text' },
{ id: 'labels', displayName: 'Labels', fieldType: 'text' },
{ id: 'tags', displayName: 'Tags', fieldType: 'text' },
{ id: 'updatedAt', displayName: 'Last Updated', fieldType: 'date' },
@@ -521,6 +541,10 @@ export const zendeskConnector: ConnectorConfig = {
result.status = metadata.status
}
if (typeof metadata.priority === 'string') {
result.priority = metadata.priority
}
const labels = joinTagArray(metadata.labels)
if (labels) {
result.labels = labels
apps/sim/tools/jira/utils.ts
+11 -2
View File
@@ -1,4 +1,5 @@
import { createLogger } from '@sim/logger'
import type { RetryOptions } from '@/lib/knowledge/documents/utils'
import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
const logger = createLogger('JiraUtils')
@@ -61,6 +62,9 @@ export function extractAdfText(content: any): string | null {
return content.map(extractAdfText).filter(Boolean).join(' ')
}
if (content.type === 'text') return content.text || ''
if (content.type === 'hardBreak') return '\n'
if (content.type === 'mention') return content.attrs?.text || ''
if (content.type === 'emoji') return content.attrs?.shortName || content.attrs?.text || ''
if (content.content) return extractAdfText(content.content)
return ''
}
@@ -163,7 +167,11 @@ export function normalizeDomain(domain: string): string {
.replace(/\/+$/, '')}`.toLowerCase()
}
export async function getJiraCloudId(domain: string, accessToken: string): Promise<string> {
export async function getJiraCloudId(
domain: string,
accessToken: string,
retryOptions?: RetryOptions
): Promise<string> {
const response = await fetchWithRetry(
'https://api.atlassian.com/oauth/token/accessible-resources',
{
@@ -172,7 +180,8 @@ export async function getJiraCloudId(domain: string, accessToken: string): Promi
Authorization: `Bearer ${accessToken}`,
Accept: 'application/json',
},
}
},
retryOptions
)
if (!response.ok) {