mirror of
https://github.com/simstudioai/sim.git
synced 2026-10-25 17:42:51 +08:00
feat(knowledge): add v1 knowledge base API, Obsidian/Evernote connectors, and docs (#3465)
* feat(knowledge): add v1 knowledge base API, Obsidian/Evernote connectors, and docs
- Add v1 REST API for knowledge bases (CRUD, document management, vector search)
- Add Obsidian and Evernote knowledge base connectors
- Add file type validation to v1 file and document upload endpoints
- Update OpenAPI spec with knowledge base endpoints and schemas
- Add connectors documentation page
- Apply query hook formatting improvements
* fix(knowledge): address PR review feedback
- Remove validateFileType from v1/files route (general file upload, not document-only)
- Reject tag filters when searching multiple KBs (tag defs are KB-specific)
- Cache tag definitions to avoid duplicate getDocumentTagDefinitions call
- Fix Obsidian connector silent empty results when syncContext is undefined
* improvement(connectors): add syncContext to getDocument, clean up caching
- Update docs to say 20+ connectors
- Add syncContext param to ConnectorConfig.getDocument interface
- Use syncContext in Evernote getDocument to cache tag/notebook maps
- Replace index-based cache check with Map keyed by KB ID in search route
* fix(knowledge): address second round of PR review feedback
- Fix Zod .default('text') overriding tag definition's actual fieldType
- Fix encodeURIComponent breaking multi-level folder paths in Obsidian
- Use 413 instead of 400 for file-too-large in document upload
- Add knowledge-bases to API reference docs navigation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(knowledge): prevent cross-workspace KB access in search
Filter accessible KBs by matching workspaceId from the request,
preventing users from querying KBs in other workspaces they have
access to but didn't specify.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(knowledge): audit resourceId, SSRF protection, recursion depth limit
- Fix recordAudit using knowledgeBaseId instead of newDocument.id
- Add SSRF validation to Obsidian connector (reject private/loopback URLs)
- Add max recursion depth (20) to listVaultFiles
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(obsidian): remove SSRF check that blocks localhost usage
The Obsidian connector is designed to connect to the Local REST API
plugin running on localhost (127.0.0.1:27124). The SSRF check was
incorrectly blocking this primary use case.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Waleed
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
13d49da8bd
commit
76486ebcc8
@@ -13,6 +13,7 @@
|
||||
"(generated)/usage",
|
||||
"(generated)/audit-logs",
|
||||
"(generated)/tables",
|
||||
"(generated)/files"
|
||||
"(generated)/files",
|
||||
"(generated)/knowledge-bases"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: Connectors
|
||||
description: Automatically sync documents from external sources into your knowledge base
|
||||
---
|
||||
|
||||
import { Callout } from 'fumadocs-ui/components/callout'
|
||||
import { Step, Steps } from 'fumadocs-ui/components/steps'
|
||||
|
||||
Connectors let you pull documents directly from external services into your knowledge base. Instead of manually uploading files, a connector continuously syncs content from sources like Notion, Google Drive, GitHub, Slack, and more — keeping your knowledge base up to date automatically.
|
||||
|
||||
## Available Connectors
|
||||
|
||||
Sim ships with 20+ built-in connectors spanning productivity tools, cloud storage, development platforms, and more.
|
||||
|
||||
| Category | Connectors |
|
||||
|----------|-----------|
|
||||
| **Productivity** | Notion, Confluence, Asana, Linear, Jira |
|
||||
| **Cloud Storage** | Google Drive, Dropbox, OneDrive, SharePoint |
|
||||
| **Documents** | Google Docs, WordPress, Webflow |
|
||||
| **Development** | GitHub |
|
||||
| **Communication** | Slack |
|
||||
| **CRM** | HubSpot, Salesforce |
|
||||
| **Data** | Airtable |
|
||||
| **Note-taking** | Evernote, Obsidian |
|
||||
| **Meetings** | Fireflies |
|
||||
|
||||
## Adding a Connector
|
||||
|
||||
<Steps>
|
||||
<Step>
|
||||
|
||||
### Select a source
|
||||
|
||||
Open a knowledge base and click **Add Connector**. You'll see the full list of available connectors — pick the service you want to sync from.
|
||||
|
||||
</Step>
|
||||
<Step>
|
||||
|
||||
### Authenticate
|
||||
|
||||
Most connectors use **OAuth** — select an existing credential from the dropdown, or click **Connect new account** to authorize through the service's login flow. Tokens are refreshed automatically, so you won't need to re-authenticate unless you revoke access.
|
||||
|
||||
A few connectors (Evernote, Obsidian, Fireflies) use **API keys** instead. Paste your key or developer token directly, and it will be stored securely.
|
||||
|
||||
<Callout type="info">
|
||||
If you rotate an API key in the external service, you'll need to update it in Sim as well. OAuth tokens are refreshed automatically, but API keys are not.
|
||||
</Callout>
|
||||
|
||||
</Step>
|
||||
<Step>
|
||||
|
||||
### Configure
|
||||
|
||||
Each connector has its own configuration fields that control what gets synced. Some examples:
|
||||
|
||||
- **Notion**: Choose between syncing an entire workspace, a specific database, or a single page tree
|
||||
- **GitHub**: Specify a repository, branch, and optional file extension filter
|
||||
- **Confluence**: Enter your Atlassian domain and optionally filter by space key or content type
|
||||
- **Obsidian**: Provide your vault URL and optionally restrict to a folder path
|
||||
|
||||
All configuration is validated when you save — if a repository doesn't exist or a domain is unreachable, you'll get an immediate error.
|
||||
|
||||
</Step>
|
||||
<Step>
|
||||
|
||||
### Choose sync frequency
|
||||
|
||||
Select how often the connector should re-sync:
|
||||
|
||||
| Frequency | Description |
|
||||
|-----------|-------------|
|
||||
| Every hour | Best for fast-moving sources |
|
||||
| Every 6 hours | Good balance for most use cases |
|
||||
| **Daily** (default) | Suitable for content that changes infrequently |
|
||||
| Weekly | For stable, rarely-updated sources |
|
||||
| Manual only | Sync only when you trigger it |
|
||||
|
||||
</Step>
|
||||
<Step>
|
||||
|
||||
### Configure metadata tags (optional)
|
||||
|
||||
If the connector supports metadata tags, you'll see checkboxes for each tag type (e.g., Labels, Last Modified, Notebook). All are enabled by default — uncheck any you don't need.
|
||||
|
||||
See the [Metadata Tags](#metadata-tags) section below for details.
|
||||
|
||||
</Step>
|
||||
<Step>
|
||||
|
||||
### Connect & Sync
|
||||
|
||||
Click **Connect & Sync** to save the connector and trigger the first sync immediately. Documents will begin appearing in your knowledge base as they are processed.
|
||||
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## How Syncing Works
|
||||
|
||||
On each sync, the connector fetches documents from the external service and compares them against what's already in your knowledge base. Only documents that have actually changed are reprocessed — new content is added, updated content is re-chunked and re-embedded, and documents that no longer exist in the source are removed.
|
||||
|
||||
This means syncing is efficient even for large document sets. A connector with thousands of documents will only do meaningful work when something changes.
|
||||
|
||||
### Handling Failures
|
||||
|
||||
If a single document fails to fetch (e.g., due to a permission issue or timeout), the sync continues with the remaining documents. The failed document will be retried on the next sync cycle.
|
||||
|
||||
If an entire sync fails (e.g., the service is down or credentials expired), the connector automatically backs off and retries later. The backoff resets as soon as a sync succeeds.
|
||||
|
||||
## Metadata Tags
|
||||
|
||||
Connectors can automatically populate [tags](/docs/knowledgebase/tags) with metadata from the source, letting you filter documents in the Knowledge block based on information from the external service.
|
||||
|
||||
For example, a Notion connector might tag documents with their **Labels**, **Last Modified** date, and **Created** date. A GitHub connector might tag documents with their **Repository** and **File Path**. This metadata becomes available for [tag-based filtering](/docs/knowledgebase/tags) in your workflows.
|
||||
|
||||
### Opting Out
|
||||
|
||||
You can disable specific metadata tags during connector setup. Disabled tags won't be populated, leaving those tag slots available for other connectors or manual tagging.
|
||||
|
||||
<Callout type="info">
|
||||
Tag slots are shared across all documents in a knowledge base. If you have multiple connectors, each one's metadata tags draw from the same pool of available slots.
|
||||
</Callout>
|
||||
|
||||
## Excluding Documents
|
||||
|
||||
You can manually exclude specific documents from a connector's sync. Excluded documents are skipped on every subsequent sync, even if they change in the source. This is useful for filtering out templates, drafts, or other content you don't want in your knowledge base.
|
||||
|
||||
## Source Links
|
||||
|
||||
Every synced document retains a link back to the original in the external service. This lets you trace any knowledge base document to its source — whether that's a Notion page, a GitHub file, a Confluence article, or a Slack conversation.
|
||||
|
||||
## Multiple Connectors
|
||||
|
||||
You can add multiple connectors to a single knowledge base. For example, you might sync internal documentation from Confluence alongside code from GitHub and meeting notes from Fireflies — all searchable together through the Knowledge block.
|
||||
|
||||
Each connector manages its own documents independently. Metadata tag slots are shared across the knowledge base, so keep an eye on slot usage if you're combining several connectors that each populate tags.
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
- **Internal knowledge base**: Sync your team's Notion workspace and Confluence spaces so AI agents can answer questions about internal processes, policies, and documentation
|
||||
- **Customer support**: Connect HubSpot or Salesforce alongside your help docs from WordPress or Google Docs to give support agents full context on customers and product information
|
||||
- **Engineering assistant**: Sync a GitHub repository and Jira or Linear issues so an AI agent can reference code, specs, and ticket history when answering developer questions
|
||||
- **Meeting intelligence**: Pull in Fireflies transcripts alongside Slack conversations to build a searchable archive of decisions and discussions
|
||||
- **Research and notes**: Sync Evernote notebooks or an Obsidian vault to make your personal notes available to AI workflows
|
||||
@@ -1,4 +1,4 @@
|
||||
{
|
||||
"title": "Knowledgebase",
|
||||
"pages": ["index", "tags"]
|
||||
"pages": ["index", "connectors", "tags"]
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,177 @@
|
||||
import { db } from '@sim/db'
|
||||
import { document } from '@sim/db/schema'
|
||||
import { and, eq, isNull } from 'drizzle-orm'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { z } from 'zod'
|
||||
import { AuditAction, AuditResourceType, recordAudit } from '@/lib/audit/log'
|
||||
import { deleteDocument } from '@/lib/knowledge/documents/service'
|
||||
import {
|
||||
authenticateRequest,
|
||||
handleError,
|
||||
resolveKnowledgeBase,
|
||||
serializeDate,
|
||||
validateSchema,
|
||||
} from '@/app/api/v1/knowledge/utils'
|
||||
|
||||
export const dynamic = 'force-dynamic'
|
||||
export const revalidate = 0
|
||||
|
||||
interface DocumentDetailRouteParams {
|
||||
params: Promise<{ id: string; documentId: string }>
|
||||
}
|
||||
|
||||
const WorkspaceIdSchema = z.object({
|
||||
workspaceId: z.string().min(1, 'workspaceId query parameter is required'),
|
||||
})
|
||||
|
||||
/** GET /api/v1/knowledge/[id]/documents/[documentId] — Get document details. */
|
||||
export async function GET(request: NextRequest, { params }: DocumentDetailRouteParams) {
|
||||
const auth = await authenticateRequest(request, 'knowledge-detail')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const { id: knowledgeBaseId, documentId } = await params
|
||||
const { searchParams } = new URL(request.url)
|
||||
|
||||
const validation = validateSchema(WorkspaceIdSchema, {
|
||||
workspaceId: searchParams.get('workspaceId'),
|
||||
})
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const result = await resolveKnowledgeBase(
|
||||
knowledgeBaseId,
|
||||
validation.data.workspaceId,
|
||||
userId,
|
||||
rateLimit
|
||||
)
|
||||
if (result instanceof NextResponse) return result
|
||||
|
||||
const docs = await db
|
||||
.select({
|
||||
id: document.id,
|
||||
knowledgeBaseId: document.knowledgeBaseId,
|
||||
filename: document.filename,
|
||||
fileSize: document.fileSize,
|
||||
mimeType: document.mimeType,
|
||||
processingStatus: document.processingStatus,
|
||||
processingError: document.processingError,
|
||||
processingStartedAt: document.processingStartedAt,
|
||||
processingCompletedAt: document.processingCompletedAt,
|
||||
chunkCount: document.chunkCount,
|
||||
tokenCount: document.tokenCount,
|
||||
characterCount: document.characterCount,
|
||||
enabled: document.enabled,
|
||||
uploadedAt: document.uploadedAt,
|
||||
connectorId: document.connectorId,
|
||||
connectorType: document.connectorType,
|
||||
sourceUrl: document.sourceUrl,
|
||||
})
|
||||
.from(document)
|
||||
.where(
|
||||
and(
|
||||
eq(document.id, documentId),
|
||||
eq(document.knowledgeBaseId, knowledgeBaseId),
|
||||
isNull(document.deletedAt)
|
||||
)
|
||||
)
|
||||
.limit(1)
|
||||
|
||||
if (docs.length === 0) {
|
||||
return NextResponse.json({ error: 'Document not found' }, { status: 404 })
|
||||
}
|
||||
|
||||
const doc = docs[0]
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
document: {
|
||||
id: doc.id,
|
||||
knowledgeBaseId: doc.knowledgeBaseId,
|
||||
filename: doc.filename,
|
||||
fileSize: doc.fileSize,
|
||||
mimeType: doc.mimeType,
|
||||
processingStatus: doc.processingStatus,
|
||||
processingError: doc.processingError,
|
||||
processingStartedAt: serializeDate(doc.processingStartedAt),
|
||||
processingCompletedAt: serializeDate(doc.processingCompletedAt),
|
||||
chunkCount: doc.chunkCount,
|
||||
tokenCount: doc.tokenCount,
|
||||
characterCount: doc.characterCount,
|
||||
enabled: doc.enabled,
|
||||
connectorId: doc.connectorId,
|
||||
connectorType: doc.connectorType,
|
||||
sourceUrl: doc.sourceUrl,
|
||||
createdAt: serializeDate(doc.uploadedAt),
|
||||
},
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to get document')
|
||||
}
|
||||
}
|
||||
|
||||
/** DELETE /api/v1/knowledge/[id]/documents/[documentId] — Delete a document. */
|
||||
export async function DELETE(request: NextRequest, { params }: DocumentDetailRouteParams) {
|
||||
const auth = await authenticateRequest(request, 'knowledge-detail')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const { id: knowledgeBaseId, documentId } = await params
|
||||
const { searchParams } = new URL(request.url)
|
||||
|
||||
const validation = validateSchema(WorkspaceIdSchema, {
|
||||
workspaceId: searchParams.get('workspaceId'),
|
||||
})
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const result = await resolveKnowledgeBase(
|
||||
knowledgeBaseId,
|
||||
validation.data.workspaceId,
|
||||
userId,
|
||||
rateLimit,
|
||||
'write'
|
||||
)
|
||||
if (result instanceof NextResponse) return result
|
||||
|
||||
const docs = await db
|
||||
.select({ id: document.id, filename: document.filename })
|
||||
.from(document)
|
||||
.where(
|
||||
and(
|
||||
eq(document.id, documentId),
|
||||
eq(document.knowledgeBaseId, knowledgeBaseId),
|
||||
isNull(document.deletedAt)
|
||||
)
|
||||
)
|
||||
.limit(1)
|
||||
|
||||
if (docs.length === 0) {
|
||||
return NextResponse.json({ error: 'Document not found' }, { status: 404 })
|
||||
}
|
||||
|
||||
await deleteDocument(documentId, requestId)
|
||||
|
||||
recordAudit({
|
||||
workspaceId: validation.data.workspaceId,
|
||||
actorId: userId,
|
||||
action: AuditAction.DOCUMENT_DELETED,
|
||||
resourceType: AuditResourceType.DOCUMENT,
|
||||
resourceId: documentId,
|
||||
resourceName: docs[0].filename,
|
||||
description: `Deleted document "${docs[0].filename}" from knowledge base via API`,
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
message: 'Document deleted successfully',
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to delete document')
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,248 @@
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { z } from 'zod'
|
||||
import { AuditAction, AuditResourceType, recordAudit } from '@/lib/audit/log'
|
||||
import {
|
||||
createSingleDocument,
|
||||
type DocumentData,
|
||||
getDocuments,
|
||||
processDocumentsWithQueue,
|
||||
} from '@/lib/knowledge/documents/service'
|
||||
import type { DocumentSortField, SortOrder } from '@/lib/knowledge/documents/types'
|
||||
import { uploadWorkspaceFile } from '@/lib/uploads/contexts/workspace'
|
||||
import { validateFileType } from '@/lib/uploads/utils/validation'
|
||||
import {
|
||||
authenticateRequest,
|
||||
handleError,
|
||||
resolveKnowledgeBase,
|
||||
serializeDate,
|
||||
validateSchema,
|
||||
} from '@/app/api/v1/knowledge/utils'
|
||||
|
||||
export const dynamic = 'force-dynamic'
|
||||
export const revalidate = 0
|
||||
|
||||
const MAX_FILE_SIZE = 100 * 1024 * 1024 // 100MB
|
||||
|
||||
interface DocumentsRouteParams {
|
||||
params: Promise<{ id: string }>
|
||||
}
|
||||
|
||||
const ListDocumentsSchema = z.object({
|
||||
workspaceId: z.string().min(1, 'workspaceId query parameter is required'),
|
||||
limit: z.coerce.number().int().min(1).max(100).default(50),
|
||||
offset: z.coerce.number().int().min(0).default(0),
|
||||
search: z.string().optional(),
|
||||
enabledFilter: z.enum(['all', 'enabled', 'disabled']).default('all'),
|
||||
sortBy: z
|
||||
.enum([
|
||||
'filename',
|
||||
'fileSize',
|
||||
'tokenCount',
|
||||
'chunkCount',
|
||||
'uploadedAt',
|
||||
'processingStatus',
|
||||
'enabled',
|
||||
])
|
||||
.default('uploadedAt'),
|
||||
sortOrder: z.enum(['asc', 'desc']).default('desc'),
|
||||
})
|
||||
|
||||
/** GET /api/v1/knowledge/[id]/documents — List documents in a knowledge base. */
|
||||
export async function GET(request: NextRequest, { params }: DocumentsRouteParams) {
|
||||
const auth = await authenticateRequest(request, 'knowledge-detail')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const { id: knowledgeBaseId } = await params
|
||||
const { searchParams } = new URL(request.url)
|
||||
|
||||
const validation = validateSchema(ListDocumentsSchema, {
|
||||
workspaceId: searchParams.get('workspaceId'),
|
||||
limit: searchParams.get('limit') ?? undefined,
|
||||
offset: searchParams.get('offset') ?? undefined,
|
||||
search: searchParams.get('search') ?? undefined,
|
||||
enabledFilter: searchParams.get('enabledFilter') ?? undefined,
|
||||
sortBy: searchParams.get('sortBy') ?? undefined,
|
||||
sortOrder: searchParams.get('sortOrder') ?? undefined,
|
||||
})
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const { workspaceId, limit, offset, search, enabledFilter, sortBy, sortOrder } = validation.data
|
||||
|
||||
const result = await resolveKnowledgeBase(knowledgeBaseId, workspaceId, userId, rateLimit)
|
||||
if (result instanceof NextResponse) return result
|
||||
|
||||
const documentsResult = await getDocuments(
|
||||
knowledgeBaseId,
|
||||
{
|
||||
enabledFilter: enabledFilter === 'all' ? undefined : enabledFilter,
|
||||
search,
|
||||
limit,
|
||||
offset,
|
||||
sortBy: sortBy as DocumentSortField,
|
||||
sortOrder: sortOrder as SortOrder,
|
||||
},
|
||||
requestId
|
||||
)
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
documents: documentsResult.documents.map((doc) => ({
|
||||
id: doc.id,
|
||||
knowledgeBaseId,
|
||||
filename: doc.filename,
|
||||
fileSize: doc.fileSize,
|
||||
mimeType: doc.mimeType,
|
||||
processingStatus: doc.processingStatus,
|
||||
chunkCount: doc.chunkCount,
|
||||
tokenCount: doc.tokenCount,
|
||||
characterCount: doc.characterCount,
|
||||
enabled: doc.enabled,
|
||||
createdAt: serializeDate(doc.uploadedAt),
|
||||
})),
|
||||
pagination: documentsResult.pagination,
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to list documents')
|
||||
}
|
||||
}
|
||||
|
||||
/** POST /api/v1/knowledge/[id]/documents — Upload a document to a knowledge base. */
|
||||
export async function POST(request: NextRequest, { params }: DocumentsRouteParams) {
|
||||
const auth = await authenticateRequest(request, 'knowledge-detail')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const { id: knowledgeBaseId } = await params
|
||||
|
||||
let formData: FormData
|
||||
try {
|
||||
formData = await request.formData()
|
||||
} catch {
|
||||
return NextResponse.json(
|
||||
{ error: 'Request body must be valid multipart form data' },
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
|
||||
const rawFile = formData.get('file')
|
||||
const file = rawFile instanceof File ? rawFile : null
|
||||
const rawWorkspaceId = formData.get('workspaceId')
|
||||
const workspaceId = typeof rawWorkspaceId === 'string' ? rawWorkspaceId : null
|
||||
|
||||
if (!workspaceId) {
|
||||
return NextResponse.json({ error: 'workspaceId form field is required' }, { status: 400 })
|
||||
}
|
||||
|
||||
if (!file) {
|
||||
return NextResponse.json({ error: 'file form field is required' }, { status: 400 })
|
||||
}
|
||||
|
||||
if (file.size > MAX_FILE_SIZE) {
|
||||
return NextResponse.json(
|
||||
{
|
||||
error: `File size exceeds 100MB limit (${(file.size / (1024 * 1024)).toFixed(2)}MB)`,
|
||||
},
|
||||
{ status: 413 }
|
||||
)
|
||||
}
|
||||
|
||||
const fileTypeError = validateFileType(file.name, file.type || '')
|
||||
if (fileTypeError) {
|
||||
return NextResponse.json({ error: fileTypeError.message }, { status: 415 })
|
||||
}
|
||||
|
||||
const result = await resolveKnowledgeBase(
|
||||
knowledgeBaseId,
|
||||
workspaceId,
|
||||
userId,
|
||||
rateLimit,
|
||||
'write'
|
||||
)
|
||||
if (result instanceof NextResponse) return result
|
||||
|
||||
const buffer = Buffer.from(await file.arrayBuffer())
|
||||
const contentType = file.type || 'application/octet-stream'
|
||||
|
||||
const uploadedFile = await uploadWorkspaceFile(
|
||||
workspaceId,
|
||||
userId,
|
||||
buffer,
|
||||
file.name,
|
||||
contentType
|
||||
)
|
||||
|
||||
const newDocument = await createSingleDocument(
|
||||
{
|
||||
filename: file.name,
|
||||
fileUrl: uploadedFile.url,
|
||||
fileSize: file.size,
|
||||
mimeType: contentType,
|
||||
},
|
||||
knowledgeBaseId,
|
||||
requestId
|
||||
)
|
||||
|
||||
const chunkingConfig = result.kb.chunkingConfig ?? { maxSize: 1024, minSize: 100, overlap: 200 }
|
||||
|
||||
const documentData: DocumentData = {
|
||||
documentId: newDocument.id,
|
||||
filename: file.name,
|
||||
fileUrl: uploadedFile.url,
|
||||
fileSize: file.size,
|
||||
mimeType: contentType,
|
||||
}
|
||||
|
||||
processDocumentsWithQueue(
|
||||
[documentData],
|
||||
knowledgeBaseId,
|
||||
{
|
||||
chunkSize: chunkingConfig.maxSize,
|
||||
minCharactersPerChunk: chunkingConfig.minSize,
|
||||
chunkOverlap: chunkingConfig.overlap,
|
||||
recipe: 'default',
|
||||
lang: 'en',
|
||||
},
|
||||
requestId
|
||||
).catch(() => {
|
||||
// Processing errors are logged internally
|
||||
})
|
||||
|
||||
recordAudit({
|
||||
workspaceId,
|
||||
actorId: userId,
|
||||
action: AuditAction.DOCUMENT_UPLOADED,
|
||||
resourceType: AuditResourceType.DOCUMENT,
|
||||
resourceId: newDocument.id,
|
||||
resourceName: file.name,
|
||||
description: `Uploaded document "${file.name}" to knowledge base via API`,
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
document: {
|
||||
id: newDocument.id,
|
||||
knowledgeBaseId,
|
||||
filename: newDocument.filename,
|
||||
fileSize: newDocument.fileSize,
|
||||
mimeType: newDocument.mimeType,
|
||||
processingStatus: 'pending',
|
||||
chunkCount: 0,
|
||||
tokenCount: 0,
|
||||
characterCount: 0,
|
||||
enabled: newDocument.enabled,
|
||||
createdAt: serializeDate(newDocument.uploadedAt),
|
||||
},
|
||||
message: 'Document uploaded successfully. Processing will begin shortly.',
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to upload document')
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,175 @@
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { z } from 'zod'
|
||||
import { AuditAction, AuditResourceType, recordAudit } from '@/lib/audit/log'
|
||||
import { deleteKnowledgeBase, updateKnowledgeBase } from '@/lib/knowledge/service'
|
||||
import {
|
||||
authenticateRequest,
|
||||
formatKnowledgeBase,
|
||||
handleError,
|
||||
parseJsonBody,
|
||||
resolveKnowledgeBase,
|
||||
validateSchema,
|
||||
} from '@/app/api/v1/knowledge/utils'
|
||||
|
||||
export const dynamic = 'force-dynamic'
|
||||
export const revalidate = 0
|
||||
|
||||
interface KnowledgeRouteParams {
|
||||
params: Promise<{ id: string }>
|
||||
}
|
||||
|
||||
const WorkspaceIdSchema = z.object({
|
||||
workspaceId: z.string().min(1, 'workspaceId query parameter is required'),
|
||||
})
|
||||
|
||||
const UpdateKBSchema = z
|
||||
.object({
|
||||
workspaceId: z.string().min(1, 'Workspace ID is required'),
|
||||
name: z.string().min(1).max(255, 'Name must be 255 characters or less').optional(),
|
||||
description: z.string().max(1000, 'Description must be 1000 characters or less').optional(),
|
||||
chunkingConfig: z
|
||||
.object({
|
||||
maxSize: z.number().min(100).max(4000),
|
||||
minSize: z.number().min(1).max(2000),
|
||||
overlap: z.number().min(0).max(500),
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
.refine(
|
||||
(data) =>
|
||||
data.name !== undefined ||
|
||||
data.description !== undefined ||
|
||||
data.chunkingConfig !== undefined,
|
||||
{ message: 'At least one of name, description, or chunkingConfig must be provided' }
|
||||
)
|
||||
|
||||
/** GET /api/v1/knowledge/[id] — Get knowledge base details. */
|
||||
export async function GET(request: NextRequest, { params }: KnowledgeRouteParams) {
|
||||
const auth = await authenticateRequest(request, 'knowledge-detail')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const { id } = await params
|
||||
const { searchParams } = new URL(request.url)
|
||||
|
||||
const validation = validateSchema(WorkspaceIdSchema, {
|
||||
workspaceId: searchParams.get('workspaceId'),
|
||||
})
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const result = await resolveKnowledgeBase(id, validation.data.workspaceId, userId, rateLimit)
|
||||
if (result instanceof NextResponse) return result
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
knowledgeBase: formatKnowledgeBase(result.kb),
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to get knowledge base')
|
||||
}
|
||||
}
|
||||
|
||||
/** PUT /api/v1/knowledge/[id] — Update a knowledge base. */
|
||||
export async function PUT(request: NextRequest, { params }: KnowledgeRouteParams) {
|
||||
const auth = await authenticateRequest(request, 'knowledge-detail')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const { id } = await params
|
||||
|
||||
const body = await parseJsonBody(request)
|
||||
if (!body.success) return body.response
|
||||
|
||||
const validation = validateSchema(UpdateKBSchema, body.data)
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const { workspaceId, name, description, chunkingConfig } = validation.data
|
||||
|
||||
const result = await resolveKnowledgeBase(id, workspaceId, userId, rateLimit, 'write')
|
||||
if (result instanceof NextResponse) return result
|
||||
|
||||
const updates: {
|
||||
name?: string
|
||||
description?: string
|
||||
chunkingConfig?: { maxSize: number; minSize: number; overlap: number }
|
||||
} = {}
|
||||
if (name !== undefined) updates.name = name
|
||||
if (description !== undefined) updates.description = description
|
||||
if (chunkingConfig !== undefined) updates.chunkingConfig = chunkingConfig
|
||||
|
||||
const updatedKb = await updateKnowledgeBase(id, updates, requestId)
|
||||
|
||||
recordAudit({
|
||||
workspaceId,
|
||||
actorId: userId,
|
||||
action: AuditAction.KNOWLEDGE_BASE_UPDATED,
|
||||
resourceType: AuditResourceType.KNOWLEDGE_BASE,
|
||||
resourceId: id,
|
||||
resourceName: updatedKb.name,
|
||||
description: `Updated knowledge base "${updatedKb.name}" via API`,
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
knowledgeBase: formatKnowledgeBase(updatedKb),
|
||||
message: 'Knowledge base updated successfully',
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to update knowledge base')
|
||||
}
|
||||
}
|
||||
|
||||
/** DELETE /api/v1/knowledge/[id] — Delete a knowledge base. */
|
||||
export async function DELETE(request: NextRequest, { params }: KnowledgeRouteParams) {
|
||||
const auth = await authenticateRequest(request, 'knowledge-detail')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const { id } = await params
|
||||
const { searchParams } = new URL(request.url)
|
||||
|
||||
const validation = validateSchema(WorkspaceIdSchema, {
|
||||
workspaceId: searchParams.get('workspaceId'),
|
||||
})
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const result = await resolveKnowledgeBase(
|
||||
id,
|
||||
validation.data.workspaceId,
|
||||
userId,
|
||||
rateLimit,
|
||||
'write'
|
||||
)
|
||||
if (result instanceof NextResponse) return result
|
||||
|
||||
await deleteKnowledgeBase(id, requestId)
|
||||
|
||||
recordAudit({
|
||||
workspaceId: validation.data.workspaceId,
|
||||
actorId: userId,
|
||||
action: AuditAction.KNOWLEDGE_BASE_DELETED,
|
||||
resourceType: AuditResourceType.KNOWLEDGE_BASE,
|
||||
resourceId: id,
|
||||
resourceName: result.kb.name,
|
||||
description: `Deleted knowledge base "${result.kb.name}" via API`,
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
message: 'Knowledge base deleted successfully',
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to delete knowledge base')
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { z } from 'zod'
|
||||
import { AuditAction, AuditResourceType, recordAudit } from '@/lib/audit/log'
|
||||
import { createKnowledgeBase, getKnowledgeBases } from '@/lib/knowledge/service'
|
||||
import {
|
||||
authenticateRequest,
|
||||
formatKnowledgeBase,
|
||||
handleError,
|
||||
parseJsonBody,
|
||||
validateSchema,
|
||||
validateWorkspaceAccess,
|
||||
} from '@/app/api/v1/knowledge/utils'
|
||||
|
||||
export const dynamic = 'force-dynamic'
|
||||
export const revalidate = 0
|
||||
|
||||
const ListKBSchema = z.object({
|
||||
workspaceId: z.string().min(1, 'workspaceId query parameter is required'),
|
||||
})
|
||||
|
||||
const ChunkingConfigSchema = z.object({
|
||||
maxSize: z.number().min(100).max(4000).default(1024),
|
||||
minSize: z.number().min(1).max(2000).default(100),
|
||||
overlap: z.number().min(0).max(500).default(200),
|
||||
})
|
||||
|
||||
const CreateKBSchema = z.object({
|
||||
workspaceId: z.string().min(1, 'Workspace ID is required'),
|
||||
name: z.string().min(1, 'Name is required').max(255, 'Name must be 255 characters or less'),
|
||||
description: z.string().max(1000, 'Description must be 1000 characters or less').optional(),
|
||||
chunkingConfig: ChunkingConfigSchema.optional().default({
|
||||
maxSize: 1024,
|
||||
minSize: 100,
|
||||
overlap: 200,
|
||||
}),
|
||||
})
|
||||
|
||||
/** GET /api/v1/knowledge — List knowledge bases in a workspace. */
|
||||
export async function GET(request: NextRequest) {
|
||||
const auth = await authenticateRequest(request, 'knowledge')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const { searchParams } = new URL(request.url)
|
||||
const validation = validateSchema(ListKBSchema, {
|
||||
workspaceId: searchParams.get('workspaceId'),
|
||||
})
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const { workspaceId } = validation.data
|
||||
|
||||
const accessError = await validateWorkspaceAccess(rateLimit, userId, workspaceId)
|
||||
if (accessError) return accessError
|
||||
|
||||
const knowledgeBases = await getKnowledgeBases(userId, workspaceId)
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
knowledgeBases: knowledgeBases.map(formatKnowledgeBase),
|
||||
totalCount: knowledgeBases.length,
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to list knowledge bases')
|
||||
}
|
||||
}
|
||||
|
||||
/** POST /api/v1/knowledge — Create a new knowledge base. */
|
||||
export async function POST(request: NextRequest) {
|
||||
const auth = await authenticateRequest(request, 'knowledge')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const body = await parseJsonBody(request)
|
||||
if (!body.success) return body.response
|
||||
|
||||
const validation = validateSchema(CreateKBSchema, body.data)
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const { workspaceId, name, description, chunkingConfig } = validation.data
|
||||
|
||||
const accessError = await validateWorkspaceAccess(rateLimit, userId, workspaceId, 'write')
|
||||
if (accessError) return accessError
|
||||
|
||||
const kb = await createKnowledgeBase(
|
||||
{
|
||||
name,
|
||||
description,
|
||||
workspaceId,
|
||||
userId,
|
||||
embeddingModel: 'text-embedding-3-small',
|
||||
embeddingDimension: 1536,
|
||||
chunkingConfig,
|
||||
},
|
||||
requestId
|
||||
)
|
||||
|
||||
recordAudit({
|
||||
workspaceId,
|
||||
actorId: userId,
|
||||
action: AuditAction.KNOWLEDGE_BASE_CREATED,
|
||||
resourceType: AuditResourceType.KNOWLEDGE_BASE,
|
||||
resourceId: kb.id,
|
||||
resourceName: kb.name,
|
||||
description: `Created knowledge base "${kb.name}" via API`,
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
knowledgeBase: formatKnowledgeBase(kb),
|
||||
message: 'Knowledge base created successfully',
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to create knowledge base')
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,268 @@
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { z } from 'zod'
|
||||
import { ALL_TAG_SLOTS } from '@/lib/knowledge/constants'
|
||||
import { getDocumentTagDefinitions } from '@/lib/knowledge/tags/service'
|
||||
import { buildUndefinedTagsError, validateTagValue } from '@/lib/knowledge/tags/utils'
|
||||
import type { StructuredFilter } from '@/lib/knowledge/types'
|
||||
import {
|
||||
generateSearchEmbedding,
|
||||
getDocumentNamesByIds,
|
||||
getQueryStrategy,
|
||||
handleTagAndVectorSearch,
|
||||
handleTagOnlySearch,
|
||||
handleVectorOnlySearch,
|
||||
type SearchResult,
|
||||
} from '@/app/api/knowledge/search/utils'
|
||||
import { checkKnowledgeBaseAccess } from '@/app/api/knowledge/utils'
|
||||
import {
|
||||
authenticateRequest,
|
||||
handleError,
|
||||
parseJsonBody,
|
||||
validateSchema,
|
||||
validateWorkspaceAccess,
|
||||
} from '@/app/api/v1/knowledge/utils'
|
||||
|
||||
export const dynamic = 'force-dynamic'
|
||||
export const revalidate = 0
|
||||
|
||||
const StructuredTagFilterSchema = z.object({
|
||||
tagName: z.string(),
|
||||
fieldType: z.enum(['text', 'number', 'date', 'boolean']).optional(),
|
||||
operator: z.string().default('eq'),
|
||||
value: z.union([z.string(), z.number(), z.boolean()]),
|
||||
valueTo: z.union([z.string(), z.number()]).optional(),
|
||||
})
|
||||
|
||||
const SearchSchema = z
|
||||
.object({
|
||||
workspaceId: z.string().min(1, 'Workspace ID is required'),
|
||||
knowledgeBaseIds: z.union([
|
||||
z.string().min(1, 'Knowledge base ID is required'),
|
||||
z
|
||||
.array(z.string().min(1))
|
||||
.min(1, 'At least one knowledge base ID is required')
|
||||
.max(20, 'Maximum 20 knowledge base IDs allowed'),
|
||||
]),
|
||||
query: z.string().optional(),
|
||||
topK: z.number().min(1).max(100).default(10),
|
||||
tagFilters: z.array(StructuredTagFilterSchema).optional(),
|
||||
})
|
||||
.refine(
|
||||
(data) => {
|
||||
const hasQuery = data.query && data.query.trim().length > 0
|
||||
const hasTagFilters = data.tagFilters && data.tagFilters.length > 0
|
||||
return hasQuery || hasTagFilters
|
||||
},
|
||||
{
|
||||
message: 'Either query or tagFilters must be provided',
|
||||
}
|
||||
)
|
||||
|
||||
/** POST /api/v1/knowledge/search — Vector search across knowledge bases. */
|
||||
export async function POST(request: NextRequest) {
|
||||
const auth = await authenticateRequest(request, 'knowledge-search')
|
||||
if (auth instanceof NextResponse) return auth
|
||||
const { requestId, userId, rateLimit } = auth
|
||||
|
||||
try {
|
||||
const body = await parseJsonBody(request)
|
||||
if (!body.success) return body.response
|
||||
|
||||
const validation = validateSchema(SearchSchema, body.data)
|
||||
if (!validation.success) return validation.response
|
||||
|
||||
const { workspaceId, topK, query, tagFilters } = validation.data
|
||||
|
||||
const accessError = await validateWorkspaceAccess(rateLimit, userId, workspaceId)
|
||||
if (accessError) return accessError
|
||||
|
||||
const knowledgeBaseIds = Array.isArray(validation.data.knowledgeBaseIds)
|
||||
? validation.data.knowledgeBaseIds
|
||||
: [validation.data.knowledgeBaseIds]
|
||||
|
||||
const accessChecks = await Promise.all(
|
||||
knowledgeBaseIds.map((kbId) => checkKnowledgeBaseAccess(kbId, userId))
|
||||
)
|
||||
const accessibleKbIds = knowledgeBaseIds.filter(
|
||||
(_, idx) =>
|
||||
accessChecks[idx]?.hasAccess &&
|
||||
accessChecks[idx]?.knowledgeBase?.workspaceId === workspaceId
|
||||
)
|
||||
|
||||
if (accessibleKbIds.length === 0) {
|
||||
return NextResponse.json(
|
||||
{ error: 'Knowledge base not found or access denied' },
|
||||
{ status: 404 }
|
||||
)
|
||||
}
|
||||
|
||||
const inaccessibleKbIds = knowledgeBaseIds.filter((id) => !accessibleKbIds.includes(id))
|
||||
if (inaccessibleKbIds.length > 0) {
|
||||
return NextResponse.json(
|
||||
{ error: `Knowledge bases not found or access denied: ${inaccessibleKbIds.join(', ')}` },
|
||||
{ status: 404 }
|
||||
)
|
||||
}
|
||||
|
||||
let structuredFilters: StructuredFilter[] = []
|
||||
const tagDefsCache = new Map<string, Awaited<ReturnType<typeof getDocumentTagDefinitions>>>()
|
||||
|
||||
if (tagFilters && tagFilters.length > 0 && accessibleKbIds.length > 1) {
|
||||
return NextResponse.json(
|
||||
{ error: 'Tag filters are only supported when searching a single knowledge base' },
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
|
||||
if (tagFilters && tagFilters.length > 0 && accessibleKbIds.length > 0) {
|
||||
const kbId = accessibleKbIds[0]
|
||||
const tagDefs = await getDocumentTagDefinitions(kbId)
|
||||
tagDefsCache.set(kbId, tagDefs)
|
||||
|
||||
const displayNameToTagDef: Record<string, { tagSlot: string; fieldType: string }> = {}
|
||||
tagDefs.forEach((def) => {
|
||||
displayNameToTagDef[def.displayName] = {
|
||||
tagSlot: def.tagSlot,
|
||||
fieldType: def.fieldType,
|
||||
}
|
||||
})
|
||||
|
||||
const undefinedTags: string[] = []
|
||||
const typeErrors: string[] = []
|
||||
|
||||
for (const filter of tagFilters) {
|
||||
const tagDef = displayNameToTagDef[filter.tagName]
|
||||
if (!tagDef) {
|
||||
undefinedTags.push(filter.tagName)
|
||||
continue
|
||||
}
|
||||
const validationError = validateTagValue(
|
||||
filter.tagName,
|
||||
String(filter.value),
|
||||
tagDef.fieldType
|
||||
)
|
||||
if (validationError) {
|
||||
typeErrors.push(validationError)
|
||||
}
|
||||
}
|
||||
|
||||
if (undefinedTags.length > 0 || typeErrors.length > 0) {
|
||||
const errorParts: string[] = []
|
||||
if (undefinedTags.length > 0) {
|
||||
errorParts.push(buildUndefinedTagsError(undefinedTags))
|
||||
}
|
||||
if (typeErrors.length > 0) {
|
||||
errorParts.push(...typeErrors)
|
||||
}
|
||||
return NextResponse.json({ error: errorParts.join('\n') }, { status: 400 })
|
||||
}
|
||||
|
||||
structuredFilters = tagFilters.map((filter) => {
|
||||
const tagDef = displayNameToTagDef[filter.tagName]!
|
||||
return {
|
||||
tagSlot: tagDef.tagSlot,
|
||||
fieldType: tagDef.fieldType,
|
||||
operator: filter.operator,
|
||||
value: filter.value,
|
||||
valueTo: filter.valueTo,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
const hasQuery = query && query.trim().length > 0
|
||||
const hasFilters = structuredFilters.length > 0
|
||||
|
||||
let results: SearchResult[]
|
||||
|
||||
if (!hasQuery && hasFilters) {
|
||||
results = await handleTagOnlySearch({
|
||||
knowledgeBaseIds: accessibleKbIds,
|
||||
topK,
|
||||
structuredFilters,
|
||||
})
|
||||
} else if (hasQuery && hasFilters) {
|
||||
const strategy = getQueryStrategy(accessibleKbIds.length, topK)
|
||||
const queryVector = JSON.stringify(
|
||||
await generateSearchEmbedding(query!, undefined, workspaceId)
|
||||
)
|
||||
results = await handleTagAndVectorSearch({
|
||||
knowledgeBaseIds: accessibleKbIds,
|
||||
topK,
|
||||
structuredFilters,
|
||||
queryVector,
|
||||
distanceThreshold: strategy.distanceThreshold,
|
||||
})
|
||||
} else if (hasQuery) {
|
||||
const strategy = getQueryStrategy(accessibleKbIds.length, topK)
|
||||
const queryVector = JSON.stringify(
|
||||
await generateSearchEmbedding(query!, undefined, workspaceId)
|
||||
)
|
||||
results = await handleVectorOnlySearch({
|
||||
knowledgeBaseIds: accessibleKbIds,
|
||||
topK,
|
||||
queryVector,
|
||||
distanceThreshold: strategy.distanceThreshold,
|
||||
})
|
||||
} else {
|
||||
return NextResponse.json(
|
||||
{ error: 'Either query or tagFilters must be provided' },
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
|
||||
const tagDefsResults = await Promise.all(
|
||||
accessibleKbIds.map(async (kbId) => {
|
||||
try {
|
||||
const tagDefs = tagDefsCache.get(kbId) ?? (await getDocumentTagDefinitions(kbId))
|
||||
const map: Record<string, string> = {}
|
||||
tagDefs.forEach((def) => {
|
||||
map[def.tagSlot] = def.displayName
|
||||
})
|
||||
return { kbId, map }
|
||||
} catch {
|
||||
return { kbId, map: {} as Record<string, string> }
|
||||
}
|
||||
})
|
||||
)
|
||||
const tagDefinitionsMap: Record<string, Record<string, string>> = {}
|
||||
tagDefsResults.forEach(({ kbId, map }) => {
|
||||
tagDefinitionsMap[kbId] = map
|
||||
})
|
||||
|
||||
const documentIds = results.map((r) => r.documentId)
|
||||
const documentNameMap = await getDocumentNamesByIds(documentIds)
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
data: {
|
||||
results: results.map((result) => {
|
||||
const kbTagMap = tagDefinitionsMap[result.knowledgeBaseId] || {}
|
||||
const tags: Record<string, string | number | boolean | Date | null> = {}
|
||||
|
||||
ALL_TAG_SLOTS.forEach((slot) => {
|
||||
const tagValue = (result as Record<string, unknown>)[slot]
|
||||
if (tagValue !== null && tagValue !== undefined) {
|
||||
const displayName = kbTagMap[slot] || slot
|
||||
tags[displayName] = tagValue as string | number | boolean | Date | null
|
||||
}
|
||||
})
|
||||
|
||||
return {
|
||||
documentId: result.documentId,
|
||||
documentName: documentNameMap[result.documentId] || undefined,
|
||||
content: result.content,
|
||||
chunkIndex: result.chunkIndex,
|
||||
metadata: tags,
|
||||
similarity: hasQuery ? 1 - result.distance : 1,
|
||||
}
|
||||
}),
|
||||
query: query || '',
|
||||
knowledgeBaseIds: accessibleKbIds,
|
||||
topK,
|
||||
totalResults: results.length,
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
return handleError(requestId, error, 'Failed to perform search')
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,187 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { z } from 'zod'
|
||||
import { generateRequestId } from '@/lib/core/utils/request'
|
||||
import { getKnowledgeBaseById } from '@/lib/knowledge/service'
|
||||
import type { KnowledgeBaseWithCounts } from '@/lib/knowledge/types'
|
||||
import { getUserEntityPermissions } from '@/lib/workspaces/permissions/utils'
|
||||
import {
|
||||
checkRateLimit,
|
||||
checkWorkspaceScope,
|
||||
createRateLimitResponse,
|
||||
type RateLimitResult,
|
||||
} from '@/app/api/v1/middleware'
|
||||
|
||||
const logger = createLogger('V1KnowledgeAPI')
|
||||
|
||||
type EndpointKey = 'knowledge' | 'knowledge-detail' | 'knowledge-search'
|
||||
|
||||
/**
|
||||
* Successful authentication result with request context
|
||||
*/
|
||||
export interface AuthorizedRequest {
|
||||
requestId: string
|
||||
userId: string
|
||||
rateLimit: RateLimitResult
|
||||
}
|
||||
|
||||
/**
|
||||
* Authenticates and rate-limits a v1 knowledge API request.
|
||||
* Returns NextResponse on failure, AuthorizedRequest on success.
|
||||
*/
|
||||
export async function authenticateRequest(
|
||||
request: NextRequest,
|
||||
endpoint: EndpointKey
|
||||
): Promise<AuthorizedRequest | NextResponse> {
|
||||
const requestId = generateRequestId()
|
||||
const rateLimit = await checkRateLimit(request, endpoint)
|
||||
if (!rateLimit.allowed) {
|
||||
return createRateLimitResponse(rateLimit)
|
||||
}
|
||||
return { requestId, userId: rateLimit.userId!, rateLimit }
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates workspace scope and user permission level.
|
||||
* Returns null on success, NextResponse on failure.
|
||||
*/
|
||||
export async function validateWorkspaceAccess(
|
||||
rateLimit: RateLimitResult,
|
||||
userId: string,
|
||||
workspaceId: string,
|
||||
level: 'read' | 'write' = 'read'
|
||||
): Promise<NextResponse | null> {
|
||||
const scopeError = checkWorkspaceScope(rateLimit, workspaceId)
|
||||
if (scopeError) return scopeError
|
||||
|
||||
const permission = await getUserEntityPermissions(userId, 'workspace', workspaceId)
|
||||
if (permission === null) {
|
||||
return NextResponse.json({ error: 'Access denied' }, { status: 403 })
|
||||
}
|
||||
if (level === 'write' && permission === 'read') {
|
||||
return NextResponse.json({ error: 'Access denied' }, { status: 403 })
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches a KB by ID, validates it exists, belongs to the workspace,
|
||||
* and the user has permission. Returns the KB or a NextResponse error.
|
||||
*/
|
||||
export async function resolveKnowledgeBase(
|
||||
id: string,
|
||||
workspaceId: string,
|
||||
userId: string,
|
||||
rateLimit: RateLimitResult,
|
||||
level: 'read' | 'write' = 'read'
|
||||
): Promise<{ kb: KnowledgeBaseWithCounts } | NextResponse> {
|
||||
const accessError = await validateWorkspaceAccess(rateLimit, userId, workspaceId, level)
|
||||
if (accessError) return accessError
|
||||
|
||||
const kb = await getKnowledgeBaseById(id)
|
||||
if (!kb) {
|
||||
return NextResponse.json({ error: 'Knowledge base not found' }, { status: 404 })
|
||||
}
|
||||
if (kb.workspaceId !== workspaceId) {
|
||||
return NextResponse.json({ error: 'Knowledge base not found' }, { status: 404 })
|
||||
}
|
||||
return { kb }
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates data against a Zod schema with consistent error response.
|
||||
*/
|
||||
export function validateSchema<T>(
|
||||
schema: z.ZodSchema<T>,
|
||||
data: unknown
|
||||
): { success: true; data: T } | { success: false; response: NextResponse } {
|
||||
const result = schema.safeParse(data)
|
||||
if (!result.success) {
|
||||
return {
|
||||
success: false,
|
||||
response: NextResponse.json(
|
||||
{ error: 'Validation error', details: result.error.errors },
|
||||
{ status: 400 }
|
||||
),
|
||||
}
|
||||
}
|
||||
return { success: true, data: result.data }
|
||||
}
|
||||
|
||||
/**
|
||||
* Safely parses a JSON request body with consistent error response.
|
||||
*/
|
||||
export async function parseJsonBody(
|
||||
request: NextRequest
|
||||
): Promise<{ success: true; data: unknown } | { success: false; response: NextResponse }> {
|
||||
try {
|
||||
const data = await request.json()
|
||||
return { success: true, data }
|
||||
} catch {
|
||||
return {
|
||||
success: false,
|
||||
response: NextResponse.json({ error: 'Request body must be valid JSON' }, { status: 400 }),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Serializes a date value for JSON responses.
|
||||
*/
|
||||
export function serializeDate(date: Date | string | null | undefined): string | null {
|
||||
if (date === null || date === undefined) return null
|
||||
if (date instanceof Date) return date.toISOString()
|
||||
return String(date)
|
||||
}
|
||||
|
||||
/**
|
||||
* Formats a KnowledgeBaseWithCounts into the API response shape.
|
||||
*/
|
||||
export function formatKnowledgeBase(kb: KnowledgeBaseWithCounts) {
|
||||
return {
|
||||
id: kb.id,
|
||||
name: kb.name,
|
||||
description: kb.description,
|
||||
tokenCount: kb.tokenCount,
|
||||
embeddingModel: kb.embeddingModel,
|
||||
embeddingDimension: kb.embeddingDimension,
|
||||
chunkingConfig: kb.chunkingConfig,
|
||||
docCount: kb.docCount,
|
||||
connectorTypes: kb.connectorTypes,
|
||||
createdAt: serializeDate(kb.createdAt),
|
||||
updatedAt: serializeDate(kb.updatedAt),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handles unexpected errors with consistent logging and response.
|
||||
*/
|
||||
export function handleError(
|
||||
requestId: string,
|
||||
error: unknown,
|
||||
defaultMessage: string
|
||||
): NextResponse {
|
||||
if (error instanceof z.ZodError) {
|
||||
return NextResponse.json({ error: 'Validation error', details: error.errors }, { status: 400 })
|
||||
}
|
||||
|
||||
if (error instanceof Error) {
|
||||
if (error.message.includes('does not have permission')) {
|
||||
return NextResponse.json({ error: 'Access denied' }, { status: 403 })
|
||||
}
|
||||
|
||||
const isStorageLimitError =
|
||||
error.message.includes('Storage limit exceeded') || error.message.includes('storage limit')
|
||||
if (isStorageLimitError) {
|
||||
return NextResponse.json({ error: 'Storage limit exceeded' }, { status: 413 })
|
||||
}
|
||||
|
||||
const isDuplicate = error.message.includes('already exists')
|
||||
if (isDuplicate) {
|
||||
return NextResponse.json({ error: 'Resource already exists' }, { status: 409 })
|
||||
}
|
||||
}
|
||||
|
||||
logger.error(`[${requestId}] ${defaultMessage}:`, error)
|
||||
return NextResponse.json({ error: defaultMessage }, { status: 500 })
|
||||
}
|
||||
@@ -33,7 +33,10 @@ export async function checkRateLimit(
|
||||
| 'table-rows'
|
||||
| 'table-row-detail'
|
||||
| 'files'
|
||||
| 'file-detail' = 'logs'
|
||||
| 'file-detail'
|
||||
| 'knowledge'
|
||||
| 'knowledge-detail'
|
||||
| 'knowledge-search' = 'logs'
|
||||
): Promise<RateLimitResult> {
|
||||
try {
|
||||
const auth = await authenticateV1Request(request)
|
||||
|
||||
@@ -0,0 +1,567 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { EvernoteIcon } from '@/components/icons'
|
||||
import { fetchWithRetry, VALIDATE_RETRY_OPTIONS } from '@/lib/knowledge/documents/utils'
|
||||
import {
|
||||
ThriftReader,
|
||||
ThriftWriter,
|
||||
TYPE_I32,
|
||||
TYPE_I64,
|
||||
TYPE_LIST,
|
||||
TYPE_STRING,
|
||||
TYPE_STRUCT,
|
||||
} from '@/app/api/tools/evernote/lib/thrift'
|
||||
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
|
||||
import { computeContentHash, htmlToPlainText, joinTagArray, parseTagDate } from '@/connectors/utils'
|
||||
|
||||
const logger = createLogger('EvernoteConnector')
|
||||
|
||||
const NOTES_PER_PAGE = 50
|
||||
|
||||
/**
|
||||
* Extracts the shard ID from an Evernote developer token.
|
||||
* Token format: "S=s1:U=12345:..." where s1 is the shard.
|
||||
*/
|
||||
function extractShardId(token: string): string {
|
||||
const match = token.match(/S=s(\d+)/)
|
||||
if (!match) {
|
||||
throw new Error('Invalid Evernote token format: cannot extract shard ID')
|
||||
}
|
||||
return `s${match[1]}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts the user ID from an Evernote developer token.
|
||||
* Token format: "S=s1:U=12345:..." where 12345 is the user ID.
|
||||
*/
|
||||
function extractUserId(token: string): string {
|
||||
const match = token.match(/:U=(\d+)/)
|
||||
if (!match) {
|
||||
throw new Error('Invalid Evernote token format: cannot extract user ID')
|
||||
}
|
||||
return match[1]
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the Evernote API host based on the token type.
|
||||
* Sandbox tokens contain `:Sandbox` and route to sandbox.evernote.com.
|
||||
*/
|
||||
function getHost(token: string): string {
|
||||
return token.includes(':Sandbox') ? 'sandbox.evernote.com' : 'www.evernote.com'
|
||||
}
|
||||
|
||||
/**
|
||||
* Derives the NoteStore URL from a developer token.
|
||||
*/
|
||||
function getNoteStoreUrl(token: string): string {
|
||||
const shardId = extractShardId(token)
|
||||
return `https://${getHost(token)}/shard/${shardId}/notestore`
|
||||
}
|
||||
|
||||
/**
|
||||
* Sends a Thrift RPC call to the Evernote NoteStore via HTTP POST.
|
||||
*/
|
||||
async function callNoteStore(
|
||||
token: string,
|
||||
writer: ThriftWriter,
|
||||
retryOptions?: Parameters<typeof fetchWithRetry>[2]
|
||||
): Promise<ThriftReader> {
|
||||
const url = getNoteStoreUrl(token)
|
||||
|
||||
const response = await fetchWithRetry(
|
||||
url,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/x-thrift',
|
||||
Accept: 'application/x-thrift',
|
||||
},
|
||||
body: new Uint8Array(writer.toBuffer()),
|
||||
},
|
||||
retryOptions
|
||||
)
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Evernote HTTP ${response.status}: ${response.statusText}`)
|
||||
}
|
||||
|
||||
const reader = new ThriftReader(await response.arrayBuffer())
|
||||
const msg = reader.readMessageBegin()
|
||||
|
||||
if (reader.isException(msg.type)) {
|
||||
const ex = reader.readException()
|
||||
throw new Error(`Evernote API error: ${ex.message}`)
|
||||
}
|
||||
|
||||
return reader
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks for Evernote-specific exceptions in response struct fields 1-3.
|
||||
*/
|
||||
function checkException(r: ThriftReader, fieldId: number, fieldType: number): boolean {
|
||||
if ((fieldId === 1 || fieldId === 2) && fieldType === TYPE_STRUCT) {
|
||||
let errorCode = 0
|
||||
let message = ''
|
||||
r.readStruct((r2, fid, ftype) => {
|
||||
if (fid === 1 && ftype === TYPE_I32) errorCode = r2.readI32()
|
||||
else if (fid === 2 && ftype === TYPE_STRING) message = r2.readString()
|
||||
else r2.skip(ftype)
|
||||
})
|
||||
throw new Error(`Evernote error (${errorCode}): ${message}`)
|
||||
}
|
||||
if (fieldId === 3 && fieldType === TYPE_STRUCT) {
|
||||
let identifier = ''
|
||||
r.readStruct((r2, fid, ftype) => {
|
||||
if (fid === 1 && ftype === TYPE_STRING) identifier = r2.readString()
|
||||
else r2.skip(ftype)
|
||||
})
|
||||
throw new Error(`Evernote not found: ${identifier}`)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
interface Notebook {
|
||||
guid: string
|
||||
name: string
|
||||
}
|
||||
|
||||
interface Tag {
|
||||
guid: string
|
||||
name: string
|
||||
}
|
||||
|
||||
interface NoteMetadata {
|
||||
guid: string
|
||||
title: string
|
||||
created: number
|
||||
updated: number
|
||||
notebookGuid: string
|
||||
tagGuids: string[]
|
||||
}
|
||||
|
||||
interface Note {
|
||||
guid: string
|
||||
title: string
|
||||
content: string
|
||||
created: number
|
||||
updated: number
|
||||
notebookGuid: string
|
||||
tagGuids: string[]
|
||||
}
|
||||
|
||||
async function apiListNotebooks(
|
||||
token: string,
|
||||
retryOptions?: Parameters<typeof fetchWithRetry>[2]
|
||||
): Promise<Notebook[]> {
|
||||
const w = new ThriftWriter()
|
||||
w.writeMessageBegin('listNotebooks', 0)
|
||||
w.writeStringField(1, token)
|
||||
w.writeFieldStop()
|
||||
|
||||
const r = await callNoteStore(token, w, retryOptions)
|
||||
const notebooks: Notebook[] = []
|
||||
|
||||
r.readStruct((r2, fid, ftype) => {
|
||||
if (fid === 0 && ftype === TYPE_LIST) {
|
||||
const { size } = r2.readListBegin()
|
||||
for (let i = 0; i < size; i++) {
|
||||
let guid = ''
|
||||
let name = ''
|
||||
r2.readStruct((r3, fid3, ftype3) => {
|
||||
if (fid3 === 1 && ftype3 === TYPE_STRING) guid = r3.readString()
|
||||
else if (fid3 === 2 && ftype3 === TYPE_STRING) name = r3.readString()
|
||||
else r3.skip(ftype3)
|
||||
})
|
||||
notebooks.push({ guid, name })
|
||||
}
|
||||
} else if (!checkException(r2, fid, ftype)) {
|
||||
r2.skip(ftype)
|
||||
}
|
||||
})
|
||||
|
||||
return notebooks
|
||||
}
|
||||
|
||||
async function apiListTags(
|
||||
token: string,
|
||||
retryOptions?: Parameters<typeof fetchWithRetry>[2]
|
||||
): Promise<Tag[]> {
|
||||
const w = new ThriftWriter()
|
||||
w.writeMessageBegin('listTags', 0)
|
||||
w.writeStringField(1, token)
|
||||
w.writeFieldStop()
|
||||
|
||||
const r = await callNoteStore(token, w, retryOptions)
|
||||
const tags: Tag[] = []
|
||||
|
||||
r.readStruct((r2, fid, ftype) => {
|
||||
if (fid === 0 && ftype === TYPE_LIST) {
|
||||
const { size } = r2.readListBegin()
|
||||
for (let i = 0; i < size; i++) {
|
||||
let guid = ''
|
||||
let name = ''
|
||||
r2.readStruct((r3, fid3, ftype3) => {
|
||||
if (fid3 === 1 && ftype3 === TYPE_STRING) guid = r3.readString()
|
||||
else if (fid3 === 2 && ftype3 === TYPE_STRING) name = r3.readString()
|
||||
else r3.skip(ftype3)
|
||||
})
|
||||
tags.push({ guid, name })
|
||||
}
|
||||
} else if (!checkException(r2, fid, ftype)) {
|
||||
r2.skip(ftype)
|
||||
}
|
||||
})
|
||||
|
||||
return tags
|
||||
}
|
||||
|
||||
/**
|
||||
* Calls NoteStore.findNotesMetadata with offset-based pagination.
|
||||
*
|
||||
* Thrift field numbers (from NoteStore.thrift):
|
||||
* findNotesMetadata(1:token, 2:NoteFilter, 3:offset, 4:maxNotes, 5:ResultSpec)
|
||||
* NoteFilter: 4:notebookGuid
|
||||
* NotesMetadataResultSpec: 2:includeTitle, 6:includeCreated, 7:includeUpdated,
|
||||
* 11:includeNotebookGuid, 12:includeTagGuids
|
||||
* NotesMetadataList: 1:startIndex, 2:totalNotes, 3:list<NoteMetadata>
|
||||
* NoteMetadata: 1:guid, 2:title, 6:created(i64), 7:updated(i64),
|
||||
* 11:notebookGuid, 12:list<tagGuids>
|
||||
*/
|
||||
async function apiFindNotesMetadata(
|
||||
token: string,
|
||||
offset: number,
|
||||
maxNotes: number,
|
||||
notebookGuid?: string
|
||||
): Promise<{ totalNotes: number; notes: NoteMetadata[] }> {
|
||||
const w = new ThriftWriter()
|
||||
w.writeMessageBegin('findNotesMetadata', 0)
|
||||
w.writeStringField(1, token)
|
||||
|
||||
w.writeFieldBegin(TYPE_STRUCT, 2) // NoteFilter
|
||||
if (notebookGuid) {
|
||||
w.writeStringField(4, notebookGuid)
|
||||
}
|
||||
w.writeFieldStop()
|
||||
|
||||
w.writeI32Field(3, offset)
|
||||
w.writeI32Field(4, maxNotes)
|
||||
|
||||
w.writeFieldBegin(TYPE_STRUCT, 5) // NotesMetadataResultSpec
|
||||
w.writeBoolField(2, true) // includeTitle
|
||||
w.writeBoolField(6, true) // includeCreated
|
||||
w.writeBoolField(7, true) // includeUpdated
|
||||
w.writeBoolField(11, true) // includeNotebookGuid
|
||||
w.writeBoolField(12, true) // includeTagGuids
|
||||
w.writeFieldStop()
|
||||
|
||||
w.writeFieldStop()
|
||||
|
||||
const r = await callNoteStore(token, w)
|
||||
|
||||
let totalNotes = 0
|
||||
const notes: NoteMetadata[] = []
|
||||
|
||||
r.readStruct((r2, fid, ftype) => {
|
||||
if (fid === 0 && ftype === TYPE_STRUCT) {
|
||||
r2.readStruct((r3, fid3, ftype3) => {
|
||||
if (fid3 === 1 && ftype3 === TYPE_I32) {
|
||||
r3.readI32()
|
||||
} else if (fid3 === 2 && ftype3 === TYPE_I32) {
|
||||
totalNotes = r3.readI32()
|
||||
} else if (fid3 === 3 && ftype3 === TYPE_LIST) {
|
||||
const { size } = r3.readListBegin()
|
||||
for (let i = 0; i < size; i++) {
|
||||
let guid = ''
|
||||
let title = ''
|
||||
let created = 0
|
||||
let updated = 0
|
||||
let nbGuid = ''
|
||||
const tagGuids: string[] = []
|
||||
|
||||
r3.readStruct((r4, fid4, ftype4) => {
|
||||
if (fid4 === 1 && ftype4 === TYPE_STRING) guid = r4.readString()
|
||||
else if (fid4 === 2 && ftype4 === TYPE_STRING) title = r4.readString()
|
||||
else if (fid4 === 6 && ftype4 === TYPE_I64) created = Number(r4.readI64())
|
||||
else if (fid4 === 7 && ftype4 === TYPE_I64) updated = Number(r4.readI64())
|
||||
else if (fid4 === 11 && ftype4 === TYPE_STRING) nbGuid = r4.readString()
|
||||
else if (fid4 === 12 && ftype4 === TYPE_LIST) {
|
||||
const { size: tagCount } = r4.readListBegin()
|
||||
for (let t = 0; t < tagCount; t++) tagGuids.push(r4.readString())
|
||||
} else {
|
||||
r4.skip(ftype4)
|
||||
}
|
||||
})
|
||||
notes.push({ guid, title, created, updated, notebookGuid: nbGuid, tagGuids })
|
||||
}
|
||||
} else {
|
||||
r3.skip(ftype3)
|
||||
}
|
||||
})
|
||||
} else if (!checkException(r2, fid, ftype)) {
|
||||
r2.skip(ftype)
|
||||
}
|
||||
})
|
||||
|
||||
return { totalNotes, notes }
|
||||
}
|
||||
|
||||
/**
|
||||
* Calls NoteStore.getNote to fetch a single note with content.
|
||||
*
|
||||
* Thrift: getNote(1:token, 2:guid, 3:withContent, 4:withResourcesData,
|
||||
* 5:withResourcesRecognition, 6:withResourcesAlternateData)
|
||||
* Note: 1:guid, 2:title, 3:content, 6:created, 7:updated, 11:notebookGuid, 12:tagGuids
|
||||
*/
|
||||
async function apiGetNote(token: string, guid: string): Promise<Note> {
|
||||
const w = new ThriftWriter()
|
||||
w.writeMessageBegin('getNote', 0)
|
||||
w.writeStringField(1, token)
|
||||
w.writeStringField(2, guid)
|
||||
w.writeBoolField(3, true) // withContent
|
||||
w.writeBoolField(4, false) // withResourcesData
|
||||
w.writeBoolField(5, false) // withResourcesRecognition
|
||||
w.writeBoolField(6, false) // withResourcesAlternateData
|
||||
w.writeFieldStop()
|
||||
|
||||
const r = await callNoteStore(token, w)
|
||||
|
||||
let noteGuid = ''
|
||||
let title = ''
|
||||
let content = ''
|
||||
let created = 0
|
||||
let updated = 0
|
||||
let notebookGuid = ''
|
||||
const tagGuids: string[] = []
|
||||
|
||||
r.readStruct((r2, fid, ftype) => {
|
||||
if (fid === 0 && ftype === TYPE_STRUCT) {
|
||||
r2.readStruct((r3, fid3, ftype3) => {
|
||||
if (fid3 === 1 && ftype3 === TYPE_STRING) noteGuid = r3.readString()
|
||||
else if (fid3 === 2 && ftype3 === TYPE_STRING) title = r3.readString()
|
||||
else if (fid3 === 3 && ftype3 === TYPE_STRING) content = r3.readString()
|
||||
else if (fid3 === 6 && ftype3 === TYPE_I64) created = Number(r3.readI64())
|
||||
else if (fid3 === 7 && ftype3 === TYPE_I64) updated = Number(r3.readI64())
|
||||
else if (fid3 === 11 && ftype3 === TYPE_STRING) notebookGuid = r3.readString()
|
||||
else if (fid3 === 12 && ftype3 === TYPE_LIST) {
|
||||
const { size } = r3.readListBegin()
|
||||
for (let t = 0; t < size; t++) tagGuids.push(r3.readString())
|
||||
} else {
|
||||
r3.skip(ftype3)
|
||||
}
|
||||
})
|
||||
} else if (!checkException(r2, fid, ftype)) {
|
||||
r2.skip(ftype)
|
||||
}
|
||||
})
|
||||
|
||||
return { guid: noteGuid || guid, title, content, created, updated, notebookGuid, tagGuids }
|
||||
}
|
||||
|
||||
export const evernoteConnector: ConnectorConfig = {
|
||||
id: 'evernote',
|
||||
name: 'Evernote',
|
||||
description: 'Sync notes from Evernote into your knowledge base',
|
||||
version: '1.0.0',
|
||||
icon: EvernoteIcon,
|
||||
|
||||
auth: {
|
||||
mode: 'apiKey',
|
||||
label: 'Developer Token',
|
||||
placeholder: 'Enter your Evernote developer token (starts with S=)',
|
||||
},
|
||||
|
||||
configFields: [
|
||||
{
|
||||
id: 'notebookGuid',
|
||||
title: 'Notebook GUID',
|
||||
type: 'short-input',
|
||||
placeholder: 'Leave empty to sync all notebooks',
|
||||
required: false,
|
||||
description: 'Sync only notes from this notebook (optional)',
|
||||
},
|
||||
],
|
||||
|
||||
listDocuments: async (
|
||||
accessToken: string,
|
||||
sourceConfig: Record<string, unknown>,
|
||||
cursor?: string,
|
||||
syncContext?: Record<string, unknown>
|
||||
): Promise<ExternalDocumentList> => {
|
||||
const notebookGuid = (sourceConfig.notebookGuid as string) || undefined
|
||||
|
||||
if (syncContext && !syncContext.tagMap) {
|
||||
const tags = await apiListTags(accessToken)
|
||||
syncContext.tagMap = Object.fromEntries(tags.map((t) => [t.guid, t.name]))
|
||||
}
|
||||
if (syncContext && !syncContext.notebookMap) {
|
||||
const notebooks = await apiListNotebooks(accessToken)
|
||||
syncContext.notebookMap = Object.fromEntries(notebooks.map((nb) => [nb.guid, nb.name]))
|
||||
}
|
||||
|
||||
const tagMap = (syncContext?.tagMap as Record<string, string>) || {}
|
||||
const notebookMap = (syncContext?.notebookMap as Record<string, string>) || {}
|
||||
const offset = cursor ? Number(cursor) : 0
|
||||
const shardId = extractShardId(accessToken)
|
||||
const userId = extractUserId(accessToken)
|
||||
const host = getHost(accessToken)
|
||||
|
||||
logger.info('Listing Evernote notes', { offset, maxNotes: NOTES_PER_PAGE })
|
||||
|
||||
const result = await apiFindNotesMetadata(accessToken, offset, NOTES_PER_PAGE, notebookGuid)
|
||||
|
||||
const documents: ExternalDocument[] = []
|
||||
|
||||
for (const meta of result.notes) {
|
||||
try {
|
||||
const note = await apiGetNote(accessToken, meta.guid)
|
||||
const plainText = htmlToPlainText(note.content)
|
||||
const contentHash = await computeContentHash(plainText)
|
||||
const tagNames = note.tagGuids.map((g) => tagMap[g]).filter(Boolean)
|
||||
|
||||
documents.push({
|
||||
externalId: note.guid,
|
||||
title: note.title || 'Untitled',
|
||||
content: plainText,
|
||||
mimeType: 'text/plain',
|
||||
sourceUrl: `https://${host}/shard/${shardId}/nl/${userId}/${note.guid}/`,
|
||||
contentHash,
|
||||
metadata: {
|
||||
tags: tagNames,
|
||||
notebook: notebookMap[note.notebookGuid] || '',
|
||||
createdAt: note.created ? new Date(note.created).toISOString() : undefined,
|
||||
updatedAt: note.updated ? new Date(note.updated).toISOString() : undefined,
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
logger.warn('Failed to fetch note content', {
|
||||
guid: meta.guid,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const nextOffset = offset + result.notes.length
|
||||
const hasMore = nextOffset < result.totalNotes
|
||||
|
||||
return {
|
||||
documents,
|
||||
nextCursor: hasMore ? String(nextOffset) : undefined,
|
||||
hasMore,
|
||||
}
|
||||
},
|
||||
|
||||
getDocument: async (
|
||||
accessToken: string,
|
||||
_sourceConfig: Record<string, unknown>,
|
||||
externalId: string,
|
||||
syncContext?: Record<string, unknown>
|
||||
): Promise<ExternalDocument | null> => {
|
||||
try {
|
||||
const note = await apiGetNote(accessToken, externalId)
|
||||
const plainText = htmlToPlainText(note.content)
|
||||
const contentHash = await computeContentHash(plainText)
|
||||
const shardId = extractShardId(accessToken)
|
||||
const userId = extractUserId(accessToken)
|
||||
const host = getHost(accessToken)
|
||||
|
||||
if (syncContext && !syncContext.tagMap) {
|
||||
const tags = await apiListTags(accessToken)
|
||||
syncContext.tagMap = Object.fromEntries(tags.map((t) => [t.guid, t.name]))
|
||||
}
|
||||
if (syncContext && !syncContext.notebookMap) {
|
||||
const notebooks = await apiListNotebooks(accessToken)
|
||||
syncContext.notebookMap = Object.fromEntries(notebooks.map((nb) => [nb.guid, nb.name]))
|
||||
}
|
||||
|
||||
let tagMap: Record<string, string>
|
||||
let notebookMap: Record<string, string>
|
||||
if (syncContext) {
|
||||
tagMap = syncContext.tagMap as Record<string, string>
|
||||
notebookMap = syncContext.notebookMap as Record<string, string>
|
||||
} else {
|
||||
const tags = await apiListTags(accessToken)
|
||||
tagMap = Object.fromEntries(tags.map((t) => [t.guid, t.name]))
|
||||
const notebooks = await apiListNotebooks(accessToken)
|
||||
notebookMap = Object.fromEntries(notebooks.map((nb) => [nb.guid, nb.name]))
|
||||
}
|
||||
|
||||
const tagNames = note.tagGuids.map((g) => tagMap[g]).filter(Boolean)
|
||||
const notebookName = notebookMap[note.notebookGuid] || ''
|
||||
|
||||
return {
|
||||
externalId,
|
||||
title: note.title || 'Untitled',
|
||||
content: plainText,
|
||||
mimeType: 'text/plain',
|
||||
sourceUrl: `https://${host}/shard/${shardId}/nl/${userId}/${externalId}/`,
|
||||
contentHash,
|
||||
metadata: {
|
||||
tags: tagNames,
|
||||
notebook: notebookName,
|
||||
createdAt: note.created ? new Date(note.created).toISOString() : undefined,
|
||||
updatedAt: note.updated ? new Date(note.updated).toISOString() : undefined,
|
||||
},
|
||||
}
|
||||
} catch (error) {
|
||||
logger.warn('Failed to get Evernote note', {
|
||||
externalId,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
})
|
||||
return null
|
||||
}
|
||||
},
|
||||
|
||||
validateConfig: async (
|
||||
accessToken: string,
|
||||
sourceConfig: Record<string, unknown>
|
||||
): Promise<{ valid: boolean; error?: string }> => {
|
||||
try {
|
||||
extractShardId(accessToken)
|
||||
} catch {
|
||||
return { valid: false, error: 'Invalid developer token format — must start with S=s{number}' }
|
||||
}
|
||||
|
||||
try {
|
||||
const notebooks = await apiListNotebooks(accessToken, VALIDATE_RETRY_OPTIONS)
|
||||
|
||||
const notebookGuid = (sourceConfig.notebookGuid as string) || ''
|
||||
if (notebookGuid.trim()) {
|
||||
const found = notebooks.some((nb) => nb.guid === notebookGuid.trim())
|
||||
if (!found) {
|
||||
return { valid: false, error: `Notebook with GUID "${notebookGuid}" not found` }
|
||||
}
|
||||
}
|
||||
|
||||
return { valid: true }
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Failed to connect to Evernote'
|
||||
return { valid: false, error: message }
|
||||
}
|
||||
},
|
||||
|
||||
tagDefinitions: [
|
||||
{ id: 'tags', displayName: 'Tags', fieldType: 'text' },
|
||||
{ id: 'notebook', displayName: 'Notebook', fieldType: 'text' },
|
||||
{ id: 'updatedAt', displayName: 'Last Updated', fieldType: 'date' },
|
||||
{ id: 'createdAt', displayName: 'Created', fieldType: 'date' },
|
||||
],
|
||||
|
||||
mapTags: (metadata: Record<string, unknown>): Record<string, unknown> => {
|
||||
const result: Record<string, unknown> = {}
|
||||
|
||||
const tags = joinTagArray(metadata.tags)
|
||||
if (tags) result.tags = tags
|
||||
|
||||
if (typeof metadata.notebook === 'string' && metadata.notebook) {
|
||||
result.notebook = metadata.notebook
|
||||
}
|
||||
|
||||
const updatedAt = parseTagDate(metadata.updatedAt)
|
||||
if (updatedAt) result.updatedAt = updatedAt
|
||||
|
||||
const createdAt = parseTagDate(metadata.createdAt)
|
||||
if (createdAt) result.createdAt = createdAt
|
||||
|
||||
return result
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
export { evernoteConnector } from '@/connectors/evernote/evernote'
|
||||
@@ -0,0 +1 @@
|
||||
export { obsidianConnector } from '@/connectors/obsidian/obsidian'
|
||||
@@ -0,0 +1,365 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { ObsidianIcon } from '@/components/icons'
|
||||
import { fetchWithRetry, VALIDATE_RETRY_OPTIONS } from '@/lib/knowledge/documents/utils'
|
||||
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
|
||||
import { computeContentHash, joinTagArray, parseTagDate } from '@/connectors/utils'
|
||||
|
||||
const logger = createLogger('ObsidianConnector')
|
||||
|
||||
const DOCS_PER_PAGE = 50
|
||||
|
||||
interface NoteJson {
|
||||
content: string
|
||||
frontmatter: Record<string, unknown>
|
||||
path: string
|
||||
stat: {
|
||||
ctime: number
|
||||
mtime: number
|
||||
size: number
|
||||
}
|
||||
tags: string[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes the vault URL by removing trailing slashes.
|
||||
*/
|
||||
function normalizeVaultUrl(url: string): string {
|
||||
return url.replace(/\/+$/, '')
|
||||
}
|
||||
|
||||
/**
|
||||
* Lists entries in a single vault directory (non-recursive).
|
||||
* Returns raw entries: files as names, subdirectories with trailing slash.
|
||||
*/
|
||||
async function listDirectory(
|
||||
baseUrl: string,
|
||||
accessToken: string,
|
||||
dirPath: string,
|
||||
retryOptions?: Parameters<typeof fetchWithRetry>[2]
|
||||
): Promise<string[]> {
|
||||
const encodedDir = dirPath
|
||||
? dirPath.split('/').map(encodeURIComponent).join('/')
|
||||
: ''
|
||||
const endpoint = encodedDir
|
||||
? `${baseUrl}/vault/${encodedDir}/`
|
||||
: `${baseUrl}/vault/`
|
||||
|
||||
const response = await fetchWithRetry(
|
||||
endpoint,
|
||||
{
|
||||
method: 'GET',
|
||||
headers: {
|
||||
Authorization: `Bearer ${accessToken}`,
|
||||
Accept: 'application/json',
|
||||
},
|
||||
},
|
||||
retryOptions
|
||||
)
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Obsidian API error: ${response.status} ${response.statusText}`)
|
||||
}
|
||||
|
||||
const data = (await response.json()) as { files: string[] }
|
||||
return data.files ?? []
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively lists all markdown files in the vault or a specific folder.
|
||||
*/
|
||||
const MAX_RECURSION_DEPTH = 20
|
||||
|
||||
async function listVaultFiles(
|
||||
baseUrl: string,
|
||||
accessToken: string,
|
||||
folderPath?: string,
|
||||
retryOptions?: Parameters<typeof fetchWithRetry>[2],
|
||||
depth = 0
|
||||
): Promise<string[]> {
|
||||
if (depth > MAX_RECURSION_DEPTH) {
|
||||
logger.warn('Max directory depth reached, skipping further recursion', { folderPath })
|
||||
return []
|
||||
}
|
||||
|
||||
const rootPath = folderPath || ''
|
||||
const entries = await listDirectory(baseUrl, accessToken, rootPath, retryOptions)
|
||||
|
||||
const mdFiles: string[] = []
|
||||
const subDirs: string[] = []
|
||||
|
||||
for (const entry of entries) {
|
||||
if (entry.endsWith('/')) {
|
||||
const fullDir = rootPath ? `${rootPath}/${entry.slice(0, -1)}` : entry.slice(0, -1)
|
||||
subDirs.push(fullDir)
|
||||
} else if (entry.endsWith('.md')) {
|
||||
const fullPath = rootPath ? `${rootPath}/${entry}` : entry
|
||||
mdFiles.push(fullPath)
|
||||
}
|
||||
}
|
||||
|
||||
for (const dir of subDirs) {
|
||||
try {
|
||||
const nested = await listVaultFiles(baseUrl, accessToken, dir, retryOptions, depth + 1)
|
||||
mdFiles.push(...nested)
|
||||
} catch (error) {
|
||||
logger.warn('Failed to list subdirectory', {
|
||||
dir,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return mdFiles
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches a single note as structured JSON with content, frontmatter, stats, and tags.
|
||||
*/
|
||||
async function fetchNote(
|
||||
baseUrl: string,
|
||||
accessToken: string,
|
||||
filePath: string,
|
||||
retryOptions?: Parameters<typeof fetchWithRetry>[2]
|
||||
): Promise<NoteJson> {
|
||||
const response = await fetchWithRetry(
|
||||
`${baseUrl}/vault/${filePath.split('/').map(encodeURIComponent).join('/')}`,
|
||||
{
|
||||
method: 'GET',
|
||||
headers: {
|
||||
Authorization: `Bearer ${accessToken}`,
|
||||
Accept: 'application/vnd.olrapi.note+json',
|
||||
},
|
||||
},
|
||||
retryOptions
|
||||
)
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Obsidian API error fetching ${filePath}: ${response.status}`)
|
||||
}
|
||||
|
||||
return (await response.json()) as NoteJson
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts a display title from a file path.
|
||||
*/
|
||||
function titleFromPath(filePath: string): string {
|
||||
const filename = filePath.split('/').pop() || filePath
|
||||
return filename.replace(/\.md$/, '')
|
||||
}
|
||||
|
||||
export const obsidianConnector: ConnectorConfig = {
|
||||
id: 'obsidian',
|
||||
name: 'Obsidian',
|
||||
description: 'Sync notes from an Obsidian vault via the Local REST API plugin',
|
||||
version: '1.0.0',
|
||||
icon: ObsidianIcon,
|
||||
|
||||
auth: {
|
||||
mode: 'apiKey',
|
||||
label: 'API Key',
|
||||
placeholder: 'Enter your Obsidian Local REST API key',
|
||||
},
|
||||
|
||||
configFields: [
|
||||
{
|
||||
id: 'vaultUrl',
|
||||
title: 'Vault URL',
|
||||
type: 'short-input',
|
||||
placeholder: 'https://127.0.0.1:27124',
|
||||
required: true,
|
||||
description: 'Base URL of your Obsidian Local REST API (default port: 27124 for HTTPS)',
|
||||
},
|
||||
{
|
||||
id: 'folderPath',
|
||||
title: 'Folder Path',
|
||||
type: 'short-input',
|
||||
placeholder: 'e.g. Projects/Notes',
|
||||
required: false,
|
||||
description: 'Only sync notes from this folder (leave empty for entire vault)',
|
||||
},
|
||||
],
|
||||
|
||||
listDocuments: async (
|
||||
accessToken: string,
|
||||
sourceConfig: Record<string, unknown>,
|
||||
cursor?: string,
|
||||
syncContext?: Record<string, unknown>
|
||||
): Promise<ExternalDocumentList> => {
|
||||
const baseUrl = normalizeVaultUrl(
|
||||
(sourceConfig.vaultUrl as string) || 'https://127.0.0.1:27124'
|
||||
)
|
||||
const folderPath = (sourceConfig.folderPath as string) || ''
|
||||
|
||||
let allFiles = syncContext?.allFiles as string[] | undefined
|
||||
if (!allFiles) {
|
||||
logger.info('Listing all vault files', { baseUrl, folderPath })
|
||||
allFiles = await listVaultFiles(baseUrl, accessToken, folderPath || undefined)
|
||||
if (syncContext) {
|
||||
syncContext.allFiles = allFiles
|
||||
}
|
||||
}
|
||||
const offset = cursor ? Number(cursor) : 0
|
||||
const pageFiles = allFiles.slice(offset, offset + DOCS_PER_PAGE)
|
||||
|
||||
const documents: ExternalDocument[] = []
|
||||
|
||||
for (const filePath of pageFiles) {
|
||||
try {
|
||||
const note = await fetchNote(baseUrl, accessToken, filePath)
|
||||
const content = note.content || ''
|
||||
const contentHash = await computeContentHash(content)
|
||||
|
||||
documents.push({
|
||||
externalId: filePath,
|
||||
title: titleFromPath(filePath),
|
||||
content,
|
||||
mimeType: 'text/plain',
|
||||
sourceUrl: `${baseUrl}/vault/${filePath.split('/').map(encodeURIComponent).join('/')}`,
|
||||
contentHash,
|
||||
metadata: {
|
||||
tags: note.tags,
|
||||
frontmatter: note.frontmatter,
|
||||
createdAt: note.stat?.ctime ? new Date(note.stat.ctime).toISOString() : undefined,
|
||||
modifiedAt: note.stat?.mtime ? new Date(note.stat.mtime).toISOString() : undefined,
|
||||
size: note.stat?.size,
|
||||
folder: filePath.includes('/') ? filePath.substring(0, filePath.lastIndexOf('/')) : '',
|
||||
},
|
||||
})
|
||||
} catch (error) {
|
||||
logger.warn('Failed to fetch note', {
|
||||
filePath,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const nextOffset = offset + pageFiles.length
|
||||
const hasMore = nextOffset < allFiles.length
|
||||
|
||||
return {
|
||||
documents,
|
||||
nextCursor: hasMore ? String(nextOffset) : undefined,
|
||||
hasMore,
|
||||
}
|
||||
},
|
||||
|
||||
getDocument: async (
|
||||
accessToken: string,
|
||||
sourceConfig: Record<string, unknown>,
|
||||
externalId: string
|
||||
): Promise<ExternalDocument | null> => {
|
||||
const baseUrl = normalizeVaultUrl(
|
||||
(sourceConfig.vaultUrl as string) || 'https://127.0.0.1:27124'
|
||||
)
|
||||
|
||||
try {
|
||||
const note = await fetchNote(baseUrl, accessToken, externalId)
|
||||
const content = note.content || ''
|
||||
const contentHash = await computeContentHash(content)
|
||||
|
||||
return {
|
||||
externalId,
|
||||
title: titleFromPath(externalId),
|
||||
content,
|
||||
mimeType: 'text/plain',
|
||||
sourceUrl: `${baseUrl}/vault/${externalId.split('/').map(encodeURIComponent).join('/')}`,
|
||||
contentHash,
|
||||
metadata: {
|
||||
tags: note.tags,
|
||||
frontmatter: note.frontmatter,
|
||||
createdAt: note.stat?.ctime ? new Date(note.stat.ctime).toISOString() : undefined,
|
||||
modifiedAt: note.stat?.mtime ? new Date(note.stat.mtime).toISOString() : undefined,
|
||||
size: note.stat?.size,
|
||||
folder: externalId.includes('/')
|
||||
? externalId.substring(0, externalId.lastIndexOf('/'))
|
||||
: '',
|
||||
},
|
||||
}
|
||||
} catch (error) {
|
||||
logger.warn('Failed to get Obsidian note', {
|
||||
externalId,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
})
|
||||
return null
|
||||
}
|
||||
},
|
||||
|
||||
validateConfig: async (
|
||||
accessToken: string,
|
||||
sourceConfig: Record<string, unknown>
|
||||
): Promise<{ valid: boolean; error?: string }> => {
|
||||
const rawUrl = (sourceConfig.vaultUrl as string) || ''
|
||||
if (!rawUrl.trim()) {
|
||||
return { valid: false, error: 'Vault URL is required' }
|
||||
}
|
||||
|
||||
const baseUrl = normalizeVaultUrl(rawUrl)
|
||||
|
||||
try {
|
||||
const response = await fetchWithRetry(
|
||||
`${baseUrl}/`,
|
||||
{
|
||||
method: 'GET',
|
||||
headers: { Authorization: `Bearer ${accessToken}` },
|
||||
},
|
||||
VALIDATE_RETRY_OPTIONS
|
||||
)
|
||||
|
||||
if (response.status === 401 || response.status === 403) {
|
||||
return {
|
||||
valid: false,
|
||||
error: 'Invalid API key — check your Obsidian Local REST API settings',
|
||||
}
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
return { valid: false, error: `Obsidian API returned status ${response.status}` }
|
||||
}
|
||||
|
||||
const folderPath = (sourceConfig.folderPath as string) || ''
|
||||
if (folderPath.trim()) {
|
||||
const files = await listVaultFiles(
|
||||
baseUrl,
|
||||
accessToken,
|
||||
folderPath.trim(),
|
||||
VALIDATE_RETRY_OPTIONS
|
||||
)
|
||||
if (files.length === 0) {
|
||||
logger.info('Folder path returned no markdown files', { folderPath })
|
||||
}
|
||||
}
|
||||
|
||||
return { valid: true }
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Failed to connect to Obsidian vault'
|
||||
return { valid: false, error: message }
|
||||
}
|
||||
},
|
||||
|
||||
tagDefinitions: [
|
||||
{ id: 'tags', displayName: 'Tags', fieldType: 'text' },
|
||||
{ id: 'folder', displayName: 'Folder', fieldType: 'text' },
|
||||
{ id: 'modifiedAt', displayName: 'Last Modified', fieldType: 'date' },
|
||||
{ id: 'createdAt', displayName: 'Created', fieldType: 'date' },
|
||||
],
|
||||
|
||||
mapTags: (metadata: Record<string, unknown>): Record<string, unknown> => {
|
||||
const result: Record<string, unknown> = {}
|
||||
|
||||
const tags = joinTagArray(metadata.tags)
|
||||
if (tags) result.tags = tags
|
||||
|
||||
if (typeof metadata.folder === 'string' && metadata.folder) {
|
||||
result.folder = metadata.folder
|
||||
}
|
||||
|
||||
const modifiedAt = parseTagDate(metadata.modifiedAt)
|
||||
if (modifiedAt) result.modifiedAt = modifiedAt
|
||||
|
||||
const createdAt = parseTagDate(metadata.createdAt)
|
||||
if (createdAt) result.createdAt = createdAt
|
||||
|
||||
return result
|
||||
},
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { airtableConnector } from '@/connectors/airtable'
|
||||
import { asanaConnector } from '@/connectors/asana'
|
||||
import { confluenceConnector } from '@/connectors/confluence'
|
||||
import { dropboxConnector } from '@/connectors/dropbox'
|
||||
import { evernoteConnector } from '@/connectors/evernote'
|
||||
import { firefliesConnector } from '@/connectors/fireflies'
|
||||
import { githubConnector } from '@/connectors/github'
|
||||
import { googleDocsConnector } from '@/connectors/google-docs'
|
||||
@@ -10,6 +11,7 @@ import { hubspotConnector } from '@/connectors/hubspot'
|
||||
import { jiraConnector } from '@/connectors/jira'
|
||||
import { linearConnector } from '@/connectors/linear'
|
||||
import { notionConnector } from '@/connectors/notion'
|
||||
import { obsidianConnector } from '@/connectors/obsidian'
|
||||
import { onedriveConnector } from '@/connectors/onedrive'
|
||||
import { salesforceConnector } from '@/connectors/salesforce'
|
||||
import { sharepointConnector } from '@/connectors/sharepoint'
|
||||
@@ -23,6 +25,7 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
|
||||
asana: asanaConnector,
|
||||
confluence: confluenceConnector,
|
||||
dropbox: dropboxConnector,
|
||||
evernote: evernoteConnector,
|
||||
fireflies: firefliesConnector,
|
||||
github: githubConnector,
|
||||
google_docs: googleDocsConnector,
|
||||
@@ -31,6 +34,7 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
|
||||
jira: jiraConnector,
|
||||
linear: linearConnector,
|
||||
notion: notionConnector,
|
||||
obsidian: obsidianConnector,
|
||||
onedrive: onedriveConnector,
|
||||
salesforce: salesforceConnector,
|
||||
sharepoint: sharepointConnector,
|
||||
|
||||
@@ -112,11 +112,16 @@ export interface ConnectorConfig {
|
||||
lastSyncAt?: Date
|
||||
) => Promise<ExternalDocumentList>
|
||||
|
||||
/** Fetch a single document by its external ID */
|
||||
/**
|
||||
* Fetch a single document by its external ID.
|
||||
* syncContext is an optional mutable object for caching expensive lookups
|
||||
* (e.g. tag maps, notebook lists) across multiple getDocument calls.
|
||||
*/
|
||||
getDocument: (
|
||||
accessToken: string,
|
||||
sourceConfig: Record<string, unknown>,
|
||||
externalId: string
|
||||
externalId: string,
|
||||
syncContext?: Record<string, unknown>
|
||||
) => Promise<ExternalDocument | null>
|
||||
|
||||
/** Validate that sourceConfig is correct and accessible (called on save) */
|
||||
|
||||
Reference in New Issue
Block a user