feat(files): upload and safely extract ZIP archives (#6782)

* feat(files): support zip extraction

* fix(files): harden zip extraction safety

* fix(files): batch extraction notifications

* fix(files): defer rollback storage cleanup

* fix(files): restore reliable drag uploads

* fix(files): use explicit archive extraction route

* fix(ci): account for archive extraction route

* refactor(files): bound every archive extraction and trim the extractor's option surface

`maxMaterializedItems` was opt-in, so only the new unzip route bounded its output
tree — the copilot `materialize_file` and `POST /api/tools/file/manage` extract
paths had no cap on folder creation at all. An archive within MAX_ARCHIVE_ENTRIES
can still imply far more folders than files, so the cap now defaults to
MAX_WORKSPACE_FILE_BULK_AFFECTED_ITEMS and applies to all three callers.

`materializedRootFolderCount` was a hand-maintained number that had to agree with
what an opaque callback would create, and the callee could not check it; drift
surfaced only as an over-limit archive slipping past the cap. It is now derived
from whether `prepareRootFolder` ran, so the contract is just "the callback
creates exactly one folder".

Also single-sources the ArchiveError -> HTTP status map (it was copied into both
the internal error policy and the tools route), drops IdempotencyService config
that only `executeWithIdempotency` reads (the extraction lease uses
atomicallyClaim/release, so no result is ever stored), hoists the duplicated
predicates in purgeCreatedWorkspaceFile and archiveWorkspaceFileFolderIfEmpty so
a lock and its write cannot diverge, and names UPLOAD_SESSION_LOCAL_PUT_MAX_BYTES
rather than overloading the multipart part size as the local single-PUT ceiling.

Adds coverage for the two guards nothing exercised: the re-validation of the
segments `prepareRootFolder` actually returned, and the default cap applying with
no caller opt-in.

UI: the drop overlay used --surface-4 unconditionally, which renders grey over the
light-mode canvas; matches the canonical overlay's --white/dark:--surface-4 and
swaps arbitrary px type sizes for named tokens.

* fix(files): bound the extraction write loop so it cannot outlive its lease

Cursor Bugbot flagged two related holes, both rooted in the write loop being
unbounded:

1. `maxDuration` is a Next.js route-segment config that serverless platforms
   enforce and self-hosted deployments do not. A slow extraction (up to 1000
   sequential uploads) could therefore outrun the six-minute lease, and
   `IdempotencyService` reclaims an expired in-progress claim — so a second unzip
   of the same archive could start beside the first.
2. Nothing rolls back a process killed mid-pass-2, so a timeout stranded the
   destination folder and every file written so far.

`decompressArchiveBufferToWorkspaceFiles` now takes an `AbortSignal` and checks it
between entries in both passes, and the extraction use case supplies a 180s
deadline. The abort unwinds through the existing all-or-nothing rollback, so the
work stops on our terms with the tree cleaned up, well inside both the route's
300s budget and the 360s lease. That closes (1) outright — the holder can no
longer outlive its lease on any platform — and converts (2) from a stranded
partial tree into a clean rollback for the slow case that actually triggers it. A
SIGKILL still cannot be caught; that needs a durable job and is out of scope here.

The overrun surfaces as a caller-fixable 413 naming the archive rather than an
opaque 500 from the raw DOMException.

* fix(files): only remap the deadline abort itself, and stop overclaiming rollback

Two follow-ups on the budget deadline, both reported by Cursor Bugbot:

`deadline.aborted` stays true for the rest of the request once the timer fires, so
it cannot decide whether *this* error was the abort. An `ArchiveError` or storage
failure thrown mid-entry after the timer fired was being relabelled as a timeout
and returned as a 413, hiding the real cause. The catch now matches the thrown
value against `deadline.reason` — `throwIfAborted()` throws exactly that object,
so the check is identity-exact and cannot capture an unrelated failure.

The message also claimed a rollback that has not necessarily happened: the budget
covers the archive download too, so it can fire before the first write, when there
is nothing to roll back. It now says the unzip was cancelled and claims nothing
about what was written. Including the download in the budget is deliberate — the
lease it has to fit inside starts earlier still — so the TSDoc says that rather
than "the extraction itself".

---------

Co-authored-by: Waleed Latif <walif6@gmail.com>
This commit is contained in:
Justin Blumencranz
2026-08-17 17:06:57 -07:00
committed by GitHub
co-authored by Waleed Latif
parent fe4480d3f7
commit 0c34e69fdc
26 changed files with 1951 additions and 112 deletions
+2 -1
View File
@@ -34,6 +34,7 @@ import {
type DecompressResult,
decompressArchiveBufferToWorkspaceFiles,
MAX_ARCHIVE_BYTES,
statusForArchiveError,
} from '@/lib/uploads/archive'
import type { getWorkspaceFile } from '@/lib/uploads/contexts/workspace/workspace-file-manager'
import {
@@ -1183,7 +1184,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
if (archiveError instanceof ArchiveError) {
// The error message is single-sourced in ArchiveError (caps included);
// only the HTTP status is mapped here.
const status = archiveError.reason === 'invalid' ? 400 : 413
const status = statusForArchiveError(archiveError)
return NextResponse.json(
{ success: false, error: `"${archive.name}": ${archiveError.message}` },
{ status }
@@ -10,6 +10,7 @@ import { processOutboxEvents } from '@/lib/core/outbox/service'
import { generateRequestId } from '@/lib/core/utils/request'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { knowledgeDocumentProcessingOutboxHandlers } from '@/lib/knowledge/documents/processing-outbox-handler'
import { workspaceFileStorageCleanupOutboxHandlers } from '@/lib/uploads/contexts/workspace/workspace-file-storage-cleanup-outbox'
import { workflowDeploymentOutboxHandlers } from '@/lib/workflows/deployment-outbox'
import { invitationMigrationOutboxHandlers } from '@/lib/workspaces/admin-move'
import { reapStaleBackgroundWork } from '@/ee/workspace-forking/lib/background-work/store'
@@ -25,6 +26,7 @@ const handlers = {
...enterpriseIssuanceOutboxHandlers,
...invitationMigrationOutboxHandlers,
...knowledgeDocumentProcessingOutboxHandlers,
...workspaceFileStorageCleanupOutboxHandlers,
...workflowDeploymentOutboxHandlers,
} as const
@@ -0,0 +1,82 @@
/**
* @vitest-environment node
*/
import { NextRequest } from 'next/server'
import { beforeEach, describe, expect, it, vi } from 'vitest'
const mocks = vi.hoisted(() => ({
getSession: vi.fn(),
extract: vi.fn(),
}))
vi.mock('@/lib/auth', () => ({ getSession: mocks.getSession }))
vi.mock('@/lib/workspace-files/application/extract-workspace-file', () => ({
extractWorkspaceFile: {
operation: { id: 'files.extract_archive', minimumRole: 'write', workspaceApiKey: 'deny' },
execute: mocks.extract,
},
}))
import { ArchiveError } from '@/lib/uploads/archive'
import { POST } from '@/app/api/workspaces/[id]/files/[fileId]/extract/route'
const WORKSPACE_ID = 'workspace-1'
const FILE_ID = 'wf_1'
const context = { params: Promise.resolve({ id: WORKSPACE_ID, fileId: FILE_ID }) }
function callExtract() {
return POST(
new NextRequest(
`http://localhost:3000/api/workspaces/${WORKSPACE_ID}/files/${FILE_ID}/extract`,
{ method: 'POST' }
),
context
)
}
describe('POST /api/workspaces/[id]/files/[fileId]/extract', () => {
beforeEach(() => {
vi.clearAllMocks()
mocks.getSession.mockResolvedValue({
user: { id: 'user-1' },
session: { id: 'session-1' },
})
mocks.extract.mockResolvedValue({ folderName: 'bundle', extractedCount: 2, skippedCount: 0 })
})
it('passes a session principal and canonical assertion to the extraction use case', async () => {
const response = await callExtract()
expect(response.status).toBe(200)
expect(await response.json()).toEqual({
success: true,
folderName: 'bundle',
extractedCount: 2,
skippedCount: 0,
})
expect(mocks.extract).toHaveBeenCalledWith({
principal: { kind: 'session', userId: 'user-1', sessionId: 'session-1' },
input: { fileId: FILE_ID, assertedWorkspaceId: WORKSPACE_ID },
request: expect.anything(),
})
})
it('authenticates before invoking extraction', async () => {
mocks.getSession.mockResolvedValue(null)
const response = await callExtract()
expect(response.status).toBe(401)
expect(mocks.extract).not.toHaveBeenCalled()
})
it('returns a caller-safe error for an invalid zip', async () => {
mocks.extract.mockRejectedValue(new ArchiveError('invalid', 'Not a valid .zip archive.'))
const response = await callExtract()
expect(response.status).toBe(400)
expect(await response.json()).toEqual({ error: 'Not a valid .zip archive.' })
})
})
@@ -0,0 +1,27 @@
import { extractWorkspaceFileContract } from '@/lib/api/contracts/workspace-files'
import {
defineInternalJsonRoute,
internalRateLimits,
internalSessionAuth,
} from '@/lib/api/server/routes'
import { internalFileErrorPolicies } from '@/lib/workspace-files/api'
import { extractWorkspaceFile } from '@/lib/workspace-files/application/extract-workspace-file'
import { fileOperations } from '@/lib/workspace-files/application/operations'
export const dynamic = 'force-dynamic'
export const maxDuration = 300
/**
* POST /api/workspaces/[id]/files/[fileId]/extract
* Unzip an archive file into a new folder beside it (requires write permission)
*/
export const POST = defineInternalJsonRoute({
contract: extractWorkspaceFileContract,
auth: internalSessionAuth,
operation: fileOperations.extractArchive,
rateLimit: internalRateLimits.none({ reason: 'Preserve existing internal file behavior' }),
errorPolicy: internalFileErrorPolicies.extractArchive,
mapInput: ({ params }) => ({ fileId: params.fileId, assertedWorkspaceId: params.id }),
useCase: extractWorkspaceFile,
present: (result) => ({ success: true, ...result }),
})
@@ -33,12 +33,14 @@ import {
formatFileSize,
getFileExtension,
getMimeTypeFromExtension,
isArchiveFileName,
isAudioFileType,
isVideoFileType,
resolveEffectiveMimeType,
} from '@/lib/uploads/utils/file-utils'
import {
isSupportedExtension,
SUPPORTED_ARCHIVE_EXTENSIONS,
SUPPORTED_AUDIO_EXTENSIONS,
SUPPORTED_CODE_EXTENSIONS,
SUPPORTED_DOCUMENT_EXTENSIONS,
@@ -119,6 +121,7 @@ import { useWorkspaceMembersQuery, type WorkspaceMember } from '@/hooks/queries/
import {
useBulkArchiveWorkspaceFileItems,
useCreateWorkspaceFileFolder,
useExtractWorkspaceFile,
useMoveWorkspaceFileItems,
useUpdateWorkspaceFileFolder,
useWorkspaceFileFolders,
@@ -174,6 +177,7 @@ const SUPPORTED_EXTENSIONS = [
...SUPPORTED_AUDIO_EXTENSIONS,
...SUPPORTED_VIDEO_EXTENSIONS,
...SUPPORTED_IMAGE_EXTENSIONS,
...SUPPORTED_ARCHIVE_EXTENSIONS,
] as const
const ACCEPT_ATTR = SUPPORTED_EXTENSIONS.map((ext) => `.${ext}`).join(',')
@@ -189,6 +193,7 @@ const COLUMNS: ResourceColumn[] = [
const MIME_TYPE_LABELS: Record<string, string> = {
'application/pdf': 'PDF',
'application/zip': 'ZIP',
'application/msword': 'Word',
'application/vnd.openxmlformats-officedocument.wordprocessingml.document': 'Word',
'application/vnd.ms-excel': 'Excel',
@@ -277,6 +282,7 @@ export function Files() {
const deleteFile = useDeleteWorkspaceFile()
const renameFile = useRenameWorkspaceFile()
const createFolder = useCreateWorkspaceFileFolder()
const extractFile = useExtractWorkspaceFile()
const updateFolder = useUpdateWorkspaceFileFolder()
const moveItems = useMoveWorkspaceFileItems()
const bulkArchiveItems = useBulkArchiveWorkspaceFileItems()
@@ -386,6 +392,8 @@ export function Files() {
})
const [showUnsavedChangesAlert, setShowUnsavedChangesAlert] = useState(false)
const [showDeleteConfirm, setShowDeleteConfirm] = useState(false)
const [extractTargetId, setExtractTargetId] = useState<string | null>(null)
const extractTarget = extractTargetId ? (fileById.get(extractTargetId) ?? null) : null
const contextMenuItemRef = useRef<FileResourceItem | null>(null)
const [deleteTarget, setDeleteTarget] = useState<{
fileIds: string[]
@@ -1430,6 +1438,11 @@ export function Files() {
void setFilesParams({ folderId: parsed.id, new: null })
return
}
const file = fileByIdRef.current.get(parsed.id)
if (file && isArchiveFileName(file.name)) {
setExtractTargetId(file.id)
return
}
router.push(
currentFolderId
? `/workspace/${workspaceId}/files/${parsed.id}?folderId=${currentFolderId}`
@@ -1440,6 +1453,21 @@ export function Files() {
[router, workspaceId, currentFolderId, setFilesParams]
)
const handleExtract = async () => {
if (!extractTarget || !canEdit) return
try {
await extractFile.mutateAsync({
workspaceId,
fileId: extractTarget.id,
fileName: extractTarget.name,
})
} catch (error) {
logger.error('Failed to unzip archive:', error)
} finally {
setExtractTargetId(null)
}
}
const handleUploadClick = useCallback(() => {
if (!canEdit || uploading) return
fileInputRef.current?.click()
@@ -1895,9 +1923,14 @@ export function Files() {
}
/>
{isDraggingOver ? (
<div className='pointer-events-none absolute inset-0 z-10 flex flex-col items-center justify-center gap-2 border border-[var(--brand-secondary)] border-dashed bg-[var(--bg)]/80 transition-colors'>
<div className='pointer-events-none absolute inset-0 z-[var(--z-dropdown)] flex flex-col items-center justify-center gap-2 border border-[var(--brand-secondary)] border-dashed bg-[var(--white)] transition-colors dark:bg-[var(--surface-4)]'>
<Upload className='size-5 text-[var(--brand-secondary)]' />
<p className='text-[var(--brand-secondary)] text-sm'>Drop to upload</p>
<div className='flex flex-col gap-0.5 text-center'>
<p className='text-[var(--brand-secondary)] text-sm'>Drop to upload</p>
<p className='text-[var(--text-tertiary)] text-xs'>
Release files here to add them to this workspace
</p>
</div>
</div>
) : null}
</>
@@ -1944,6 +1977,25 @@ export function Files() {
isPending={deleteFile.isPending || bulkArchiveItems.isPending}
/>
<ChipConfirmModal
open={Boolean(extractTarget)}
onOpenChange={(open) => !open && setExtractTargetId(null)}
title='Unzip archive?'
text={[
'This will unzip ',
{ text: extractTarget?.name ?? 'this archive', bold: true },
' into a new folder beside it.',
]}
confirm={{
label: 'Unzip',
onClick: () => void handleExtract(),
variant: 'primary',
pending: extractFile.isPending,
pendingLabel: 'Unzipping...',
disabled: !canEdit,
}}
/>
{shareModal}
<input
@@ -0,0 +1,55 @@
/**
* @vitest-environment node
*/
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { ApiClientError } from '@/lib/api/client/errors'
import { useExtractWorkspaceFile } from '@/hooks/queries/workspace-file-folders'
const { queryClient } = vi.hoisted(() => ({
queryClient: {
invalidateQueries: vi.fn(),
},
}))
vi.mock('@sim/emcn', () => ({
toast: { error: vi.fn(), success: vi.fn() },
}))
vi.mock('@tanstack/react-query', () => ({
keepPreviousData: {},
useMutation: vi.fn((options) => options),
useQuery: vi.fn(),
useQueryClient: vi.fn(() => queryClient),
}))
vi.mock('@/lib/api/client/request', () => ({ requestJson: vi.fn() }))
const variables = { workspaceId: 'workspace-1', fileId: 'file-1', fileName: 'archive.zip' }
describe('useExtractWorkspaceFile reconciliation', () => {
beforeEach(() => {
vi.clearAllMocks()
})
it('invalidates file browsers after success', () => {
const mutation = useExtractWorkspaceFile()
mutation.onSuccess(
{ success: true, folderName: 'archive', extractedCount: 2, skippedCount: 0 },
variables
)
mutation.onSettled(undefined, undefined, variables)
expect(queryClient.invalidateQueries).toHaveBeenCalledTimes(3)
})
it('invalidates file browsers after an API error response', () => {
const mutation = useExtractWorkspaceFile()
const error = new ApiClientError({ status: 409, message: 'Folder exists', body: {} })
mutation.onError(error, variables)
mutation.onSettled(undefined, error, variables)
expect(queryClient.invalidateQueries).toHaveBeenCalledTimes(3)
})
})
@@ -11,6 +11,7 @@ import {
updateWorkspaceFileFolderContract,
type WorkspaceFileFolderApi,
} from '@/lib/api/contracts/workspace-file-folders'
import { extractWorkspaceFileContract } from '@/lib/api/contracts/workspace-files'
import {
buildWorkspaceFileFolderDisplayPath,
parseWorkspaceFileFolderDisplayPath,
@@ -87,6 +88,25 @@ export function useCreateWorkspaceFileFolder() {
})
}
export function useExtractWorkspaceFile() {
const queryClient = useQueryClient()
return useMutation({
mutationFn: async (variables: { workspaceId: string; fileId: string; fileName: string }) =>
requestJson(extractWorkspaceFileContract, {
params: { id: variables.workspaceId, fileId: variables.fileId },
}),
onSuccess: (data, variables) => {
toast.success(`Unzipped "${variables.fileName}" into "${data.folderName}"`)
},
onError: (error) => {
toast.error(toError(error).message)
},
onSettled: (_data, _error, variables) => {
invalidateWorkspaceFileBrowsers(queryClient, variables.workspaceId)
},
})
}
export function useUpdateWorkspaceFileFolder() {
const queryClient = useQueryClient()
return useMutation({
@@ -145,6 +145,14 @@ const listWorkspaceFilesResponseSchema = workspaceFileSuccessSchema.extend({
export type ListWorkspaceFilesResponse = z.output<typeof listWorkspaceFilesResponseSchema>
export const extractWorkspaceFileResponseSchema = workspaceFileSuccessSchema.extend({
folderName: z.string(),
extractedCount: z.number().int().nonnegative(),
skippedCount: z.number().int().nonnegative(),
})
export type ExtractWorkspaceFileResponse = z.output<typeof extractWorkspaceFileResponseSchema>
export const listWorkspaceFilesContract = defineRouteContract({
method: 'GET',
path: '/api/workspaces/[id]/files',
@@ -184,6 +192,16 @@ export const renameWorkspaceFileContract = defineRouteContract({
error: renameWorkspaceFileErrorSchema,
})
export const extractWorkspaceFileContract = defineRouteContract({
method: 'POST',
path: '/api/workspaces/[id]/files/[fileId]/extract',
params: workspaceFileParamsSchema,
response: {
mode: 'json',
schema: extractWorkspaceFileResponseSchema,
},
})
export const updateWorkspaceFileDimensionsContract = defineRouteContract({
method: 'PATCH',
path: '/api/workspaces/[id]/files/[fileId]/dimensions',
+278 -52
View File
@@ -16,41 +16,50 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'
* - `exactName: true` throws `FileConflictError` on a duplicate leaf name, while
* `exactName: false` auto-suffixes, mirroring `uploadWorkspaceFile`.
*/
const { store, mockUpload, mockDelete, mockEnsureFolder, mockDeleteFolder } = vi.hoisted(() => ({
store: {
folderIdByPath: new Map<string, string>(),
fileKeys: new Set<string>(),
/** Paths passed to the folder-delete operation, in call order. */
deletedFolderPaths: [] as string[],
sequence: 0,
},
mockUpload: vi.fn(),
mockDelete: vi.fn(),
mockEnsureFolder: vi.fn(),
mockDeleteFolder: vi.fn(),
}))
const { store, mockUpload, mockPurge, mockEnsureFolder, mockArchiveFolderIfEmpty, mockNotify } =
vi.hoisted(() => ({
store: {
folderIdByPath: new Map<string, string>(),
fileKeys: new Set<string>(),
blockedFolderIds: new Set<string>(),
/** Paths passed to the folder-delete operation, in call order. */
deletedFolderPaths: [] as string[],
sequence: 0,
},
mockUpload: vi.fn(),
mockPurge: vi.fn(),
mockEnsureFolder: vi.fn(),
mockArchiveFolderIfEmpty: vi.fn(),
mockNotify: vi.fn(),
}))
vi.mock('@/lib/realtime/notify', () => ({ notifyWorkspaceFilesChanged: mockNotify }))
vi.mock('@/lib/workspace-files/application/workspace-file-folders', () => ({
ensureWorkspaceFileFolderPathOperation: { execute: mockEnsureFolder },
deleteWorkspaceFileFolderOperation: { execute: mockDeleteFolder },
}))
vi.mock('@/lib/workspace-files/application/create-workspace-file', () => ({
createWorkspaceFileFromBuffer: {
execute: mockUpload,
},
}))
vi.mock('@/lib/workspace-files/application/delete-workspace-file', () => ({
deleteWorkspaceFileOperation: {
execute: mockDelete,
},
vi.mock('@/lib/uploads/contexts/workspace/workspace-file-manager', () => ({
purgeCreatedWorkspaceFile: mockPurge,
}))
vi.mock('@/lib/uploads/contexts/workspace/workspace-file-folder-manager', () => ({
archiveWorkspaceFileFolderIfEmpty: mockArchiveFolderIfEmpty,
}))
import { buildFolderPath } from '@/lib/folders/paths'
import {
buildFolderPath,
MAX_FOLDER_PATH_BYTES,
MAX_FOLDER_PATH_SEGMENTS,
} from '@/lib/folders/paths'
import {
decompressArchiveBufferToWorkspaceFiles,
MAX_ARCHIVE_CENTRAL_DIR_EXTRA_BYTES,
MAX_ARCHIVE_CENTRAL_DIR_RECORDS,
MAX_ARCHIVE_ENTRY_BYTES,
} from '@/lib/uploads/archive'
import { MAX_WORKSPACE_FILE_BULK_AFFECTED_ITEMS } from '@/lib/workspace-files/limits'
const TEST_PRINCIPAL = {
kind: 'session',
@@ -137,6 +146,7 @@ beforeEach(() => {
vi.clearAllMocks()
store.folderIdByPath.clear()
store.fileKeys.clear()
store.blockedFolderIds.clear()
store.deletedFolderPaths.length = 0
store.sequence = 0
@@ -161,24 +171,20 @@ beforeEach(() => {
return { folderId, createdFolderIds }
})
mockDeleteFolder.mockImplementation(
async ({ input }: { input: { folderId?: string; recursive?: boolean } }) => {
const path = folderPathById(input.folderId)
// Mirrors `deleteWorkspaceFileFolderOperation`, which raises `not_found` when
// nothing was archived — deleting a parent before its children would make the
// child's own delete hit this.
if (!path) throw new Error('Folder not found')
store.deletedFolderPaths.push(path)
for (const [candidate] of store.folderIdByPath) {
if (candidate === path || candidate.startsWith(`${path}/`)) {
store.folderIdByPath.delete(candidate)
}
}
return { deletedItems: { files: 0, folders: 1 } }
}
)
mockArchiveFolderIfEmpty.mockImplementation(async ({ folderId }: { folderId: string }) => {
const path = folderPathById(folderId)
if (!path) throw new Error('Folder not found')
const hasChild = [...store.folderIdByPath.keys()].some((candidate) =>
candidate.startsWith(`${path}/`)
)
if (store.blockedFolderIds.has(folderId) || hasChild) throw new Error('Folder is not empty')
store.deletedFolderPaths.push(path)
store.folderIdByPath.delete(path)
return true
})
mockDelete.mockResolvedValue(undefined)
mockPurge.mockResolvedValue(true)
mockNotify.mockResolvedValue(undefined)
mockUpload.mockImplementation(
async ({
input,
@@ -223,6 +229,8 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
expect(result.extracted).toHaveLength(2)
expect(result.skippedUnsafePaths).toEqual([])
expect(mockUpload).toHaveBeenCalledTimes(2)
expect(mockNotify).toHaveBeenCalledOnce()
expect(mockNotify).toHaveBeenCalledWith('ws')
const leafNames = mockUpload.mock.calls.map(([args]) => args.input.name).sort()
expect(leafNames).toEqual(['report.txt', 'sheet.csv'])
// Entries are rooted under the archive's folder; nested paths are preserved.
@@ -319,7 +327,7 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
'other.txt',
'report (1).txt',
])
expect(mockDelete).not.toHaveBeenCalled()
expect(mockPurge).not.toHaveBeenCalled()
})
it('marks extracted files unknown when an archive has secret provenance', async () => {
@@ -338,7 +346,10 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
expect(mockUpload).toHaveBeenCalledTimes(2)
for (const call of mockUpload.mock.calls) {
expect(call[0].input).toEqual(
expect.objectContaining({ secretProvenance: { status: 'unknown' } })
expect.objectContaining({
secretProvenance: { status: 'unknown' },
notifyWorkspaceChange: false,
})
)
}
})
@@ -459,12 +470,29 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
// (storage/DB error, quota crossed). Every file written before the failure
// must be deleted so callers and retries never observe a partial tree.
const buffer = await buildZip({ 'a.txt': 'first', 'b.txt': 'second', 'c.txt': 'third' })
const uploadedAt = new Date('2026-08-17T12:00:00.000Z')
mockUpload
.mockResolvedValueOnce({
file: { id: 'f_a', name: 'a.txt', url: '/a', key: 'k/a', size: 5 },
file: {
id: 'f_a',
name: 'a.txt',
url: '/a',
key: 'k/a',
size: 5,
folderId: 'folder_archive',
updatedAt: uploadedAt,
},
})
.mockResolvedValueOnce({
file: { id: 'f_b', name: 'b.txt', url: '/b', key: 'k/b', size: 6 },
file: {
id: 'f_b',
name: 'b.txt',
url: '/b',
key: 'k/b',
size: 6,
folderId: 'folder_archive',
updatedAt: uploadedAt,
},
})
.mockRejectedValueOnce(new Error('storage quota exceeded'))
@@ -475,13 +503,25 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
})
).rejects.toThrow('storage quota exceeded')
expect(mockDelete).toHaveBeenCalledTimes(2)
expect(mockDelete).toHaveBeenCalledWith(
expect.objectContaining({ input: { fileId: 'f_a', assertedWorkspaceId: 'ws' } })
)
expect(mockDelete).toHaveBeenCalledWith(
expect.objectContaining({ input: { fileId: 'f_b', assertedWorkspaceId: 'ws' } })
)
expect(mockPurge).toHaveBeenCalledTimes(2)
expect(mockNotify).toHaveBeenCalledOnce()
expect(mockNotify).toHaveBeenCalledWith('ws')
expect(mockPurge).toHaveBeenCalledWith({
workspaceId: 'ws',
fileId: 'f_a',
key: 'k/a',
expectedName: 'a.txt',
expectedFolderId: 'folder_archive',
expectedUpdatedAt: uploadedAt,
})
expect(mockPurge).toHaveBeenCalledWith({
workspaceId: 'ws',
fileId: 'f_b',
key: 'k/b',
expectedName: 'b.txt',
expectedFolderId: 'folder_archive',
expectedUpdatedAt: uploadedAt,
})
})
it('rolls back the folders it created when an upload fails mid-extraction', async () => {
@@ -504,10 +544,8 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
).rejects.toThrow('storage quota exceeded')
expect([...store.folderIdByPath.keys()]).toEqual([])
expect(mockDeleteFolder).toHaveBeenCalledWith(
expect.objectContaining({
input: expect.objectContaining({ workspaceId: 'ws', recursive: true }),
})
expect(mockArchiveFolderIfEmpty).toHaveBeenCalledWith(
expect.objectContaining({ workspaceId: 'ws' })
)
})
@@ -533,7 +571,7 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
expect([...store.folderIdByPath.keys()].sort()).toEqual(['/bundle', '/bundle/keep'])
expect(store.deletedFolderPaths).toEqual(['/bundle/fresh'])
const deletedIds = mockDeleteFolder.mock.calls.map(([args]) => args.input.folderId)
const deletedIds = mockArchiveFolderIfEmpty.mock.calls.map(([args]) => args.folderId)
for (const preexistingId of preexistingIds) {
expect(deletedIds).not.toContain(preexistingId)
}
@@ -562,6 +600,27 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
expect([...store.folderIdByPath.keys()]).toEqual([])
})
it('preserves created folders that gain collaborator content before rollback', async () => {
const buffer = await buildZip({ 'nested/one.txt': 'first', 'nested/two.txt': 'second' })
mockUpload
.mockImplementationOnce(async ({ input }) => {
store.blockedFolderIds.add(input.folderId)
return { file: { id: 'f_one', name: 'one.txt', url: '/one', key: 'k/one', size: 5 } }
})
.mockRejectedValueOnce(new Error('storage quota exceeded'))
await expect(
decompressArchiveBufferToWorkspaceFiles(buffer, {
workspaceId: 'ws',
principal: TEST_PRINCIPAL,
rootFolderSegments: ['bundle'],
})
).rejects.toThrow('storage quota exceeded')
expect([...store.folderIdByPath.keys()]).toEqual(['/bundle', '/bundle/nested'])
expect(store.deletedFolderPaths).toEqual([])
})
it('does not count noise entries toward the extraction cap when they are being skipped', async () => {
// macOS Finder zips carry a __MACOSX/._* shadow per file, doubling the raw
// entry count. 501 files + 501 shadows = 1002 raw entries — over the
@@ -584,6 +643,173 @@ describe('decompressArchiveBufferToWorkspaceFiles', () => {
expect(result.skipped).toBe(501)
})
it('rejects a materialized tree above the bulk-operation limit before creating its root', async () => {
const buffer = await buildZip({ 'nested/file.txt': 'x' })
const prepareRootFolder = vi.fn(async () => ['bundle'])
await expect(
decompressArchiveBufferToWorkspaceFiles(buffer, {
workspaceId: 'ws',
principal: TEST_PRINCIPAL,
prepareRootFolder,
maxMaterializedItems: 2,
})
).rejects.toMatchObject({ name: 'ArchiveError', reason: 'too_many_entries' })
expect(prepareRootFolder).not.toHaveBeenCalled()
expect(mockEnsureFolder).not.toHaveBeenCalled()
expect(mockUpload).not.toHaveBeenCalled()
await expect(
decompressArchiveBufferToWorkspaceFiles(buffer, {
workspaceId: 'ws',
principal: TEST_PRINCIPAL,
prepareRootFolder,
maxMaterializedItems: 3,
})
).resolves.toMatchObject({ extracted: [expect.objectContaining({ name: 'file.txt' })] })
expect(prepareRootFolder).toHaveBeenCalledOnce()
})
it.each([
{
label: 'too many folder segments',
entryName: `${Array.from({ length: MAX_FOLDER_PATH_SEGMENTS + 1 }, () => 'x').join('/')}/file.txt`,
expectedMessage: `Folder paths cannot exceed ${MAX_FOLDER_PATH_SEGMENTS} segments`,
},
{
label: 'too many encoded folder-path bytes',
entryName: `${'x'.repeat(MAX_FOLDER_PATH_BYTES)}/file.txt`,
expectedMessage: `Folder paths cannot exceed ${MAX_FOLDER_PATH_BYTES} bytes`,
},
])(
'rejects $label before enumerating or creating folders',
async ({ entryName, expectedMessage }) => {
const buffer = await buildZip({ [entryName]: 'x' })
const prepareRootFolder = vi.fn(async () => ['bundle'])
await expect(
decompressArchiveBufferToWorkspaceFiles(buffer, {
workspaceId: 'ws',
principal: TEST_PRINCIPAL,
prepareRootFolder,
maxMaterializedItems: 5000,
})
).rejects.toMatchObject({
name: 'ArchiveError',
reason: 'invalid',
message: expect.stringContaining(expectedMessage),
})
expect(prepareRootFolder).not.toHaveBeenCalled()
expect(mockEnsureFolder).not.toHaveBeenCalled()
expect(mockUpload).not.toHaveBeenCalled()
}
)
it('includes the destination prefix when validating archive folder paths', async () => {
const buffer = await buildZip({ 'nested/file.txt': 'x' })
const prepareRootFolder = vi.fn(async () => ['unused'])
await expect(
decompressArchiveBufferToWorkspaceFiles(buffer, {
workspaceId: 'ws',
principal: TEST_PRINCIPAL,
rootFolderSegments: Array.from(
{ length: MAX_FOLDER_PATH_SEGMENTS },
(_, index) => `existing-${index}`
),
prepareRootFolder,
})
).rejects.toMatchObject({
name: 'ArchiveError',
reason: 'invalid',
message: expect.stringContaining(
`Folder paths cannot exceed ${MAX_FOLDER_PATH_SEGMENTS} segments`
),
})
expect(prepareRootFolder).not.toHaveBeenCalled()
expect(mockEnsureFolder).not.toHaveBeenCalled()
expect(mockUpload).not.toHaveBeenCalled()
})
it('aborts mid-extraction on the caller signal and rolls the partial tree back', async () => {
const buffer = await buildZip({ 'a.txt': 'x', 'b.txt': 'y', 'c.txt': 'z' })
const controller = new AbortController()
const commit = mockUpload.getMockImplementation()!
mockUpload.mockImplementation(async (args: any) => {
const uploaded = await commit(args)
// Abort once the first file is committed, so rollback has something to undo.
controller.abort()
return uploaded
})
await expect(
decompressArchiveBufferToWorkspaceFiles(buffer, {
workspaceId: 'ws',
principal: TEST_PRINCIPAL,
signal: controller.signal,
})
).rejects.toMatchObject({ name: 'AbortError' })
expect(mockUpload).toHaveBeenCalledOnce()
expect(mockPurge).toHaveBeenCalledOnce()
expect(mockPurge).toHaveBeenCalledWith(expect.objectContaining({ expectedName: 'a.txt' }))
})
it('applies the workspace bulk limit by default, so every caller is bounded', async () => {
// 1000 files, each under six folders unique to it: 7000 materialized items from an entry
// count that is itself within MAX_ARCHIVE_ENTRIES. No caller opts in to this cap.
const entries: Record<string, string> = {}
for (let index = 0; index < 1000; index += 1) {
entries[`a${index}/b/c/d/e/f/file.txt`] = 'x'
}
const buffer = await buildZip(entries)
await expect(
decompressArchiveBufferToWorkspaceFiles(buffer, {
workspaceId: 'ws',
principal: TEST_PRINCIPAL,
})
).rejects.toMatchObject({
name: 'ArchiveError',
reason: 'too_many_entries',
message: expect.stringContaining(`the maximum is ${MAX_WORKSPACE_FILE_BULK_AFFECTED_ITEMS}`),
})
expect(mockEnsureFolder).not.toHaveBeenCalled()
expect(mockUpload).not.toHaveBeenCalled()
})
it('re-validates the segments prepareRootFolder actually returned, before any upload', async () => {
const buffer = await buildZip({ 'nested/file.txt': 'x' })
// A callback that validates one path and returns a different, invalid one. The callee
// cannot trust the callback to have checked what it returns, so it re-checks itself.
const prepareRootFolder = vi.fn(async (validate: (segments: string[]) => void) => {
validate(['bundle'])
return Array.from({ length: MAX_FOLDER_PATH_SEGMENTS + 1 }, (_, index) => `deep-${index}`)
})
await expect(
decompressArchiveBufferToWorkspaceFiles(buffer, {
workspaceId: 'ws',
principal: TEST_PRINCIPAL,
prepareRootFolder,
})
).rejects.toMatchObject({
name: 'ArchiveError',
reason: 'invalid',
message: expect.stringContaining(
`Folder paths cannot exceed ${MAX_FOLDER_PATH_SEGMENTS} segments`
),
})
expect(prepareRootFolder).toHaveBeenCalledOnce()
expect(mockEnsureFolder).not.toHaveBeenCalled()
expect(mockUpload).not.toHaveBeenCalled()
})
it('throws ArchiveError invalid for a non-zip buffer (no files written)', async () => {
await expect(
decompressArchiveBufferToWorkspaceFiles(Buffer.from('not a zip at all'), {
+133 -19
View File
@@ -1,16 +1,21 @@
import { Buffer } from 'buffer'
import type { Readable } from 'stream'
import type { Principal } from '@sim/auth/principal'
import { createLogger } from '@sim/logger'
import JSZip from 'jszip'
import { readZipCentralDirectoryStats } from '@/lib/file-parsers/zip-guard'
import { buildFolderPath, FolderPathError } from '@/lib/folders/paths'
import { notifyWorkspaceFilesChanged } from '@/lib/realtime/notify'
import { archiveWorkspaceFileFolderIfEmpty } from '@/lib/uploads/contexts/workspace/workspace-file-folder-manager'
import {
purgeCreatedWorkspaceFile,
type WorkspaceFileRecord,
} from '@/lib/uploads/contexts/workspace/workspace-file-manager'
import type { WorkspaceFileSecretProvenance } from '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance'
import { getFileExtension, getMimeTypeFromExtension } from '@/lib/uploads/utils/file-utils'
import { createWorkspaceFileFromBuffer } from '@/lib/workspace-files/application/create-workspace-file'
import { deleteWorkspaceFileOperation } from '@/lib/workspace-files/application/delete-workspace-file'
import {
deleteWorkspaceFileFolderOperation,
ensureWorkspaceFileFolderPathOperation,
} from '@/lib/workspace-files/application/workspace-file-folders'
import { ensureWorkspaceFileFolderPathOperation } from '@/lib/workspace-files/application/workspace-file-folders'
import { MAX_WORKSPACE_FILE_BULK_AFFECTED_ITEMS } from '@/lib/workspace-files/limits'
import type { UserFile } from '@/executor/types'
/**
@@ -31,6 +36,8 @@ import type { UserFile } from '@/executor/types'
* entry in both passes.
*/
const logger = createLogger('ArchiveExtraction')
/** Input archive download/size cap. */
export const MAX_ARCHIVE_BYTES = 100 * 1024 * 1024
/** Maximum number of entries extracted from a single archive. */
@@ -81,6 +88,15 @@ export class ArchiveError extends Error {
}
}
/**
* The caller-facing HTTP status for an {@link ArchiveError}. A malformed archive is the
* caller's request being wrong (400); every other reason is a cap the payload exceeded (413).
* Single-sourced beside the reason union so a new variant is classified in exactly one place.
*/
export function statusForArchiveError(error: ArchiveError): number {
return error.reason === 'invalid' ? 400 : 413
}
const MB = 1024 * 1024
/**
@@ -255,6 +271,23 @@ function throwInflateCapError(reason: 'entry' | 'total', entryName: string): nev
* re-inflates and uploads one entry at a time. Peak memory stays ~one entry in
* both passes; the cost is inflating twice (CPU only, bounded by the caps).
*
* `signal` aborts between entries in both passes. Callers that hold a lease or run
* under a request deadline must pass one: the write loop is otherwise unbounded, and a
* process killed mid-pass-2 strands a partial tree that no `catch` can roll back.
* Aborting instead unwinds through the same all-or-nothing rollback as any other failure.
*
* When `prepareRootFolder` is provided it takes precedence over
* `rootFolderSegments`: it runs once, only after the caps have been proven and
* only when at least one safe entry exists, and extraction lands under the
* segments it returns. It must create exactly one folder, and must pass its
* final segments to the supplied validator before inserting it so the complete
* destination path is rejected before any folder mutation. That one folder is
* counted by the `maxMaterializedItems` pre-check (files + implied folders +
* root folder), which rejects an over-limit archive before the callback
* materializes anything. That cap always applies — it defaults to
* {@link MAX_WORKSPACE_FILE_BULK_AFFECTED_ITEMS} — because the file count alone
* does not bound folder creation.
*
* Filesystem-noise entries (`__MACOSX/`, `.DS_Store`, `Thumbs.db`) are extracted
* verbatim unless `skipNoiseEntries` is set — the HTTP decompress route preserves
* them; the agent-facing extract path drops them. Decompression is not byte-preserving,
@@ -267,16 +300,26 @@ export async function decompressArchiveBufferToWorkspaceFiles(
workspaceId: string
principal: Principal
rootFolderSegments?: string[]
prepareRootFolder?: (
validateRootFolderSegments: (rootFolderSegments: string[]) => void
) => Promise<string[]>
signal?: AbortSignal
maxMaterializedItems?: number
skipNoiseEntries?: boolean
secretProvenance?: WorkspaceFileSecretProvenance
notifyWorkspaceChange?: boolean
}
): Promise<DecompressResult> {
const {
workspaceId,
principal,
rootFolderSegments = [],
prepareRootFolder,
signal,
maxMaterializedItems = MAX_WORKSPACE_FILE_BULK_AFFECTED_ITEMS,
skipNoiseEntries = false,
secretProvenance = { status: 'unknown' },
notifyWorkspaceChange = true,
} = opts
const extractedSecretProvenance: WorkspaceFileSecretProvenance =
secretProvenance.status === 'exact' && secretProvenance.entries.length === 0
@@ -323,6 +366,46 @@ export async function decompressArchiveBufferToWorkspaceFiles(
)
}
const validateRootFolderSegments = (candidateRootFolderSegments: string[]): void => {
for (const { entry, segments } of safeEntries) {
try {
buildFolderPath([...candidateRootFolderSegments, ...segments.slice(0, -1)])
} catch (error) {
if (!(error instanceof FolderPathError)) throw error
throw new ArchiveError(
'invalid',
`Archive contains an invalid folder path: ${error.message}`,
entry.name
)
}
}
}
validateRootFolderSegments(rootFolderSegments)
const impliedFolderPaths = new Set<string>()
for (const { segments } of safeEntries) {
let prefix = ''
for (let depth = 0; depth < segments.length - 1; depth += 1) {
prefix += `\0${segments[depth]}`
impliedFolderPaths.add(prefix)
}
}
/**
* Bounds the whole output tree, not just the file count: 1000 entries nested 64 deep imply
* far more folders than files, and nothing else caps folder creation. `prepareRootFolder`
* contributes exactly one more folder when it runs.
*/
const materializedItems =
safeEntries.length +
impliedFolderPaths.size +
(safeEntries.length > 0 && prepareRootFolder ? 1 : 0)
if (materializedItems > maxMaterializedItems) {
throw new ArchiveError(
'too_many_entries',
`Archive would create ${materializedItems} files and folders; the maximum is ${maxMaterializedItems}.`
)
}
// Cheap declared-size fast-reject for honestly-declared archives.
let declaredTotal = 0
for (const { entry } of safeEntries) {
@@ -337,6 +420,7 @@ export async function decompressArchiveBufferToWorkspaceFiles(
// persisting anything, so a lying header aborts before any upload happens.
let validatedTotal = 0
for (const { entry } of safeEntries) {
signal?.throwIfAborted()
const result = await inflateEntryWithinCaps(
entry,
MAX_ARCHIVE_TOTAL_BYTES - validatedTotal,
@@ -346,6 +430,15 @@ export async function decompressArchiveBufferToWorkspaceFiles(
validatedTotal += result.size
}
const resolvedRootFolderSegments =
safeEntries.length > 0 && prepareRootFolder
? await prepareRootFolder(validateRootFolderSegments)
: rootFolderSegments
// Re-check what the callback actually returned; identical segments were proven above.
if (resolvedRootFolderSegments !== rootFolderSegments) {
validateRootFolderSegments(resolvedRootFolderSegments)
}
// Pass 2 — extract: the archive is proven within caps; inflate again and upload.
// Uploads themselves can still fail mid-loop (storage/DB errors, quota crossed
// by another writer), so a failure rolls back every file written so far *and*
@@ -356,17 +449,19 @@ export async function decompressArchiveBufferToWorkspaceFiles(
const folderIdCache = new Map<string, string | null>()
/** Only folders this call inserted, in creation order — never a reused one. */
const createdFolderIds: string[] = []
const createdFiles: WorkspaceFileRecord[] = []
const extracted: UserFile[] = []
let totalBytes = 0
try {
for (const { entry, segments } of safeEntries) {
signal?.throwIfAborted()
const result = await inflateEntryWithinCaps(entry, MAX_ARCHIVE_TOTAL_BYTES - totalBytes, true)
if (!result.ok) throwInflateCapError(result.reason, entry.name)
totalBytes += result.size
const entryBuffer = result.buffer as Buffer
const leafName = segments[segments.length - 1]
const folderSegments = [...rootFolderSegments, ...segments.slice(0, -1)]
const folderSegments = [...resolvedRootFolderSegments, ...segments.slice(0, -1)]
const folderKey = folderSegments.join('/')
let folderId = folderIdCache.get(folderKey)
if (folderId === undefined) {
@@ -396,9 +491,11 @@ export async function decompressArchiveBufferToWorkspaceFiles(
// roll back an otherwise valid extraction.
exactName: false,
secretProvenance: extractedSecretProvenance,
notifyWorkspaceChange: false,
},
})
).file
createdFiles.push(uploaded)
extracted.push({
id: uploaded.id,
name: uploaded.name,
@@ -410,32 +507,49 @@ export async function decompressArchiveBufferToWorkspaceFiles(
})
}
} catch (error) {
for (const file of extracted) {
for (const file of createdFiles) {
try {
await deleteWorkspaceFileOperation.execute({
principal,
input: { fileId: file.id, assertedWorkspaceId: workspaceId },
await purgeCreatedWorkspaceFile({
workspaceId,
fileId: file.id,
key: file.key,
expectedName: file.name,
expectedFolderId: file.folderId ?? null,
expectedUpdatedAt: file.updatedAt,
})
} catch (cleanupError) {
// Best-effort cleanup never masks the extraction failure.
logger.error('Failed to purge extracted file during rollback', {
workspaceId,
fileId: file.id,
key: file.key,
cleanupError,
})
} catch {
// Best-effort: a file whose cleanup fails is still soft-deletable by hand;
// the original error is what the caller needs to see.
}
}
// Deepest-first (creation order records parents before children), so a parent is
// never removed out from under a child that is still being cleaned up.
for (let index = createdFolderIds.length - 1; index >= 0; index--) {
try {
await deleteWorkspaceFileFolderOperation.execute({
principal,
input: { workspaceId, folderId: createdFolderIds[index], recursive: true },
await archiveWorkspaceFileFolderIfEmpty({
workspaceId,
folderId: createdFolderIds[index],
})
} catch (cleanupError) {
// Best-effort cleanup never masks the extraction failure.
logger.warn('Failed to archive created folder during rollback', {
workspaceId,
folderId: createdFolderIds[index],
cleanupError,
})
} catch {
// Best-effort: a folder whose cleanup fails is still deletable by hand;
// the original error is what the caller needs to see.
}
}
if (notifyWorkspaceChange) await notifyWorkspaceFilesChanged(workspaceId)
throw error
}
if (notifyWorkspaceChange && extracted.length > 0) {
await notifyWorkspaceFilesChanged(workspaceId)
}
return { extracted, skipped, skippedUnsafePaths }
}
@@ -2,11 +2,28 @@
* @vitest-environment node
*/
import { describe, expect, it } from 'vitest'
import { dbChainMockFns, resetDbChainMock } from '@sim/testing'
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockAcquireFolderMutationLock, mockDeduplicateFolderName } = vi.hoisted(() => ({
mockAcquireFolderMutationLock: vi.fn(),
mockDeduplicateFolderName: vi.fn(),
}))
vi.mock('@/lib/folders/locks', () => ({
acquireFolderMutationLock: mockAcquireFolderMutationLock,
}))
vi.mock('@/lib/folders/naming', () => ({
deduplicateFolderName: mockDeduplicateFolderName,
}))
import { asOrchestrationError, statusForOrchestrationError } from '@/lib/core/orchestration/types'
import { MAX_FOLDER_PATH_SEGMENTS } from '@/lib/folders/paths'
import {
archiveWorkspaceFileFolderIfEmpty,
buildWorkspaceFileFolderPathMap,
createWorkspaceFileFolder,
ensureWorkspaceFileFolderPath,
normalizeWorkspaceFileItemName,
WorkspaceFileFolderConflictError,
@@ -14,6 +31,58 @@ import {
WorkspaceFileMoveConflictError,
} from '@/lib/uploads/contexts/workspace/workspace-file-folder-manager'
describe('createWorkspaceFileFolder', () => {
beforeEach(() => {
resetDbChainMock()
mockAcquireFolderMutationLock.mockReset()
mockDeduplicateFolderName.mockReset()
})
it('uses the shared numeric suffix allocator when exact naming is disabled', async () => {
const now = new Date('2026-08-17T12:00:00.000Z')
const inserted = {
id: 'folder-archive-3',
resourceType: 'file',
workspaceId: 'workspace-1',
userId: 'user-1',
name: 'Archive (3)',
parentId: null,
sortOrder: 0,
deletedAt: null,
createdAt: now,
updatedAt: now,
}
const validateResolvedName = vi.fn()
mockDeduplicateFolderName.mockResolvedValueOnce('Archive (3)')
dbChainMockFns.returning.mockResolvedValueOnce([inserted])
await expect(
createWorkspaceFileFolder({
workspaceId: 'workspace-1',
userId: 'user-1',
name: 'Archive',
exactName: false,
validateResolvedName,
})
).resolves.toMatchObject({ name: 'Archive (3)' })
expect(mockDeduplicateFolderName).toHaveBeenCalledWith(
expect.anything(),
'workspace-1',
null,
'Archive',
'file'
)
expect(dbChainMockFns.values).toHaveBeenCalledWith(
expect.objectContaining({ name: 'Archive (3)' })
)
expect(validateResolvedName).toHaveBeenCalledWith('Archive (3)')
expect(validateResolvedName.mock.invocationCallOrder[0]).toBeLessThan(
dbChainMockFns.values.mock.invocationCallOrder[0]
)
})
})
describe('workspace file folder paths', () => {
it('builds nested paths from parent relationships', () => {
const paths = buildWorkspaceFileFolderPathMap([
@@ -98,3 +167,62 @@ describe('workspace file folder failure classification', () => {
expect(asOrchestrationError(wrapped)?.code).toBe('conflict')
})
})
describe('archiveWorkspaceFileFolderIfEmpty', () => {
beforeEach(() => {
resetDbChainMock()
mockAcquireFolderMutationLock.mockReset()
})
it('archives an empty folder under the folder mutation lock', async () => {
dbChainMockFns.limit
.mockResolvedValueOnce([{ id: 'folder-1' }])
.mockResolvedValueOnce([])
.mockResolvedValueOnce([])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'folder-1' }])
await expect(
archiveWorkspaceFileFolderIfEmpty({ workspaceId: 'workspace-1', folderId: 'folder-1' })
).resolves.toBe(true)
expect(mockAcquireFolderMutationLock).toHaveBeenCalledWith(
expect.anything(),
'workspace-1',
'file'
)
})
it('returns false without archiving when the folder is missing or already archived', async () => {
dbChainMockFns.limit.mockResolvedValueOnce([])
await expect(
archiveWorkspaceFileFolderIfEmpty({ workspaceId: 'workspace-1', folderId: 'folder-1' })
).resolves.toBe(false)
expect(dbChainMockFns.returning).not.toHaveBeenCalled()
})
it('refuses to archive a folder that still holds an active file', async () => {
dbChainMockFns.limit
.mockResolvedValueOnce([{ id: 'folder-1' }])
.mockResolvedValueOnce([])
.mockResolvedValueOnce([{ id: 'file-1' }])
await expect(
archiveWorkspaceFileFolderIfEmpty({ workspaceId: 'workspace-1', folderId: 'folder-1' })
).rejects.toMatchObject({ code: 'conflict', message: 'Folder is not empty' })
expect(dbChainMockFns.returning).not.toHaveBeenCalled()
})
it('refuses to archive a folder with an active child folder', async () => {
dbChainMockFns.limit
.mockResolvedValueOnce([{ id: 'folder-1' }])
.mockResolvedValueOnce([{ id: 'child-1' }])
.mockResolvedValueOnce([])
await expect(
archiveWorkspaceFileFolderIfEmpty({ workspaceId: 'workspace-1', folderId: 'folder-1' })
).rejects.toMatchObject({ code: 'conflict' })
})
})
@@ -493,8 +493,11 @@ export async function createWorkspaceFileFolder(params: {
name: string
parentId?: string | null
sortOrder?: number
exactName?: boolean
/** Validates the exact post-deduplication name before the folder row is inserted. */
validateResolvedName?: (name: string) => void
}): Promise<WorkspaceFileFolderRecord> {
const name = normalizeWorkspaceFileItemName(params.name, 'Folder')
const requestedName = normalizeWorkspaceFileItemName(params.name, 'Folder')
const folder = await db.transaction(async (tx) => {
await acquireWorkspaceFileFolderMutationLock(tx, params.workspaceId)
@@ -519,22 +522,37 @@ export async function createWorkspaceFileFolder(params: {
}
}
const existingFolders = await tx
.select({ id: folderTable.id })
.from(folderTable)
.where(
and(
eq(folderTable.workspaceId, params.workspaceId),
isFileFolder,
eq(folderTable.name, name),
folderParentCondition(parentId),
isNull(folderTable.deletedAt)
const deduplicate = params.exactName === false
const name = deduplicate
? await deduplicateFolderName(
tx,
params.workspaceId,
parentId,
requestedName,
FILE_FOLDER_RESOURCE_TYPE
)
)
.limit(1)
: requestedName
if (existingFolders.length > 0) {
throw new WorkspaceFileFolderConflictError(name)
params.validateResolvedName?.(name)
if (!deduplicate) {
const existingFolders = await tx
.select({ id: folderTable.id })
.from(folderTable)
.where(
and(
eq(folderTable.workspaceId, params.workspaceId),
isFileFolder,
eq(folderTable.name, name),
folderParentCondition(parentId),
isNull(folderTable.deletedAt)
)
)
.limit(1)
if (existingFolders.length > 0) {
throw new WorkspaceFileFolderConflictError(name)
}
}
const [sortOrderResult] = await tx
@@ -1570,3 +1588,59 @@ export async function deleteWorkspaceFileFolderByPath(params: {
return { folders: archivedFolders.length, files: archivedFiles.length }
})
}
/** Archives an exact folder only while it has no active files or child folders. */
export async function archiveWorkspaceFileFolderIfEmpty(params: {
workspaceId: string
folderId: string
}): Promise<boolean> {
const isTargetFolder = and(
eq(folderTable.id, params.folderId),
eq(folderTable.workspaceId, params.workspaceId),
isFileFolder,
isNull(folderTable.deletedAt)
)
return db.transaction(async (tx) => {
await acquireWorkspaceFileFolderMutationLock(tx, params.workspaceId)
const [folder] = await tx
.select({ id: folderTable.id })
.from(folderTable)
.where(isTargetFolder)
.limit(1)
if (!folder) return false
const [childFolder] = await tx
.select({ id: folderTable.id })
.from(folderTable)
.where(
and(
eq(folderTable.parentId, params.folderId),
eq(folderTable.workspaceId, params.workspaceId),
isFileFolder,
isNull(folderTable.deletedAt)
)
)
.limit(1)
const [file] = await tx
.select({ id: workspaceFiles.id })
.from(workspaceFiles)
.where(
and(
eq(workspaceFiles.folderId, params.folderId),
eq(workspaceFiles.workspaceId, params.workspaceId),
eq(workspaceFiles.context, 'workspace'),
isNull(workspaceFiles.deletedAt)
)
)
.limit(1)
if (childFolder || file) throw new OrchestrationError('conflict', 'Folder is not empty')
const [archived] = await tx
.update(folderTable)
.set({ deletedAt: new Date(), updatedAt: new Date() })
.where(isTargetFolder)
.returning({ id: folderTable.id })
return Boolean(archived)
})
}
@@ -57,6 +57,10 @@ import {
type WorkspaceFileSecretProvenance,
type WorkspaceFileSecretProvenancePolicy,
} from '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance'
import {
enqueueWorkspaceFileStorageCleanup,
processWorkspaceFileStorageCleanupNow,
} from '@/lib/uploads/contexts/workspace/workspace-file-storage-cleanup-outbox'
import { buildStorageKeySegment } from '@/lib/uploads/core/storage-key'
import {
deleteFile,
@@ -390,6 +394,7 @@ export async function uploadWorkspaceFile(
folderPath?: string
exactName?: boolean
secretProvenance?: WorkspaceFileSecretProvenance
notifyWorkspaceChange?: boolean
}
): Promise<UploadedWorkspaceFileRecord> {
logger.info(`Uploading workspace file: ${fileName} for workspace ${workspaceId}`)
@@ -512,9 +517,9 @@ export async function uploadWorkspaceFile(
`Successfully uploaded workspace file: ${uniqueName} with key: ${uploadResult.key}`
)
// Fan out the live-tree signal for this server-buffered path. Upload-session
// finalization sends its own notification after registering metadata.
await notifyWorkspaceFilesChanged(workspaceId)
if (options?.notifyWorkspaceChange !== false) {
await notifyWorkspaceFilesChanged(workspaceId)
}
return mapUploadedWorkspaceFileRecord(finalized.inserted, workspaceId, folderPath)
} catch (error) {
@@ -2048,6 +2053,87 @@ export async function deleteWorkspaceFile(workspaceId: string, fileId: string):
}
}
/**
* Permanently removes a file created by an in-flight archive extraction only while
* its name, folder, and update timestamp still match the creation result. The matching
* metadata deletion, accounting update, and durable storage-cleanup event commit together. This
* is rollback-only: ordinary user deletion remains recoverable through
* {@link deleteWorkspaceFile}.
*/
export async function purgeCreatedWorkspaceFile(params: {
workspaceId: string
fileId: string
key: string
expectedName: string
expectedFolderId: string | null
expectedUpdatedAt: Date
}): Promise<boolean> {
const storageBillingContext = await resolveStorageBillingContext(params.workspaceId)
const expectedFolder =
params.expectedFolderId === null
? isNull(workspaceFiles.folderId)
: eq(workspaceFiles.folderId, params.expectedFolderId)
/** The full creation identity. Shared so the lock and the delete can never diverge. */
const matchesCreatedFile = and(
eq(workspaceFiles.id, params.fileId),
eq(workspaceFiles.workspaceId, params.workspaceId),
eq(workspaceFiles.key, params.key),
eq(workspaceFiles.originalName, params.expectedName),
expectedFolder,
eq(workspaceFiles.updatedAt, params.expectedUpdatedAt),
eq(workspaceFiles.context, 'workspace'),
isNull(workspaceFiles.deletedAt)
)
const cleanupEventId = await db.transaction(async (tx) => {
const [lockedFile] = await tx
.select({
id: workspaceFiles.id,
key: workspaceFiles.key,
size: workspaceFiles.size,
sizeBytes: workspaceFiles.sizeBytes,
})
.from(workspaceFiles)
.where(matchesCreatedFile)
.for('update')
.limit(1)
if (!lockedFile) return null
const [deleted] = await tx
.delete(workspaceFiles)
.where(matchesCreatedFile)
.returning({ id: workspaceFiles.id })
if (!deleted) throw new Error('Locked archive-created file could not be deleted')
await decrementStorageUsageForBillingContextInTx(
tx,
storageBillingContext,
workspaceFileSize(lockedFile)
)
return enqueueWorkspaceFileStorageCleanup(tx, { key: lockedFile.key })
})
if (!cleanupEventId) return false
try {
const result = await processWorkspaceFileStorageCleanupNow(cleanupEventId)
if (result !== 'completed') {
logger.warn('Archive rollback storage cleanup deferred to outbox retry', {
workspaceId: params.workspaceId,
fileId: params.fileId,
cleanupEventId,
result,
})
}
} catch (error) {
logger.warn('Archive rollback storage cleanup deferred after inline processing error', {
workspaceId: params.workspaceId,
fileId: params.fileId,
cleanupEventId,
error: getErrorMessage(error),
})
}
return true
}
/**
* Restore a soft-deleted workspace file.
*/
@@ -1,13 +1,16 @@
/**
* @vitest-environment node
*/
import { workspaceFiles } from '@sim/db/schema'
import { dbChainMockFns, resetDbChainMock } from '@sim/testing'
import { describeError } from '@sim/utils/errors'
import { eq } from 'drizzle-orm'
import { beforeEach, describe, expect, it, vi } from 'vitest'
const {
mockDecrementStorageUsageForBillingContextInTx,
mockDeleteFile,
mockEnqueueWorkspaceFileStorageCleanup,
mockGetWorkspaceWithOwner,
mockHasCloudStorage,
mockHeadObject,
@@ -19,6 +22,7 @@ const {
mockMaybeNotifyStorageLimitForBillingContext,
mockMergeEditIntoLiveFileDoc,
mockNotifyWorkspaceFilesChanged,
mockProcessWorkspaceFileStorageCleanupNow,
mockResolveStorageBillingContext,
mockResolveFolderPathFromIndex,
mockResolveWorkspaceFileFolderTarget,
@@ -27,6 +31,7 @@ const {
} = vi.hoisted(() => ({
mockDecrementStorageUsageForBillingContextInTx: vi.fn(),
mockDeleteFile: vi.fn(),
mockEnqueueWorkspaceFileStorageCleanup: vi.fn(),
mockGetWorkspaceWithOwner: vi.fn(),
mockHasCloudStorage: vi.fn(),
mockHeadObject: vi.fn(),
@@ -38,6 +43,7 @@ const {
mockMaybeNotifyStorageLimitForBillingContext: vi.fn(),
mockMergeEditIntoLiveFileDoc: vi.fn(),
mockNotifyWorkspaceFilesChanged: vi.fn(),
mockProcessWorkspaceFileStorageCleanupNow: vi.fn(),
mockResolveStorageBillingContext: vi.fn(),
mockResolveFolderPathFromIndex: vi.fn(),
mockResolveWorkspaceFileFolderTarget: vi.fn(),
@@ -76,6 +82,11 @@ vi.mock('@/lib/uploads/core/storage-service', () => ({
uploadFile: mockUploadFile,
}))
vi.mock('@/lib/uploads/contexts/workspace/workspace-file-storage-cleanup-outbox', () => ({
enqueueWorkspaceFileStorageCleanup: mockEnqueueWorkspaceFileStorageCleanup,
processWorkspaceFileStorageCleanupNow: mockProcessWorkspaceFileStorageCleanupNow,
}))
vi.mock('@/lib/uploads/contexts/workspace/workspace-file-folder-manager', () => ({
assertWorkspaceFileFolderTarget: mockAssertWorkspaceFileFolderTarget,
buildWorkspaceFileFolderPathMap: vi.fn(() => new Map()),
@@ -103,6 +114,7 @@ vi.mock('@/lib/workspaces/permissions/utils', () => ({
import {
ContentVersionConflictError,
deleteWorkspaceFile,
purgeCreatedWorkspaceFile,
registerUploadedWorkspaceFile,
restoreWorkspaceFile,
updateWorkspaceFileContent,
@@ -152,8 +164,10 @@ describe('workspace file metadata and storage accounting', () => {
mockDecrementStorageUsageForBillingContextInTx.mockResolvedValue(undefined)
mockMaybeNotifyStorageLimitForBillingContext.mockResolvedValue(undefined)
mockDeleteFile.mockResolvedValue(undefined)
mockEnqueueWorkspaceFileStorageCleanup.mockResolvedValue('cleanup-event-1')
mockMergeEditIntoLiveFileDoc.mockResolvedValue(undefined)
mockNotifyWorkspaceFilesChanged.mockResolvedValue(undefined)
mockProcessWorkspaceFileStorageCleanupNow.mockResolvedValue('completed')
mockReplaceWorkspaceFileSecretProvenanceInTx.mockResolvedValue(undefined)
})
@@ -262,6 +276,111 @@ describe('workspace file metadata and storage accounting', () => {
)
})
it('atomically purges exact archive-created metadata and accounting before storage cleanup', async () => {
const extractedRow = { ...FILE_ROW, folderId: 'folder-archive' }
dbChainMockFns.limit.mockResolvedValueOnce([extractedRow])
dbChainMockFns.returning.mockResolvedValueOnce([extractedRow])
await expect(
purgeCreatedWorkspaceFile({
workspaceId: FILE_ROW.workspaceId,
fileId: FILE_ROW.id,
key: FILE_ROW.key,
expectedName: FILE_ROW.originalName,
expectedFolderId: extractedRow.folderId,
expectedUpdatedAt: FILE_ROW.updatedAt,
})
).resolves.toBe(true)
expect(eq).toHaveBeenCalledWith(workspaceFiles.originalName, FILE_ROW.originalName)
expect(eq).toHaveBeenCalledWith(workspaceFiles.folderId, extractedRow.folderId)
expect(eq).toHaveBeenCalledWith(workspaceFiles.updatedAt, FILE_ROW.updatedAt)
expect(mockDecrementStorageUsageForBillingContextInTx).toHaveBeenCalledWith(
expect.any(Object),
STORAGE_CONTEXT,
FILE_ROW.size
)
expect(mockEnqueueWorkspaceFileStorageCleanup).toHaveBeenCalledWith(expect.any(Object), {
key: FILE_ROW.key,
})
expect(dbChainMockFns.delete.mock.invocationCallOrder[0]).toBeLessThan(
mockDecrementStorageUsageForBillingContextInTx.mock.invocationCallOrder[0]
)
expect(mockDecrementStorageUsageForBillingContextInTx.mock.invocationCallOrder[0]).toBeLessThan(
mockEnqueueWorkspaceFileStorageCleanup.mock.invocationCallOrder[0]
)
expect(mockProcessWorkspaceFileStorageCleanupNow).toHaveBeenCalledWith('cleanup-event-1')
expect(mockDeleteFile).not.toHaveBeenCalled()
})
it('leaves an extracted file untouched when its creation identity no longer matches', async () => {
dbChainMockFns.limit.mockResolvedValueOnce([])
await expect(
purgeCreatedWorkspaceFile({
workspaceId: FILE_ROW.workspaceId,
fileId: FILE_ROW.id,
key: FILE_ROW.key,
expectedName: FILE_ROW.originalName,
expectedFolderId: 'folder-archive',
expectedUpdatedAt: FILE_ROW.updatedAt,
})
).resolves.toBe(false)
expect(mockDecrementStorageUsageForBillingContextInTx).not.toHaveBeenCalled()
expect(mockEnqueueWorkspaceFileStorageCleanup).not.toHaveBeenCalled()
expect(mockProcessWorkspaceFileStorageCleanupNow).not.toHaveBeenCalled()
expect(mockDeleteFile).not.toHaveBeenCalled()
})
it('does not touch storage when the metadata and accounting transaction fails', async () => {
const extractedRow = { ...FILE_ROW, folderId: 'folder-archive' }
dbChainMockFns.limit.mockResolvedValueOnce([extractedRow])
dbChainMockFns.returning.mockResolvedValueOnce([extractedRow])
mockDecrementStorageUsageForBillingContextInTx.mockRejectedValueOnce(
new Error('accounting unavailable')
)
await expect(
purgeCreatedWorkspaceFile({
workspaceId: FILE_ROW.workspaceId,
fileId: FILE_ROW.id,
key: FILE_ROW.key,
expectedName: FILE_ROW.originalName,
expectedFolderId: extractedRow.folderId,
expectedUpdatedAt: FILE_ROW.updatedAt,
})
).rejects.toThrow('accounting unavailable')
expect(mockEnqueueWorkspaceFileStorageCleanup).not.toHaveBeenCalled()
expect(mockProcessWorkspaceFileStorageCleanupNow).not.toHaveBeenCalled()
expect(mockDeleteFile).not.toHaveBeenCalled()
})
it('keeps deferred cleanup durable when immediate processing fails', async () => {
const extractedRow = { ...FILE_ROW, folderId: 'folder-archive' }
dbChainMockFns.limit.mockResolvedValueOnce([extractedRow])
dbChainMockFns.returning.mockResolvedValueOnce([extractedRow])
mockProcessWorkspaceFileStorageCleanupNow.mockRejectedValueOnce(
new Error('outbox processor unavailable')
)
await expect(
purgeCreatedWorkspaceFile({
workspaceId: FILE_ROW.workspaceId,
fileId: FILE_ROW.id,
key: FILE_ROW.key,
expectedName: FILE_ROW.originalName,
expectedFolderId: extractedRow.folderId,
expectedUpdatedAt: FILE_ROW.updatedAt,
})
).resolves.toBe(true)
expect(mockEnqueueWorkspaceFileStorageCleanup).toHaveBeenCalledOnce()
expect(mockProcessWorkspaceFileStorageCleanupNow).toHaveBeenCalledWith('cleanup-event-1')
expect(mockDeleteFile).not.toHaveBeenCalled()
})
it('preserves the driver cause so the SQLSTATE survives the upload wrapper', async () => {
const driver = Object.assign(
new Error('cannot execute SELECT FOR UPDATE in a read-only transaction'),
@@ -302,6 +421,21 @@ describe('workspace file metadata and storage accounting', () => {
expect(mockReplaceWorkspaceFileSecretProvenanceInTx).not.toHaveBeenCalled()
})
it('allows extraction to batch the workspace notification', async () => {
dbChainMockFns.returning.mockResolvedValueOnce([FILE_ROW])
await uploadWorkspaceFile(
FILE_ROW.workspaceId,
FILE_ROW.userId,
Buffer.from('hello'),
FILE_ROW.originalName,
FILE_ROW.contentType,
{ notifyWorkspaceChange: false }
)
expect(mockNotifyWorkspaceFilesChanged).not.toHaveBeenCalled()
})
it('persists explicitly supplied workspace upload provenance', async () => {
dbChainMockFns.returning.mockResolvedValueOnce([FILE_ROW])
@@ -0,0 +1,74 @@
/**
* @vitest-environment node
*/
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockDeleteFile } = vi.hoisted(() => ({
mockDeleteFile: vi.fn(),
}))
vi.mock('@/lib/uploads/core/storage-service', () => ({
deleteFile: mockDeleteFile,
}))
import type { OutboxEventContext } from '@/lib/core/outbox/service'
import {
WORKSPACE_FILE_STORAGE_CLEANUP_OUTBOX_EVENT,
workspaceFileStorageCleanupOutboxHandlers,
} from '@/lib/uploads/contexts/workspace/workspace-file-storage-cleanup-outbox'
function context(): OutboxEventContext {
return {
eventId: 'cleanup-event-1',
eventType: WORKSPACE_FILE_STORAGE_CLEANUP_OUTBOX_EVENT,
attempts: 0,
maxAttempts: 10,
signal: new AbortController().signal,
checkpointPayload: vi.fn(),
}
}
function handler() {
const registered =
workspaceFileStorageCleanupOutboxHandlers[WORKSPACE_FILE_STORAGE_CLEANUP_OUTBOX_EVENT]
if (!registered) throw new Error('Workspace file storage cleanup handler is not registered')
return registered
}
describe('workspace file storage cleanup outbox', () => {
beforeEach(() => {
vi.clearAllMocks()
mockDeleteFile.mockResolvedValue(undefined)
})
it('deletes the deferred workspace object', async () => {
await handler()({ key: 'workspace/ws/file.txt' }, context())
expect(mockDeleteFile).toHaveBeenCalledWith({
key: 'workspace/ws/file.txt',
context: 'workspace',
})
})
it('treats an already-missing local object as completed', async () => {
mockDeleteFile.mockRejectedValueOnce(Object.assign(new Error('missing'), { code: 'ENOENT' }))
await expect(handler()({ key: 'workspace/ws/file.txt' }, context())).resolves.toBeUndefined()
})
it('rejects malformed payloads without touching storage', async () => {
await expect(handler()({ key: '' }, context())).rejects.toThrow(
'Workspace file storage cleanup outbox payload is missing key'
)
expect(mockDeleteFile).not.toHaveBeenCalled()
})
it('propagates storage failures for retry', async () => {
mockDeleteFile.mockRejectedValueOnce(new Error('storage unavailable'))
await expect(handler()({ key: 'workspace/ws/file.txt' }, context())).rejects.toThrow(
'storage unavailable'
)
})
})
@@ -0,0 +1,54 @@
import type { db } from '@sim/db'
import { describeError } from '@sim/utils/errors'
import {
enqueueOutboxEvent,
type OutboxHandler,
type OutboxHandlerRegistry,
processOutboxEventById,
} from '@/lib/core/outbox/service'
import { deleteFile } from '@/lib/uploads/core/storage-service'
export const WORKSPACE_FILE_STORAGE_CLEANUP_OUTBOX_EVENT = 'workspace-file.storage.cleanup'
interface WorkspaceFileStorageCleanupPayload {
key: string
}
function parsePayload(payload: unknown): WorkspaceFileStorageCleanupPayload {
if (!payload || typeof payload !== 'object' || Array.isArray(payload)) {
throw new Error('Workspace file storage cleanup outbox payload must be an object')
}
const key = (payload as Record<string, unknown>).key
if (typeof key !== 'string' || key.trim().length === 0) {
throw new Error('Workspace file storage cleanup outbox payload is missing key')
}
return { key }
}
const cleanupWorkspaceFileStorage: OutboxHandler<unknown> = async (rawPayload, context) => {
const payload = parsePayload(rawPayload)
context.signal.throwIfAborted()
try {
await deleteFile({ key: payload.key, context: 'workspace' })
} catch (error) {
if (describeError(error).code === 'ENOENT') return
throw error
}
}
export const workspaceFileStorageCleanupOutboxHandlers = {
[WORKSPACE_FILE_STORAGE_CLEANUP_OUTBOX_EVENT]: cleanupWorkspaceFileStorage,
} satisfies OutboxHandlerRegistry
/** Enqueues storage deletion in the transaction that removes the corresponding metadata. */
export function enqueueWorkspaceFileStorageCleanup(
executor: Pick<typeof db, 'insert'>,
payload: WorkspaceFileStorageCleanupPayload
): Promise<string> {
return enqueueOutboxEvent(executor, WORKSPACE_FILE_STORAGE_CLEANUP_OUTBOX_EVENT, payload)
}
/** Attempts a newly committed cleanup immediately; the outbox worker retries incomplete work. */
export function processWorkspaceFileStorageCleanupNow(eventId: string) {
return processOutboxEventById(eventId, workspaceFileStorageCleanupOutboxHandlers)
}
@@ -52,7 +52,8 @@ export type { UploadStorageProvider } from '@/lib/uploads/upload-session/types'
* Multipart parts are re-signed on demand by the per-surface `.../parts`
* endpoints, so a long-lived multipart session always outlives its part URLs
* and recovers by asking for new ones. A whole-object PUT has no such endpoint
* and needs none: it is size-capped by `UPLOAD_SESSION_PUT_MAX_BYTES`, is
* and needs none: it is size-capped by the provider's single-PUT ceiling
* (`UPLOAD_SESSION_PUT_MAX_BYTES`, or `UPLOAD_SESSION_LOCAL_PUT_MAX_BYTES` for `local`), is
* issued and used within one client call, and is not resumable — an expired PUT
* URL and an interrupted PUT have the identical recovery of starting a new
* session. Nothing durable is written in either case, because the transfer is
@@ -17,6 +17,7 @@ const {
mockInitiateMultipart,
mockListMultipartParts,
mockResolveBillingContext,
mockUploadStorageProvider,
} = vi.hoisted(() => ({
mockAbortProviderUpload: vi.fn(),
mockCheckStorageQuota: vi.fn(),
@@ -27,6 +28,7 @@ const {
mockInitiateMultipart: vi.fn(),
mockListMultipartParts: vi.fn(),
mockResolveBillingContext: vi.fn(),
mockUploadStorageProvider: vi.fn(() => 's3' as const),
}))
vi.mock('@/lib/billing/storage', () => ({
@@ -63,7 +65,7 @@ vi.mock('@/lib/uploads/upload-session/provider', () => ({
headProviderObject: mockHeadObject,
initiateMultipartProviderUpload: mockInitiateMultipart,
listMultipartProviderParts: mockListMultipartParts,
uploadStorageProvider: vi.fn(() => 's3'),
uploadStorageProvider: mockUploadStorageProvider,
}))
import { OrchestrationError } from '@/lib/core/orchestration/types'
@@ -109,6 +111,7 @@ describe('upload sessions', () => {
resetDbChainMock()
mockResolveBillingContext.mockResolvedValue({ workspaceId: WORKSPACE_ID })
mockCheckStorageQuota.mockResolvedValue({ allowed: true })
mockUploadStorageProvider.mockReturnValue('s3')
mockCreatePutTransfer.mockResolvedValue({
method: 'put',
url: 'https://storage.example/upload',
@@ -624,6 +627,30 @@ describe('upload sessions', () => {
expect(mockCreatePutTransfer).not.toHaveBeenCalled()
})
it('uses proxy-safe multipart parts for large local uploads', async () => {
const fileSize = UPLOAD_SESSION_PART_SIZE + 1
mockUploadStorageProvider.mockReturnValue('local')
mockInitiateMultipart.mockResolvedValueOnce({ provider: 'local', providerUploadId: null })
dbChainMockFns.returning.mockResolvedValueOnce([
uploadRow({
fileSize,
method: 'multipart',
storageProvider: 'local',
partSize: UPLOAD_SESSION_PART_SIZE,
partCount: 2,
}),
])
const created = await createWorkspaceUpload(fileSize)
expect(created.transfer).toEqual({
method: 'multipart',
partSize: UPLOAD_SESSION_PART_SIZE,
partCount: 2,
})
expect(mockCreatePutTransfer).not.toHaveBeenCalled()
})
it('preserves multipart request bounds before provider signing', async () => {
const multipart = sessionRecord({
method: 'multipart',
@@ -714,8 +741,8 @@ describe('upload sessions', () => {
partCount: 2,
})
const parts = [
{ partNumber: 1, etag: 'etag-1', size: UPLOAD_SESSION_PART_SIZE },
{ partNumber: 2, etag: 'etag-2', size: 3 },
{ partNumber: 1, etag: 'etag-1', size: UPLOAD_SESSION_PART_SIZE },
]
mockListMultipartParts.mockResolvedValue(parts)
mockHeadObject
@@ -733,7 +760,13 @@ describe('upload sessions', () => {
expect.objectContaining({ key: FINAL_KEY, providerUploadId: 'provider-upload-1' })
)
expect(mockCompleteMultipart).toHaveBeenCalledWith(
expect.objectContaining({ key: FINAL_KEY, parts })
expect.objectContaining({
key: FINAL_KEY,
parts: [
{ partNumber: 1, etag: 'etag-1', size: UPLOAD_SESSION_PART_SIZE },
{ partNumber: 2, etag: 'etag-2', size: 3 },
],
})
)
expect(finalize).toHaveBeenCalledOnce()
})
+21 -8
View File
@@ -45,6 +45,14 @@ import type {
export const UPLOAD_SESSION_PUT_MAX_BYTES = 50 * 1024 * 1024
export const UPLOAD_SESSION_PART_SIZE = 8 * 1024 * 1024
/**
* Single-PUT ceiling for the `local` provider. A local PUT is proxied through an app route
* rather than sent to object storage, so it must stay under the route's body limit; anything
* larger goes multipart, whose parts are already sized to fit. Kept equal to
* {@link UPLOAD_SESSION_PART_SIZE} but named separately so tuning part size for cloud
* throughput cannot silently move the local proxy threshold.
*/
export const UPLOAD_SESSION_LOCAL_PUT_MAX_BYTES = UPLOAD_SESSION_PART_SIZE
export const UPLOAD_SESSION_MAX_PART_URLS = 100
export const UPLOAD_SESSION_TTL_MS = 24 * 60 * 60 * 1000
export const UPLOAD_SESSION_ASSET_MAX_BYTES = 5 * 1024 * 1024
@@ -207,11 +215,6 @@ export async function createUploadSession(
})
}
const { storageContext, finalKey } = resolveUploadStorage(params, id)
const method: UploadTransferMethod =
params.fileSize <= UPLOAD_SESSION_PUT_MAX_BYTES ? 'put' : 'multipart'
const partSize = method === 'multipart' ? UPLOAD_SESSION_PART_SIZE : null
const partCount =
method === 'multipart' ? Math.ceil(params.fileSize / UPLOAD_SESSION_PART_SIZE) : null
if (requiresStorageQuota(params.purpose)) {
if (!workspaceId) throw new Error(`${params.purpose} upload is missing workspaceId`)
@@ -223,6 +226,12 @@ export async function createUploadSession(
}
const provider = uploadStorageProvider()
const putMaxBytes =
provider === 'local' ? UPLOAD_SESSION_LOCAL_PUT_MAX_BYTES : UPLOAD_SESSION_PUT_MAX_BYTES
const method: UploadTransferMethod = params.fileSize <= putMaxBytes ? 'put' : 'multipart'
const partSize = method === 'multipart' ? UPLOAD_SESSION_PART_SIZE : null
const partCount =
method === 'multipart' ? Math.ceil(params.fileSize / UPLOAD_SESSION_PART_SIZE) : null
if (provider === 'local') await maybeCleanupLocalUploadArtifacts()
const objectMetadata = uploadSessionObjectMetadata({
id,
@@ -627,14 +636,14 @@ export async function completeUploadSession<T>(params: {
if (claimed.method === 'put') {
throw new UploadSessionError('conflict', 'Uploaded object not found')
}
const parts = await listMultipartProviderParts({
const providerParts = await listMultipartProviderParts({
provider: claimed.storageProvider,
providerUploadId: claimed.providerUploadId,
uploadId: claimed.id,
key: claimed.finalKey,
context: claimed.storageContext,
})
validateProviderParts(claimed, parts)
const parts = validatedSortedProviderParts(claimed, providerParts)
try {
await completeMultipartProviderUpload({
provider: claimed.storageProvider,
@@ -1036,7 +1045,10 @@ async function claimSession(
return sessionFromRow(row, '')
}
function validateProviderParts(session: UploadSessionRecord, parts: CompletedUploadPart[]): void {
function validatedSortedProviderParts(
session: UploadSessionRecord,
parts: CompletedUploadPart[]
): CompletedUploadPart[] {
if (!session.partCount) throw new Error('Multipart upload is missing partCount')
if (parts.length !== session.partCount) {
throw new UploadSessionError(
@@ -1067,6 +1079,7 @@ function validateProviderParts(session: UploadSessionRecord, parts: CompletedUpl
)
}
}
return sorted
}
function assertObjectIdentity(
@@ -4,6 +4,7 @@
import { describe, expect, it } from 'vitest'
import { StorageLimitExceededError } from '@/lib/billing/storage'
import { OrchestrationError } from '@/lib/core/orchestration/types'
import { ArchiveError } from '@/lib/uploads/archive'
import { internalFileErrorPolicies } from '@/lib/workspace-files/api/internal-error-policies'
import {
CompiledCheckTooLargeError,
@@ -45,4 +46,25 @@ describe('internal file error policies', () => {
headers: undefined,
})
})
it('maps archive extraction failures onto caller-safe statuses', () => {
expect(
internalFileErrorPolicies.extractArchive.project(
new ArchiveError('invalid', 'Not a valid .zip archive.')
)
).toEqual({
status: 400,
body: { error: 'Not a valid .zip archive.' },
headers: undefined,
})
expect(
internalFileErrorPolicies.extractArchive.project(
new ArchiveError('too_many_entries', 'Archive has 1001 files; the maximum is 1000.')
)
).toEqual({
status: 413,
body: { error: 'Archive has 1001 files; the maximum is 1000.' },
headers: undefined,
})
})
})
@@ -8,6 +8,7 @@ import {
} from '@/lib/api/server/routes'
import { StorageLimitExceededError } from '@/lib/billing/storage'
import { asOrchestrationError, statusForOrchestrationError } from '@/lib/core/orchestration/types'
import { ArchiveError, statusForArchiveError } from '@/lib/uploads/archive'
import {
CompiledCheckTooLargeError,
CompiledCheckUnsupportedError,
@@ -81,6 +82,16 @@ const inline: InternalErrorPolicy = {
const FILE_NOT_FOUND_MESSAGE = 'File not found'
const concealResourceAuthorization = createInternalResourceConcealmentPolicy({
base: internalOrchestrationErrorPolicy,
notFoundMessage: FILE_NOT_FOUND_MESSAGE,
})
const extractArchive = extendInternalErrorPolicy(concealResourceAuthorization, (error) => {
if (!(error instanceof ArchiveError)) return null
return internalErrorResponse(statusForArchiveError(error), { error: error.message })
})
export const internalFileErrorPolicies = {
default: internalOrchestrationErrorPolicy,
content,
@@ -88,10 +99,7 @@ export const internalFileErrorPolicies = {
* Single-file internal routes reach the same use cases as the concealing v2
* file routes, so they withhold the same cross-tenant existence signal.
*/
concealResourceAuthorization: createInternalResourceConcealmentPolicy({
base: internalOrchestrationErrorPolicy,
notFoundMessage: FILE_NOT_FOUND_MESSAGE,
}),
concealResourceAuthorization,
concealContentAuthorization: createInternalResourceConcealmentPolicy({
base: content,
notFoundMessage: FILE_NOT_FOUND_MESSAGE,
@@ -100,5 +108,6 @@ export const internalFileErrorPolicies = {
compiledCheck,
downloadUrl,
downloadArchive,
extractArchive,
inline,
} as const
@@ -36,6 +36,7 @@ export interface CreateWorkspaceFileResult {
export interface CreateWorkspaceFileBufferInput
extends Omit<CreateWorkspaceFileInput, 'content' | 'encoding'> {
content: Buffer
notifyWorkspaceChange?: boolean
}
async function resolveCreateWorkspaceFileContext(workspaceId: string) {
@@ -51,7 +52,7 @@ async function createAuthorizedWorkspaceFile({
workspace,
}: {
principal: Principal
input: Omit<CreateWorkspaceFileInput, 'content' | 'encoding'>
input: Omit<CreateWorkspaceFileBufferInput, 'content'>
content: Buffer
workspace: Awaited<ReturnType<typeof resolveCreateWorkspaceFileContext>>
}): Promise<CreateWorkspaceFileResult> {
@@ -71,6 +72,7 @@ async function createAuthorizedWorkspaceFile({
folderPath: input.folderPath,
exactName: input.exactName,
secretProvenance: input.secretProvenance ?? EXACT_EMPTY_WORKSPACE_FILE_SECRET_PROVENANCE,
notifyWorkspaceChange: input.notifyWorkspaceChange,
}
)
} catch (error) {
@@ -0,0 +1,372 @@
/**
* @vitest-environment node
*/
import { Buffer } from 'buffer'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { OrchestrationError } from '@/lib/core/orchestration/types'
const mocks = vi.hoisted(() => ({
archiveFolderIfEmpty: vi.fn(),
atomicallyClaim: vi.fn(),
createFolder: vi.fn(),
decompress: vi.fn(),
fetchBuffer: vi.fn(),
getFile: vi.fn(),
getSecretProvenance: vi.fn(),
loadContext: vi.fn(),
notify: vi.fn(),
releaseLease: vi.fn(),
resolvePermission: vi.fn(),
}))
vi.mock('@sim/platform-authz/workspace', () => ({
permissionSatisfies: () => true,
resolveEffectiveWorkspacePermission: mocks.resolvePermission,
}))
vi.mock('@/lib/realtime/notify', () => ({ notifyWorkspaceFilesChanged: mocks.notify }))
vi.mock('@/lib/core/idempotency/service', () => ({
IdempotencyService: class MockIdempotencyService {
atomicallyClaim(...args: unknown[]) {
return mocks.atomicallyClaim(...args)
}
release(...args: unknown[]) {
return mocks.releaseLease(...args)
}
},
}))
vi.mock('@/lib/uploads/archive', () => ({
decompressArchiveBufferToWorkspaceFiles: mocks.decompress,
MAX_ARCHIVE_BYTES: 100 * 1024 * 1024,
}))
vi.mock('@/lib/uploads/contexts/workspace/workspace-file-folder-manager', () => ({
archiveWorkspaceFileFolderIfEmpty: mocks.archiveFolderIfEmpty,
createWorkspaceFileFolder: mocks.createFolder,
}))
vi.mock('@/lib/uploads/contexts/workspace/workspace-file-manager', () => ({
fetchWorkspaceFileBuffer: mocks.fetchBuffer,
getWorkspaceFile: mocks.getFile,
loadActiveWorkspaceFileContext: mocks.loadContext,
}))
vi.mock('@/lib/uploads/contexts/workspace/workspace-file-secret-provenance', () => ({
getBoundWorkspaceFileSecretProvenance: mocks.getSecretProvenance,
}))
import { extractWorkspaceFile } from '@/lib/workspace-files/application/extract-workspace-file'
const principal = { kind: 'session', userId: 'user-1', sessionId: 'session-1' } as const
const context = {
fileId: 'file-1',
workspaceId: 'workspace-1',
workspaceOrganizationId: null,
allowPersonalApiKeys: true,
billedAccountUserId: 'billing-owner',
}
const file = {
id: 'file-1',
workspaceId: 'workspace-1',
name: 'bundle.zip',
key: 'workspace/workspace-1/bundle.zip',
size: 256,
folderPath: 'Projects/Imports',
storageContext: 'workspace' as const,
folderId: 'folder-imports',
}
describe('extractWorkspaceFile', () => {
beforeEach(() => {
vi.clearAllMocks()
mocks.loadContext.mockResolvedValue(context)
mocks.resolvePermission.mockResolvedValue('write')
mocks.getFile.mockResolvedValue(file)
mocks.createFolder.mockResolvedValue({
id: 'folder-bundle',
name: 'bundle',
path: 'Projects/Imports/bundle',
})
mocks.archiveFolderIfEmpty.mockResolvedValue(true)
mocks.atomicallyClaim.mockResolvedValue({
claimed: true,
normalizedKey: 'workspace-file:extract:workspace-1:file-1',
storageMethod: 'database',
claimToken: 'claim-1',
})
mocks.fetchBuffer.mockResolvedValue(Buffer.from('zip'))
mocks.getSecretProvenance.mockResolvedValue({ status: 'exact', entries: [] })
mocks.decompress.mockImplementation(async (_content, options) => {
await options.prepareRootFolder(vi.fn())
return {
extracted: [{ id: 'extracted-1' }, { id: 'extracted-2' }],
skipped: 1,
skippedUnsafePaths: [],
}
})
mocks.notify.mockResolvedValue(undefined)
mocks.releaseLease.mockResolvedValue(undefined)
})
it('extracts into a same-name folder beside the archive', async () => {
const validateRootFolderSegments = vi.fn()
mocks.decompress.mockImplementationOnce(async (_content, options) => {
await options.prepareRootFolder(validateRootFolderSegments)
return {
extracted: [{ id: 'extracted-1' }, { id: 'extracted-2' }],
skipped: 1,
skippedUnsafePaths: [],
}
})
mocks.createFolder.mockImplementationOnce(async (options) => {
options.validateResolvedName('bundle')
return {
id: 'folder-bundle',
name: 'bundle',
path: 'Projects/Imports/bundle',
}
})
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).resolves.toEqual({ folderName: 'bundle', extractedCount: 2, skippedCount: 1 })
expect(mocks.createFolder).toHaveBeenCalledWith({
workspaceId: 'workspace-1',
userId: 'user-1',
name: 'bundle',
parentId: 'folder-imports',
exactName: false,
validateResolvedName: expect.any(Function),
})
expect(validateRootFolderSegments).toHaveBeenCalledWith(['Projects', 'Imports', 'bundle'])
expect(mocks.fetchBuffer).toHaveBeenCalledWith(file, { maxBytes: 100 * 1024 * 1024 })
expect(mocks.decompress).toHaveBeenCalledWith(Buffer.from('zip'), {
workspaceId: 'workspace-1',
principal,
rootFolderSegments: ['Projects', 'Imports', 'bundle'],
prepareRootFolder: expect.any(Function),
signal: expect.any(AbortSignal),
skipNoiseEntries: true,
secretProvenance: { status: 'exact', entries: [] },
notifyWorkspaceChange: false,
})
expect(mocks.atomicallyClaim).toHaveBeenCalledWith('extract', 'workspace-1:file-1')
expect(mocks.releaseLease).toHaveBeenCalledWith(
'workspace-file:extract:workspace-1:file-1',
'database',
'claim-1'
)
expect(mocks.notify).toHaveBeenCalledWith('workspace-1')
})
it('rejects non-zip files before reading storage', async () => {
mocks.getFile.mockResolvedValue({ ...file, name: 'bundle.txt' })
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).rejects.toMatchObject({ code: 'validation', message: 'Only .zip files can be unzipped' })
expect(mocks.fetchBuffer).not.toHaveBeenCalled()
expect(mocks.decompress).not.toHaveBeenCalled()
})
it('rejects a second extraction while the same archive is already being extracted', async () => {
mocks.atomicallyClaim
.mockResolvedValueOnce({
claimed: true,
normalizedKey: 'workspace-file:extract:workspace-1:file-1',
storageMethod: 'database',
claimToken: 'claim-1',
})
.mockResolvedValueOnce({
claimed: false,
normalizedKey: 'workspace-file:extract:workspace-1:file-1',
storageMethod: 'database',
existingResult: { status: 'in-progress' },
})
let releaseFetch: ((value: Buffer) => void) | undefined
mocks.fetchBuffer.mockImplementationOnce(
() =>
new Promise<Buffer>((resolve) => {
releaseFetch = resolve
})
)
const firstExtraction = extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
await vi.waitFor(() => expect(mocks.fetchBuffer).toHaveBeenCalledOnce())
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).rejects.toMatchObject({
code: 'conflict',
message: 'This archive is already being unzipped',
})
releaseFetch?.(Buffer.from('zip'))
await expect(firstExtraction).resolves.toMatchObject({ extractedCount: 2 })
expect(mocks.atomicallyClaim).toHaveBeenCalledTimes(2)
expect(mocks.releaseLease).toHaveBeenCalledOnce()
})
it('rejects extraction when another server owns the archive lease', async () => {
mocks.atomicallyClaim.mockResolvedValueOnce({
claimed: false,
normalizedKey: 'workspace-file:extract:workspace-1:file-1',
storageMethod: 'database',
existingResult: { status: 'in-progress' },
})
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).rejects.toMatchObject({
code: 'conflict',
message: 'This archive is already being unzipped',
})
expect(mocks.getFile).not.toHaveBeenCalled()
expect(mocks.releaseLease).not.toHaveBeenCalled()
})
it('uses a suffixed destination instead of merging into a stranded folder', async () => {
mocks.createFolder.mockResolvedValueOnce({
id: 'folder-bundle-3',
name: 'bundle (3)',
path: 'Projects/Imports/bundle (3)',
})
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).resolves.toEqual({ folderName: 'bundle (3)', extractedCount: 2, skippedCount: 1 })
expect(mocks.createFolder).toHaveBeenCalledWith(
expect.objectContaining({ name: 'bundle', exactName: false })
)
})
it('only removes the destination folder when it is still empty after extraction fails', async () => {
mocks.decompress.mockImplementationOnce(async (_content, options) => {
await options.prepareRootFolder()
throw new Error('invalid archive')
})
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).rejects.toThrow('invalid archive')
expect(mocks.archiveFolderIfEmpty).toHaveBeenCalledWith({
workspaceId: 'workspace-1',
folderId: 'folder-bundle',
})
expect(mocks.notify).toHaveBeenCalledOnce()
expect(mocks.notify).toHaveBeenCalledWith('workspace-1')
})
/** Fires the deadline the way `AbortSignal.timeout` does: `reason` is what gets thrown. */
function expireDeadline(signal: AbortSignal): unknown {
const reason = new DOMException('The operation was aborted due to timeout', 'TimeoutError')
Object.defineProperty(signal, 'aborted', { value: true })
Object.defineProperty(signal, 'reason', { value: reason })
return reason
}
it('reports a budget overrun as a caller-fixable error, not the raw abort', async () => {
mocks.decompress.mockImplementationOnce(async (_content, options) => {
await options.prepareRootFolder()
throw expireDeadline(options.signal)
})
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).rejects.toMatchObject({
code: 'payload_too_large',
message: expect.stringContaining('took too long and was cancelled'),
})
expect(mocks.archiveFolderIfEmpty).toHaveBeenCalledOnce()
})
it('keeps the real cause when a failure races the deadline', async () => {
// The signal stays aborted for the rest of the request, so an unrelated mid-entry
// failure after the timer fires must not be relabelled as a timeout.
mocks.decompress.mockImplementationOnce(async (_content, options) => {
await options.prepareRootFolder()
expireDeadline(options.signal)
throw Object.assign(new Error('Archive entry "a.txt" could not be decompressed'), {
name: 'ArchiveError',
reason: 'invalid',
})
})
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).rejects.toMatchObject({
name: 'ArchiveError',
message: expect.stringContaining('could not be decompressed'),
})
})
it('leaves a destination folder that gained collaborators content during rollback', async () => {
mocks.decompress.mockImplementationOnce(async (_content, options) => {
await options.prepareRootFolder()
throw new Error('storage quota exceeded')
})
mocks.archiveFolderIfEmpty.mockRejectedValueOnce(
new OrchestrationError('conflict', 'Folder is not empty')
)
await expect(
extractWorkspaceFile.execute({
principal,
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).rejects.toThrow('storage quota exceeded')
expect(mocks.archiveFolderIfEmpty).toHaveBeenCalledWith(
expect.objectContaining({ folderId: 'folder-bundle' })
)
expect(mocks.notify).toHaveBeenCalledOnce()
})
it('rejects non-session principals before loading the file', async () => {
await expect(
extractWorkspaceFile.execute({
principal: { kind: 'personal_api_key', userId: 'user-1', keyId: 'key-1' },
input: { fileId: 'file-1', assertedWorkspaceId: 'workspace-1' },
})
).rejects.toMatchObject({ code: 'forbidden' })
expect(mocks.loadContext).not.toHaveBeenCalled()
expect(mocks.fetchBuffer).not.toHaveBeenCalled()
})
})
@@ -0,0 +1,234 @@
import { AuditAction, AuditResourceType } from '@sim/audit'
import { resolvePrincipalAttribution } from '@sim/auth/principal'
import { createLogger } from '@sim/logger'
import { getErrorMessage } from '@sim/utils/errors'
import type { AuthorizedWorkspaceUseCaseContext } from '@/lib/core/application'
import { IdempotencyService } from '@/lib/core/idempotency/service'
import { OrchestrationError } from '@/lib/core/orchestration/types'
import { notifyWorkspaceFilesChanged } from '@/lib/realtime/notify'
import { decompressArchiveBufferToWorkspaceFiles, MAX_ARCHIVE_BYTES } from '@/lib/uploads/archive'
import {
archiveWorkspaceFileFolderIfEmpty,
createWorkspaceFileFolder,
} from '@/lib/uploads/contexts/workspace/workspace-file-folder-manager'
import {
type ActiveWorkspaceFileContext,
fetchWorkspaceFileBuffer,
getWorkspaceFile,
} from '@/lib/uploads/contexts/workspace/workspace-file-manager'
import { getBoundWorkspaceFileSecretProvenance } from '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance'
import { isArchiveFileName } from '@/lib/uploads/utils/file-utils'
import { defineAuthorizedWorkspaceFileUseCase } from '@/lib/workspace-files/application/authorized-workspace-file-use-case'
import { fileOperations } from '@/lib/workspace-files/application/operations'
import { resolveActiveWorkspaceFileContext } from '@/lib/workspace-files/application/workspace-file-context'
import { parseWorkspaceFileFolderDisplayPath } from '@/lib/workspace-files/folder-display-path'
const logger = createLogger('ExtractWorkspaceFile')
/**
* Wall-clock budget for the whole operation — archive download included, because the lease
* this must fit inside starts earlier still. Deliberately shorter than the route's
* `maxDuration` so the work stops on our terms, with the all-or-nothing rollback still able
* to run, rather than the platform killing the process mid-write and stranding a partial
* tree. Self-hosted deployments do not enforce `maxDuration` at all, so this is the only
* thing bounding the write loop there.
*/
const EXTRACTION_BUDGET_MS = 180 * 1000
/**
* Must exceed {@link EXTRACTION_BUDGET_MS} plus the worst-case rollback, because
* `IdempotencyService` reclaims an expired in-progress claim: a holder that outlives its
* own lease would let a second unzip of the same archive start beside it.
*/
const EXTRACTION_LEASE_TTL_SECONDS = 6 * 60
/**
* Used only through `atomicallyClaim`/`release`, never `executeWithIdempotency`, so no
* result is ever stored — this is a lease, not a memoized operation.
*/
const extractionLeases = new IdempotencyService({
namespace: 'workspace-file',
ttlSeconds: EXTRACTION_LEASE_TTL_SECONDS,
forceStorage: 'database',
})
export interface ExtractWorkspaceFileInput {
fileId: string
assertedWorkspaceId?: string
}
export interface ExtractWorkspaceFileResult {
folderName: string
extractedCount: number
skippedCount: number
}
type ExtractWorkspaceFileUseCaseContext = AuthorizedWorkspaceUseCaseContext<
typeof fileOperations.extractArchive,
ExtractWorkspaceFileInput,
ActiveWorkspaceFileContext
>
function archiveFolderName(fileName: string): string {
const stripped = fileName
.replace(/\.zip$/i, '')
.normalize('NFC')
.replace(/[\x00-\x1f\x7f]/g, '')
.replace(/[/\\]/g, '-')
.trim()
return stripped && stripped !== '.' && stripped !== '..' ? stripped : 'archive'
}
async function withExtractionLease<T>(
workspaceId: string,
fileId: string,
extract: () => Promise<T>
): Promise<T> {
const claim = await extractionLeases.atomicallyClaim('extract', `${workspaceId}:${fileId}`)
if (!claim.claimed) {
throw new OrchestrationError('conflict', 'This archive is already being unzipped')
}
if (!claim.claimToken) throw new Error('Archive extraction lease is missing its fencing token')
try {
return await extract()
} finally {
await extractionLeases
.release(claim.normalizedKey, claim.storageMethod, claim.claimToken)
.catch((error) => {
logger.warn('Failed to release archive extraction lease; TTL will expire it', {
workspaceId,
fileId,
error: getErrorMessage(error),
})
})
}
}
async function executeExtractWorkspaceFile(
useCaseContext: ExtractWorkspaceFileUseCaseContext
): Promise<ExtractWorkspaceFileResult> {
const { context } = useCaseContext
return withExtractionLease(context.workspaceId, context.fileId, () =>
extractWorkspaceFileContents(useCaseContext)
)
}
async function extractWorkspaceFileContents({
principal,
context,
}: ExtractWorkspaceFileUseCaseContext): Promise<ExtractWorkspaceFileResult> {
const file = await getWorkspaceFile(context.workspaceId, context.fileId, { throwOnError: true })
if (!file) throw new OrchestrationError('not_found', 'File not found')
if (!isArchiveFileName(file.name)) {
throw new OrchestrationError('validation', 'Only .zip files can be unzipped')
}
if (file.size > MAX_ARCHIVE_BYTES) {
throw new OrchestrationError(
'payload_too_large',
`Archive exceeds the ${MAX_ARCHIVE_BYTES / 1024 / 1024} MB unzip limit`
)
}
const deadline = AbortSignal.timeout(EXTRACTION_BUDGET_MS)
const folderName = archiveFolderName(file.name)
const parentFolderSegments = file.folderPath
? parseWorkspaceFileFolderDisplayPath(file.folderPath)
: []
const [content, secretProvenance] = await Promise.all([
fetchWorkspaceFileBuffer(file, { maxBytes: MAX_ARCHIVE_BYTES }),
getBoundWorkspaceFileSecretProvenance(context.workspaceId, {
fileId: file.id,
key: file.key,
context: file.storageContext ?? 'workspace',
}),
])
let rootFolder: Awaited<ReturnType<typeof createWorkspaceFileFolder>> | undefined
try {
const result = await decompressArchiveBufferToWorkspaceFiles(content, {
workspaceId: context.workspaceId,
principal,
rootFolderSegments: [...parentFolderSegments, folderName],
prepareRootFolder: async (validateRootFolderSegments) => {
const attribution = resolvePrincipalAttribution(principal, {
workspaceBillingOwnerUserId: context.billedAccountUserId,
})
rootFolder = await createWorkspaceFileFolder({
workspaceId: context.workspaceId,
userId: attribution.attributedUserId,
name: folderName,
parentId: file.folderId,
exactName: false,
validateResolvedName: (resolvedName) =>
validateRootFolderSegments([...parentFolderSegments, resolvedName]),
})
return parseWorkspaceFileFolderDisplayPath(rootFolder.path)
},
signal: deadline,
skipNoiseEntries: true,
secretProvenance,
notifyWorkspaceChange: false,
})
if (result.extracted.length === 0) {
throw new OrchestrationError('validation', `No files could be unzipped from "${file.name}"`)
}
if (result.skippedUnsafePaths.length > 0) {
logger.warn('Skipped unsafe archive entries', {
workspaceId: context.workspaceId,
fileId: file.id,
entryNames: result.skippedUnsafePaths,
})
}
return {
folderName: rootFolder?.name ?? folderName,
extractedCount: result.extracted.length,
skippedCount: result.skipped,
}
} catch (error) {
if (rootFolder) {
try {
await archiveWorkspaceFileFolderIfEmpty({
workspaceId: context.workspaceId,
folderId: rootFolder.id,
})
} catch (cleanupError) {
logger.warn('Left non-empty archive destination folder after extraction error', {
workspaceId: context.workspaceId,
folderId: rootFolder.id,
cleanupError,
})
}
await notifyWorkspaceFilesChanged(context.workspaceId)
}
if (deadline.aborted && error === deadline.reason) {
logger.warn('Archive extraction exceeded its budget', {
workspaceId: context.workspaceId,
fileId: file.id,
budgetMs: EXTRACTION_BUDGET_MS,
wroteAnything: Boolean(rootFolder),
})
throw new OrchestrationError(
'payload_too_large',
`Unzipping "${file.name}" took too long and was cancelled. Try a smaller archive.`
)
}
throw error
}
}
export const extractWorkspaceFile = defineAuthorizedWorkspaceFileUseCase({
operation: fileOperations.extractArchive,
resolveContext: ({ input }) => resolveActiveWorkspaceFileContext(input),
execute: executeExtractWorkspaceFile,
projectAudit: ({ context, result }) => ({
action: AuditAction.FILE_UPDATED,
resourceType: AuditResourceType.FILE,
resourceId: context.fileId,
description: `Unzipped workspace file ${context.fileId}`,
metadata: {
destinationFolder: result.folderName,
extractedCount: result.extractedCount,
skippedCount: result.skippedCount,
},
}),
afterSuccess: ({ context }) => notifyWorkspaceFilesChanged(context.workspaceId),
})
@@ -59,6 +59,12 @@ export const fileOperations = {
workspaceApiKey: 'allow',
...ALL_COPILOT_PRINCIPAL_POLICY,
}),
extractArchive: defineWorkspaceOperation({
id: 'files.extract_archive',
minimumRole: 'write',
workspaceApiKey: 'deny',
principalKinds: ['session'],
}),
updateContent: defineWorkspaceOperation({
id: 'files.update_content',
minimumRole: 'write',
+2 -2
View File
@@ -9,8 +9,8 @@ const QUERY_HOOKS_DIR = path.join(ROOT, 'apps/sim/hooks/queries')
const SELECTOR_HOOKS_DIR = path.join(ROOT, 'apps/sim/hooks/selectors')
const BASELINE = {
totalRoutes: 1119,
zodRoutes: 1119,
totalRoutes: 1120,
zodRoutes: 1120,
nonZodRoutes: 0,
} as const