feat: add batch image MVP

2026-09-24 16:05:44 +08:00 · 2026-07-04 05:30:50 +08:00
parent 28c2d20d9f
commit a994fbd77a
79 changed files with 34113 additions and 36 deletions
@@ -0,0 +1,287 @@
+# Batch Image MVP
+
+Sub2API Batch Image MVP provides asynchronous Gemini image batch generation through a unified API surface backed by Redis workers, PostgreSQL state, and provider-specific batch backends.
+
+Supported providers:
+
+- `gemini_api`
+- `vertex`
+
+API users do not see Gemini file names, Vertex job names, GCS paths, signed URLs, API keys, or service account material. Downloads are proxied through Sub2API.
+
+## API Routes
+
+```text
+POST   /v1/images/batches
+GET    /v1/images/batches/{id}
+GET    /v1/images/batches/{id}/items
+GET    /v1/images/batches/{id}/items/{custom_id}/content
+GET    /v1/images/batches/{id}/download
+POST   /v1/images/batches/{id}/cancel
+DELETE /v1/images/batches/{id}/outputs
+```
+
+Submit request:
+
+```json
+{
+  "model": "gemini-2.5-flash-image",
+  "provider": "gemini_api",
+  "items": [
+    {
+      "custom_id": "cover_001",
+      "prompt": "A clean product hero image..."
+    }
+  ],
+  "image_size": "1K",
+  "response_mime_type": "image/png"
+}
+```
+
+Public batch response:
+
+```json
+{
+  "id": "imgbatch_0123456789abcdef0123456789abcdef",
+  "object": "image.batch",
+  "status": "queued",
+  "model": "gemini-2.5-flash-image",
+  "provider": "gemini_api",
+  "item_count": 1,
+  "success_count": 0,
+  "fail_count": 0,
+  "estimated_cost": 0.25,
+  "actual_cost": null,
+  "created_at": 1783123200,
+  "submitted_at": 1783123201,
+  "settled_at": null
+}
+```
+
+Public items response:
+
+```json
+{
+  "object": "list",
+  "data": [
+    {
+      "custom_id": "cover_001",
+      "status": "succeeded",
+      "mime_type": "image/png",
+      "file_extension": "png",
+      "image_count": 1,
+      "error": null
+    }
+  ],
+  "has_more": false
+}
+```
+
+## Lifecycle
+
+Internal lifecycle:
+
+```text
+created -> uploading -> submitted -> running -> indexing -> settling -> completed
+```
+
+Terminal and cleanup statuses:
+
+```text
+failed
+cancelled
+completed -> output_deleted
+```
+
+Public status mapping:
+
+```text
+created/uploading/submitted -> queued
+running                    -> running
+indexing                   -> processing_results
+settling                   -> settling
+completed                  -> completed
+failed                     -> failed
+cancelled                  -> cancelled
+output_deleted             -> output_deleted
+```
+
+`completed -> output_deleted` happens after manual output deletion or TTL cleanup.
+
+## Redis
+
+Redis is used for wakeups, retries, worker coordination, per-job locks, and download limiting. PostgreSQL remains the source of truth.
+
+`batch_image.queue_enabled` defaults to `false`. When it is set to `true`, app startup starts `BatchImageWorker` runtime loops for the Redis ready queue, delayed queue mover, and stale active recovery. The worker reserves jobs from the Redis ready queue and blocks there when no job is available.
+
+Redis structures:
+
+- Ready queue: `batch_image.queue_ready_key`
+- Delayed queue: `batch_image.queue_delayed_key`
+- Active set: `batch_image.queue_active_key`
+- Inflight keys: `batch_image.inflight_key_prefix`
+- Per-job lock keys: `batch_image.lock_key_prefix`
+- Queue idempotency keys: `batch_image.idempotency_key_prefix`
+- Download limiter keys managed by the download limiter
+
+Workers should reserve from Redis. They are not expected to run as a database scan loop.
+
+The worker does not perform DB scan polling. Database reads happen only after a Redis queue reservation yields a specific batch id.
+
+## Billing
+
+MVP billing rules:
+
+- Submit may estimate cost.
+- Settlement runs after result indexing.
+- Only successful images are charged.
+- Failed items are not charged.
+- Settlement request id is `batch_image_settlement:{batch_id}`.
+- Settlement is idempotent; re-running settlement must not double charge.
+
+Exact production pricing is resolved through model pricing configuration and is not defined here.
+
+## Cleanup
+
+Defaults:
+
+- Input retention after terminal status: 24 hours.
+- Output retention after terminal status: 72 hours.
+- Maximum output retention: 7 days.
+- Cleanup interval: 30 minutes.
+- Cleanup batch size: 100.
+
+Manual output deletion:
+
+```text
+DELETE /v1/images/batches/{id}/outputs
+```
+
+After output cleanup, downloads return `410 Gone` with `BATCH_IMAGE_OUTPUT_DELETED`.
+
+Cleanup never accepts user-supplied provider paths. Provider cleanup must use server-generated refs and prefix-safe deletion.
+
+For the managed Vertex/GCS batch bucket, disable Cloud Storage soft delete or configure lifecycle carefully to avoid hidden retained storage cost.
+
+## Provider Notes
+
+`gemini_api`:
+
+- Uses Gemini Batch API with JSONL file mode.
+- Result file refs are internal.
+- API keys are never returned.
+
+`vertex`:
+
+- Uses Vertex `BatchPredictionJob` with managed GCS JSONL.
+- GCS bucket and prefix are server-managed.
+- Vertex job name and GCS paths are internal.
+- Batch image output should be treated as `1K`/default only in MVP.
+- Do not promise `2K` or `4K`.
+
+## Config
+
+These keys exist in `backend/internal/config/config.go`:
+
+```yaml
+batch_image:
+  enabled: false
+  max_items_per_job_default: 500
+  max_items_per_job_trial: 50
+  max_prompt_chars_per_item: 8000
+  default_response_mime_type: "image/png"
+  default_image_size: "1K"
+
+  max_download_items_zip: 1000
+  max_download_bytes_per_request: 2147483648
+  max_download_duration_seconds: 600
+  max_download_concurrency_per_user: 2
+
+  input_retention_after_terminal_hours: 24
+  output_retention_after_terminal_hours: 72
+  output_retention_max_days: 7
+  cleanup_interval_minutes: 30
+  cleanup_batch_size: 100
+
+  queue_enabled: false
+  queue_ready_key: "batch_image:queue:ready"
+  queue_delayed_key: "batch_image:queue:delayed"
+  queue_active_key: "batch_image:queue:active"
+  inflight_key_prefix: "batch_image:queue:inflight:"
+  lock_key_prefix: "batch_image:queue:lock:"
+  idempotency_key_prefix: "batch_image:queue:idem:"
+  inflight_ttl_seconds: 604800
+  job_lock_ttl_seconds: 300
+  default_requeue_delay_seconds: 30
+  error_retry_delay_seconds: 60
+  lock_conflict_delay_seconds: 5
+  stale_active_after_seconds: 600
+  delayed_mover_interval_seconds: 5
+  recovery_interval_seconds: 300
+  delayed_move_limit: 100
+  recover_limit: 100
+
+  vertex_enabled: false
+  vertex_project_id: ""
+  vertex_location: "global"
+  vertex_managed_gcs_bucket: ""
+  vertex_managed_gcs_prefix: "batch-image/{env}/{batch_id}"
+  vertex_input_retention_hours: 24
+  vertex_output_retention_hours: 72
+  vertex_batch_prediction_base_url: ""
+  vertex_gcs_base_url: ""
+```
+
+Feature flags default to disabled.
+
+## Operations Checklist
+
+- Enable `batch_image.enabled`.
+- Configure Redis.
+- Enable `batch_image.queue_enabled` when workers should consume queue jobs.
+- Configure provider accounts.
+- Configure the Vertex managed GCS bucket if using Vertex.
+- Ensure bucket permissions are correct.
+- Disable or manage GCS soft delete.
+- Configure cleanup worker settings.
+- Configure max items per job.
+- Configure download concurrency.
+- Confirm billing pricing.
+- Run smoke tests before enabling.
+
+## Security Checklist
+
+- No provider refs in public responses.
+- No GCS URI exposure.
+- No signed URL exposure.
+- No service account exposure.
+- No API key exposure.
+- No image bytes/base64 in PostgreSQL.
+- No base64 in logs.
+- Owner-scoped status, item, download, cancel, and delete routes.
+- Output deletion is owner-scoped.
+- Cleanup paths are server-generated only.
+
+## Test Commands
+
+Core smoke and compile commands:
+
+```bash
+go test -tags=unit ./internal/service -run 'BatchImage' -count=1
+go test -tags=unit ./internal/config ./internal/service ./internal/repository -count=1
+go test ./internal/config ./internal/service ./internal/repository ./internal/handler ./internal/server/routes -run '^$'
+go test ./... -run '^$'
+```
+
+These commands should not require Docker, testcontainers, Redis, GCP, Gemini, Vertex, or GCS.
+
+## PR Hygiene Checklist
+
+- Do not accidentally commit `rfcs/batch-image-issue-draft.md` unless maintainers explicitly want it.
+- Keep migrations ordered: `159_batch_image_foundation.sql`, then `160_batch_image_provider_refs.sql`, then later migrations.
+- Include generated Ent code if generated code is committed in this repository.
+- Keep generated server and wire files updated.
+- Keep feature flags disabled by default unless maintainers ask otherwise.
+- Do not commit real secrets, API keys, service account JSON, or local machine paths.
+- Keep fixtures tiny and fake; no real cloud refs or credentials.
+- Do not add new public routes, providers, dashboards, queues, or billing behavior in this stabilization PR.