mirror of
https://github.com/HKUDS/CLI-Anything.git
synced 2026-08-29 07:30:51 +08:00
test: add calibre harness smoke validation
This commit is contained in:
yuhao
@@ -0,0 +1,60 @@
|
||||
# FIX_NOTES.md - PR #223 Calibre Harness Validation
|
||||
|
||||
## Blocker Status
|
||||
|
||||
- No code blocker was identified in the Calibre harness.
|
||||
- Remaining validation/documentation blocker addressed by adding no-Calibre subprocess smoke coverage and explicit real-backend validation steps.
|
||||
|
||||
## No-Calibre Smoke Validation
|
||||
|
||||
These commands validate importability, Click entrypoint behavior, and missing-library
|
||||
error handling without requiring `calibredb`, `ebook-convert`, or `ebook-meta`.
|
||||
|
||||
```bash
|
||||
cd calibre/agent-harness
|
||||
python -m py_compile \
|
||||
cli_anything/calibre/calibre_cli.py \
|
||||
cli_anything/calibre/core/*.py \
|
||||
cli_anything/calibre/utils/*.py
|
||||
python -m pytest cli_anything/calibre/tests/test_core.py -v
|
||||
```
|
||||
|
||||
To require the installed console script for smoke validation:
|
||||
|
||||
```bash
|
||||
cd calibre/agent-harness
|
||||
pip install -e .
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
|
||||
cli_anything/calibre/tests/test_core.py::TestCLISubprocessSmoke -v
|
||||
```
|
||||
|
||||
## Real Calibre Backend Validation
|
||||
|
||||
Install Calibre first and confirm all wrapped commands resolve:
|
||||
|
||||
```bash
|
||||
which calibredb
|
||||
which ebook-convert
|
||||
which ebook-meta
|
||||
```
|
||||
|
||||
Then run the E2E suite:
|
||||
|
||||
```bash
|
||||
cd calibre/agent-harness
|
||||
pip install -e .
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
|
||||
cli_anything/calibre/tests/test_full_e2e.py -v -s
|
||||
```
|
||||
|
||||
Expected E2E coverage:
|
||||
|
||||
- Temporary Calibre libraries are created for test isolation.
|
||||
- Generated EPUB fixtures are imported through `calibredb`.
|
||||
- Metadata changes are round-tripped through the real backend.
|
||||
- EPUB to TXT/MOBI conversion runs through `ebook-convert`.
|
||||
- Exported and converted artifacts are checked for existence and nonzero size.
|
||||
|
||||
## Remaining Gaps
|
||||
|
||||
- Catalog generation, file-level metadata embedding, and real custom-column workflows are documented as E2E gaps in `cli_anything/calibre/tests/TEST.md`.
|
||||
@@ -177,12 +177,38 @@ Override with the `CALIBRE_LIBRARY` environment variable.
|
||||
```bash
|
||||
cd agent-harness
|
||||
|
||||
# Unit tests (no Calibre required)
|
||||
pytest cli_anything/calibre/tests/test_core.py -v
|
||||
# Syntax check
|
||||
python -m py_compile \
|
||||
cli_anything/calibre/calibre_cli.py \
|
||||
cli_anything/calibre/core/*.py \
|
||||
cli_anything/calibre/utils/*.py
|
||||
|
||||
# Unit and CLI smoke tests (no Calibre required)
|
||||
python -m pytest cli_anything/calibre/tests/test_core.py -v
|
||||
|
||||
# Installed-command smoke tests (no Calibre required)
|
||||
pip install -e .
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
|
||||
cli_anything/calibre/tests/test_core.py::TestCLISubprocessSmoke -v
|
||||
|
||||
# Full E2E tests (Calibre required)
|
||||
pytest cli_anything/calibre/tests/test_full_e2e.py -v -s
|
||||
python -m pytest cli_anything/calibre/tests/test_full_e2e.py -v -s
|
||||
|
||||
# All tests with installed binary
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 pytest cli_anything/calibre/tests/ -v -s
|
||||
# All tests with installed binary and real Calibre backend
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest cli_anything/calibre/tests/ -v -s
|
||||
```
|
||||
|
||||
### Real Backend Validation
|
||||
|
||||
Before running E2E validation, verify Calibre's commands are on PATH:
|
||||
|
||||
```bash
|
||||
which calibredb
|
||||
which ebook-convert
|
||||
which ebook-meta
|
||||
```
|
||||
|
||||
The E2E suite creates temporary Calibre libraries, imports generated EPUB files,
|
||||
updates metadata with `calibredb`, converts EPUB files with `ebook-convert`, and
|
||||
checks exported/converted artifacts for real output files. These tests require a
|
||||
real Calibre installation; `test_core.py` remains the no-Calibre validation path.
|
||||
|
||||
@@ -5,7 +5,13 @@
|
||||
This document covers the test plan and results for `cli-anything-calibre`, a CLI harness
|
||||
wrapping the real Calibre tools (`calibredb`, `ebook-convert`, `ebook-meta`).
|
||||
|
||||
**Hard dependency:** Calibre must be installed (`calibredb` in PATH).
|
||||
**Unit/smoke dependency:** `test_core.py` does not require Calibre. It includes
|
||||
subprocess smoke checks for help, version, and missing-library behavior using the
|
||||
installed `cli-anything-calibre` command when available, or `python -m
|
||||
cli_anything.calibre` as a development fallback.
|
||||
|
||||
**E2E hard dependency:** `test_full_e2e.py` requires Calibre (`calibredb`,
|
||||
`ebook-convert`, and `ebook-meta` in PATH) for real backend validation.
|
||||
|
||||
---
|
||||
|
||||
@@ -13,8 +19,8 @@ wrapping the real Calibre tools (`calibredb`, `ebook-convert`, `ebook-meta`).
|
||||
|
||||
| File | Tests Planned | Description |
|
||||
|------|--------------|-------------|
|
||||
| `test_core.py` | 38 | Unit tests: synthetic data, no external deps, no Calibre needed |
|
||||
| `test_full_e2e.py` | 20 | E2E tests: real Calibre library operations + subprocess CLI tests |
|
||||
| `test_core.py` | 41 | Unit tests: synthetic data, no external deps, no Calibre needed, plus subprocess smoke |
|
||||
| `test_full_e2e.py` | 21 | E2E tests: real Calibre library operations + subprocess CLI tests |
|
||||
|
||||
---
|
||||
|
||||
@@ -72,13 +78,36 @@ wrapping the real Calibre tools (`calibredb`, `ebook-convert`, `ebook-meta`).
|
||||
| `test_find_ebook_meta_missing` | find_ebook_meta() raises RuntimeError if not in PATH |
|
||||
| `test_error_message_contains_install_hint` | Error messages include install instructions |
|
||||
|
||||
### `calibre_cli.py` — 3 tests
|
||||
### `calibre_cli.py` — 6 tests
|
||||
|
||||
| Test | Description |
|
||||
|------|-------------|
|
||||
| `test_cli_help_exits_zero` | `--help` prints usage and returns 0 |
|
||||
| `test_cli_version` | `--version` returns version string |
|
||||
| `test_cli_missing_library_error` | Commands without library set print clear error |
|
||||
| `test_installed_or_module_help_smoke` | Subprocess `--help` smoke test; uses installed entry point if present |
|
||||
| `test_installed_or_module_version_smoke` | Subprocess `--version` smoke test; uses installed entry point if present |
|
||||
| `test_missing_library_error_without_calibre` | Subprocess missing-library error works without Calibre installed |
|
||||
|
||||
Run no-backend validation:
|
||||
|
||||
```bash
|
||||
cd calibre/agent-harness
|
||||
python -m py_compile \
|
||||
cli_anything/calibre/calibre_cli.py \
|
||||
cli_anything/calibre/core/*.py \
|
||||
cli_anything/calibre/utils/*.py
|
||||
python -m pytest cli_anything/calibre/tests/test_core.py -v
|
||||
```
|
||||
|
||||
Installed-command smoke mode:
|
||||
|
||||
```bash
|
||||
cd calibre/agent-harness
|
||||
pip install -e .
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
|
||||
cli_anything/calibre/tests/test_core.py::TestCLISubprocessSmoke -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -91,14 +120,15 @@ These tests invoke the **real Calibre** tools and verify the output.
|
||||
E2E tests use a temporary Calibre library created with `calibredb add` from a real EPUB.
|
||||
A minimal EPUB is generated programmatically (valid ZIP structure) for reproducibility.
|
||||
|
||||
### `TestLibraryOperations` — 5 tests
|
||||
### `TestLibraryOperations` — 6 tests
|
||||
|
||||
| Test | Description | Verified |
|
||||
|------|-------------|---------|
|
||||
| `test_create_library_and_add_epub` | Add EPUB to new library | Book ID returned, book count > 0 |
|
||||
| fixture setup | Add EPUB to new library | Book ID returned, book count > 0 |
|
||||
| `test_list_books` | list_books() returns book entries | ID, title, authors present |
|
||||
| `test_list_books_custom_fields` | list_books() honors explicit field list | Requested fields present, omitted fields absent |
|
||||
| `test_search_books` | search_books() with query | Returns matching IDs |
|
||||
| `test_show_metadata` | get_metadata() returns parsed OPF | title, authors fields present |
|
||||
| `test_get_metadata` | get_metadata() returns parsed OPF | title, authors fields present |
|
||||
| `test_export_books` | export_books() exports to directory | Files exported, cover.jpg present |
|
||||
|
||||
### `TestMetadataOperations` — 3 tests
|
||||
@@ -182,7 +212,39 @@ Tests the installed `cli-anything-calibre` command directly via subprocess.
|
||||
|
||||
## Test Results
|
||||
|
||||
Run command:
|
||||
No-backend validation run:
|
||||
|
||||
```bash
|
||||
cd calibre/agent-harness
|
||||
python -m py_compile \
|
||||
cli_anything/calibre/calibre_cli.py \
|
||||
cli_anything/calibre/core/*.py \
|
||||
cli_anything/calibre/utils/*.py
|
||||
python -m pytest cli_anything/calibre/tests/test_core.py -v
|
||||
```
|
||||
|
||||
Current no-backend result:
|
||||
|
||||
```text
|
||||
41 passed in 0.74s
|
||||
```
|
||||
|
||||
Installed-command smoke run:
|
||||
|
||||
```bash
|
||||
cd calibre/agent-harness
|
||||
pip install -e .
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
|
||||
cli_anything/calibre/tests/test_core.py::TestCLISubprocessSmoke -v
|
||||
```
|
||||
|
||||
Current installed-command smoke result:
|
||||
|
||||
```text
|
||||
3 passed in 0.63s
|
||||
```
|
||||
|
||||
Historical real-backend run:
|
||||
```bash
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest cli_anything/calibre/tests/ -v --tb=no
|
||||
```
|
||||
@@ -250,14 +312,38 @@ cli_anything/calibre/tests/test_full_e2e.py::TestCLISubprocess::test_version PAS
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total tests | 50 |
|
||||
| Passed | 50 |
|
||||
| Total tests expected now | 62 |
|
||||
| No-backend tests passed in current run | 41 |
|
||||
| Installed smoke tests passed in current run | 3 |
|
||||
| Historical full-suite tests passed | 50 |
|
||||
| Failed | 0 |
|
||||
| Pass rate | 100% |
|
||||
| Execution time | 15.93s |
|
||||
| Current no-backend execution time | 0.74s |
|
||||
| Current installed-smoke execution time | 0.63s |
|
||||
| Historical full-suite execution time | 15.93s |
|
||||
| Calibre version | 7.6 |
|
||||
| Subprocess backend | `/home/orgleaf/py-base-venv/bin/cli-anything-calibre` (installed) |
|
||||
|
||||
## Real Backend Validation Steps
|
||||
|
||||
Use these steps to validate the harness against a real Calibre install:
|
||||
|
||||
```bash
|
||||
cd calibre/agent-harness
|
||||
pip install -e .
|
||||
which calibredb
|
||||
which ebook-convert
|
||||
which ebook-meta
|
||||
CLI_ANYTHING_FORCE_INSTALLED=1 python -m pytest \
|
||||
cli_anything/calibre/tests/test_full_e2e.py -v -s
|
||||
```
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- Tests create temporary Calibre libraries and seed them with generated EPUB files.
|
||||
- `calibredb` is used for add/list/search/metadata/export operations.
|
||||
- `ebook-convert` is used for EPUB to TXT/MOBI conversion and output files are checked for existence and nonzero size.
|
||||
- Subprocess E2E tests require the installed `cli-anything-calibre` entry point when `CLI_ANYTHING_FORCE_INSTALLED=1` is set.
|
||||
|
||||
## Coverage Notes
|
||||
|
||||
- All session persistence operations are tested (save, load, env override, error paths)
|
||||
|
||||
@@ -5,6 +5,8 @@ All tests use synthetic data — no real Calibre installation required.
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
@@ -614,5 +616,59 @@ class TestCLIHelp(unittest.TestCase):
|
||||
)
|
||||
|
||||
|
||||
class TestCLISubprocessSmoke(unittest.TestCase):
|
||||
"""Subprocess smoke tests that do not require Calibre to be installed."""
|
||||
|
||||
@staticmethod
|
||||
def _resolve_cli():
|
||||
force = os.environ.get("CLI_ANYTHING_FORCE_INSTALLED", "").strip() == "1"
|
||||
installed = shutil.which("cli-anything-calibre")
|
||||
if installed:
|
||||
return [installed]
|
||||
if force:
|
||||
raise RuntimeError(
|
||||
"cli-anything-calibre not found in PATH. Install with: pip install -e ."
|
||||
)
|
||||
return [sys.executable, "-m", "cli_anything.calibre"]
|
||||
|
||||
def _run(self, args, *, home=None):
|
||||
env = os.environ.copy()
|
||||
harness_root = Path(__file__).resolve().parents[3]
|
||||
env["PYTHONPATH"] = (
|
||||
str(harness_root)
|
||||
if not env.get("PYTHONPATH")
|
||||
else f"{harness_root}{os.pathsep}{env['PYTHONPATH']}"
|
||||
)
|
||||
env.pop("CALIBRE_LIBRARY", None)
|
||||
if home:
|
||||
env["HOME"] = home
|
||||
return subprocess.run(
|
||||
self._resolve_cli() + args,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
check=False,
|
||||
)
|
||||
|
||||
def test_installed_or_module_help_smoke(self):
|
||||
result = self._run(["--help"])
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("Usage:", result.stdout)
|
||||
self.assertIn("library", result.stdout)
|
||||
|
||||
def test_installed_or_module_version_smoke(self):
|
||||
result = self._run(["--version"])
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("cli-anything-calibre", result.stdout)
|
||||
|
||||
def test_missing_library_error_without_calibre(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
result = self._run(["books", "list"], home=tmp)
|
||||
combined = result.stdout + result.stderr
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("No Calibre library connected", combined)
|
||||
self.assertIn("CALIBRE_LIBRARY", combined)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user