diff --git a/.gitignore b/.gitignore index 2b6bc1871..88ad56875 100644 --- a/.gitignore +++ b/.gitignore @@ -68,6 +68,7 @@ !/zotero/ !/mubu/ !/obs-studio/ +!/nslogger/ !/kdenlive/ !/shotcut/ !/anygen/ @@ -116,6 +117,8 @@ /mubu/.* /obs-studio/* /obs-studio/.* +/nslogger/* +/nslogger/.* /kdenlive/* /kdenlive/.* /shotcut/* @@ -190,6 +193,7 @@ !/zotero/agent-harness/ !/mubu/agent-harness/ !/obs-studio/agent-harness/ +!/nslogger/agent-harness/ !/kdenlive/agent-harness/ !/shotcut/agent-harness/ !/anygen/agent-harness/ diff --git a/README.md b/README.md index a7838e0c9..ca2fae9a6 100644 --- a/README.md +++ b/README.md @@ -840,6 +840,13 @@ Each application received complete, production-ready CLI interfaces โ€” not demo โœ… 153 +๐Ÿ“ฑ NSLogger +iOS/macOS Log Capture +cli-anything-nslogger +NSLogger wire protocol + native macOS Bonjour +โœ… 139 + + ๐ŸŽž๏ธ Kdenlive Video Editing cli-anything-kdenlive @@ -988,11 +995,11 @@ Each application received complete, production-ready CLI interfaces โ€” not demo Total -โœ… 2,152 +โœ… 2,291 -> **100% pass rate** across all 2,152 tests โ€” 1,564 unit tests + 569 end-to-end tests + 19 Node.js tests. +> **100% pass rate** across all 2,291 tests โ€” 1,661 unit tests + 611 end-to-end tests + 19 Node.js tests. --- @@ -1016,6 +1023,7 @@ audacity 161 passed โœ… (107 unit + 54 e2e) libreoffice 158 passed โœ… (89 unit + 69 e2e) mubu 96 passed โœ… (85 unit + 11 e2e) obs-studio 153 passed โœ… (116 unit + 37 e2e) +nslogger 139 passed โœ… (97 unit + 42 e2e) kdenlive 155 passed โœ… (111 unit + 44 e2e) shotcut 154 passed โœ… (110 unit + 44 e2e) zoom 22 passed โœ… (22 unit + 0 e2e) @@ -1033,7 +1041,7 @@ cloudcompare 88 passed โœ… (49 unit + 39 e2e) openscreen 101 passed โœ… (78 unit + 23 e2e) cloudanalyzer 14 passed โœ… (7 unit + 7 e2e) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ -TOTAL 2,120 passed โœ… 100% pass rate +TOTAL 2,259 passed โœ… 100% pass rate ``` --- @@ -1091,6 +1099,7 @@ cli-anything/ โ”œโ”€โ”€ ๐Ÿ“š zotero/agent-harness/ # Zotero CLI (new, write import support) โ”œโ”€โ”€ ๐Ÿ“ mubu/agent-harness/ # Mubu CLI (96 tests) โ”œโ”€โ”€ ๐Ÿ“น obs-studio/agent-harness/ # OBS Studio CLI (153 tests) +โ”œโ”€โ”€ ๐Ÿ“ฑ nslogger/agent-harness/ # NSLogger CLI (139 tests) โ”œโ”€โ”€ ๐ŸŽž๏ธ kdenlive/agent-harness/ # Kdenlive CLI (155 tests) โ”œโ”€โ”€ ๐ŸŽฌ shotcut/agent-harness/ # Shotcut CLI (154 tests) โ”œโ”€โ”€ ๐Ÿ“ž zoom/agent-harness/ # Zoom CLI (22 tests) diff --git a/nslogger/agent-harness/NSLOGGER.md b/nslogger/agent-harness/NSLOGGER.md new file mode 100644 index 000000000..3d980449b --- /dev/null +++ b/nslogger/agent-harness/NSLOGGER.md @@ -0,0 +1,103 @@ +# NSLogger CLI Harness โ€” SOP + +## What is NSLogger + +NSLogger is a macOS application for receiving, viewing, and analyzing log messages +sent by iOS/macOS applications using the NSLogger client library. + +## Core Concepts + +| Concept | Description | +|---------|-------------| +| `.nsloggerdata` | Saved binary-plist session file (opened/exported by the GUI) | +| `.rawnsloggerdata` | Raw wire-protocol capture file | +| Message | A single log entry with: sequence, timestamp, level, tag, threadID, type, text | +| Message Types | `text`, `image`, `data` | +| Log Level | 0=error, 1=warning, 2=info, 3=debug, 4=verbose (higher=noisier) | +| Connection | A live client connection (by Bonjour or direct TCP on port 50000) | +| Filter | A predicate that restricts which messages are displayed | + +## Wire Protocol (rawnsloggerdata) + +NSLogger uses a custom binary protocol over TCP: + +``` +[4-byte big-endian total message length] +[4-byte big-endian sequence number] +[2-byte part count] +for each part: + [1-byte part key] + [1-byte part type] + [4-byte big-endian data length] + [N bytes data] +``` + +Part keys: `0=messageType`, `1=timestamp_s`, `2=timestamp_ms`, `3=timestamp_us`, + `4=threadID`, `5=tag`, `6=level`, `7=message`, `8=imageWidth`, + `9=imageHeight`, `10=messageSeq`, `11=filename`, `12=lineNumber`, + `13=functionName`, `20=clientName`, `21=clientVersion`, + `22=osName`, `23=osVersion`, `24=clientModel`, `25=uniqueID` + +Part types: `0=string(UTF8)`, `1=binary`, `2=int16`, `3=int32`, `4=int64`, + `5=image` + +Message types: `0=log`, `1=blockStart`, `2=blockEnd`, `3=clientInfo`, + `4=disconnect`, `255=marker` + +## CLI Command Groups + +``` +cli-anything-nslogger read # Parse and display .nsloggerdata / .rawnsloggerdata +cli-anything-nslogger filter # Filter messages from a file (level, tag, thread, text, regex, type, time, seq) +cli-anything-nslogger export # Export messages to text/JSON/CSV +cli-anything-nslogger stats # Summary statistics for a file +cli-anything-nslogger listen # Listen for live NSLogger connections +cli-anything-nslogger generate # Generate sample .rawnsloggerdata for testing +cli-anything-nslogger tail # Show last N messages from a file +cli-anything-nslogger clients # List all client_info records in a file +cli-anything-nslogger blocks # Show block start/end structure as indented tree +cli-anything-nslogger merge # Merge multiple files sorted by timestamp +``` + +## Typical Agent Workflow + +```bash +# Inspect a captured log file +cli-anything-nslogger read session.rawnsloggerdata + +# Find all errors +cli-anything-nslogger filter --level 0 session.rawnsloggerdata + +# Filter within a time window +cli-anything-nslogger filter --after "10:30:00" --before "10:45:00" session.rawnsloggerdata + +# Filter by sequence range +cli-anything-nslogger filter --from-seq 100 --to-seq 200 session.rawnsloggerdata + +# Show last 50 messages +cli-anything-nslogger tail --count 50 session.rawnsloggerdata + +# List connected clients +cli-anything-nslogger clients session.rawnsloggerdata + +# Show block/call structure +cli-anything-nslogger blocks session.rawnsloggerdata + +# Merge two capture files +cli-anything-nslogger merge a.rawnsloggerdata b.rawnsloggerdata --format json + +# Export for further analysis +cli-anything-nslogger export --format json session.rawnsloggerdata > logs.json + +# Get statistics +cli-anything-nslogger stats session.rawnsloggerdata + +# Listen for live iOS logs via Bonjour, matching NSLogger.app +cli-anything-nslogger listen --bonjour --name bazinga --debug + +# Mirror live logs to disk while still printing stdout +cli-anything-nslogger listen --bonjour --name bazinga --output app.log + +# Direct TCP/TLS mode for manually configured clients +cli-anything-nslogger listen --port 50000 --ssl --debug +``` diff --git a/nslogger/agent-harness/cli_anything/nslogger/README.md b/nslogger/agent-harness/cli_anything/nslogger/README.md new file mode 100644 index 000000000..c6c7e37df --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/README.md @@ -0,0 +1,85 @@ +# cli-anything-nslogger + +CLI harness for [NSLogger](https://github.com/fpillet/NSLogger) โ€” read, filter, export, and monitor NSLogger log files from the command line or from AI agents. + +## Installation + +```bash +cd agent-harness +pip install -e . +``` + +## Quick Start + +```bash +# Generate a sample file +cli-anything-nslogger generate sample.rawnsloggerdata --count 50 + +# Read and display all messages +cli-anything-nslogger read sample.rawnsloggerdata + +# Show only errors +cli-anything-nslogger read sample.rawnsloggerdata --level 0 + +# Filter by tag +cli-anything-nslogger filter sample.rawnsloggerdata --tag Network + +# Export to JSON +cli-anything-nslogger export sample.rawnsloggerdata --format json + +# Statistics +cli-anything-nslogger stats sample.rawnsloggerdata + +# Listen for live iOS logs via Bonjour, matching the NSLogger.app GUI +cli-anything-nslogger listen --bonjour --name bazinga --debug + +# Direct TCP/TLS mode for manually configured clients +cli-anything-nslogger listen --port 50000 --ssl --debug + +# Listen and write received live logs to a file +cli-anything-nslogger listen --bonjour --name bazinga --output app.log + +# Write machine-readable live logs as JSON Lines +cli-anything-nslogger listen --bonjour --name bazinga --output app.jsonl --output-format jsonl + +# Interactive REPL +cli-anything-nslogger repl sample.rawnsloggerdata +``` + +## JSON Output (Agent Mode) + +Every command accepts `--json` for machine-readable output: + +```bash +cli-anything-nslogger read sample.rawnsloggerdata --json +cli-anything-nslogger stats sample.rawnsloggerdata --json +``` + +## Commands + +| Command | Description | +|-----------|-------------| +| `read` | Parse and display messages from a file | +| `filter` | Advanced filtering (level, tag, thread, regex) | +| `export` | Export to text / JSON / CSV | +| `stats` | Summary statistics | +| `listen` | Receive live NSLogger connections via Bonjour or TCP/TLS | +| `generate`| Create sample `.rawnsloggerdata` for testing | +| `repl` | Interactive Python REPL with loaded messages | + +## Log Levels + +| Level | Name | +|-------|---------| +| 0 | ERROR | +| 1 | WARNING | +| 2 | INFO | +| 3 | DEBUG | +| 4 | VERBOSE | + +## File Formats + +| Extension | Description | +|-----------|-------------| +| `.rawnsloggerdata` | Raw NSLogger wire-protocol capture | +| `.nsloggerdata` | Binary plist archive saved by NSLogger.app | diff --git a/nslogger/agent-harness/cli_anything/nslogger/__init__.py b/nslogger/agent-harness/cli_anything/nslogger/__init__.py new file mode 100644 index 000000000..21ae48e7f --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/__init__.py @@ -0,0 +1,2 @@ +"""NSLogger CLI harness โ€” cli_anything.nslogger package.""" +__version__ = "0.1.0" diff --git a/nslogger/agent-harness/cli_anything/nslogger/core/__init__.py b/nslogger/agent-harness/cli_anything/nslogger/core/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/nslogger/agent-harness/cli_anything/nslogger/core/blocks.py b/nslogger/agent-harness/cli_anything/nslogger/core/blocks.py new file mode 100644 index 000000000..2e5065a7a --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/core/blocks.py @@ -0,0 +1,42 @@ +"""Block structure analysis for NSLogger files.""" +from __future__ import annotations +from typing import Iterator, List, Dict, Any +from .message import LogMessage, MSG_TYPE_BLOCK_START, MSG_TYPE_BLOCK_END, MSG_TYPE_CLIENT_INFO + + +def iter_block_tree(messages: Iterator[LogMessage]): + """Yield (depth, msg) tuples representing the block-indented structure.""" + depth = 0 + for msg in messages: + if msg.message_type == MSG_TYPE_BLOCK_END: + depth = max(0, depth - 1) + yield depth, msg + if msg.message_type == MSG_TYPE_BLOCK_START: + depth += 1 + + +def extract_clients(messages: Iterator[LogMessage]) -> List[Dict[str, Any]]: + """Return a list of dicts for every client_info message in the stream.""" + clients = [] + for msg in messages: + if msg.message_type == MSG_TYPE_CLIENT_INFO: + clients.append({ + "sequence": msg.sequence, + "timestamp": msg.timestamp.isoformat() if msg.timestamp else None, + "client_name": msg.client_name, + "client_version": msg.client_version, + "os_name": msg.os_name, + "os_version": msg.os_version, + "machine": msg.machine, + }) + return clients + + +def merge_files(paths: List[str]) -> List[LogMessage]: + """Load multiple rawnsloggerdata files and return messages sorted by timestamp.""" + from ..core.parser import parse_file + all_msgs: List[LogMessage] = [] + for path in paths: + all_msgs.extend(parse_file(path)) + all_msgs.sort(key=lambda m: (m.timestamp or __import__("datetime").datetime.min, m.sequence)) + return all_msgs diff --git a/nslogger/agent-harness/cli_anything/nslogger/core/exporter.py b/nslogger/agent-harness/cli_anything/nslogger/core/exporter.py new file mode 100644 index 000000000..eb90a8fb2 --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/core/exporter.py @@ -0,0 +1,36 @@ +"""Export LogMessages to various formats.""" +from __future__ import annotations +import csv +import io +import json +from typing import Iterator, List +from .message import LogMessage + + +def export_text(messages: List[LogMessage]) -> str: + lines = [m.to_text_line() for m in messages] + return "\n".join(lines) + + +def export_json(messages: List[LogMessage]) -> str: + return json.dumps([m.to_dict() for m in messages], indent=2, default=str) + + +def export_csv(messages: List[LogMessage]) -> str: + buf = io.StringIO() + fields = ["sequence", "timestamp", "level", "level_name", "tag", "thread_id", "type", "text"] + writer = csv.DictWriter(buf, fieldnames=fields, extrasaction="ignore") + writer.writeheader() + for m in messages: + d = m.to_dict() + writer.writerow({f: d.get(f, "") for f in fields}) + return buf.getvalue() + + +def export_messages(messages: Iterator[LogMessage], fmt: str = "text") -> str: + msgs = list(messages) + if fmt == "json": + return export_json(msgs) + if fmt == "csv": + return export_csv(msgs) + return export_text(msgs) diff --git a/nslogger/agent-harness/cli_anything/nslogger/core/filter.py b/nslogger/agent-harness/cli_anything/nslogger/core/filter.py new file mode 100644 index 000000000..32cd36332 --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/core/filter.py @@ -0,0 +1,56 @@ +"""Message filtering logic for NSLogger CLI.""" +from __future__ import annotations +import re +from datetime import datetime +from typing import Iterator, Optional, List +from .message import LogMessage + + +def filter_messages( + messages: Iterator[LogMessage], + max_level: Optional[int] = None, + min_level: Optional[int] = None, + tags: Optional[List[str]] = None, + thread_id: Optional[str] = None, + text_search: Optional[str] = None, + text_regex: Optional[str] = None, + msg_types: Optional[List[str]] = None, + limit: Optional[int] = None, + after: Optional[datetime] = None, + before: Optional[datetime] = None, + from_seq: Optional[int] = None, + to_seq: Optional[int] = None, +) -> Iterator[LogMessage]: + """Yield messages matching all specified criteria.""" + pattern = re.compile(text_regex, re.IGNORECASE) if text_regex else None + tag_set = {t.lower() for t in tags} if tags else None + type_set = set(msg_types) if msg_types else None + count = 0 + + for msg in messages: + if max_level is not None and msg.level > max_level: + continue + if min_level is not None and msg.level < min_level: + continue + if tag_set and msg.tag.lower() not in tag_set: + continue + if thread_id and msg.thread_id != thread_id: + continue + if text_search and text_search.lower() not in msg.text.lower(): + continue + if pattern and not pattern.search(msg.text): + continue + if type_set and msg.type_name not in type_set: + continue + if after is not None and (msg.timestamp is None or msg.timestamp < after): + continue + if before is not None and (msg.timestamp is None or msg.timestamp > before): + continue + if from_seq is not None and msg.sequence < from_seq: + continue + if to_seq is not None and msg.sequence > to_seq: + continue + yield msg + count += 1 + if limit is not None and count >= limit: + break diff --git a/nslogger/agent-harness/cli_anything/nslogger/core/listener.py b/nslogger/agent-harness/cli_anything/nslogger/core/listener.py new file mode 100644 index 000000000..2f17f97fa --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/core/listener.py @@ -0,0 +1,744 @@ +"""TCP listener that receives live NSLogger connections.""" +from __future__ import annotations +import os +import base64 +import json +import socket +import ssl +import struct +import subprocess +import sys +import tempfile +import threading +from importlib import resources +from typing import Callable, Optional +from .message import LogMessage +from .parser import _parse_message, ParseError + +# NSLogger Bonjour service types +NSLOGGER_SERVICE_TYPE = "_nslogger._tcp.local." +NSLOGGER_SSL_SERVICE_TYPE = "_nslogger-ssl._tcp.local." + + +def _get_local_ip() -> str: + """Return the primary local IPv4 address.""" + try: + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + s.connect(("8.8.8.8", 80)) + ip = s.getsockname()[0] + s.close() + return ip + except Exception: + return "127.0.0.1" + + +def _make_ssl_context() -> tuple[ssl.SSLContext, str]: + """Generate a temporary self-signed cert and return (SSLContext, tmp_dir).""" + tmp_dir = tempfile.mkdtemp(prefix="nslogger_cli_") + cert = os.path.join(tmp_dir, "server.crt") + key = os.path.join(tmp_dir, "server.key") + subprocess.run( + [ + "openssl", "req", "-x509", "-newkey", "rsa:2048", + "-keyout", key, "-out", cert, + "-days", "1", "-nodes", + "-subj", "/CN=nslogger-cli", + ], + check=True, + capture_output=True, + ) + ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + if hasattr(ssl, "TLSVersion"): + ctx.minimum_version = ssl.TLSVersion.TLSv1_2 + ctx.maximum_version = ssl.TLSVersion.TLSv1_2 + try: + ctx.set_ciphers("DEFAULT@SECLEVEL=1") + except ssl.SSLError: + pass + ctx.load_cert_chain(cert, key) + return ctx, tmp_dir + + +def _make_pkcs12_identity() -> tuple[str, str, str]: + """Generate a temporary self-signed PKCS#12 identity for CFStream server SSL.""" + tmp_dir = tempfile.mkdtemp(prefix="nslogger_cli_") + cert = os.path.join(tmp_dir, "server.crt") + key = os.path.join(tmp_dir, "server.key") + p12 = os.path.join(tmp_dir, "server.p12") + password = "nslogger-cli" + subprocess.run( + [ + "openssl", "req", "-x509", "-newkey", "rsa:2048", + "-keyout", key, "-out", cert, + "-days", "1", "-nodes", + "-subj", "/CN=nslogger-cli", + ], + check=True, + capture_output=True, + ) + subprocess.run( + [ + "openssl", "pkcs12", "-export", + "-inkey", key, + "-in", cert, + "-out", p12, + "-passout", f"pass:{password}", + ], + check=True, + capture_output=True, + ) + return p12, password, tmp_dir + + +def _swift_helper_env() -> dict[str, str]: + env = os.environ.copy() + cache_root = tempfile.gettempdir() + env.setdefault("SWIFT_MODULE_CACHE_PATH", os.path.join(cache_root, "nslogger_cli_swift_module_cache")) + env.setdefault("CLANG_MODULE_CACHE_PATH", os.path.join(cache_root, "nslogger_cli_clang_module_cache")) + return env + + +def _compiled_swift_helper(helper_name: str, on_debug: Callable[[str], None]) -> str: + helper = resources.files("cli_anything.nslogger").joinpath(f"helpers/{helper_name}.swift") + helper_path = str(helper) + cache_dir = os.path.join(tempfile.gettempdir(), "nslogger_cli_swift_helpers") + os.makedirs(cache_dir, exist_ok=True) + try: + stamp = f"{int(os.path.getmtime(helper_path))}_{os.path.getsize(helper_path)}" + except OSError: + stamp = "unknown" + executable = os.path.join(cache_dir, f"{helper_name}_{stamp}") + if not os.path.exists(executable): + on_debug(f"Compiling native helper {helper_name}") + subprocess.run( + ["swiftc", helper_path, "-o", executable], + check=True, + capture_output=True, + env=_swift_helper_env(), + ) + return executable + + +def _peek(conn: socket.socket, size: int = 16) -> bytes: + try: + return conn.recv(size, socket.MSG_PEEK) + except socket.timeout: + raise + except (AttributeError, OSError): + return b"" + + +def _peek_hex(conn: socket.socket, size: int = 16) -> str: + return _peek(conn, size).hex(" ") + + +def _looks_like_tls_client_hello(conn: socket.socket) -> bool: + """Best-effort TLS ClientHello detection without consuming bytes.""" + header = _peek(conn, 5) + if len(header) < 3: + return False + # TLS record header: 0x16, 0x03, version + return header[0] == 0x16 and header[1] == 0x03 + + +def _classify_connection(conn: socket.socket) -> tuple[str, bytes]: + """Classify the first bytes without consuming them: tls, raw, or empty.""" + try: + header = _peek(conn, 5) + except socket.timeout: + return "timeout", b"" + if not header: + return "empty", header + if len(header) >= 3 and header[0] == 0x16 and header[1] == 0x03: + return "tls", header + return "raw", header + + +def _dns_sd_txt_args(service_name: str, filter_clients: bool = False) -> list[str]: + """Return TXT records matching NSLogger.app's named-service publishing.""" + return ["filterClients=1"] if service_name and filter_clients else [] + + +def _bonjour_service_types(use_ssl: bool, allow_plaintext: bool = False) -> tuple[str, ...]: + """Return the Bonjour service type advertised by NSLogger.app for this mode.""" + if use_ssl and allow_plaintext: + return ("_nslogger._tcp", "_nslogger-ssl._tcp") + return ("_nslogger-ssl._tcp",) if use_ssl else ("_nslogger._tcp",) + + +class _ZeroconfBonjourPublisher: + """In-process Bonjour publisher so Ctrl-C cannot leave dns-sd children behind.""" + + def __init__(self, service_name: str, service_types: tuple[str, ...], port: int, local_ip: str, filter_clients: bool): + from zeroconf import ServiceInfo, Zeroconf + + self._zeroconf = Zeroconf() + self._infos = [] + properties = {"filterClients": "1"} if service_name and filter_clients else {} + addresses = [] if local_ip == "127.0.0.1" else [socket.inet_aton(local_ip)] + hostname = socket.gethostname().split(".")[0] + server = f"{hostname}.local." + + for service_type in service_types: + type_domain = f"{service_type}.local." + info = ServiceInfo( + type_domain, + f"{service_name}.{type_domain}", + addresses=addresses, + port=port, + properties=properties, + server=server, + ) + self._zeroconf.register_service(info) + self._infos.append(info) + + def close(self): + for info in self._infos: + try: + self._zeroconf.unregister_service(info) + except Exception: + pass + self._zeroconf.close() + + +class _DnsSdBonjourPublisher: + """Fallback Bonjour publisher using macOS dns-sd.""" + + def __init__( + self, + service_name: str, + service_types: tuple[str, ...], + port: int, + filter_clients: bool, + on_debug: Callable[[str], None], + ): + self._procs = [] + txt_args = _dns_sd_txt_args(service_name, filter_clients) + for service_type in service_types: + command = ["dns-sd", "-R", service_name, service_type, "local", str(port), *txt_args] + proc = subprocess.Popen( + command, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + bufsize=1, + ) + self._procs.append(proc) + on_debug(f"Started dns-sd publisher pid={proc.pid} command={' '.join(command)}") + threading.Thread( + target=self._drain_output, + args=(proc, service_type, on_debug), + daemon=True, + ).start() + + @staticmethod + def _drain_output(proc: subprocess.Popen, service_type: str, on_debug: Callable[[str], None]): + if proc.stdout is None: + return + try: + for line in proc.stdout: + line = line.strip() + if line: + on_debug(f"dns-sd[{service_type}] {line}") + finally: + code = proc.poll() + if code is not None: + on_debug(f"dns-sd[{service_type}] exited with code {code}") + + def close(self): + for proc in self._procs: + try: + proc.terminate() + proc.wait(timeout=2.0) + except Exception: + try: + proc.kill() + except Exception: + pass + + +class _NativeBonjourPublisher: + """macOS Bonjour publisher backed by Foundation.NetService, matching NSLogger.app more closely.""" + + def __init__( + self, + service_name: str, + service_types: tuple[str, ...], + port: int, + filter_clients: bool, + on_debug: Callable[[str], None], + ): + helper = _compiled_swift_helper("native_bonjour_publisher", on_debug) + command = [ + helper, + "--name", + service_name, + "--port", + str(port), + "--types", + ",".join(service_types), + ] + if filter_clients: + command.extend(["--txt", "filterClients=1"]) + + self._proc = subprocess.Popen( + command, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + bufsize=1, + env=_swift_helper_env(), + ) + on_debug(f"Started native Bonjour publisher pid={self._proc.pid} command={' '.join(command)}") + threading.Thread( + target=self._drain_output, + args=(self._proc, on_debug), + daemon=True, + ).start() + + @staticmethod + def _drain_output(proc: subprocess.Popen, on_debug: Callable[[str], None]): + if proc.stdout is None: + return + try: + for line in proc.stdout: + line = line.strip() + if line: + on_debug(f"native-bonjour {line}") + finally: + code = proc.poll() + if code is not None: + on_debug(f"native-bonjour exited with code {code}") + + def close(self): + try: + self._proc.terminate() + self._proc.wait(timeout=2.0) + except Exception: + try: + self._proc.kill() + except Exception: + pass + + +class _NativeBonjourListenerProcess: + """macOS NetService listener using NSNetServiceListenForConnections.""" + + def __init__( + self, + service_name: str, + service_type: str, + port: int, + filter_clients: bool, + secure: bool, + p12_path: Optional[str], + p12_password: Optional[str], + on_debug: Callable[[str], None], + ): + helper = _compiled_swift_helper("native_bonjour_listener", on_debug) + command = [ + helper, + "--name", + service_name, + "--port", + str(port), + "--type", + service_type, + ] + if filter_clients: + command.extend(["--txt", "filterClients=1"]) + if secure: + command.append("--secure") + if p12_path: + command.extend(["--p12", p12_path]) + if p12_password: + command.extend(["--p12-pass", p12_password]) + + self._proc = subprocess.Popen( + command, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + bufsize=1, + env=_swift_helper_env(), + start_new_session=True, # don't propagate terminal Ctrl-C SIGINT to this helper + ) + on_debug(f"Started native Bonjour listener pid={self._proc.pid} command={' '.join(command)}") + + @property + def stdout(self): + return self._proc.stdout + + def poll(self): + return self._proc.poll() + + def close(self): + try: + self._proc.terminate() + self._proc.wait(timeout=2.0) + except Exception: + try: + self._proc.kill() + except Exception: + pass + + +class NSLoggerListener: + """Listen on a TCP port for NSLogger client connections.""" + + def __init__( + self, + port: int = 50001, + timeout: Optional[float] = None, + on_message: Optional[Callable[[LogMessage], None]] = None, + on_connect: Optional[Callable[[str, int], None]] = None, + on_disconnect: Optional[Callable[[str, int], None]] = None, + on_bonjour_ready: Optional[Callable[[str, int], None]] = None, + on_parse_error: Optional[Callable[[str, int, bytes, Exception], None]] = None, + on_debug: Optional[Callable[[str], None]] = None, + use_ssl: Optional[bool] = None, + allow_plaintext: Optional[bool] = None, + bonjour: bool = False, + bonjour_name: Optional[str] = None, + filter_clients: Optional[bool] = None, + bonjour_publisher: str = "native", + advertise_host: Optional[str] = None, + ): + self.port = port + self.timeout = timeout + self.on_message = on_message or (lambda m: None) + self.on_connect = on_connect or (lambda h, p: None) + self.on_disconnect = on_disconnect or (lambda h, p: None) + self.on_bonjour_ready = on_bonjour_ready or (lambda name, port: None) + self.on_parse_error = on_parse_error or (lambda h, p, raw, e: None) + self.on_debug = on_debug or (lambda message: None) + # Bonjour mode mirrors NSLogger.app: publish the SSL service by default. + self.use_ssl = bonjour if use_ssl is None else use_ssl + self.allow_plaintext = False if allow_plaintext is None else allow_plaintext + self.bonjour = bonjour + self.bonjour_name = bonjour_name if bonjour_name is not None else "" + self.filter_clients = bool(self.bonjour_name) if filter_clients is None else filter_clients + self.bonjour_publisher = bonjour_publisher + self.advertise_host = advertise_host + self._stop = threading.Event() + self._ssl_ctx: Optional[ssl.SSLContext] = None + self.messages: list[LogMessage] = [] + + def stop(self): + self._stop.set() + + def _handle_client(self, conn: socket.socket, addr: tuple): + host, port = addr[0], addr[1] + saw_first_byte = False + if self._ssl_ctx: + try: + conn.settimeout(10.0) + except OSError: + pass + while not self._stop.is_set(): + mode, initial = _classify_connection(conn) + if mode == "timeout": + self.on_debug(f"Waiting for first TLS/raw byte from {host}:{port}") + continue + break + else: + conn.close() + return + initial_hex = initial.hex(" ") + if mode == "empty": + self.on_debug(f"Ignoring connection closed before NSLogger data from {host}:{port}") + conn.close() + return + if mode == "raw" and self.allow_plaintext: + self.on_debug(f"Raw NSLogger connection from {host}:{port} first_bytes={initial_hex}") + elif mode == "tls": + self.on_debug(f"Starting TLS handshake for {host}:{port} client_hello={initial_hex}") + try: + conn = self._ssl_ctx.wrap_socket(conn, server_side=True) + self.on_debug( + f"TLS handshake completed for {host}:{port}" + f" protocol={conn.version()} cipher={conn.cipher()}" + ) + except (ssl.SSLError, OSError) as exc: + self.on_debug(f"TLS handshake failed for {host}:{port}: {exc!r}") + conn.close() + return + else: + self.on_debug( + f"Expected TLS ClientHello from {host}:{port}, got first_bytes={initial_hex}" + ) + conn.close() + return + else: + self.on_debug(f"Raw NSLogger connection from {host}:{port}") + self.on_connect(host, port) + try: + conn.settimeout(2.0) + while not self._stop.is_set(): + try: + header = b"" + while len(header) < 4: + chunk = conn.recv(4 - len(header)) + if not chunk: + if not saw_first_byte: + self.on_debug(f"No data before disconnect from {host}:{port}") + return + saw_first_byte = True + header += chunk + msg_len = struct.unpack(">I", header)[0] + self.on_debug(f"Frame header from {host}:{port}: len={msg_len} bytes") + if msg_len == 0: + continue + raw = b"" + while len(raw) < msg_len: + chunk = conn.recv(msg_len - len(raw)) + if not chunk: + return + raw += chunk + try: + msg = _parse_message(raw) + self.messages.append(msg) + self.on_message(msg) + except ParseError as exc: + self.on_parse_error(host, port, raw, exc) + except socket.timeout: + if not saw_first_byte: + self.on_debug(f"Waiting for first frame from {host}:{port}") + continue + except OSError as exc: + self.on_debug(f"Socket error from {host}:{port}: {exc}") + return + finally: + conn.close() + self.on_disconnect(host, port) + + def _start_bonjour(self, local_ip: str) -> object: + """Advertise NSLogger services via Bonjour/mDNS.""" + service_types = _bonjour_service_types(self.use_ssl, self.allow_plaintext) + self.on_debug( + "Bonjour service types: " + f"{', '.join(service_types)} filter_clients={int(self.filter_clients)}" + ) + if self.bonjour_publisher == "native" and sys.platform == "darwin": + self.on_debug("Advertising Bonjour with macOS NetService") + publisher = _NativeBonjourPublisher( + self.bonjour_name, + service_types, + self.port, + self.filter_clients, + self.on_debug, + ) + self.on_bonjour_ready(self.bonjour_name, self.port) + return publisher + + if self.bonjour_publisher == "dns-sd" and sys.platform == "darwin": + self.on_debug("Advertising Bonjour with macOS dns-sd") + publisher = _DnsSdBonjourPublisher( + self.bonjour_name, + service_types, + self.port, + self.filter_clients, + self.on_debug, + ) + self.on_bonjour_ready(self.bonjour_name, self.port) + return publisher + + try: + self.on_debug(f"Advertising Bonjour with zeroconf address={local_ip}") + publisher = _ZeroconfBonjourPublisher( + self.bonjour_name, + service_types, + self.port, + local_ip, + self.filter_clients, + ) + except ImportError: + self.on_debug("zeroconf package unavailable; falling back to macOS dns-sd") + publisher = _DnsSdBonjourPublisher( + self.bonjour_name, + service_types, + self.port, + self.filter_clients, + self.on_debug, + ) + self.on_bonjour_ready(self.bonjour_name, self.port) + return publisher + + def _should_use_native_bonjour_listener(self) -> bool: + if not (self.bonjour and self.bonjour_publisher == "native" and sys.platform == "darwin"): + return False + # NSNetServiceListenForConnections owns the listening socket, so it can only + # publish one service type on the port. The explicit "auto" mode still uses + # Python's socket listener plus a publisher so it can advertise raw+SSL. + return len(_bonjour_service_types(self.use_ssl, self.allow_plaintext)) == 1 + + def _listen_native_bonjour(self): + """Use macOS NetService as both Bonjour publisher and listener.""" + import select + import shutil + import time + + p12_path = None + tmp_dir = None + if self.use_ssl: + p12_path, p12_password, tmp_dir = _make_pkcs12_identity() + else: + p12_password = None + + service_type = _bonjour_service_types(self.use_ssl, self.allow_plaintext)[0] + listener = _NativeBonjourListenerProcess( + self.bonjour_name, + service_type, + self.port, + self.filter_clients, + self.use_ssl, + p12_path, + p12_password, + self.on_debug, + ) + + deadline = None + if self.timeout is not None: + deadline = time.monotonic() + self.timeout + + def _drain(stdout, timeout_s: float = 1.0): + """Read any frames already buffered in the pipe before closing.""" + drain_deadline = time.monotonic() + timeout_s + while time.monotonic() < drain_deadline: + try: + readable, _, _ = select.select([stdout], [], [], 0.1) + except (OSError, ValueError): + break + if not readable: + break + line = stdout.readline() + if not line: + break + self._handle_native_bonjour_event(line) + + try: + stdout = listener.stdout + while not self._stop.is_set(): + if deadline and time.monotonic() > deadline: + break + if stdout is None: + break + readable, _, _ = select.select([stdout], [], [], 0.2) + if not readable: + code = listener.poll() + if code is not None: + self.on_debug(f"native-bonjour listener exited with code {code} (no pending output)") + break + continue + line = stdout.readline() + if not line: + code = listener.poll() + if code is not None: + self.on_debug(f"native-bonjour listener exited with code {code}") + break + continue + self._handle_native_bonjour_event(line) + except KeyboardInterrupt: + self._stop.set() + if stdout is not None: + _drain(stdout, timeout_s=1.0) + raise + finally: + listener.close() + if tmp_dir: + shutil.rmtree(tmp_dir, ignore_errors=True) + + return self.messages + + def _handle_native_bonjour_event(self, line: str): + try: + event = json.loads(line) + except json.JSONDecodeError: + self.on_debug(f"native-bonjour {line.strip()}") + return + + event_type = event.get("event") + if event_type == "ready": + self.port = int(event.get("port") or self.port) + self.on_bonjour_ready(event.get("name", self.bonjour_name), self.port) + elif event_type == "debug": + self.on_debug(f"native-bonjour {event.get('message', '')}") + elif event_type == "connect": + self.on_connect("native-bonjour", 0) + elif event_type == "disconnect": + self.on_disconnect("native-bonjour", 0) + elif event_type == "error": + details = " ".join( + str(event.get(key, "")) + for key in ("message", "error", "status") + if event.get(key, "") != "" + ) + self.on_debug(f"native-bonjour error: {details}") + elif event_type == "frame": + try: + raw = base64.b64decode(event.get("payload", ""), validate=True) + self.on_debug( + f"Frame #{len(self.messages) + 1}: {len(raw)} bytes" + f" head={raw[:8].hex(' ')}" + ) + msg = _parse_message(raw) + self.messages.append(msg) + self.on_message(msg) + except (ValueError, ParseError) as exc: + raw_bytes = base64.b64decode(event.get("payload", "") or "", validate=False) + self.on_parse_error("native-bonjour", 0, raw_bytes, exc) + + def listen(self): + """Block until timeout or stop() is called. Returns collected messages.""" + if self._should_use_native_bonjour_listener(): + return self._listen_native_bonjour() + + tmp_dir = None + if self.use_ssl: + self._ssl_ctx, tmp_dir = _make_ssl_context() + + server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + server.bind(("0.0.0.0", self.port)) + server.listen(5) + server.settimeout(1.0) + + publisher = None + if self.bonjour: + local_ip = self.advertise_host or _get_local_ip() + publisher = self._start_bonjour(local_ip) + + deadline = None + if self.timeout is not None: + import time + deadline = time.monotonic() + self.timeout + + threads = [] + try: + import time + while not self._stop.is_set(): + if deadline and time.monotonic() > deadline: + break + try: + conn, addr = server.accept() + t = threading.Thread( + target=self._handle_client, + args=(conn, addr), + daemon=True, + ) + t.start() + threads.append(t) + except socket.timeout: + continue + finally: + server.close() + self._stop.set() + for t in threads: + t.join(timeout=2.0) + if publisher: + publisher.close() + if tmp_dir: + import shutil + shutil.rmtree(tmp_dir, ignore_errors=True) + + return self.messages diff --git a/nslogger/agent-harness/cli_anything/nslogger/core/message.py b/nslogger/agent-harness/cli_anything/nslogger/core/message.py new file mode 100644 index 000000000..6d25c10a9 --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/core/message.py @@ -0,0 +1,135 @@ +"""NSLogger message model and constants.""" +from __future__ import annotations +import dataclasses +from datetime import datetime +from typing import Optional + +# Message type constants (wire protocol) +MSG_TYPE_LOG = 0 +MSG_TYPE_BLOCK_START = 1 +MSG_TYPE_BLOCK_END = 2 +MSG_TYPE_CLIENT_INFO = 3 +MSG_TYPE_DISCONNECT = 4 +MSG_TYPE_MARKER = 255 + +# Part key constants from the official NSLogger wire protocol. +PART_KEY_MESSAGE_TYPE = 0 +PART_KEY_TIMESTAMP_S = 1 +PART_KEY_TIMESTAMP_MS = 2 +PART_KEY_TIMESTAMP_US = 3 +PART_KEY_THREAD_ID = 4 +PART_KEY_TAG = 5 +PART_KEY_LEVEL = 6 +PART_KEY_MESSAGE = 7 +PART_KEY_IMAGE_WIDTH = 8 +PART_KEY_IMAGE_HEIGHT = 9 +PART_KEY_MESSAGE_SEQ = 10 +PART_KEY_FILENAME = 11 +PART_KEY_LINENUMBER = 12 +PART_KEY_FUNCTIONNAME = 13 +PART_KEY_CLIENT_NAME = 20 +PART_KEY_CLIENT_VERSION = 21 +PART_KEY_OS_NAME = 22 +PART_KEY_OS_VERSION = 23 +PART_KEY_CLIENT_MODEL = 24 +PART_KEY_UNIQUEID = 25 + +# Part type constants +PART_TYPE_STRING = 0 +PART_TYPE_BINARY = 1 +PART_TYPE_INT16 = 2 +PART_TYPE_INT32 = 3 +PART_TYPE_INT64 = 4 +PART_TYPE_IMAGE = 5 + +LEVEL_NAMES = { + 0: "ERROR", + 1: "WARNING", + 2: "INFO", + 3: "DEBUG", + 4: "VERBOSE", + 5: "NOISE", +} + + +@dataclasses.dataclass +class LogMessage: + sequence: int = 0 + timestamp: Optional[datetime] = None + timestamp_ms: int = 0 + thread_id: str = "" + tag: str = "" + level: int = 2 + message_type: int = MSG_TYPE_LOG + text: str = "" + image_width: int = 0 + image_height: int = 0 + image_data: Optional[bytes] = None + binary_data: Optional[bytes] = None + client_name: str = "" + client_version: str = "" + os_name: str = "" + os_version: str = "" + machine: str = "" + + @property + def level_name(self) -> str: + return LEVEL_NAMES.get(self.level, f"LEVEL{self.level}") + + @property + def type_name(self) -> str: + if self.message_type == MSG_TYPE_LOG: + if self.image_data: + return "image" + if self.binary_data: + return "data" + return "text" + type_map = { + MSG_TYPE_BLOCK_START: "block_start", + MSG_TYPE_BLOCK_END: "block_end", + MSG_TYPE_CLIENT_INFO: "client_info", + MSG_TYPE_DISCONNECT: "disconnect", + MSG_TYPE_MARKER: "marker", + } + return type_map.get(self.message_type, f"type_{self.message_type}") + + def to_dict(self) -> dict: + ts = self.timestamp.isoformat() if self.timestamp else None + return { + "sequence": self.sequence, + "timestamp": ts, + "timestamp_ms": self.timestamp_ms, + "thread_id": self.thread_id, + "tag": self.tag, + "level": self.level, + "level_name": self.level_name, + "type": self.type_name, + "text": self.text, + "image_width": self.image_width, + "image_height": self.image_height, + "client_name": self.client_name, + "client_version": self.client_version, + "os_name": self.os_name, + "os_version": self.os_version, + "machine": self.machine, + } + + def to_text_line(self) -> str: + ts = self.timestamp.strftime("%H:%M:%S.") + f"{self.timestamp_ms:03d}" if self.timestamp else "??:??:??.???" + tag_part = f"[{self.tag}] " if self.tag else "" + thread_part = f"({self.thread_id}) " if self.thread_id else "" + level_part = f"{self.level_name:<7} " + if self.type_name == "image": + content = f"" + elif self.type_name == "data": + size = len(self.binary_data) if self.binary_data else 0 + content = f"" + elif self.message_type == MSG_TYPE_CLIENT_INFO: + content = f"CLIENT: {self.client_name} {self.client_version} on {self.os_name} {self.os_version} ({self.machine})" + elif self.message_type == MSG_TYPE_BLOCK_START: + content = f">>> {self.text}" + elif self.message_type == MSG_TYPE_BLOCK_END: + content = f"<<< {self.text}" + else: + content = self.text + return f"{ts} {level_part}{thread_part}{tag_part}{content}" diff --git a/nslogger/agent-harness/cli_anything/nslogger/core/parser.py b/nslogger/agent-harness/cli_anything/nslogger/core/parser.py new file mode 100644 index 000000000..99c98cfc4 --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/core/parser.py @@ -0,0 +1,258 @@ +"""Parse NSLogger raw wire-protocol (.rawnsloggerdata) files.""" +from __future__ import annotations +import struct +from datetime import datetime, timezone +from typing import Iterator, BinaryIO +from .message import ( + LogMessage, + PART_KEY_MESSAGE_TYPE, PART_KEY_TIMESTAMP_S, PART_KEY_TIMESTAMP_MS, + PART_KEY_TIMESTAMP_US, PART_KEY_THREAD_ID, PART_KEY_TAG, PART_KEY_LEVEL, PART_KEY_MESSAGE, + PART_KEY_IMAGE_WIDTH, PART_KEY_IMAGE_HEIGHT, PART_KEY_MESSAGE_SEQ, + PART_KEY_CLIENT_NAME, PART_KEY_CLIENT_VERSION, + PART_KEY_OS_NAME, PART_KEY_OS_VERSION, PART_KEY_CLIENT_MODEL, + PART_TYPE_STRING, PART_TYPE_BINARY, PART_TYPE_INT16, + PART_TYPE_INT32, PART_TYPE_INT64, PART_TYPE_IMAGE, + MSG_TYPE_LOG, +) + + +class ParseError(Exception): + pass + + +def _read_exactly(f: BinaryIO, n: int) -> bytes: + data = f.read(n) + if len(data) != n: + raise ParseError(f"Expected {n} bytes, got {len(data)}") + return data + + +def _decode_part_value(part_type: int, data: bytes): + if part_type == PART_TYPE_STRING: + return data.decode("utf-8", errors="replace") + if part_type == PART_TYPE_INT16: + if len(data) != 2: + raise ParseError(f"Expected 2 bytes for int16, got {len(data)}") + return struct.unpack(">H", data)[0] + if part_type == PART_TYPE_INT32: + if len(data) != 4: + raise ParseError(f"Expected 4 bytes for int32, got {len(data)}") + return struct.unpack(">I", data)[0] + if part_type == PART_TYPE_INT64: + if len(data) != 8: + raise ParseError(f"Expected 8 bytes for int64, got {len(data)}") + return struct.unpack(">Q", data)[0] + if part_type in (PART_TYPE_BINARY, PART_TYPE_IMAGE): + return data + return data + + +def _part_data(raw: bytes, offset: int, part_type: int, *, implicit_int_sizes: bool) -> tuple[bytes, int]: + if implicit_int_sizes and part_type in (PART_TYPE_INT16, PART_TYPE_INT32, PART_TYPE_INT64): + part_len = {PART_TYPE_INT16: 2, PART_TYPE_INT32: 4, PART_TYPE_INT64: 8}[part_type] + if offset + part_len > len(raw): + raise ParseError("Truncated integer part") + return raw[offset:offset + part_len], offset + part_len + + if offset + 4 > len(raw): + raise ParseError("Truncated variable-length part") + part_len = struct.unpack(">I", raw[offset:offset + 4])[0] + offset += 4 + if offset + part_len > len(raw): + raise ParseError("Truncated part data") + return raw[offset:offset + part_len], offset + part_len + + +MESSAGE_TEXT_KEYS = {PART_KEY_MESSAGE, 6} +IMAGE_WIDTH_KEYS = {PART_KEY_IMAGE_WIDTH, 7} +IMAGE_HEIGHT_KEYS = {PART_KEY_IMAGE_HEIGHT, 8} +CLIENT_NAME_KEYS = {PART_KEY_CLIENT_NAME, 9} +CLIENT_VERSION_KEYS = {PART_KEY_CLIENT_VERSION, 10} +OS_NAME_KEYS = {PART_KEY_OS_NAME, 11} +OS_VERSION_KEYS = {PART_KEY_OS_VERSION, 12} +CLIENT_MODEL_KEYS = {PART_KEY_CLIENT_MODEL, 13} + + +def _is_int_value(value) -> bool: + return isinstance(value, int) + + +def _is_text_value(part_type: int) -> bool: + return part_type in (PART_TYPE_STRING, PART_TYPE_BINARY, PART_TYPE_IMAGE) + + +def _parse_message_payload( + raw: bytes, + offset: int, + *, + initial_sequence: int = 0, + implicit_int_sizes: bool = True, +) -> LogMessage: + """Parse a message payload starting at `offset`, where the payload begins with part_count.""" + if len(raw) - offset < 2: + raise ParseError("Message too short") + + part_count = struct.unpack(">H", raw[offset:offset + 2])[0] + msg = LogMessage(sequence=initial_sequence, message_type=MSG_TYPE_LOG) + ts_s = None + ts_ms = 0 + offset += 2 + + for _ in range(part_count): + if offset + 2 > len(raw): + raise ParseError("Truncated part header") + part_key = raw[offset] + part_type = raw[offset + 1] + offset += 2 + part_data, offset = _part_data(raw, offset, part_type, implicit_int_sizes=implicit_int_sizes) + + value = _decode_part_value(part_type, part_data) + + if part_key == PART_KEY_MESSAGE_TYPE: + msg.message_type = value if isinstance(value, int) else int.from_bytes(part_data, "big") + elif part_key == PART_KEY_TIMESTAMP_S: + ts_s = value + elif part_key == PART_KEY_TIMESTAMP_MS: + ts_ms = value if isinstance(value, int) else 0 + elif part_key == PART_KEY_TIMESTAMP_US: + ts_ms = (value // 1000) if isinstance(value, int) else 0 + elif part_key == PART_KEY_THREAD_ID: + msg.thread_id = str(value) if not isinstance(value, str) else value + elif part_key == PART_KEY_TAG: + msg.tag = str(value) if not isinstance(value, str) else value + elif part_key == PART_KEY_LEVEL: + msg.level = value if isinstance(value, int) else 0 + elif part_key in MESSAGE_TEXT_KEYS and _is_text_value(part_type): + if isinstance(value, bytes) and part_type in (PART_TYPE_BINARY, PART_TYPE_IMAGE): + msg.image_data = value + else: + msg.text = str(value) + elif part_key in IMAGE_WIDTH_KEYS and _is_int_value(value): + msg.image_width = value + elif part_key in IMAGE_HEIGHT_KEYS and _is_int_value(value): + msg.image_height = value + elif part_key == PART_KEY_MESSAGE_SEQ and _is_int_value(value): + msg.sequence = value + elif part_key in CLIENT_NAME_KEYS and part_type == PART_TYPE_STRING: + msg.client_name = str(value) + elif part_key in CLIENT_VERSION_KEYS and part_type == PART_TYPE_STRING: + msg.client_version = str(value) + elif part_key in OS_NAME_KEYS and part_type == PART_TYPE_STRING: + msg.os_name = str(value) + elif part_key in OS_VERSION_KEYS and part_type == PART_TYPE_STRING: + msg.os_version = str(value) + elif part_key in CLIENT_MODEL_KEYS and part_type == PART_TYPE_STRING: + msg.machine = str(value) + + if ts_s is not None: + ts_s_int = ts_s if isinstance(ts_s, int) else int(ts_s) + msg.timestamp = datetime.fromtimestamp(ts_s_int, tz=timezone.utc) + msg.timestamp_ms = ts_ms if isinstance(ts_ms, int) else 0 + + # Detect binary data vs image + if msg.image_data and not (msg.image_width or msg.image_height): + msg.binary_data = msg.image_data + msg.image_data = None + + return msg + + +def _parse_message(raw: bytes) -> LogMessage: + """Parse a single wire-protocol message from raw bytes. + + NSLogger's native wire format is: + [partCount][parts...] + + Older local test fixtures in this repo used: + [sequence][partCount][parts...] + We keep a best-effort fallback for those fixtures. + """ + if len(raw) < 2: + raise ParseError("Message too short") + + # Native NSLogger format: payload begins with partCount. + try: + msg = _parse_message_payload(raw, 0, implicit_int_sizes=True) + if msg.message_type != MSG_TYPE_LOG or msg.text or msg.client_name or msg.image_data or msg.binary_data: + return msg + except ParseError: + pass + + # Older generated fixtures used official part keys but still wrote a + # redundant 4-byte size before integer values. + try: + msg = _parse_message_payload(raw, 0, implicit_int_sizes=False) + if msg.message_type != MSG_TYPE_LOG or msg.text or msg.client_name or msg.image_data or msg.binary_data: + return msg + except ParseError: + pass + + # Backward-compatible fallback for historical local fixtures. + if len(raw) < 6: + raise ParseError("Message too short") + seq = struct.unpack(">I", raw[0:4])[0] + try: + return _parse_message_payload(raw, 4, initial_sequence=seq, implicit_int_sizes=False) + except ParseError: + return _parse_message_payload(raw, 4, initial_sequence=seq, implicit_int_sizes=True) + + +def parse_raw_file(path: str) -> Iterator[LogMessage]: + """Yield LogMessage objects from a .rawnsloggerdata file.""" + with open(path, "rb") as f: + while True: + header = f.read(4) + if not header: + break + if len(header) < 4: + raise ParseError(f"Truncated file: got {len(header)} bytes in length header") + msg_len = struct.unpack(">I", header)[0] + if msg_len == 0: + continue + raw = f.read(msg_len) + if len(raw) < msg_len: + break + try: + yield _parse_message(raw) + except ParseError: + continue + + +def parse_file(path: str) -> Iterator[LogMessage]: + """Auto-detect format and parse .rawnsloggerdata or .nsloggerdata files.""" + if path.endswith(".rawnsloggerdata"): + yield from parse_raw_file(path) + else: + # .nsloggerdata is a binary plist wrapping archived messages + # Fall back to raw parser which handles both (raw starts with length) + yield from _parse_nsloggerdata(path) + + +def _parse_nsloggerdata(path: str) -> Iterator[LogMessage]: + """Parse .nsloggerdata binary plist files via Python plistlib.""" + import plistlib + try: + with open(path, "rb") as f: + data = plistlib.load(f) + # NSLogger saves as a plist dict with 'messages' array + messages = data if isinstance(data, list) else data.get("messages", []) + for i, m in enumerate(messages): + if not isinstance(m, dict): + continue + msg = LogMessage(sequence=i) + ts = m.get("timestamp") + if ts is not None: + try: + msg.timestamp = datetime.fromtimestamp(float(ts), tz=timezone.utc) + frac = float(ts) - int(float(ts)) + msg.timestamp_ms = int(frac * 1000) + except (TypeError, ValueError): + pass + msg.tag = str(m.get("tag", "")) + msg.level = int(m.get("level", 2)) + msg.thread_id = str(m.get("threadID", "")) + msg.text = str(m.get("message", m.get("messageText", ""))) + yield msg + except Exception: + # Try raw protocol as fallback + yield from parse_raw_file(path) diff --git a/nslogger/agent-harness/cli_anything/nslogger/core/stats.py b/nslogger/agent-harness/cli_anything/nslogger/core/stats.py new file mode 100644 index 000000000..8a03b6f5c --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/core/stats.py @@ -0,0 +1,54 @@ +"""Compute statistics over a sequence of LogMessages.""" +from __future__ import annotations +from collections import Counter +from typing import Iterator, Dict, Any +from .message import LogMessage, LEVEL_NAMES + + +def compute_stats(messages: Iterator[LogMessage]) -> Dict[str, Any]: + msgs = list(messages) + if not msgs: + return {"total": 0} + + level_counts: Counter = Counter() + tag_counts: Counter = Counter() + thread_counts: Counter = Counter() + type_counts: Counter = Counter() + client_names: set = set() + + first_ts = None + last_ts = None + + for m in msgs: + level_counts[m.level] += 1 + if m.tag: + tag_counts[m.tag] += 1 + if m.thread_id: + thread_counts[m.thread_id] += 1 + type_counts[m.type_name] += 1 + if m.client_name: + client_names.add(m.client_name) + if m.timestamp: + if first_ts is None or m.timestamp < first_ts: + first_ts = m.timestamp + if last_ts is None or m.timestamp > last_ts: + last_ts = m.timestamp + + duration_s = None + if first_ts and last_ts: + duration_s = (last_ts - first_ts).total_seconds() + + return { + "total": len(msgs), + "by_level": { + LEVEL_NAMES.get(k, f"level_{k}"): v + for k, v in sorted(level_counts.items()) + }, + "by_tag": dict(tag_counts.most_common(20)), + "by_thread": dict(thread_counts.most_common(10)), + "by_type": dict(type_counts), + "clients": sorted(client_names), + "first_timestamp": first_ts.isoformat() if first_ts else None, + "last_timestamp": last_ts.isoformat() if last_ts else None, + "duration_seconds": duration_s, + } diff --git a/nslogger/agent-harness/cli_anything/nslogger/helpers/native_bonjour_listener.swift b/nslogger/agent-harness/cli_anything/nslogger/helpers/native_bonjour_listener.swift new file mode 100644 index 000000000..e12686a03 --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/helpers/native_bonjour_listener.swift @@ -0,0 +1,270 @@ +import Darwin +import Foundation +import Security + +func emit(_ fields: [String: Any]) { + if let data = try? JSONSerialization.data(withJSONObject: fields, options: []), + let line = String(data: data, encoding: .utf8) { + print(line) + fflush(stdout) + } +} + +struct Arguments { + var name = "" + var port: Int32 = 50000 + var type = "_nslogger-ssl._tcp." + var txt: [String: Data] = [:] + var secure = false + var p12Path: String? + var p12Password = "" +} + +func normalizeServiceType(_ type: String) -> String { + if type.isEmpty { + return type + } + return type.hasSuffix(".") ? type : "\(type)." +} + +func parseArguments(_ args: [String]) -> Arguments { + var parsed = Arguments() + var index = 1 + while index < args.count { + let arg = args[index] + switch arg { + case "--name": + index += 1 + if index < args.count { + parsed.name = args[index] + } + case "--port": + index += 1 + if index < args.count, let port = Int32(args[index]) { + parsed.port = port + } + case "--type": + index += 1 + if index < args.count { + parsed.type = normalizeServiceType(args[index]) + } + case "--txt": + index += 1 + if index < args.count { + let parts = args[index].split(separator: "=", maxSplits: 1).map(String.init) + let key = parts.first ?? "" + let value = parts.count > 1 ? parts[1] : "" + if !key.isEmpty { + parsed.txt[key] = value.data(using: .utf8) ?? Data() + } + } + case "--secure": + parsed.secure = true + case "--p12": + index += 1 + if index < args.count { + parsed.p12Path = args[index] + } + case "--p12-pass": + index += 1 + if index < args.count { + parsed.p12Password = args[index] + } + default: + break + } + index += 1 + } + return parsed +} + +func loadIdentity(path: String?, password: String) -> SecIdentity? { + guard let path else { + return nil + } + guard let data = try? Data(contentsOf: URL(fileURLWithPath: path)) else { + emit(["event": "error", "message": "failed to read pkcs12 identity"]) + return nil + } + let options = [kSecImportExportPassphrase as String: password] + var items: CFArray? + let status = SecPKCS12Import(data as CFData, options as CFDictionary, &items) + guard status == errSecSuccess, + let imported = items as? [[String: Any]], + let identityValue = imported.first?[kSecImportItemIdentity as String] else { + emit(["event": "error", "message": "failed to import pkcs12 identity", "status": Int(status)]) + return nil + } + return (identityValue as! SecIdentity) +} + +final class Connection: NSObject, StreamDelegate { + private let input: InputStream + private let output: OutputStream + private let secure: Bool + private let identity: SecIdentity? + private var buffer = Data() + private var connected = false + + init(input: InputStream, output: OutputStream, secure: Bool, identity: SecIdentity?) { + self.input = input + self.output = output + self.secure = secure + self.identity = identity + } + + func open() { + if secure { + guard let identity else { + emit(["event": "error", "message": "missing ssl identity"]) + return + } + let settings: [String: Any] = [ + kCFStreamSSLLevel as String: kCFStreamSocketSecurityLevelNegotiatedSSL, + kCFStreamSSLValidatesCertificateChain as String: false, + kCFStreamSSLIsServer as String: true, + kCFStreamSSLCertificates as String: [identity], + ] + input.setProperty(settings, forKey: Stream.PropertyKey(rawValue: kCFStreamPropertySSLSettings as String)) + } + + input.delegate = self + output.delegate = self + input.schedule(in: .current, forMode: .default) + output.schedule(in: .current, forMode: .default) + input.open() + output.open() + } + + func close() { + input.close() + output.close() + input.remove(from: .current, forMode: .default) + output.remove(from: .current, forMode: .default) + } + + func stream(_ aStream: Stream, handle eventCode: Stream.Event) { + switch eventCode { + case .openCompleted: + if !connected { + connected = true + emit(["event": "connect"]) + } + case .hasBytesAvailable: + readAvailableBytes() + case .endEncountered: + emit(["event": "disconnect"]) + close() + case .errorOccurred: + let message = aStream.streamError?.localizedDescription ?? "stream error" + emit(["event": "error", "message": message]) + close() + default: + break + } + } + + private func readAvailableBytes() { + var chunk = [UInt8](repeating: 0, count: 64 * 1024) + while input.hasBytesAvailable { + let count = input.read(&chunk, maxLength: chunk.count) + if count <= 0 { + break + } + buffer.append(chunk, count: count) + processFrames() + } + } + + private func processFrames() { + while buffer.count >= 4 { + let headerStart = buffer.startIndex + let headerEnd = buffer.index(headerStart, offsetBy: 4) + let length = buffer[headerStart.. Arguments { + var parsed = Arguments() + var index = 1 + while index < args.count { + let arg = args[index] + switch arg { + case "--name": + index += 1 + if index < args.count { + parsed.name = args[index] + } + case "--port": + index += 1 + if index < args.count, let port = Int32(args[index]) { + parsed.port = port + } + case "--types": + index += 1 + if index < args.count { + parsed.types = args[index] + .split(separator: ",") + .map { normalizeServiceType(String($0)) } + .filter { !$0.isEmpty } + } + case "--txt": + index += 1 + if index < args.count { + let parts = args[index].split(separator: "=", maxSplits: 1).map(String.init) + let key = parts.first ?? "" + let value = parts.count > 1 ? parts[1] : "" + if !key.isEmpty { + parsed.txt[key] = value.data(using: .utf8) ?? Data() + } + } + default: + break + } + index += 1 + } + return parsed +} + +func normalizeServiceType(_ type: String) -> String { + if type.isEmpty { + return type + } + return type.hasSuffix(".") ? type : "\(type)." +} + +signal(SIGTERM) { _ in + exit(0) +} +signal(SIGINT) { _ in + exit(0) +} + +setvbuf(stdout, nil, _IONBF, 0) + +let arguments = parseArguments(CommandLine.arguments) +let delegate = PublisherDelegate() +var services: [NetService] = [] + +for type in arguments.types { + let service = NetService(domain: "", type: type, name: arguments.name, port: arguments.port) + service.includesPeerToPeer = true + service.delegate = delegate + if !arguments.txt.isEmpty { + service.setTXTRecord(NetService.data(fromTXTRecord: arguments.txt)) + } + service.publish() + services.append(service) +} + +if services.isEmpty { + print("publish-failed error=no-service-types") + exit(2) +} + +RunLoop.main.run() diff --git a/nslogger/agent-harness/cli_anything/nslogger/nslogger_cli.py b/nslogger/agent-harness/cli_anything/nslogger/nslogger_cli.py new file mode 100644 index 000000000..468624d67 --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/nslogger_cli.py @@ -0,0 +1,555 @@ +"""cli-anything-nslogger โ€” CLI harness for NSLogger.""" +from __future__ import annotations +import json +import sys +import os +from datetime import datetime, timezone +from typing import Optional + +import click + +from .core.parser import parse_file +from .core.filter import filter_messages +from .core.stats import compute_stats +from .core.exporter import export_messages +from .core.message import LEVEL_NAMES, MSG_TYPE_CLIENT_INFO + + +CONTEXT_SETTINGS = {"help_option_names": ["-h", "--help"]} + + +def _level_option(): + return click.option( + "--level", "-l", + type=int, default=None, + help="Maximum log level to show (0=error โ€ฆ 4=verbose)", + ) + + +def _json_option(): + return click.option("--json", "as_json", is_flag=True, help="Output as JSON") + + +def _parse_dt(value: Optional[str]) -> Optional[datetime]: + """Parse ISO-like datetime string to aware UTC datetime.""" + if value is None: + return None + for fmt in ("%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M:%S", "%H:%M:%S"): + try: + dt = datetime.strptime(value, fmt) + if dt.year == 1900: + today = datetime.now(tz=timezone.utc).date() + dt = dt.replace(year=today.year, month=today.month, day=today.day) + return dt.replace(tzinfo=timezone.utc) + except ValueError: + continue + raise click.BadParameter(f"Cannot parse datetime: {value!r}. Use HH:MM:SS or YYYY-MM-DDTHH:MM:SS") + + +def _time_range_options(): + def decorator(f): + f = click.option("--after", default=None, + help="Show messages after this time (HH:MM:SS or YYYY-MM-DDTHH:MM:SS)")(f) + f = click.option("--before", default=None, + help="Show messages before this time (HH:MM:SS or YYYY-MM-DDTHH:MM:SS)")(f) + return f + return decorator + + +def _listen_waiting_message(port: int, bonjour: bool) -> str: + if bonjour: + return f"[Bonjour] Waiting for an iOS client to connect on port {port}โ€ฆ" + return f"Waiting for a client connection on port {port}โ€ฆ" + + +def _format_live_output_message(msg, fmt: str) -> str: + if fmt == "jsonl": + return json.dumps(msg.to_dict(), default=str) + return msg.to_text_line() + + +def _open_live_output_file(path: str, append: bool): + parent = os.path.dirname(os.path.abspath(path)) + if parent: + os.makedirs(parent, exist_ok=True) + mode = "a" if append else "w" + return open(path, mode, encoding="utf-8", buffering=1) + + +@click.group(context_settings=CONTEXT_SETTINGS) +@click.version_option(package_name="cli-anything-nslogger") +def cli(): + """NSLogger CLI โ€” read, filter, export, and monitor NSLogger log files. + + \b + Use COMMAND -h to see command-specific options, for example: + cli-anything-nslogger listen -h + + \b + Live logs can be mirrored to a file with: + cli-anything-nslogger listen --bonjour --name bazinga --output app.log + + \b + Live listen file output options: + -o, --output FILE Write live logs to FILE while printing stdout + --output-format text|jsonl Write text lines or JSON Lines + --append Append instead of replacing FILE on startup + """ + + +# --------------------------------------------------------------------------- +# read +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("file", type=click.Path(exists=True)) +@_level_option() +@click.option("--tag", "-t", multiple=True, help="Filter by tag (repeatable)") +@click.option("--thread", help="Filter by thread ID") +@click.option("--search", "-s", help="Text search (case-insensitive)") +@click.option("--limit", "-n", type=int, default=None, help="Max messages to show") +@click.option("--after", default=None, help="Show messages after this time (HH:MM:SS or YYYY-MM-DDTHH:MM:SS)") +@click.option("--before", default=None, help="Show messages before this time (HH:MM:SS or YYYY-MM-DDTHH:MM:SS)") +@_json_option() +def read(file, level, tag, thread, search, limit, after, before, as_json): + """Parse and display messages from a .rawnsloggerdata or .nsloggerdata file.""" + msgs = parse_file(file) + msgs = filter_messages( + msgs, + max_level=level, + tags=list(tag) if tag else None, + thread_id=thread, + text_search=search, + limit=limit, + after=_parse_dt(after), + before=_parse_dt(before), + ) + result = list(msgs) + if as_json: + click.echo(json.dumps([m.to_dict() for m in result], indent=2, default=str)) + else: + for m in result: + click.echo(m.to_text_line()) + + +# --------------------------------------------------------------------------- +# filter +# --------------------------------------------------------------------------- + +@cli.command(name="filter") +@click.argument("file", type=click.Path(exists=True)) +@_level_option() +@click.option("--min-level", type=int, default=None, help="Minimum log level") +@click.option("--tag", "-t", multiple=True, help="Filter by tag") +@click.option("--thread", help="Filter by thread ID") +@click.option("--search", "-s", help="Substring search in message text") +@click.option("--regex", "-r", help="Regex search in message text") +@click.option("--type", "msg_type", multiple=True, + type=click.Choice(["text", "image", "data", "client_info", "block_start", "block_end"]), + help="Filter by message type") +@click.option("--limit", "-n", type=int, default=None, help="Max messages") +@click.option("--after", default=None, help="Show messages after this time (HH:MM:SS or YYYY-MM-DDTHH:MM:SS)") +@click.option("--before", default=None, help="Show messages before this time") +@click.option("--from-seq", type=int, default=None, help="Start from sequence number (inclusive)") +@click.option("--to-seq", type=int, default=None, help="End at sequence number (inclusive)") +@_json_option() +def filter_cmd(file, level, min_level, tag, thread, search, regex, msg_type, limit, + after, before, from_seq, to_seq, as_json): + """Filter messages from a file with advanced criteria.""" + msgs = parse_file(file) + msgs = filter_messages( + msgs, + max_level=level, + min_level=min_level, + tags=list(tag) if tag else None, + thread_id=thread, + text_search=search, + text_regex=regex, + msg_types=list(msg_type) if msg_type else None, + limit=limit, + after=_parse_dt(after), + before=_parse_dt(before), + from_seq=from_seq, + to_seq=to_seq, + ) + result = list(msgs) + if as_json: + click.echo(json.dumps([m.to_dict() for m in result], indent=2, default=str)) + else: + for m in result: + click.echo(m.to_text_line()) + + +# --------------------------------------------------------------------------- +# export +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("file", type=click.Path(exists=True)) +@click.option("--format", "-f", "fmt", + type=click.Choice(["text", "json", "csv"]), default="text", + show_default=True, help="Output format") +@click.option("--output", "-o", type=click.Path(), default=None, + help="Output file (default: stdout)") +@_level_option() +@click.option("--tag", "-t", multiple=True, help="Filter by tag before export") +@click.option("--search", "-s", help="Filter by text before export") +@click.option("--limit", "-n", type=int, default=None, help="Max messages") +def export(file, fmt, output, level, tag, search, limit): + """Export messages to text, JSON, or CSV.""" + msgs = parse_file(file) + msgs = filter_messages( + msgs, + max_level=level, + tags=list(tag) if tag else None, + text_search=search, + limit=limit, + ) + result_str = export_messages(msgs, fmt=fmt) + if output: + with open(output, "w", encoding="utf-8") as f: + f.write(result_str) + click.echo(f"Exported to {output}", err=True) + else: + click.echo(result_str, nl=False) + + +# --------------------------------------------------------------------------- +# stats +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("file", type=click.Path(exists=True)) +@_json_option() +def stats(file, as_json): + """Show statistics for a NSLogger file.""" + msgs = parse_file(file) + s = compute_stats(msgs) + if as_json: + click.echo(json.dumps(s, indent=2, default=str)) + return + + click.echo(f"Total messages : {s['total']}") + if s.get("first_timestamp"): + click.echo(f"First message : {s['first_timestamp']}") + click.echo(f"Last message : {s['last_timestamp']}") + if s.get("duration_seconds") is not None: + click.echo(f"Duration : {s['duration_seconds']:.1f}s") + if s.get("clients"): + click.echo(f"Clients : {', '.join(s['clients'])}") + click.echo("") + click.echo("By level:") + for name, count in s.get("by_level", {}).items(): + click.echo(f" {name:<10} {count}") + click.echo("") + click.echo("By type:") + for name, count in s.get("by_type", {}).items(): + click.echo(f" {name:<15} {count}") + if s.get("by_tag"): + click.echo("") + click.echo("Top tags:") + for tag, count in list(s["by_tag"].items())[:10]: + click.echo(f" {tag:<20} {count}") + if s.get("by_thread"): + click.echo("") + click.echo("Top threads:") + for thread, count in list(s["by_thread"].items())[:5]: + click.echo(f" {thread:<25} {count}") + + +# --------------------------------------------------------------------------- +# listen +# --------------------------------------------------------------------------- + +@cli.command(short_help="Listen for live logs; use --output FILE to mirror them to disk.") +@click.option("--port", "-p", type=int, default=50000, show_default=True, + help="TCP port to listen on") +@click.option("--timeout", "-t", type=float, default=None, + help="Stop after N seconds (default: run until Ctrl-C)") +@click.option("--level", "-l", type=int, default=None, + help="Maximum level to display while listening") +@click.option("--bonjour", "-b", is_flag=True, default=False, + help="Advertise via Bonjour/mDNS (iOS app auto-discovers, no IP config needed)") +@click.option("--name", "-n", default=None, + help="Bonjour service name (default: system-selected name)") +@click.option("--ssl", "force_ssl", is_flag=True, + help="Use SSL/TLS for direct TCP mode; Bonjour uses SSL by default") +@click.option("--no-ssl", is_flag=True, help="Advertise/use the legacy non-SSL NSLogger Bonjour service") +@click.option("--bonjour-mode", type=click.Choice(["auto", "ssl", "raw"]), default="ssl", show_default=True, + help="Bonjour service mode: ssl matches NSLogger GUI default, auto publishes raw+SSL, raw publishes legacy raw only") +@click.option("--bonjour-publisher", type=click.Choice(["native", "dns-sd", "zeroconf"]), default="native", show_default=True, + help="Bonjour publisher backend") +@click.option("--advertise-host", default=None, + help="IP address to publish when using --bonjour-publisher zeroconf") +@click.option("--filter-clients/--no-filter-clients", default=None, + help="Advertise filterClients=1; defaults to on when --name is non-empty, matching NSLogger GUI") +@click.option("--output", "-o", type=click.Path(dir_okay=False, path_type=str), + help="Write received live logs to this file while still printing to stdout") +@click.option("--output-format", type=click.Choice(["text", "jsonl"]), default="text", show_default=True, + help="Format used for --output") +@click.option("--append", is_flag=True, + help="Append to --output instead of replacing it at listener startup") +@click.option("--debug", is_flag=True, help="Print live frame diagnostics to stderr") +@_json_option() +def listen( + port, timeout, level, bonjour, name, force_ssl, no_ssl, bonjour_mode, + bonjour_publisher, advertise_host, filter_clients, output, output_format, + append, debug, as_json +): + """Listen for live NSLogger connections. + + \b + TCP mode (default): + cli-anything-nslogger listen --port 50000 + + Bonjour mode (iOS auto-discovers on same WiFi): + cli-anything-nslogger listen --bonjour --name bazinga + """ + from .core.listener import NSLoggerListener + + collected = [] + output_file = _open_live_output_file(output, append) if output else None + + def on_message(msg): + if level is not None and msg.level > level: + return + collected.append(msg) + if output_file: + output_file.write(_format_live_output_message(msg, output_format) + "\n") + if as_json: + click.echo(json.dumps(msg.to_dict(), default=str)) + else: + click.echo(msg.to_text_line()) + + def on_connect(host, p): + click.echo(f"[+] Client connected: {host}:{p}", err=True) + + def on_disconnect(host, p): + click.echo(f"[-] Client disconnected: {host}:{p}", err=True) + + def on_bonjour_ready(svc_name, svc_port): + click.echo(f"[Bonjour] Advertising as '{svc_name}' on port {svc_port}", err=True) + click.echo(f"[Bonjour] iOS app will auto-discover โ€” no IP config needed", err=True) + + def on_parse_error(host, p, raw, exc): + if not debug: + return + head = raw[:32].hex(" ") + click.echo( + f"[debug] Dropped frame from {host}:{p}: len={len(raw)} head={head} error={exc}", + err=True, + ) + + def on_debug(message): + if debug: + click.echo(f"[debug] {message}", err=True) + + if no_ssl: + bonjour_mode = "raw" + use_ssl = force_ssl + if bonjour: + use_ssl = False if bonjour_mode == "raw" else None + allow_plaintext = bonjour_mode != "ssl" + + listener = NSLoggerListener( + port=port, + timeout=timeout, + on_message=on_message, + on_connect=on_connect, + on_disconnect=on_disconnect, + on_bonjour_ready=on_bonjour_ready, + on_parse_error=on_parse_error, + on_debug=on_debug, + use_ssl=use_ssl, + allow_plaintext=allow_plaintext, + bonjour=bonjour, + bonjour_name=name, + filter_clients=filter_clients, + bonjour_publisher=bonjour_publisher, + advertise_host=advertise_host, + ) + + if bonjour: + click.echo(f"Starting Bonjour listener on port {port}โ€ฆ (Ctrl-C to stop)", err=True) + else: + click.echo(f"Listening on TCP port {port}โ€ฆ (Ctrl-C to stop)", err=True) + click.echo(_listen_waiting_message(port, bonjour), err=True) + if output: + action = "Appending" if append else "Writing" + click.echo(f"[output] {action} live logs to {output} ({output_format})", err=True) + + try: + listener.listen() + except KeyboardInterrupt: + pass + finally: + if output_file: + output_file.close() + click.echo(f"\nCaptured {len(collected)} messages.", err=True) + + +# --------------------------------------------------------------------------- +# generate +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("output", type=click.Path()) +@click.option("--count", "-n", type=int, default=20, show_default=True, + help="Number of log messages to generate") +def generate(output, count): + """Generate a sample .rawnsloggerdata file for testing.""" + from .utils.generate import generate_sample_file + generate_sample_file(output, count=count) + click.echo(f"Generated {count} messages โ†’ {output}") + + +# --------------------------------------------------------------------------- +# tail +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("file", type=click.Path(exists=True)) +@click.option("--count", "-n", type=int, default=20, show_default=True, + help="Number of messages from the end to show") +@_level_option() +@click.option("--tag", "-t", multiple=True, help="Filter by tag before tailing") +@_json_option() +def tail(file, count, level, tag, as_json): + """Show the last N messages from a file (reverse of --limit in read).""" + msgs = parse_file(file) + msgs = filter_messages( + msgs, + max_level=level, + tags=list(tag) if tag else None, + ) + all_msgs = list(msgs) + result = all_msgs[-count:] + if as_json: + click.echo(json.dumps([m.to_dict() for m in result], indent=2, default=str)) + else: + for m in result: + click.echo(m.to_text_line()) + + +# --------------------------------------------------------------------------- +# clients +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("file", type=click.Path(exists=True)) +@_json_option() +def clients(file, as_json): + """List all client connections recorded in a NSLogger file.""" + from .core.blocks import extract_clients + msgs = parse_file(file) + client_list = extract_clients(msgs) + if as_json: + click.echo(json.dumps(client_list, indent=2, default=str)) + else: + if not client_list: + click.echo("No client_info messages found.") + return + for c in client_list: + ts = c.get("timestamp") or "?" + name = c.get("client_name") or "unknown" + ver = c.get("client_version") or "" + os_ = f"{c.get('os_name', '')} {c.get('os_version', '')}".strip() + machine = c.get("machine") or "" + click.echo(f"[{ts}] {name} {ver} {os_} {machine}".strip()) + + +# --------------------------------------------------------------------------- +# blocks +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("file", type=click.Path(exists=True)) +@click.option("--indent", type=int, default=2, show_default=True, + help="Spaces per indent level") +@_json_option() +def blocks(file, indent, as_json): + """Show the block start/end structure from a NSLogger file as an indented tree.""" + from .core.blocks import iter_block_tree + msgs = parse_file(file) + entries = list(iter_block_tree(msgs)) + if as_json: + result = [ + {"depth": depth, **m.to_dict()} + for depth, m in entries + ] + click.echo(json.dumps(result, indent=2, default=str)) + else: + for depth, m in entries: + prefix = " " * (depth * indent) + click.echo(f"{prefix}{m.to_text_line()}") + + +# --------------------------------------------------------------------------- +# merge +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("files", nargs=-1, required=True, type=click.Path(exists=True)) +@click.option("--output", "-o", type=click.Path(), default=None, + help="Write merged output to file (default: stdout)") +@click.option("--format", "-f", "fmt", + type=click.Choice(["text", "json", "csv"]), default="text", + show_default=True) +@_level_option() +def merge(files, output, fmt, level): + """Merge multiple NSLogger files, sorted by timestamp.""" + from .core.blocks import merge_files + all_msgs = merge_files(list(files)) + if level is not None: + all_msgs = [m for m in all_msgs if m.level <= level] + from .core.exporter import export_messages + result_str = export_messages(iter(all_msgs), fmt=fmt) + if output: + with open(output, "w", encoding="utf-8") as f: + f.write(result_str) + click.echo(f"Merged {len(files)} files โ†’ {output}", err=True) + else: + click.echo(result_str, nl=False) + + +# --------------------------------------------------------------------------- +# repl (interactive shell) +# --------------------------------------------------------------------------- + +@cli.command() +@click.argument("file", type=click.Path(exists=True), required=False) +def repl(file): + """Start an interactive REPL for exploring NSLogger files.""" + try: + import code + import readline # noqa: F401 + except ImportError: + pass + + context = {} + if file: + msgs = list(parse_file(file)) + context["messages"] = msgs + context["file"] = file + click.echo(f"Loaded {len(msgs)} messages from {file}") + click.echo("Available: messages, filter_messages, compute_stats, export_messages") + else: + click.echo("NSLogger REPL โ€” no file loaded. Use: messages = list(parse_file('x.rawnsloggerdata'))") + + context.update({ + "parse_file": parse_file, + "filter_messages": filter_messages, + "compute_stats": compute_stats, + "export_messages": export_messages, + }) + + import code as _code + _code.interact(local=context, banner="") + + +def main(): + cli() + + +if __name__ == "__main__": + main() diff --git a/nslogger/agent-harness/cli_anything/nslogger/skills/SKILL.md b/nslogger/agent-harness/cli_anything/nslogger/skills/SKILL.md new file mode 100644 index 000000000..5ead7b0fb --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/skills/SKILL.md @@ -0,0 +1,188 @@ +--- +name: cli-anything-nslogger +description: CLI harness for NSLogger โ€” parse, filter, export, and monitor NSLogger log files (.rawnsloggerdata / .nsloggerdata) +version: 0.1.0 +install: pip install git+https://github.com/HKUDS/CLI-Anything.git#subdirectory=nslogger/agent-harness +binary: cli-anything-nslogger +tags: [logging, ios, macos, debugging, nslogger] +--- + +# cli-anything-nslogger + +A complete CLI harness for [NSLogger](https://github.com/fpillet/NSLogger), the macOS log viewer for iOS/macOS apps. + +## Installation + +```bash +cd nslogger/agent-harness +pip install -e . +# Verify +cli-anything-nslogger --help +``` + +## Command Reference + +### `generate` โ€” Create sample files for testing + +```bash +cli-anything-nslogger generate sample.rawnsloggerdata --count 50 +``` + +### `read` โ€” Display messages from a file + +```bash +# All messages +cli-anything-nslogger read session.rawnsloggerdata + +# Errors only (level 0) +cli-anything-nslogger read session.rawnsloggerdata --level 0 + +# Filter by tag and text search +cli-anything-nslogger read session.rawnsloggerdata --tag Network --search "timeout" + +# First 20 messages as JSON +cli-anything-nslogger read session.rawnsloggerdata --limit 20 --json +``` + +### `filter` โ€” Advanced filtering + +```bash +# Errors and warnings only +cli-anything-nslogger filter session.rawnsloggerdata --level 1 + +# By tag +cli-anything-nslogger filter session.rawnsloggerdata --tag Auth --tag Network + +# Regex search +cli-anything-nslogger filter session.rawnsloggerdata --regex "(timeout|failed|error)" + +# By thread +cli-anything-nslogger filter session.rawnsloggerdata --thread "main" + +# JSON output +cli-anything-nslogger filter session.rawnsloggerdata --level 0 --json +``` + +### `export` โ€” Export to text/JSON/CSV + +```bash +# JSON to stdout +cli-anything-nslogger export session.rawnsloggerdata --format json + +# CSV to file +cli-anything-nslogger export session.rawnsloggerdata --format csv --output logs.csv + +# Filtered text export +cli-anything-nslogger export session.rawnsloggerdata --format text --level 1 --tag Network +``` + +### `stats` โ€” Summary statistics + +```bash +# Human-readable summary +cli-anything-nslogger stats session.rawnsloggerdata + +# JSON for agent consumption +cli-anything-nslogger stats session.rawnsloggerdata --json +``` + +JSON output shape: +```json +{ + "total": 342, + "by_level": {"ERROR": 12, "WARNING": 34, "INFO": 200, "DEBUG": 96}, + "by_tag": {"Network": 89, "Auth": 45, "UI": 120}, + "by_thread": {"main": 200, "bg-queue": 142}, + "by_type": {"text": 340, "client_info": 1, "disconnect": 1}, + "clients": ["MyApp"], + "first_timestamp": "2024-01-01T10:00:00+00:00", + "last_timestamp": "2024-01-01T10:05:30+00:00", + "duration_seconds": 330.0 +} +``` + +### `listen` โ€” Receive live connections + +```bash +# Match the NSLogger.app GUI Bonjour behavior for iOS auto-discovery +cli-anything-nslogger listen --bonjour --name bazinga --debug + +# Mirror live logs to a text file while still printing stdout +cli-anything-nslogger listen --bonjour --name bazinga --output app.log + +# Write machine-readable JSON Lines +cli-anything-nslogger listen --bonjour --name bazinga --output app.jsonl --output-format jsonl + +# Direct TCP/TLS mode for manually configured clients +cli-anything-nslogger listen --port 50000 --ssl --debug + +# Show only errors while listening, output as JSON stream +cli-anything-nslogger listen --bonjour --name bazinga --level 0 --json + +# Run until Ctrl-C +cli-anything-nslogger listen --bonjour --name bazinga +``` + +Use Bonjour mode first for iOS apps because it matches the desktop NSLogger GUI: +the CLI publishes a native macOS `NetService` with `_nslogger-ssl._tcp` and +accepts TLS NSLogger frames. Use direct TCP/TLS only when the app is manually +configured with the Mac host and port. + +### `repl` โ€” Interactive Python REPL + +```bash +cli-anything-nslogger repl session.rawnsloggerdata +# Available: messages, parse_file, filter_messages, compute_stats, export_messages +``` + +## Log Levels + +| Value | Name | Use for | +|-------|---------|---------| +| 0 | ERROR | Unrecoverable failures | +| 1 | WARNING | Recoverable issues | +| 2 | INFO | Normal operation | +| 3 | DEBUG | Developer details | +| 4 | VERBOSE | Trace-level noise | + +## Message JSON Shape + +```json +{ + "sequence": 42, + "timestamp": "2024-01-01T10:01:23+00:00", + "timestamp_ms": 456, + "thread_id": "main", + "tag": "Network", + "level": 0, + "level_name": "ERROR", + "type": "text", + "text": "Connection timed out after 30s", + "image_width": 0, + "image_height": 0, + "client_name": "MyApp", + "client_version": "2.1.0", + "os_name": "iOS", + "os_version": "17.0", + "machine": "iPhone15,2" +} +``` + +## Agent Workflow Examples + +```bash +# 1. Inspect a captured crash session +cli-anything-nslogger stats crash.rawnsloggerdata --json + +# 2. Find all errors in the 5 minutes before crash +cli-anything-nslogger filter crash.rawnsloggerdata --level 0 --json + +# 3. Get network failures only +cli-anything-nslogger filter crash.rawnsloggerdata --tag Network --regex "fail|timeout|error" --json + +# 4. Export full log for offline analysis +cli-anything-nslogger export crash.rawnsloggerdata --format json --output crash_log.json + +# 5. Monitor an iOS app live and keep a local copy +cli-anything-nslogger listen --bonjour --name bazinga --output app.log --debug +``` diff --git a/nslogger/agent-harness/cli_anything/nslogger/tests/TEST.md b/nslogger/agent-harness/cli_anything/nslogger/tests/TEST.md new file mode 100644 index 000000000..bc5ad42c7 --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/tests/TEST.md @@ -0,0 +1,137 @@ +# TEST.md โ€” cli-anything-nslogger + +## Test Plan + +### Unit Tests (`test_core.py`) + +All tests use synthetic in-memory data โ€” no external files or network required. + +| Class | Coverage | +|-------|----------| +| `TestLogMessage` | `to_dict()`, `to_text_line()`, level/type name derivation for all types | +| `TestFilterMessages` | level, min-level, tag (case-insensitive), thread, text search, regex, limit, combined | +| `TestComputeStats` | totals, by_level, by_tag, by_thread, by_type, duration, timestamps, empty input | +| `TestExporter` | text, JSON (shape + validity), CSV (header + rows), `export_messages()` dispatcher | +| `TestWireProtocol` | encodeโ†’decode round-trip for text, timestamp, client-info; length prefix format | +| `TestGenerateSampleFile` | file creation, parseability, level variety | +| `TestParseRawFile` | single message, multiple messages, empty file | + +### E2E Tests (`test_full_e2e.py`) + +All tests use real files created by `generate_sample_file()` and real subprocess invocations. + +| Class | Coverage | +|-------|----------| +| `TestGenerateCommand` | file creation, count in output, parseable result | +| `TestReadCommand` | output lines, `--json` shape, `--level` filter, `--limit`, `--search` | +| `TestFilterCommand` | `--level`, no-results, `--regex` | +| `TestExportCommand` | text/JSON/CSV stdout, `--output` file, `--level` pre-filter | +| `TestStatsCommand` | text summary, JSON shape, `by_level`, `by_tag` | +| `TestWorkflow` | generateโ†’filterโ†’export pipeline, stats on generated file, help output | +| `TestCLISubprocess` | installed entrypoint via `_resolve_cli()`: help, generate+read, stats JSON, export CSV | + +### Scenarios NOT covered by automated tests + +- `listen` command (requires live TCP client; integration test would need a network fixture) +- `repl` command (interactive terminal; tested manually) +- `.nsloggerdata` binary-plist format (requires a real NSLogger.app saved file) +- SSL/TLS listener mode + +--- + +## Test Results + +Run: `python3 -m pytest cli_anything/nslogger/tests/ -v --tb=no` + +Platform: darwin / Python 3.13.2 / pytest 9.0.3 + +``` +============================= test session starts ============================== +platform darwin -- Python 3.13.2, pytest-9.0.3, pluggy-1.6.0 +collected 80 items + +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_level_name_known PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_level_name_unknown PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_type_name_text PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_type_name_image PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_type_name_data PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_type_name_client_info PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_to_dict_keys PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_to_dict_timestamp_isoformat PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_to_text_line_contains_text PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_to_text_line_contains_level PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_to_text_line_contains_tag PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_to_text_line_no_timestamp PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_to_text_line_image PASSED +cli_anything/nslogger/tests/test_core.py::TestLogMessage::test_to_text_line_binary PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_no_filter_passes_all PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_max_level PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_min_level PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_tag_filter PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_tag_case_insensitive PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_thread_filter PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_text_search PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_text_search_case_insensitive PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_regex_filter PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_limit PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_combined_filters PASSED +cli_anything/nslogger/tests/test_core.py::TestFilterMessages::test_empty_input PASSED +cli_anything/nslogger/tests/test_core.py::TestComputeStats::test_total PASSED +cli_anything/nslogger/tests/test_core.py::TestComputeStats::test_by_level PASSED +cli_anything/nslogger/tests/test_core.py::TestComputeStats::test_by_tag PASSED +cli_anything/nslogger/tests/test_core.py::TestComputeStats::test_by_thread PASSED +cli_anything/nslogger/tests/test_core.py::TestComputeStats::test_duration PASSED +cli_anything/nslogger/tests/test_core.py::TestComputeStats::test_timestamps PASSED +cli_anything/nslogger/tests/test_core.py::TestComputeStats::test_empty PASSED +cli_anything/nslogger/tests/test_core.py::TestComputeStats::test_by_type PASSED +cli_anything/nslogger/tests/test_core.py::TestExporter::test_export_text PASSED +cli_anything/nslogger/tests/test_core.py::TestExporter::test_export_json_valid PASSED +cli_anything/nslogger/tests/test_core.py::TestExporter::test_export_json_has_all_fields PASSED +cli_anything/nslogger/tests/test_core.py::TestExporter::test_export_csv_has_header PASSED +cli_anything/nslogger/tests/test_core.py::TestExporter::test_export_csv_has_data PASSED +cli_anything/nslogger/tests/test_core.py::TestExporter::test_export_messages_text PASSED +cli_anything/nslogger/tests/test_core.py::TestExporter::test_export_messages_json PASSED +cli_anything/nslogger/tests/test_core.py::TestExporter::test_export_messages_csv PASSED +cli_anything/nslogger/tests/test_core.py::TestWireProtocol::test_round_trip_text PASSED +cli_anything/nslogger/tests/test_core.py::TestWireProtocol::test_round_trip_timestamp PASSED +cli_anything/nslogger/tests/test_core.py::TestWireProtocol::test_round_trip_client_info PASSED +cli_anything/nslogger/tests/test_core.py::TestWireProtocol::test_encode_message_has_length_prefix PASSED +cli_anything/nslogger/tests/test_core.py::TestGenerateSampleFile::test_creates_file PASSED +cli_anything/nslogger/tests/test_core.py::TestGenerateSampleFile::test_parseable PASSED +cli_anything/nslogger/tests/test_core.py::TestGenerateSampleFile::test_varied_levels PASSED +cli_anything/nslogger/tests/test_core.py::TestParseRawFile::test_single_message PASSED +cli_anything/nslogger/tests/test_core.py::TestParseRawFile::test_multiple_messages PASSED +cli_anything/nslogger/tests/test_core.py::TestParseRawFile::test_empty_file PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestGenerateCommand::test_generate_creates_file PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestGenerateCommand::test_generate_output_mentions_count PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestGenerateCommand::test_generate_parseable PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestReadCommand::test_read_outputs_messages PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestReadCommand::test_read_json_valid PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestReadCommand::test_read_json_message_shape PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestReadCommand::test_read_level_filter PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestReadCommand::test_read_limit PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestReadCommand::test_read_search PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestFilterCommand::test_filter_by_level PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestFilterCommand::test_filter_no_results PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestFilterCommand::test_filter_regex PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestExportCommand::test_export_text_stdout PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestExportCommand::test_export_json_stdout PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestExportCommand::test_export_csv_stdout PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestExportCommand::test_export_to_file PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestExportCommand::test_export_with_level_filter PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestStatsCommand::test_stats_text_output PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestStatsCommand::test_stats_json_output PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestStatsCommand::test_stats_json_has_by_level PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestStatsCommand::test_stats_json_has_by_tag PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestWorkflow::test_generate_filter_export_pipeline PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestWorkflow::test_stats_on_generated_file PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestWorkflow::test_cli_help_shows_commands PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestCLISubprocess::test_installed_cli_help PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestCLISubprocess::test_installed_generate_and_read PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestCLISubprocess::test_installed_stats_json PASSED +cli_anything/nslogger/tests/test_full_e2e.py::TestCLISubprocess::test_installed_export_csv PASSED + +============================== 80 passed in 3.55s ============================== +``` + +**Result: 80 passed, 0 failed (100% pass rate)** diff --git a/nslogger/agent-harness/cli_anything/nslogger/tests/__init__.py b/nslogger/agent-harness/cli_anything/nslogger/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/nslogger/agent-harness/cli_anything/nslogger/tests/test_core.py b/nslogger/agent-harness/cli_anything/nslogger/tests/test_core.py new file mode 100644 index 000000000..45167bda3 --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/tests/test_core.py @@ -0,0 +1,886 @@ +"""Unit tests for NSLogger CLI core modules (no external deps, synthetic data).""" +import io +import json +import struct +import tempfile +import os +import time +from datetime import datetime, timezone +from unittest.mock import patch + +import pytest + +from cli_anything.nslogger.core.message import ( + LogMessage, MSG_TYPE_LOG, MSG_TYPE_CLIENT_INFO, MSG_TYPE_BLOCK_START, MSG_TYPE_BLOCK_END, + LEVEL_NAMES, +) +from cli_anything.nslogger.core.filter import filter_messages +from cli_anything.nslogger.core.stats import compute_stats +from cli_anything.nslogger.core.exporter import export_text, export_json, export_csv, export_messages +from cli_anything.nslogger.core.blocks import iter_block_tree, extract_clients, merge_files +from cli_anything.nslogger.core.listener import ( + NSLoggerListener, _bonjour_service_types, _dns_sd_txt_args, _looks_like_tls_client_hello, + _classify_connection, _swift_helper_env, +) +from cli_anything.nslogger.nslogger_cli import ( + _format_live_output_message, + _listen_waiting_message, + _open_live_output_file, +) +from cli_anything.nslogger.utils.generate import encode_message, generate_sample_file +from cli_anything.nslogger.core.parser import _parse_message, parse_raw_file + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def make_msg(**kwargs) -> LogMessage: + defaults = dict( + sequence=0, + timestamp=datetime(2024, 1, 1, 12, 0, 0, tzinfo=timezone.utc), + timestamp_ms=500, + level=2, + tag="Test", + thread_id="main", + text="hello world", + message_type=MSG_TYPE_LOG, + ) + defaults.update(kwargs) + return LogMessage(**defaults) + + +# --------------------------------------------------------------------------- +# LogMessage model +# --------------------------------------------------------------------------- + +class TestLogMessage: + def test_level_name_known(self): + msg = make_msg(level=0) + assert msg.level_name == "ERROR" + + def test_level_name_unknown(self): + msg = make_msg(level=99) + assert "99" in msg.level_name + + def test_type_name_text(self): + msg = make_msg(text="hello") + assert msg.type_name == "text" + + def test_type_name_image(self): + msg = make_msg(image_data=b"\xff\xd8\xff", image_width=100, image_height=200) + assert msg.type_name == "image" + + def test_type_name_data(self): + msg = make_msg(binary_data=b"\x00\x01") + assert msg.type_name == "data" + + def test_type_name_client_info(self): + msg = make_msg(message_type=MSG_TYPE_CLIENT_INFO) + assert msg.type_name == "client_info" + + def test_to_dict_keys(self): + msg = make_msg() + d = msg.to_dict() + assert "sequence" in d + assert "timestamp" in d + assert "level_name" in d + assert "type" in d + assert "text" in d + + def test_to_dict_timestamp_isoformat(self): + msg = make_msg() + d = msg.to_dict() + assert "2024-01-01" in d["timestamp"] + + def test_to_text_line_contains_text(self): + msg = make_msg(text="SAMPLE_TEXT") + line = msg.to_text_line() + assert "SAMPLE_TEXT" in line + + def test_to_text_line_contains_level(self): + msg = make_msg(level=0) + line = msg.to_text_line() + assert "ERROR" in line + + def test_to_text_line_contains_tag(self): + msg = make_msg(tag="NetworkOps") + line = msg.to_text_line() + assert "NetworkOps" in line + + def test_to_text_line_no_timestamp(self): + msg = make_msg(timestamp=None) + line = msg.to_text_line() + assert "??" in line + + def test_to_text_line_image(self): + msg = make_msg(image_data=b"x", image_width=320, image_height=240) + line = msg.to_text_line() + assert "image" in line + assert "320" in line + + def test_to_text_line_binary(self): + msg = make_msg(binary_data=b"\x00" * 10) + line = msg.to_text_line() + assert "10" in line + + +# --------------------------------------------------------------------------- +# filter_messages +# --------------------------------------------------------------------------- + +class TestFilterMessages: + def _msgs(self): + return [ + make_msg(sequence=1, level=0, tag="Auth", thread_id="main", text="error occurred"), + make_msg(sequence=2, level=2, tag="Network", thread_id="bg", text="request sent"), + make_msg(sequence=3, level=3, tag="Auth", thread_id="main", text="token refreshed"), + make_msg(sequence=4, level=4, tag="UI", thread_id="main", text="view loaded"), + ] + + def test_no_filter_passes_all(self): + result = list(filter_messages(iter(self._msgs()))) + assert len(result) == 4 + + def test_max_level(self): + result = list(filter_messages(iter(self._msgs()), max_level=1)) + assert all(m.level <= 1 for m in result) + assert len(result) == 1 + + def test_min_level(self): + result = list(filter_messages(iter(self._msgs()), min_level=3)) + assert all(m.level >= 3 for m in result) + assert len(result) == 2 + + def test_tag_filter(self): + result = list(filter_messages(iter(self._msgs()), tags=["auth"])) + assert all(m.tag == "Auth" for m in result) + assert len(result) == 2 + + def test_tag_case_insensitive(self): + result = list(filter_messages(iter(self._msgs()), tags=["AUTH"])) + assert len(result) == 2 + + def test_thread_filter(self): + result = list(filter_messages(iter(self._msgs()), thread_id="bg")) + assert len(result) == 1 + assert result[0].sequence == 2 + + def test_text_search(self): + result = list(filter_messages(iter(self._msgs()), text_search="token")) + assert len(result) == 1 + assert "token" in result[0].text.lower() + + def test_text_search_case_insensitive(self): + result = list(filter_messages(iter(self._msgs()), text_search="ERROR")) + assert len(result) == 1 + + def test_regex_filter(self): + result = list(filter_messages(iter(self._msgs()), text_regex=r"re(quest|freshed)")) + assert len(result) == 2 + + def test_limit(self): + result = list(filter_messages(iter(self._msgs()), limit=2)) + assert len(result) == 2 + + def test_combined_filters(self): + result = list(filter_messages(iter(self._msgs()), max_level=2, tags=["auth"])) + assert len(result) == 1 + assert result[0].level == 0 + + def test_empty_input(self): + result = list(filter_messages(iter([]))) + assert result == [] + + +# --------------------------------------------------------------------------- +# compute_stats +# --------------------------------------------------------------------------- + +class TestComputeStats: + def _msgs(self): + return [ + make_msg(level=0, tag="Auth", thread_id="main", + timestamp=datetime(2024, 1, 1, 10, 0, tzinfo=timezone.utc)), + make_msg(level=2, tag="Network", thread_id="bg", + timestamp=datetime(2024, 1, 1, 10, 1, tzinfo=timezone.utc)), + make_msg(level=2, tag="Auth", thread_id="main", + timestamp=datetime(2024, 1, 1, 10, 2, tzinfo=timezone.utc)), + ] + + def test_total(self): + s = compute_stats(iter(self._msgs())) + assert s["total"] == 3 + + def test_by_level(self): + s = compute_stats(iter(self._msgs())) + assert s["by_level"]["ERROR"] == 1 + assert s["by_level"]["INFO"] == 2 + + def test_by_tag(self): + s = compute_stats(iter(self._msgs())) + assert s["by_tag"]["Auth"] == 2 + assert s["by_tag"]["Network"] == 1 + + def test_by_thread(self): + s = compute_stats(iter(self._msgs())) + assert s["by_thread"]["main"] == 2 + + def test_duration(self): + s = compute_stats(iter(self._msgs())) + assert s["duration_seconds"] == 120.0 + + def test_timestamps(self): + s = compute_stats(iter(self._msgs())) + assert "2024-01-01T10:00:00" in s["first_timestamp"] + assert "2024-01-01T10:02:00" in s["last_timestamp"] + + def test_empty(self): + s = compute_stats(iter([])) + assert s["total"] == 0 + + def test_by_type(self): + s = compute_stats(iter(self._msgs())) + assert "text" in s["by_type"] + + +# --------------------------------------------------------------------------- +# exporter +# --------------------------------------------------------------------------- + +class TestExporter: + def _msgs(self): + return [ + make_msg(sequence=1, level=0, tag="A", text="first message"), + make_msg(sequence=2, level=2, tag="B", text="second message"), + ] + + def test_export_text(self): + out = export_text(self._msgs()) + assert "first message" in out + assert "second message" in out + + def test_export_json_valid(self): + out = export_json(self._msgs()) + data = json.loads(out) + assert isinstance(data, list) + assert len(data) == 2 + assert data[0]["sequence"] == 1 + + def test_export_json_has_all_fields(self): + out = export_json(self._msgs()) + data = json.loads(out) + for key in ("sequence", "timestamp", "level", "level_name", "tag", "text", "type"): + assert key in data[0] + + def test_export_csv_has_header(self): + out = export_csv(self._msgs()) + assert "sequence" in out.splitlines()[0] + + def test_export_csv_has_data(self): + out = export_csv(self._msgs()) + lines = out.strip().splitlines() + assert len(lines) == 3 # header + 2 rows + + def test_export_messages_text(self): + out = export_messages(iter(self._msgs()), fmt="text") + assert "first message" in out + + def test_export_messages_json(self): + out = export_messages(iter(self._msgs()), fmt="json") + json.loads(out) # must be valid JSON + + def test_export_messages_csv(self): + out = export_messages(iter(self._msgs()), fmt="csv") + assert "," in out + + +# --------------------------------------------------------------------------- +# wire protocol encoder / parser +# --------------------------------------------------------------------------- + +class TestWireProtocol: + def _encode_and_parse(self, **kwargs): + raw_file_bytes = encode_message(sequence=1, **kwargs) + # raw_file_bytes = [4-byte length][body] + body = raw_file_bytes[4:] + return _parse_message(body) + + def test_round_trip_text(self): + msg = self._encode_and_parse(text="hello", level=2, tag="TAG", thread_id="main") + assert msg.text == "hello" + assert msg.level == 2 + assert msg.tag == "TAG" + assert msg.thread_id == "main" + assert msg.sequence == 1 + + def test_round_trip_timestamp(self): + ts = 1700000000.5 + msg = self._encode_and_parse(timestamp=ts, text="x") + assert msg.timestamp is not None + assert abs(msg.timestamp.timestamp() - int(ts)) < 1 + + def test_round_trip_client_info(self): + msg = self._encode_and_parse( + msg_type=3, + client_name="TestApp", + client_version="2.0", + os_name="iOS", + os_version="17.0", + machine="iPhone15,2", + ) + assert msg.client_name == "TestApp" + assert msg.os_name == "iOS" + + def test_encode_message_has_length_prefix(self): + raw = encode_message(sequence=0, text="x") + declared_len = struct.unpack(">I", raw[:4])[0] + assert declared_len == len(raw) - 4 + + def test_official_integer_parts_do_not_have_length_fields(self): + body = encode_message(sequence=7, text="official", level=1)[4:] + msg = _parse_message(body) + assert msg.sequence == 7 + assert msg.level == 1 + assert msg.text == "official" + + def test_legacy_lengthful_integer_parts_still_parse(self): + parts = b"" + parts += bytes([0, 3]) + struct.pack(">I", 4) + struct.pack(">I", 0) + parts += bytes([10, 3]) + struct.pack(">I", 4) + struct.pack(">I", 8) + text = b"legacy" + parts += bytes([7, 0]) + struct.pack(">I", len(text)) + text + body = struct.pack(">H", 3) + parts + msg = _parse_message(body) + assert msg.sequence == 8 + assert msg.text == "legacy" + + +# --------------------------------------------------------------------------- +# generate_sample_file +# --------------------------------------------------------------------------- + +class TestGenerateSampleFile: + def test_creates_file(self, tmp_path): + path = str(tmp_path / "sample.rawnsloggerdata") + generate_sample_file(path, count=5) + assert os.path.exists(path) + assert os.path.getsize(path) > 0 + + def test_parseable(self, tmp_path): + path = str(tmp_path / "sample.rawnsloggerdata") + generate_sample_file(path, count=10) + msgs = list(parse_raw_file(path)) + assert len(msgs) >= 10 # 10 log + 1 client_info + + def test_varied_levels(self, tmp_path): + path = str(tmp_path / "sample.rawnsloggerdata") + generate_sample_file(path, count=50) + msgs = list(parse_raw_file(path)) + levels = {m.level for m in msgs} + assert len(levels) > 1 + + +# --------------------------------------------------------------------------- +# parse_raw_file (file I/O) +# --------------------------------------------------------------------------- + +class TestParseRawFile: + def _write_file(self, tmp_path, *messages_kwargs): + path = str(tmp_path / "test.rawnsloggerdata") + with open(path, "wb") as f: + for i, kw in enumerate(messages_kwargs): + f.write(encode_message(sequence=i, **kw)) + return path + + def test_single_message(self, tmp_path): + path = self._write_file(tmp_path, {"text": "only one"}) + msgs = list(parse_raw_file(path)) + assert len(msgs) == 1 + assert msgs[0].text == "only one" + + def test_multiple_messages(self, tmp_path): + path = self._write_file( + tmp_path, + {"text": "first", "level": 0}, + {"text": "second", "level": 2}, + {"text": "third", "level": 4}, + ) + msgs = list(parse_raw_file(path)) + assert len(msgs) == 3 + assert msgs[0].text == "first" + assert msgs[2].text == "third" + + def test_empty_file(self, tmp_path): + path = str(tmp_path / "empty.rawnsloggerdata") + open(path, "wb").close() + msgs = list(parse_raw_file(path)) + assert msgs == [] + + +# --------------------------------------------------------------------------- +# filter_messages โ€” new time-range and sequence-range options +# --------------------------------------------------------------------------- + +class TestFilterMessagesExtended: + def _msgs_with_timestamps(self): + t = lambda h, m: datetime(2024, 1, 1, h, m, 0, tzinfo=timezone.utc) + return [ + make_msg(sequence=10, timestamp=t(10, 0), level=0, text="early error"), + make_msg(sequence=20, timestamp=t(10, 30), level=2, text="mid info"), + make_msg(sequence=30, timestamp=t(11, 0), level=3, text="late debug"), + ] + + def test_after_filter(self): + after = datetime(2024, 1, 1, 10, 15, tzinfo=timezone.utc) + result = list(filter_messages(iter(self._msgs_with_timestamps()), after=after)) + assert len(result) == 2 + assert result[0].sequence == 20 + + def test_before_filter(self): + before = datetime(2024, 1, 1, 10, 45, tzinfo=timezone.utc) + result = list(filter_messages(iter(self._msgs_with_timestamps()), before=before)) + assert len(result) == 2 + assert result[-1].sequence == 20 + + def test_after_and_before_window(self): + after = datetime(2024, 1, 1, 10, 15, tzinfo=timezone.utc) + before = datetime(2024, 1, 1, 10, 45, tzinfo=timezone.utc) + result = list(filter_messages(iter(self._msgs_with_timestamps()), after=after, before=before)) + assert len(result) == 1 + assert result[0].sequence == 20 + + def test_from_seq(self): + result = list(filter_messages(iter(self._msgs_with_timestamps()), from_seq=20)) + assert len(result) == 2 + assert result[0].sequence == 20 + + def test_to_seq(self): + result = list(filter_messages(iter(self._msgs_with_timestamps()), to_seq=20)) + assert len(result) == 2 + assert result[-1].sequence == 20 + + def test_seq_range(self): + result = list(filter_messages(iter(self._msgs_with_timestamps()), from_seq=20, to_seq=20)) + assert len(result) == 1 + assert result[0].sequence == 20 + + def test_no_timestamp_excluded_by_after(self): + msgs = [make_msg(sequence=1, timestamp=None, text="no ts")] + after = datetime(2024, 1, 1, tzinfo=timezone.utc) + result = list(filter_messages(iter(msgs), after=after)) + assert result == [] + + +# --------------------------------------------------------------------------- +# blocks module +# --------------------------------------------------------------------------- + +class TestIterBlockTree: + def _block_msgs(self): + return [ + make_msg(sequence=1, message_type=MSG_TYPE_LOG, text="before block"), + make_msg(sequence=2, message_type=MSG_TYPE_BLOCK_START, text="enter"), + make_msg(sequence=3, message_type=MSG_TYPE_LOG, text="inside"), + make_msg(sequence=4, message_type=MSG_TYPE_BLOCK_END, text="exit"), + make_msg(sequence=5, message_type=MSG_TYPE_LOG, text="after block"), + ] + + def test_top_level_messages_have_depth_zero(self): + pairs = list(iter_block_tree(iter(self._block_msgs()))) + assert pairs[0][0] == 0 # before block + assert pairs[4][0] == 0 # after block + + def test_block_start_at_zero_before_increment(self): + pairs = list(iter_block_tree(iter(self._block_msgs()))) + # block_start emitted at depth 0, then depth increments + assert pairs[1][0] == 0 + assert pairs[1][1].message_type == MSG_TYPE_BLOCK_START + + def test_inside_block_has_depth_one(self): + pairs = list(iter_block_tree(iter(self._block_msgs()))) + assert pairs[2][0] == 1 + + def test_block_end_has_depth_zero_after_decrement(self): + pairs = list(iter_block_tree(iter(self._block_msgs()))) + # block_end decrements first, so emitted at depth 0 + assert pairs[3][0] == 0 + + def test_empty_input(self): + assert list(iter_block_tree(iter([]))) == [] + + +class TestExtractClients: + def _msgs_with_client(self): + client = LogMessage( + sequence=0, + message_type=MSG_TYPE_CLIENT_INFO, + client_name="MyApp", + client_version="3.1", + os_name="iOS", + os_version="17.0", + machine="iPhone16,1", + ) + log = make_msg(sequence=1, text="regular log") + return [client, log] + + def test_returns_only_client_info(self): + result = extract_clients(iter(self._msgs_with_client())) + assert len(result) == 1 + + def test_client_fields(self): + result = extract_clients(iter(self._msgs_with_client())) + c = result[0] + assert c["client_name"] == "MyApp" + assert c["client_version"] == "3.1" + assert c["os_name"] == "iOS" + assert c["machine"] == "iPhone16,1" + + def test_no_clients_returns_empty(self): + result = extract_clients(iter([make_msg(text="log")])) + assert result == [] + + +class TestMergeFiles: + def _write(self, tmp_path, name, *messages_kwargs): + path = str(tmp_path / name) + with open(path, "wb") as f: + for i, kw in enumerate(messages_kwargs): + f.write(encode_message(sequence=i, **kw)) + return path + + def test_merge_two_files_sorted(self, tmp_path): + ts_early = 1700000000.0 + ts_late = 1700000100.0 + path_a = self._write(tmp_path, "a.rawnsloggerdata", + {"text": "late msg", "timestamp": ts_late}) + path_b = self._write(tmp_path, "b.rawnsloggerdata", + {"text": "early msg", "timestamp": ts_early}) + result = merge_files([path_a, path_b]) + assert len(result) == 2 + assert result[0].text == "early msg" + assert result[1].text == "late msg" + + def test_merge_single_file(self, tmp_path): + path = self._write(tmp_path, "c.rawnsloggerdata", + {"text": "only msg"}) + result = merge_files([path]) + assert len(result) == 1 + + def test_merge_preserves_all_messages(self, tmp_path): + path_a = self._write(tmp_path, "d.rawnsloggerdata", + {"text": "a1"}, {"text": "a2"}) + path_b = self._write(tmp_path, "e.rawnsloggerdata", + {"text": "b1"}, {"text": "b2"}) + result = merge_files([path_a, path_b]) + assert len(result) == 4 + + +# --------------------------------------------------------------------------- +# listener +# --------------------------------------------------------------------------- + +class TestNSLoggerListener: + def test_swift_helper_env_uses_temp_module_caches(self): + env = _swift_helper_env() + + assert env["SWIFT_MODULE_CACHE_PATH"].startswith(tempfile.gettempdir()) + assert env["CLANG_MODULE_CACHE_PATH"].startswith(tempfile.gettempdir()) + + def test_named_bonjour_service_uses_filter_txt_record(self): + assert _dns_sd_txt_args("bazinga", filter_clients=True) == ["filterClients=1"] + + def test_named_bonjour_service_does_not_filter_by_default(self): + assert _dns_sd_txt_args("bazinga") == [] + + def test_empty_bonjour_service_has_no_filter_txt_record(self): + assert _dns_sd_txt_args("") == [] + + def test_ssl_bonjour_advertises_only_ssl_service(self): + assert _bonjour_service_types(True) == ("_nslogger-ssl._tcp",) + + def test_auto_bonjour_advertises_raw_and_ssl_services(self): + assert _bonjour_service_types(True, allow_plaintext=True) == ("_nslogger._tcp", "_nslogger-ssl._tcp") + + def test_non_ssl_bonjour_advertises_only_legacy_service(self): + assert _bonjour_service_types(False) == ("_nslogger._tcp",) + + def test_macos_bonjour_prefers_native_netservice(self): + listener = NSLoggerListener(bonjour=True, bonjour_name="bazinga") + + class DummyNativePublisher: + def __init__(self, service_name, service_types, port, filter_clients, on_debug): + self.service_name = service_name + self.service_types = service_types + self.port = port + self.filter_clients = filter_clients + self.on_debug = on_debug + + with patch("cli_anything.nslogger.core.listener.sys.platform", "darwin"), \ + patch("cli_anything.nslogger.core.listener._NativeBonjourPublisher", DummyNativePublisher), \ + patch("cli_anything.nslogger.core.listener._DnsSdBonjourPublisher") as dns_sd_publisher, \ + patch("cli_anything.nslogger.core.listener._ZeroconfBonjourPublisher") as zeroconf_publisher: + publisher = listener._start_bonjour("192.168.10.1") + + assert isinstance(publisher, DummyNativePublisher) + assert publisher.service_name == "bazinga" + assert publisher.service_types == ("_nslogger-ssl._tcp",) + assert publisher.filter_clients is True + assert dns_sd_publisher.call_count == 0 + assert zeroconf_publisher.call_count == 0 + + def test_bonjour_auto_mode_can_publish_raw_and_ssl_services(self): + listener = NSLoggerListener( + bonjour=True, + bonjour_name="bazinga", + allow_plaintext=True, + ) + + class DummyNativePublisher: + def __init__(self, service_name, service_types, port, filter_clients, on_debug): + self.service_types = service_types + + with patch("cli_anything.nslogger.core.listener.sys.platform", "darwin"), \ + patch("cli_anything.nslogger.core.listener._NativeBonjourPublisher", DummyNativePublisher): + publisher = listener._start_bonjour("192.168.10.1") + + assert publisher.service_types == ("_nslogger._tcp", "_nslogger-ssl._tcp") + + def test_dns_sd_publisher_can_be_forced_on_macos(self): + listener = NSLoggerListener( + bonjour=True, + bonjour_name="bazinga", + bonjour_publisher="dns-sd", + ) + + class DummyDnsSdPublisher: + def __init__(self, service_name, service_types, port, filter_clients, on_debug): + self.service_name = service_name + self.service_types = service_types + self.port = port + self.filter_clients = filter_clients + + with patch("cli_anything.nslogger.core.listener.sys.platform", "darwin"), \ + patch("cli_anything.nslogger.core.listener._NativeBonjourPublisher") as native_publisher, \ + patch("cli_anything.nslogger.core.listener._DnsSdBonjourPublisher", DummyDnsSdPublisher): + publisher = listener._start_bonjour("192.168.10.1") + + assert isinstance(publisher, DummyDnsSdPublisher) + assert native_publisher.call_count == 0 + + def test_zeroconf_publisher_can_be_forced_on_macos(self): + listener = NSLoggerListener( + bonjour=True, + bonjour_name="bazinga", + bonjour_publisher="zeroconf", + ) + + class DummyZeroconfPublisher: + def __init__(self, service_name, service_types, port, local_ip, filter_clients): + self.service_name = service_name + self.service_types = service_types + self.port = port + self.local_ip = local_ip + self.filter_clients = filter_clients + + with patch("cli_anything.nslogger.core.listener.sys.platform", "darwin"), \ + patch("cli_anything.nslogger.core.listener._ZeroconfBonjourPublisher", DummyZeroconfPublisher), \ + patch("cli_anything.nslogger.core.listener._DnsSdBonjourPublisher") as dns_sd_publisher: + publisher = listener._start_bonjour("192.168.10.5") + + assert isinstance(publisher, DummyZeroconfPublisher) + assert publisher.local_ip == "192.168.10.5" + assert dns_sd_publisher.call_count == 0 + + def test_ssl_handshake_failure_is_silent(self): + class DummySSLContext: + def wrap_socket(self, conn, server_side=True): + raise OSError(22, "Invalid argument") + + class DummyConn: + def __init__(self): + self.closed = False + self.peeked = False + + def recv(self, n, flags=0): + if flags: + self.peeked = True + return b"\x16\x03\x01\x00\x2a" + return b"" + + def settimeout(self, timeout): + pass + + def close(self): + self.closed = True + + connect_calls = [] + disconnect_calls = [] + + listener = NSLoggerListener( + on_connect=lambda host, port: connect_calls.append((host, port)), + on_disconnect=lambda host, port: disconnect_calls.append((host, port)), + ) + listener._ssl_ctx = DummySSLContext() + conn = DummyConn() + + listener._handle_client(conn, ("127.0.0.1", 50000)) + + assert conn.closed is True + assert connect_calls == [] + assert disconnect_calls == [] + + def test_raw_connection_without_ssl_context_is_not_wrapped(self): + class DummyConn: + def __init__(self, payload: bytes): + self.payload = payload + self.closed = False + + def recv(self, n, flags=0): + if flags == getattr(__import__("socket"), "MSG_PEEK", 0): + return self.payload[:n] + return b"" + + def settimeout(self, timeout): + pass + + def close(self): + self.closed = True + + listener = NSLoggerListener(use_ssl=False) + conn = DummyConn(b"\x00\x00\x00\x10\x00") + + assert _looks_like_tls_client_hello(conn) is False + listener._handle_client(conn, ("127.0.0.1", 50000)) + assert conn.closed is True + + def test_tls_client_hello_is_detected(self): + class DummyConn: + def recv(self, n, flags=0): + if flags == getattr(__import__("socket"), "MSG_PEEK", 0): + return b"\x16\x03\x01\x00\x2a" + return b"" + + assert _looks_like_tls_client_hello(DummyConn()) is True + + def test_connection_classifier_distinguishes_tls_raw_and_empty(self): + class DummyConn: + def __init__(self, payload: bytes): + self.payload = payload + + def recv(self, n, flags=0): + return self.payload[:n] + + assert _classify_connection(DummyConn(b"\x16\x03\x01\x00\x2a"))[0] == "tls" + assert _classify_connection(DummyConn(b"\x00\x00\x00\x10\x00"))[0] == "raw" + assert _classify_connection(DummyConn(b""))[0] == "empty" + + def test_connection_classifier_distinguishes_timeout_from_empty_probe(self): + class DummyConn: + def recv(self, n, flags=0): + raise __import__("socket").timeout() + + assert _classify_connection(DummyConn())[0] == "timeout" + + def test_auto_ssl_context_accepts_plaintext_live_packet(self): + class DummySSLContext: + def wrap_socket(self, conn, server_side=True): + raise AssertionError("raw packet should not be TLS-wrapped") + + server, client = __import__("socket").socketpair() + messages = [] + listener = NSLoggerListener( + on_message=messages.append, + use_ssl=True, + allow_plaintext=True, + ) + listener._ssl_ctx = DummySSLContext() + thread = __import__("threading").Thread( + target=listener._handle_client, + args=(server, ("127.0.0.1", 50000)), + daemon=True, + ) + + thread.start() + client.sendall(encode_message(sequence=43, text="auto raw packet", tag="Network", level=1)) + client.close() + thread.join(timeout=2.0) + + assert len(messages) == 1 + assert messages[0].sequence == 43 + assert messages[0].text == "auto raw packet" + + def test_handle_client_parses_official_live_packet(self): + server, client = __import__("socket").socketpair() + messages = [] + listener = NSLoggerListener(on_message=messages.append) + thread = __import__("threading").Thread( + target=listener._handle_client, + args=(server, ("127.0.0.1", 50000)), + daemon=True, + ) + + thread.start() + client.sendall(encode_message(sequence=42, text="live packet", tag="Network", level=1)) + client.close() + thread.join(timeout=2.0) + + assert len(messages) == 1 + assert messages[0].sequence == 42 + assert messages[0].text == "live packet" + assert messages[0].tag == "Network" + + def test_handle_client_reports_parse_errors(self): + server, client = __import__("socket").socketpair() + errors = [] + listener = NSLoggerListener(on_parse_error=lambda host, port, raw, exc: errors.append((raw, exc))) + thread = __import__("threading").Thread( + target=listener._handle_client, + args=(server, ("127.0.0.1", 50000)), + daemon=True, + ) + + thread.start() + client.sendall(struct.pack(">I", 2) + b"\x00\x01") + client.close() + thread.join(timeout=2.0) + + assert len(errors) == 1 + assert errors[0][0] == b"\x00\x01" + + +class TestListenWaitingMessage: + def test_non_bonjour_message(self): + assert _listen_waiting_message(50000, False) == "Waiting for a client connection on port 50000โ€ฆ" + + def test_bonjour_message(self): + assert _listen_waiting_message(50000, True) == "[Bonjour] Waiting for an iOS client to connect on port 50000โ€ฆ" + + +class TestListenOutputFile: + def test_format_live_output_text(self): + assert _format_live_output_message(make_msg(text="live text"), "text").endswith("live text") + + def test_format_live_output_jsonl(self): + line = _format_live_output_message(make_msg(text="live json"), "jsonl") + data = json.loads(line) + + assert data["text"] == "live json" + + def test_open_live_output_file_creates_parent_and_replaces(self, tmp_path): + path = tmp_path / "nested" / "live.log" + + with _open_live_output_file(str(path), append=False) as f: + f.write("new\n") + + assert path.read_text(encoding="utf-8") == "new\n" + + def test_open_live_output_file_appends(self, tmp_path): + path = tmp_path / "nested" / "live.log" + + with _open_live_output_file(str(path), append=False) as f: + f.write("one\n") + with _open_live_output_file(str(path), append=True) as f: + f.write("two\n") + + assert path.read_text(encoding="utf-8") == "one\ntwo\n" diff --git a/nslogger/agent-harness/cli_anything/nslogger/tests/test_full_e2e.py b/nslogger/agent-harness/cli_anything/nslogger/tests/test_full_e2e.py new file mode 100644 index 000000000..eb086f33f --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/tests/test_full_e2e.py @@ -0,0 +1,408 @@ +"""End-to-end tests for cli-anything-nslogger using real files and subprocess.""" +from __future__ import annotations +import json +import os +import subprocess +import sys +import tempfile + +import pytest + +from cli_anything.nslogger.utils.generate import generate_sample_file +from cli_anything.nslogger.core.parser import parse_raw_file +from cli_anything.nslogger.core.filter import filter_messages +from cli_anything.nslogger.core.stats import compute_stats +from cli_anything.nslogger.core.exporter import export_messages + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _resolve_cli(name: str) -> list[str]: + """Return argv prefix for the CLI, respecting test-mode env var.""" + if os.environ.get("CLI_ANYTHING_FORCE_INSTALLED"): + return [name] + # When not installed, run via python -m + return [sys.executable, "-m", f"cli_anything.nslogger.nslogger_cli"] + + +def run_cli(*args, expect_ok=True) -> subprocess.CompletedProcess: + cmd = _resolve_cli("cli-anything-nslogger") + list(args) + result = subprocess.run(cmd, capture_output=True, text=True) + if expect_ok and result.returncode != 0: + pytest.fail(f"CLI failed:\nSTDOUT: {result.stdout}\nSTDERR: {result.stderr}") + return result + + +@pytest.fixture(scope="module") +def sample_file(tmp_path_factory): + path = str(tmp_path_factory.mktemp("data") / "sample.rawnsloggerdata") + generate_sample_file(path, count=30) + return path + + +# --------------------------------------------------------------------------- +# generate command +# --------------------------------------------------------------------------- + +class TestGenerateCommand: + def test_generate_creates_file(self, tmp_path): + out = str(tmp_path / "gen.rawnsloggerdata") + result = run_cli("generate", out, "--count", "10") + assert os.path.exists(out) + assert os.path.getsize(out) > 0 + + def test_generate_output_mentions_count(self, tmp_path): + out = str(tmp_path / "gen.rawnsloggerdata") + result = run_cli("generate", out, "--count", "10") + assert "10" in result.stdout + + def test_generate_parseable(self, tmp_path): + out = str(tmp_path / "gen.rawnsloggerdata") + run_cli("generate", out, "--count", "15") + msgs = list(parse_raw_file(out)) + assert len(msgs) >= 15 + + +# --------------------------------------------------------------------------- +# read command +# --------------------------------------------------------------------------- + +class TestReadCommand: + def test_read_outputs_messages(self, sample_file): + result = run_cli("read", sample_file) + lines = [l for l in result.stdout.strip().splitlines() if l] + assert len(lines) > 0 + + def test_read_json_valid(self, sample_file): + result = run_cli("read", sample_file, "--json") + data = json.loads(result.stdout) + assert isinstance(data, list) + assert len(data) > 0 + + def test_read_json_message_shape(self, sample_file): + result = run_cli("read", sample_file, "--json") + data = json.loads(result.stdout) + msg = data[0] + for key in ("sequence", "level", "level_name", "type", "text"): + assert key in msg + + def test_read_level_filter(self, sample_file): + result = run_cli("read", sample_file, "--level", "0", "--json") + data = json.loads(result.stdout) + assert all(m["level"] <= 0 for m in data) + + def test_read_limit(self, sample_file): + result = run_cli("read", sample_file, "--limit", "5", "--json") + data = json.loads(result.stdout) + assert len(data) <= 5 + + def test_read_search(self, sample_file): + result = run_cli("read", sample_file, "--search", "error", "--json") + data = json.loads(result.stdout) + for m in data: + assert "error" in m["text"].lower() or m["level"] == 0 + + +# --------------------------------------------------------------------------- +# filter command +# --------------------------------------------------------------------------- + +class TestFilterCommand: + def test_filter_by_level(self, sample_file): + result = run_cli("filter", sample_file, "--level", "1", "--json") + data = json.loads(result.stdout) + assert all(m["level"] <= 1 for m in data) + + def test_filter_no_results(self, sample_file, tmp_path): + # Generate file with no level-99 messages + out = str(tmp_path / "g.rawnsloggerdata") + generate_sample_file(out, count=5) + result = run_cli("filter", out, "--search", "XYZZY_NEVER_MATCHES_ANYTHING") + assert result.stdout.strip() == "" + + def test_filter_regex(self, sample_file): + result = run_cli("filter", sample_file, "--regex", r"(error|failed)", "--json") + data = json.loads(result.stdout) + import re + pattern = re.compile(r"(error|failed)", re.IGNORECASE) + for m in data: + assert pattern.search(m["text"]), f"No match in: {m['text']!r}" + + +# --------------------------------------------------------------------------- +# export command +# --------------------------------------------------------------------------- + +class TestExportCommand: + def test_export_text_stdout(self, sample_file): + result = run_cli("export", sample_file, "--format", "text") + assert len(result.stdout) > 0 + + def test_export_json_stdout(self, sample_file): + result = run_cli("export", sample_file, "--format", "json") + data = json.loads(result.stdout) + assert isinstance(data, list) + + def test_export_csv_stdout(self, sample_file): + result = run_cli("export", sample_file, "--format", "csv") + lines = result.stdout.strip().splitlines() + assert "sequence" in lines[0] + assert len(lines) > 1 + + def test_export_to_file(self, sample_file, tmp_path): + out = str(tmp_path / "export.json") + run_cli("export", sample_file, "--format", "json", "--output", out) + assert os.path.exists(out) + with open(out) as f: + data = json.load(f) + assert isinstance(data, list) + + def test_export_with_level_filter(self, sample_file): + result = run_cli("export", sample_file, "--format", "json", "--level", "1") + data = json.loads(result.stdout) + assert all(m["level"] <= 1 for m in data) + + +# --------------------------------------------------------------------------- +# stats command +# --------------------------------------------------------------------------- + +class TestStatsCommand: + def test_stats_text_output(self, sample_file): + result = run_cli("stats", sample_file) + assert "Total" in result.stdout or "total" in result.stdout.lower() + + def test_stats_json_output(self, sample_file): + result = run_cli("stats", sample_file, "--json") + data = json.loads(result.stdout) + assert "total" in data + assert data["total"] > 0 + + def test_stats_json_has_by_level(self, sample_file): + result = run_cli("stats", sample_file, "--json") + data = json.loads(result.stdout) + assert "by_level" in data + + def test_stats_json_has_by_tag(self, sample_file): + result = run_cli("stats", sample_file, "--json") + data = json.loads(result.stdout) + assert "by_tag" in data + + +# --------------------------------------------------------------------------- +# Full pipeline workflow test +# --------------------------------------------------------------------------- + +class TestWorkflow: + def test_generate_filter_export_pipeline(self, tmp_path): + """Generate โ†’ filter errors โ†’ export JSON.""" + log_file = str(tmp_path / "app.rawnsloggerdata") + generate_sample_file(log_file, count=50) + + msgs = list(parse_raw_file(log_file)) + assert len(msgs) > 0 + + errors = list(filter_messages(iter(msgs), max_level=0)) + assert isinstance(errors, list) + + out = export_messages(iter(errors), fmt="json") + data = json.loads(out) + for m in data: + assert m["level"] <= 0 + + def test_stats_on_generated_file(self, tmp_path): + log_file = str(tmp_path / "app.rawnsloggerdata") + generate_sample_file(log_file, count=40) + msgs = list(parse_raw_file(log_file)) + s = compute_stats(iter(msgs)) + assert s["total"] >= 40 + assert "by_level" in s + assert "by_tag" in s + + def test_cli_help_shows_commands(self): + result = run_cli("--help") + for cmd in ("read", "filter", "export", "stats", "listen", "generate"): + assert cmd in result.stdout + + +# --------------------------------------------------------------------------- +# TestCLISubprocess โ€” installed entrypoint tests +# --------------------------------------------------------------------------- + +class TestCLISubprocess: + def test_installed_cli_help(self): + cmd = _resolve_cli("cli-anything-nslogger") + ["--help"] + result = subprocess.run(cmd, capture_output=True, text=True) + assert result.returncode == 0 + assert "NSLogger" in result.stdout + + def test_installed_generate_and_read(self, tmp_path): + out = str(tmp_path / "sub.rawnsloggerdata") + run_cli("generate", out, "--count", "5") + result = run_cli("read", out, "--json") + data = json.loads(result.stdout) + assert len(data) >= 5 + + def test_installed_stats_json(self, tmp_path): + out = str(tmp_path / "sub2.rawnsloggerdata") + run_cli("generate", out, "--count", "8") + result = run_cli("stats", out, "--json") + data = json.loads(result.stdout) + assert data["total"] >= 8 + + def test_installed_export_csv(self, tmp_path): + out = str(tmp_path / "sub3.rawnsloggerdata") + run_cli("generate", out, "--count", "5") + result = run_cli("export", out, "--format", "csv") + assert "sequence" in result.stdout.splitlines()[0] + + +# --------------------------------------------------------------------------- +# tail command +# --------------------------------------------------------------------------- + +class TestTailCommand: + def test_tail_returns_last_n(self, tmp_path): + out = str(tmp_path / "tail.rawnsloggerdata") + run_cli("generate", out, "--count", "20") + result_all = run_cli("read", out, "--json") + result_tail = run_cli("tail", out, "--count", "5", "--json") + all_msgs = json.loads(result_all.stdout) + tail_msgs = json.loads(result_tail.stdout) + assert len(tail_msgs) == 5 + # Last 5 of all should match tail + assert [m["sequence"] for m in tail_msgs] == [m["sequence"] for m in all_msgs[-5:]] + + def test_tail_default_count(self, tmp_path): + out = str(tmp_path / "tail2.rawnsloggerdata") + run_cli("generate", out, "--count", "30") + result = run_cli("tail", out, "--json") + data = json.loads(result.stdout) + assert len(data) == 20 # default is 20 + + def test_tail_text_output(self, tmp_path): + out = str(tmp_path / "tail3.rawnsloggerdata") + run_cli("generate", out, "--count", "5") + result = run_cli("tail", out) + assert len(result.stdout.strip().splitlines()) > 0 + + +# --------------------------------------------------------------------------- +# clients command +# --------------------------------------------------------------------------- + +class TestClientsCommand: + def test_clients_json_output(self, sample_file): + result = run_cli("clients", sample_file, "--json") + data = json.loads(result.stdout) + assert isinstance(data, list) + # sample file should have at least one client_info + if data: + assert "client_name" in data[0] + + def test_clients_text_output(self, sample_file): + result = run_cli("clients", sample_file) + assert result.returncode == 0 # may output "No client_info" or real clients + + def test_clients_empty_file(self, tmp_path): + # File with only plain log messages, no client_info + from cli_anything.nslogger.utils.generate import encode_message + path = str(tmp_path / "no_clients.rawnsloggerdata") + with open(path, "wb") as f: + f.write(encode_message(sequence=0, text="plain log")) + result = run_cli("clients", path) + assert result.returncode == 0 + assert "No client_info" in result.stdout + + +# --------------------------------------------------------------------------- +# blocks command +# --------------------------------------------------------------------------- + +class TestBlocksCommand: + def test_blocks_text_output(self, sample_file): + result = run_cli("blocks", sample_file) + assert result.returncode == 0 + assert len(result.stdout.strip()) > 0 + + def test_blocks_json_output(self, sample_file): + result = run_cli("blocks", sample_file, "--json") + data = json.loads(result.stdout) + assert isinstance(data, list) + for entry in data: + assert "depth" in entry + assert "sequence" in entry + + def test_blocks_indent_applied(self, tmp_path): + from cli_anything.nslogger.utils.generate import encode_message + from cli_anything.nslogger.core.message import MSG_TYPE_BLOCK_START, MSG_TYPE_BLOCK_END + path = str(tmp_path / "blk.rawnsloggerdata") + with open(path, "wb") as f: + f.write(encode_message(sequence=0, text="before")) + f.write(encode_message(sequence=1, msg_type=MSG_TYPE_BLOCK_START, text="enter")) + f.write(encode_message(sequence=2, text="inside")) + f.write(encode_message(sequence=3, msg_type=MSG_TYPE_BLOCK_END, text="exit")) + result = run_cli("blocks", path, "--indent", "4") + lines = result.stdout.splitlines() + # "inside" should be indented (4 spaces) + inside_lines = [l for l in lines if "inside" in l] + assert inside_lines and inside_lines[0].startswith(" ") + + +# --------------------------------------------------------------------------- +# merge command +# --------------------------------------------------------------------------- + +class TestMergeCommand: + def test_merge_two_files(self, tmp_path): + a = str(tmp_path / "a.rawnsloggerdata") + b = str(tmp_path / "b.rawnsloggerdata") + run_cli("generate", a, "--count", "5") + run_cli("generate", b, "--count", "5") + result = run_cli("merge", a, b, "--format", "json") + data = json.loads(result.stdout) + assert len(data) >= 10 + + def test_merge_to_file(self, tmp_path): + a = str(tmp_path / "c.rawnsloggerdata") + b = str(tmp_path / "d.rawnsloggerdata") + out = str(tmp_path / "merged.json") + run_cli("generate", a, "--count", "5") + run_cli("generate", b, "--count", "5") + run_cli("merge", a, b, "--format", "json", "--output", out) + assert os.path.exists(out) + with open(out) as f: + data = json.load(f) + assert len(data) >= 10 + + def test_merge_csv_format(self, tmp_path): + a = str(tmp_path / "e.rawnsloggerdata") + run_cli("generate", a, "--count", "5") + result = run_cli("merge", a, "--format", "csv") + assert "sequence" in result.stdout.splitlines()[0] + + +# --------------------------------------------------------------------------- +# filter โ€” new options (time-range, seq-range) +# --------------------------------------------------------------------------- + +class TestFilterExtendedOptions: + def test_filter_from_seq(self, tmp_path): + out = str(tmp_path / "seqtest.rawnsloggerdata") + run_cli("generate", out, "--count", "10") + all_data = json.loads(run_cli("read", out, "--json").stdout) + mid_seq = all_data[5]["sequence"] + result = run_cli("filter", out, "--from-seq", str(mid_seq), "--json") + data = json.loads(result.stdout) + assert all(m["sequence"] >= mid_seq for m in data) + + def test_filter_to_seq(self, tmp_path): + out = str(tmp_path / "seqtest2.rawnsloggerdata") + run_cli("generate", out, "--count", "10") + all_data = json.loads(run_cli("read", out, "--json").stdout) + mid_seq = all_data[5]["sequence"] + result = run_cli("filter", out, "--to-seq", str(mid_seq), "--json") + data = json.loads(result.stdout) + assert all(m["sequence"] <= mid_seq for m in data) diff --git a/nslogger/agent-harness/cli_anything/nslogger/utils/__init__.py b/nslogger/agent-harness/cli_anything/nslogger/utils/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/nslogger/agent-harness/cli_anything/nslogger/utils/generate.py b/nslogger/agent-harness/cli_anything/nslogger/utils/generate.py new file mode 100644 index 000000000..31f0e67db --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/utils/generate.py @@ -0,0 +1,151 @@ +"""Generate sample .rawnsloggerdata files for testing.""" +from __future__ import annotations +import struct +import time +from typing import List, Optional + +from ..core.message import ( + PART_KEY_MESSAGE_TYPE, PART_KEY_TIMESTAMP_S, PART_KEY_TIMESTAMP_MS, + PART_KEY_THREAD_ID, PART_KEY_TAG, PART_KEY_LEVEL, PART_KEY_MESSAGE, + PART_KEY_IMAGE_WIDTH, PART_KEY_IMAGE_HEIGHT, PART_KEY_MESSAGE_SEQ, + PART_KEY_CLIENT_NAME, PART_KEY_CLIENT_VERSION, + PART_KEY_OS_NAME, PART_KEY_OS_VERSION, PART_KEY_CLIENT_MODEL, + PART_TYPE_STRING, PART_TYPE_INT16, PART_TYPE_INT32, +) + + +def _encode_string_part(key: int, value: str) -> bytes: + encoded = value.encode("utf-8") + return bytes([key, PART_TYPE_STRING]) + struct.pack(">I", len(encoded)) + encoded + + +def _encode_int32_part(key: int, value: int) -> bytes: + return bytes([key, PART_TYPE_INT32]) + struct.pack(">I", value & 0xFFFFFFFF) + + +def _encode_int16_part(key: int, value: int) -> bytes: + return bytes([key, PART_TYPE_INT16]) + struct.pack(">H", value & 0xFFFF) + + +def encode_message( + sequence: int, + msg_type: int = 0, + timestamp: Optional[float] = None, + thread_id: str = "main", + tag: str = "", + level: int = 2, + text: str = "", + client_name: str = "", + client_version: str = "", + os_name: str = "", + os_version: str = "", + machine: str = "", +) -> bytes: + if timestamp is None: + timestamp = time.time() + + parts = b"" + parts += _encode_int32_part(PART_KEY_MESSAGE_TYPE, msg_type) # message type + parts += _encode_int32_part(PART_KEY_TIMESTAMP_S, int(timestamp)) # timestamp seconds + ms = int((timestamp - int(timestamp)) * 1000) + parts += _encode_int16_part(PART_KEY_TIMESTAMP_MS, ms) # timestamp ms + if thread_id: + parts += _encode_string_part(PART_KEY_THREAD_ID, thread_id) + if tag: + parts += _encode_string_part(PART_KEY_TAG, tag) + parts += _encode_int16_part(PART_KEY_LEVEL, level) # level + parts += _encode_int32_part(PART_KEY_MESSAGE_SEQ, sequence) + if text: + parts += _encode_string_part(PART_KEY_MESSAGE, text) + if client_name: + parts += _encode_string_part(PART_KEY_CLIENT_NAME, client_name) + if client_version: + parts += _encode_string_part(PART_KEY_CLIENT_VERSION, client_version) + if os_name: + parts += _encode_string_part(PART_KEY_OS_NAME, os_name) + if os_version: + parts += _encode_string_part(PART_KEY_OS_VERSION, os_version) + if machine: + parts += _encode_string_part(PART_KEY_CLIENT_MODEL, machine) + + # Count parts actually encoded. Integer part sizes are implicit in the + # official NSLogger protocol, variable-size parts carry a 4-byte length. + part_count = 0 + offset = 0 + temp = parts + while offset < len(temp): + if offset + 2 > len(temp): + break + part_type = temp[offset + 1] + offset += 2 + if part_type == PART_TYPE_INT16: + offset += 2 + elif part_type == PART_TYPE_INT32: + offset += 4 + else: + if offset + 4 > len(temp): + break + part_len = struct.unpack(">I", temp[offset:offset + 4])[0] + offset += 4 + part_len + part_count += 1 + + body = struct.pack(">H", part_count) + parts + return struct.pack(">I", len(body)) + body + + +def generate_sample_file(path: str, count: int = 20): + """Write a sample .rawnsloggerdata file with synthetic messages.""" + import random + + tags = ["Network", "UI", "Database", "Auth", "Cache", ""] + levels = [0, 0, 1, 1, 2, 2, 2, 3, 3, 4] + threads = ["main", "main", "background", "network-queue", "io-queue"] + base_ts = time.time() - count + + messages_text = [ + "Starting application", + "Fetching user data from API", + "User authenticated successfully", + "Cache miss for key: user_profile", + "Database query took 45ms", + "Network request failed: timeout", + "Retry attempt 1/3", + "View did appear: HomeViewController", + "Decoding JSON response", + "Background sync completed", + "Memory warning received", + "Connection pool exhausted", + "SSL handshake completed", + "Pushing notification", + "Saving to CoreData", + "Failed to parse response body", + "Token refreshed successfully", + "WebSocket connected", + "User tapped login button", + "App did enter background", + ] + + with open(path, "wb") as f: + # Write client info message first + f.write(encode_message( + sequence=0, + msg_type=3, # client info + timestamp=base_ts, + client_name="SampleApp", + client_version="1.0.0", + os_name="iOS", + os_version="17.0", + machine="iPhone15,2", + )) + for i in range(1, count + 1): + ts = base_ts + i * 0.5 + random.uniform(0, 0.4) + text = messages_text[i % len(messages_text)] + f.write(encode_message( + sequence=i, + msg_type=0, + timestamp=ts, + thread_id=random.choice(threads), + tag=random.choice(tags), + level=random.choice(levels), + text=text, + )) diff --git a/nslogger/agent-harness/cli_anything/nslogger/utils/repl_skin.py b/nslogger/agent-harness/cli_anything/nslogger/utils/repl_skin.py new file mode 100644 index 000000000..bc1fb6d1d --- /dev/null +++ b/nslogger/agent-harness/cli_anything/nslogger/utils/repl_skin.py @@ -0,0 +1,567 @@ +"""cli-anything REPL Skin โ€” Unified terminal interface for all CLI harnesses. + +Copy this file into your CLI package at: + cli_anything//utils/repl_skin.py + +Usage: + from cli_anything..utils.repl_skin import ReplSkin + + skin = ReplSkin("shotcut", version="1.0.0") + skin.print_banner() # auto-detects repo-root or packaged SKILL.md + prompt_text = skin.prompt(project_name="my_video.mlt", modified=True) + skin.success("Project saved") + skin.error("File not found") + skin.warning("Unsaved changes") + skin.info("Processing 24 clips...") + skin.status("Track 1", "3 clips, 00:02:30") + skin.table(headers, rows) + skin.print_goodbye() +""" + +import os +import sys +from pathlib import Path + +# โ”€โ”€ ANSI color codes (no external deps for core styling) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +_RESET = "\033[0m" +_BOLD = "\033[1m" +_DIM = "\033[2m" +_ITALIC = "\033[3m" +_UNDERLINE = "\033[4m" + +# Brand colors +_CYAN = "\033[38;5;80m" # cli-anything brand cyan +_CYAN_BG = "\033[48;5;80m" +_WHITE = "\033[97m" +_GRAY = "\033[38;5;245m" +_DARK_GRAY = "\033[38;5;240m" +_LIGHT_GRAY = "\033[38;5;250m" + +# Software accent colors โ€” each software gets a unique accent +_ACCENT_COLORS = { + "gimp": "\033[38;5;214m", # warm orange + "blender": "\033[38;5;208m", # deep orange + "inkscape": "\033[38;5;39m", # bright blue + "audacity": "\033[38;5;33m", # navy blue + "libreoffice": "\033[38;5;40m", # green + "obs_studio": "\033[38;5;55m", # purple + "kdenlive": "\033[38;5;69m", # slate blue + "shotcut": "\033[38;5;35m", # teal green +} +_DEFAULT_ACCENT = "\033[38;5;75m" # default sky blue + +# Status colors +_GREEN = "\033[38;5;78m" +_YELLOW = "\033[38;5;220m" +_RED = "\033[38;5;196m" +_BLUE = "\033[38;5;75m" +_MAGENTA = "\033[38;5;176m" + +_SKILL_SOURCE_REPO = os.environ.get("CLI_ANYTHING_SKILL_REPO", "HKUDS/CLI-Anything") + +# โ”€โ”€ Brand icon โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +# The cli-anything icon: a small colored diamond/chevron mark +_ICON = f"{_CYAN}{_BOLD}โ—†{_RESET}" +_ICON_SMALL = f"{_CYAN}โ–ธ{_RESET}" + +# โ”€โ”€ Box drawing characters โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +_H_LINE = "โ”€" +_V_LINE = "โ”‚" +_TL = "โ•ญ" +_TR = "โ•ฎ" +_BL = "โ•ฐ" +_BR = "โ•ฏ" +_T_DOWN = "โ”ฌ" +_T_UP = "โ”ด" +_T_RIGHT = "โ”œ" +_T_LEFT = "โ”ค" +_CROSS = "โ”ผ" + + +def _strip_ansi(text: str) -> str: + """Remove ANSI escape codes for length calculation.""" + import re + return re.sub(r"\033\[[^m]*m", "", text) + + +def _visible_len(text: str) -> int: + """Get visible length of text (excluding ANSI codes).""" + return len(_strip_ansi(text)) + + +def _display_home_path(path: str) -> str: + """Display a path relative to the home directory when possible.""" + expanded = Path(path).expanduser().resolve() + home = Path.home().resolve() + try: + relative = expanded.relative_to(home) + return f"~/{relative.as_posix()}" + except ValueError: + return str(expanded) + + +class ReplSkin: + """Unified REPL skin for cli-anything CLIs. + + Provides consistent branding, prompts, and message formatting + across all CLI harnesses built with the cli-anything methodology. + """ + + def __init__(self, software: str, version: str = "1.0.0", + history_file: str | None = None, skill_path: str | None = None): + """Initialize the REPL skin. + + Args: + software: Software name (e.g., "gimp", "shotcut", "blender"). + version: CLI version string. + history_file: Path for persistent command history. + Defaults to ~/.cli-anything-/history + skill_path: Path to the SKILL.md file for agent discovery. + Auto-detected from the repo-root skills/ tree when present, + otherwise from the package's skills/ directory. + Displayed in banner for AI agents to know where to read skill info. + """ + self.software = software.lower().replace("-", "_") + self.display_name = software.replace("_", " ").title() + self.version = version + software_aliases = {"iterm2_ctl": "iterm2"} + self.skill_slug = software_aliases.get(self.software, self.software).replace("_", "-") + self.skill_id = f"cli-anything-{self.skill_slug}" + self.skill_install_cmd = ( + f"npx skills add {_SKILL_SOURCE_REPO} --skill {self.skill_id} -g -y" + ) + global_skill_root = Path( + os.environ.get("CLI_ANYTHING_GLOBAL_SKILLS_DIR", str(Path.home() / ".agents" / "skills")) + ).expanduser() + self.global_skill_path = str(global_skill_root / self.skill_id / "SKILL.md") + + # Prefer repo-root canonical skills//SKILL.md when running + # inside the CLI-Anything monorepo. Fall back to the packaged + # cli_anything//skills/SKILL.md for installed harnesses. + if skill_path is None: + package_skill = Path(__file__).resolve().parent.parent / "skills" / "SKILL.md" + repo_skill = None + for parent in Path(__file__).resolve().parents: + candidate = parent / "skills" / self.skill_id / "SKILL.md" + if candidate.is_file(): + repo_skill = candidate + break + if repo_skill and repo_skill.is_file(): + skill_path = str(repo_skill) + elif package_skill.is_file(): + skill_path = str(package_skill) + self.skill_path = skill_path + self.accent = _ACCENT_COLORS.get(self.software, _DEFAULT_ACCENT) + + # History file + if history_file is None: + hist_dir = Path.home() / f".cli-anything-{self.software}" + hist_dir.mkdir(parents=True, exist_ok=True) + self.history_file = str(hist_dir / "history") + else: + self.history_file = history_file + + # Detect terminal capabilities + self._color = self._detect_color_support() + + def _detect_color_support(self) -> bool: + """Check if terminal supports color.""" + if os.environ.get("NO_COLOR"): + return False + if os.environ.get("CLI_ANYTHING_NO_COLOR"): + return False + if not hasattr(sys.stdout, "isatty"): + return False + return sys.stdout.isatty() + + def _c(self, code: str, text: str) -> str: + """Apply color code if colors are supported.""" + if not self._color: + return text + return f"{code}{text}{_RESET}" + + # โ”€โ”€ Banner โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def print_banner(self): + """Print the startup banner with branding.""" + import textwrap + + inner = 72 + + def _box_line(content: str) -> str: + """Wrap content in box drawing, padding to inner width.""" + pad = inner - _visible_len(content) + vl = self._c(_DARK_GRAY, _V_LINE) + return f"{vl}{content}{' ' * max(0, pad)}{vl}" + + def _meta_lines(label: str, value: str) -> list[str]: + """Wrap a metadata line for the banner box.""" + icon = self._c(_MAGENTA, "โ—‡") + label_text = self._c(_DARK_GRAY, label) + prefix = f" {icon} {label_text} " + available = max(12, inner - _visible_len(prefix)) + wrapped = textwrap.wrap( + value, + width=available, + break_long_words=True, + break_on_hyphens=False, + ) or [""] + lines = [f"{prefix}{self._c(_LIGHT_GRAY, wrapped[0])}"] + continuation_prefix = " " * _visible_len(prefix) + for chunk in wrapped[1:]: + lines.append(f"{continuation_prefix}{self._c(_LIGHT_GRAY, chunk)}") + return lines + + top = self._c(_DARK_GRAY, f"{_TL}{_H_LINE * inner}{_TR}") + bot = self._c(_DARK_GRAY, f"{_BL}{_H_LINE * inner}{_BR}") + + # Title: โ—† cli-anything ยท Shotcut + icon = self._c(_CYAN + _BOLD, "โ—†") + brand = self._c(_CYAN + _BOLD, "cli-anything") + dot = self._c(_DARK_GRAY, "ยท") + name = self._c(self.accent + _BOLD, self.display_name) + title = f" {icon} {brand} {dot} {name}" + + ver = f" {self._c(_DARK_GRAY, f' v{self.version}')}" + tip = f" {self._c(_DARK_GRAY, ' Type help for commands, quit to exit')}" + empty = "" + + meta_lines: list[str] = [] + meta_lines.extend(_meta_lines("Install:", self.skill_install_cmd)) + meta_lines.extend(_meta_lines("Global skill:", _display_home_path(self.global_skill_path))) + print(top) + print(_box_line(title)) + print(_box_line(ver)) + for line in meta_lines: + print(_box_line(line)) + print(_box_line(empty)) + print(_box_line(tip)) + print(bot) + print() + + # โ”€โ”€ Prompt โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def prompt(self, project_name: str = "", modified: bool = False, + context: str = "") -> str: + """Build a styled prompt string for prompt_toolkit or input(). + + Args: + project_name: Current project name (empty if none open). + modified: Whether the project has unsaved changes. + context: Optional extra context to show in prompt. + + Returns: + Formatted prompt string. + """ + parts = [] + + # Icon + if self._color: + parts.append(f"{_CYAN}โ—†{_RESET} ") + else: + parts.append("> ") + + # Software name + parts.append(self._c(self.accent + _BOLD, self.software)) + + # Project context + if project_name or context: + ctx = context or project_name + mod = "*" if modified else "" + parts.append(f" {self._c(_DARK_GRAY, '[')}") + parts.append(self._c(_LIGHT_GRAY, f"{ctx}{mod}")) + parts.append(self._c(_DARK_GRAY, ']')) + + parts.append(self._c(_GRAY, " โฏ ")) + + return "".join(parts) + + def prompt_tokens(self, project_name: str = "", modified: bool = False, + context: str = ""): + """Build prompt_toolkit formatted text tokens for the prompt. + + Use with prompt_toolkit's FormattedText for proper ANSI handling. + + Returns: + list of (style, text) tuples for prompt_toolkit. + """ + accent_hex = _ANSI_256_TO_HEX.get(self.accent, "#5fafff") + tokens = [] + + tokens.append(("class:icon", "โ—† ")) + tokens.append(("class:software", self.software)) + + if project_name or context: + ctx = context or project_name + mod = "*" if modified else "" + tokens.append(("class:bracket", " [")) + tokens.append(("class:context", f"{ctx}{mod}")) + tokens.append(("class:bracket", "]")) + + tokens.append(("class:arrow", " โฏ ")) + + return tokens + + def get_prompt_style(self): + """Get a prompt_toolkit Style object matching the skin. + + Returns: + prompt_toolkit.styles.Style + """ + try: + from prompt_toolkit.styles import Style + except ImportError: + return None + + accent_hex = _ANSI_256_TO_HEX.get(self.accent, "#5fafff") + + return Style.from_dict({ + "icon": "#5fdfdf bold", # cyan brand color + "software": f"{accent_hex} bold", + "bracket": "#585858", + "context": "#bcbcbc", + "arrow": "#808080", + # Completion menu + "completion-menu.completion": "bg:#303030 #bcbcbc", + "completion-menu.completion.current": f"bg:{accent_hex} #000000", + "completion-menu.meta.completion": "bg:#303030 #808080", + "completion-menu.meta.completion.current": f"bg:{accent_hex} #000000", + # Auto-suggest + "auto-suggest": "#585858", + # Bottom toolbar + "bottom-toolbar": "bg:#1c1c1c #808080", + "bottom-toolbar.text": "#808080", + }) + + # โ”€โ”€ Messages โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def success(self, message: str): + """Print a success message with green checkmark.""" + icon = self._c(_GREEN + _BOLD, "โœ“") + print(f" {icon} {self._c(_GREEN, message)}") + + def error(self, message: str): + """Print an error message with red cross.""" + icon = self._c(_RED + _BOLD, "โœ—") + print(f" {icon} {self._c(_RED, message)}", file=sys.stderr) + + def warning(self, message: str): + """Print a warning message with yellow triangle.""" + icon = self._c(_YELLOW + _BOLD, "โš ") + print(f" {icon} {self._c(_YELLOW, message)}") + + def info(self, message: str): + """Print an info message with blue dot.""" + icon = self._c(_BLUE, "โ—") + print(f" {icon} {self._c(_LIGHT_GRAY, message)}") + + def hint(self, message: str): + """Print a subtle hint message.""" + print(f" {self._c(_DARK_GRAY, message)}") + + def section(self, title: str): + """Print a section header.""" + print() + print(f" {self._c(self.accent + _BOLD, title)}") + print(f" {self._c(_DARK_GRAY, _H_LINE * len(title))}") + + # โ”€โ”€ Status display โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def status(self, label: str, value: str): + """Print a key-value status line.""" + lbl = self._c(_GRAY, f" {label}:") + val = self._c(_WHITE, f" {value}") + print(f"{lbl}{val}") + + def status_block(self, items: dict[str, str], title: str = ""): + """Print a block of status key-value pairs. + + Args: + items: Dict of label -> value pairs. + title: Optional title for the block. + """ + if title: + self.section(title) + + max_key = max(len(k) for k in items) if items else 0 + for label, value in items.items(): + lbl = self._c(_GRAY, f" {label:<{max_key}}") + val = self._c(_WHITE, f" {value}") + print(f"{lbl}{val}") + + def progress(self, current: int, total: int, label: str = ""): + """Print a simple progress indicator. + + Args: + current: Current step number. + total: Total number of steps. + label: Optional label for the progress. + """ + pct = int(current / total * 100) if total > 0 else 0 + bar_width = 20 + filled = int(bar_width * current / total) if total > 0 else 0 + bar = "โ–ˆ" * filled + "โ–‘" * (bar_width - filled) + text = f" {self._c(_CYAN, bar)} {self._c(_GRAY, f'{pct:3d}%')}" + if label: + text += f" {self._c(_LIGHT_GRAY, label)}" + print(text) + + # โ”€โ”€ Table display โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def table(self, headers: list[str], rows: list[list[str]], + max_col_width: int = 40): + """Print a formatted table with box-drawing characters. + + Args: + headers: Column header strings. + rows: List of rows, each a list of cell strings. + max_col_width: Maximum column width before truncation. + """ + if not headers: + return + + # Calculate column widths + col_widths = [min(len(h), max_col_width) for h in headers] + for row in rows: + for i, cell in enumerate(row): + if i < len(col_widths): + col_widths[i] = min( + max(col_widths[i], len(str(cell))), max_col_width + ) + + def pad(text: str, width: int) -> str: + t = str(text)[:width] + return t + " " * (width - len(t)) + + # Header + header_cells = [ + self._c(_CYAN + _BOLD, pad(h, col_widths[i])) + for i, h in enumerate(headers) + ] + sep = self._c(_DARK_GRAY, f" {_V_LINE} ") + header_line = f" {sep.join(header_cells)}" + print(header_line) + + # Separator + sep_parts = [self._c(_DARK_GRAY, _H_LINE * w) for w in col_widths] + sep_line = self._c(_DARK_GRAY, f" {'โ”€โ”€โ”€'.join([_H_LINE * w for w in col_widths])}") + print(sep_line) + + # Rows + for row in rows: + cells = [] + for i, cell in enumerate(row): + if i < len(col_widths): + cells.append(self._c(_LIGHT_GRAY, pad(str(cell), col_widths[i]))) + row_sep = self._c(_DARK_GRAY, f" {_V_LINE} ") + print(f" {row_sep.join(cells)}") + + # โ”€โ”€ Help display โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def help(self, commands: dict[str, str]): + """Print a formatted help listing. + + Args: + commands: Dict of command -> description pairs. + """ + self.section("Commands") + max_cmd = max(len(c) for c in commands) if commands else 0 + for cmd, desc in commands.items(): + cmd_styled = self._c(self.accent, f" {cmd:<{max_cmd}}") + desc_styled = self._c(_GRAY, f" {desc}") + print(f"{cmd_styled}{desc_styled}") + print() + + # โ”€โ”€ Goodbye โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def print_goodbye(self): + """Print a styled goodbye message.""" + print(f"\n {_ICON_SMALL} {self._c(_GRAY, 'Goodbye!')}\n") + + # โ”€โ”€ Prompt toolkit session factory โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def create_prompt_session(self): + """Create a prompt_toolkit PromptSession with skin styling. + + Returns: + A configured PromptSession, or None if prompt_toolkit unavailable. + """ + try: + from prompt_toolkit import PromptSession + from prompt_toolkit.history import FileHistory + from prompt_toolkit.auto_suggest import AutoSuggestFromHistory + from prompt_toolkit.formatted_text import FormattedText + + style = self.get_prompt_style() + + session = PromptSession( + history=FileHistory(self.history_file), + auto_suggest=AutoSuggestFromHistory(), + style=style, + enable_history_search=True, + ) + return session + except ImportError: + return None + + def get_input(self, pt_session, project_name: str = "", + modified: bool = False, context: str = "") -> str: + """Get input from user using prompt_toolkit or fallback. + + Args: + pt_session: A prompt_toolkit PromptSession (or None). + project_name: Current project name. + modified: Whether project has unsaved changes. + context: Optional context string. + + Returns: + User input string (stripped). + """ + if pt_session is not None: + from prompt_toolkit.formatted_text import FormattedText + tokens = self.prompt_tokens(project_name, modified, context) + return pt_session.prompt(FormattedText(tokens)).strip() + else: + raw_prompt = self.prompt(project_name, modified, context) + return input(raw_prompt).strip() + + # โ”€โ”€ Toolbar builder โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + def bottom_toolbar(self, items: dict[str, str]): + """Create a bottom toolbar callback for prompt_toolkit. + + Args: + items: Dict of label -> value pairs to show in toolbar. + + Returns: + A callable that returns FormattedText for the toolbar. + """ + def toolbar(): + from prompt_toolkit.formatted_text import FormattedText + parts = [] + for i, (k, v) in enumerate(items.items()): + if i > 0: + parts.append(("class:bottom-toolbar.text", " โ”‚ ")) + parts.append(("class:bottom-toolbar.text", f" {k}: ")) + parts.append(("class:bottom-toolbar", v)) + return FormattedText(parts) + return toolbar + + +# โ”€โ”€ ANSI 256-color to hex mapping (for prompt_toolkit styles) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +_ANSI_256_TO_HEX = { + "\033[38;5;33m": "#0087ff", # audacity navy blue + "\033[38;5;35m": "#00af5f", # shotcut teal + "\033[38;5;39m": "#00afff", # inkscape bright blue + "\033[38;5;40m": "#00d700", # libreoffice green + "\033[38;5;55m": "#5f00af", # obs purple + "\033[38;5;69m": "#5f87ff", # kdenlive slate blue + "\033[38;5;75m": "#5fafff", # default sky blue + "\033[38;5;80m": "#5fd7d7", # brand cyan + "\033[38;5;208m": "#ff8700", # blender deep orange + "\033[38;5;214m": "#ffaf00", # gimp warm orange +} diff --git a/nslogger/agent-harness/pyproject.toml b/nslogger/agent-harness/pyproject.toml new file mode 100644 index 000000000..7b959378a --- /dev/null +++ b/nslogger/agent-harness/pyproject.toml @@ -0,0 +1,3 @@ +[build-system] +requires = ["setuptools>=64", "wheel"] +build-backend = "setuptools.build_meta" diff --git a/nslogger/agent-harness/setup.py b/nslogger/agent-harness/setup.py new file mode 100644 index 000000000..92710cfa1 --- /dev/null +++ b/nslogger/agent-harness/setup.py @@ -0,0 +1,31 @@ +"""PyPI setup for cli-anything-nslogger.""" +from setuptools import setup, find_namespace_packages + +setup( + name="cli-anything-nslogger", + version="0.1.0", + description="CLI harness for NSLogger โ€” read, filter, export, and monitor NSLogger log files", + long_description=open("cli_anything/nslogger/README.md", encoding="utf-8").read(), + long_description_content_type="text/markdown", + author="cli-anything", + python_requires=">=3.10", + packages=find_namespace_packages(include=["cli_anything.*"]), + package_data={ + "cli_anything.nslogger": ["helpers/*.swift"], + }, + install_requires=[ + "click>=8.0", + "rich>=13.0", + "zeroconf>=0.38.0", + ], + entry_points={ + "console_scripts": [ + "cli-anything-nslogger=cli_anything.nslogger.nslogger_cli:main", + ], + }, + classifiers=[ + "Programming Language :: Python :: 3", + "License :: OSI Approved :: MIT License", + "Operating System :: MacOS", + ], +) diff --git a/registry.json b/registry.json index ddcba5913..be2fba108 100644 --- a/registry.json +++ b/registry.json @@ -24,6 +24,25 @@ } ] }, + { + "name": "nslogger", + "display_name": "NSLogger", + "version": "0.1.0", + "description": "Capture, parse, filter, export, and mirror NSLogger iOS/macOS logs from the CLI", + "requires": "macOS for native Bonjour live capture; Python 3.10+", + "homepage": "https://github.com/fpillet/NSLogger", + "source_url": null, + "install_cmd": "pip install git+https://github.com/HKUDS/CLI-Anything.git#subdirectory=nslogger/agent-harness", + "entry_point": "cli-anything-nslogger", + "skill_md": "skills/cli-anything-nslogger/SKILL.md", + "category": "devops", + "contributors": [ + { + "name": "bazinga8023", + "url": "https://github.com/bazinga8023" + } + ] + }, { "name": "anygen", "display_name": "AnyGen", diff --git a/skills/cli-anything-nslogger/SKILL.md b/skills/cli-anything-nslogger/SKILL.md new file mode 100644 index 000000000..5ead7b0fb --- /dev/null +++ b/skills/cli-anything-nslogger/SKILL.md @@ -0,0 +1,188 @@ +--- +name: cli-anything-nslogger +description: CLI harness for NSLogger โ€” parse, filter, export, and monitor NSLogger log files (.rawnsloggerdata / .nsloggerdata) +version: 0.1.0 +install: pip install git+https://github.com/HKUDS/CLI-Anything.git#subdirectory=nslogger/agent-harness +binary: cli-anything-nslogger +tags: [logging, ios, macos, debugging, nslogger] +--- + +# cli-anything-nslogger + +A complete CLI harness for [NSLogger](https://github.com/fpillet/NSLogger), the macOS log viewer for iOS/macOS apps. + +## Installation + +```bash +cd nslogger/agent-harness +pip install -e . +# Verify +cli-anything-nslogger --help +``` + +## Command Reference + +### `generate` โ€” Create sample files for testing + +```bash +cli-anything-nslogger generate sample.rawnsloggerdata --count 50 +``` + +### `read` โ€” Display messages from a file + +```bash +# All messages +cli-anything-nslogger read session.rawnsloggerdata + +# Errors only (level 0) +cli-anything-nslogger read session.rawnsloggerdata --level 0 + +# Filter by tag and text search +cli-anything-nslogger read session.rawnsloggerdata --tag Network --search "timeout" + +# First 20 messages as JSON +cli-anything-nslogger read session.rawnsloggerdata --limit 20 --json +``` + +### `filter` โ€” Advanced filtering + +```bash +# Errors and warnings only +cli-anything-nslogger filter session.rawnsloggerdata --level 1 + +# By tag +cli-anything-nslogger filter session.rawnsloggerdata --tag Auth --tag Network + +# Regex search +cli-anything-nslogger filter session.rawnsloggerdata --regex "(timeout|failed|error)" + +# By thread +cli-anything-nslogger filter session.rawnsloggerdata --thread "main" + +# JSON output +cli-anything-nslogger filter session.rawnsloggerdata --level 0 --json +``` + +### `export` โ€” Export to text/JSON/CSV + +```bash +# JSON to stdout +cli-anything-nslogger export session.rawnsloggerdata --format json + +# CSV to file +cli-anything-nslogger export session.rawnsloggerdata --format csv --output logs.csv + +# Filtered text export +cli-anything-nslogger export session.rawnsloggerdata --format text --level 1 --tag Network +``` + +### `stats` โ€” Summary statistics + +```bash +# Human-readable summary +cli-anything-nslogger stats session.rawnsloggerdata + +# JSON for agent consumption +cli-anything-nslogger stats session.rawnsloggerdata --json +``` + +JSON output shape: +```json +{ + "total": 342, + "by_level": {"ERROR": 12, "WARNING": 34, "INFO": 200, "DEBUG": 96}, + "by_tag": {"Network": 89, "Auth": 45, "UI": 120}, + "by_thread": {"main": 200, "bg-queue": 142}, + "by_type": {"text": 340, "client_info": 1, "disconnect": 1}, + "clients": ["MyApp"], + "first_timestamp": "2024-01-01T10:00:00+00:00", + "last_timestamp": "2024-01-01T10:05:30+00:00", + "duration_seconds": 330.0 +} +``` + +### `listen` โ€” Receive live connections + +```bash +# Match the NSLogger.app GUI Bonjour behavior for iOS auto-discovery +cli-anything-nslogger listen --bonjour --name bazinga --debug + +# Mirror live logs to a text file while still printing stdout +cli-anything-nslogger listen --bonjour --name bazinga --output app.log + +# Write machine-readable JSON Lines +cli-anything-nslogger listen --bonjour --name bazinga --output app.jsonl --output-format jsonl + +# Direct TCP/TLS mode for manually configured clients +cli-anything-nslogger listen --port 50000 --ssl --debug + +# Show only errors while listening, output as JSON stream +cli-anything-nslogger listen --bonjour --name bazinga --level 0 --json + +# Run until Ctrl-C +cli-anything-nslogger listen --bonjour --name bazinga +``` + +Use Bonjour mode first for iOS apps because it matches the desktop NSLogger GUI: +the CLI publishes a native macOS `NetService` with `_nslogger-ssl._tcp` and +accepts TLS NSLogger frames. Use direct TCP/TLS only when the app is manually +configured with the Mac host and port. + +### `repl` โ€” Interactive Python REPL + +```bash +cli-anything-nslogger repl session.rawnsloggerdata +# Available: messages, parse_file, filter_messages, compute_stats, export_messages +``` + +## Log Levels + +| Value | Name | Use for | +|-------|---------|---------| +| 0 | ERROR | Unrecoverable failures | +| 1 | WARNING | Recoverable issues | +| 2 | INFO | Normal operation | +| 3 | DEBUG | Developer details | +| 4 | VERBOSE | Trace-level noise | + +## Message JSON Shape + +```json +{ + "sequence": 42, + "timestamp": "2024-01-01T10:01:23+00:00", + "timestamp_ms": 456, + "thread_id": "main", + "tag": "Network", + "level": 0, + "level_name": "ERROR", + "type": "text", + "text": "Connection timed out after 30s", + "image_width": 0, + "image_height": 0, + "client_name": "MyApp", + "client_version": "2.1.0", + "os_name": "iOS", + "os_version": "17.0", + "machine": "iPhone15,2" +} +``` + +## Agent Workflow Examples + +```bash +# 1. Inspect a captured crash session +cli-anything-nslogger stats crash.rawnsloggerdata --json + +# 2. Find all errors in the 5 minutes before crash +cli-anything-nslogger filter crash.rawnsloggerdata --level 0 --json + +# 3. Get network failures only +cli-anything-nslogger filter crash.rawnsloggerdata --tag Network --regex "fail|timeout|error" --json + +# 4. Export full log for offline analysis +cli-anything-nslogger export crash.rawnsloggerdata --format json --output crash_log.json + +# 5. Monitor an iOS app live and keep a local copy +cli-anything-nslogger listen --bonjour --name bazinga --output app.log --debug +```