Separate stage1 scan from stage2 verification

This commit is contained in:
kim
2026-03-30 15:53:53 +08:00
parent 7e74de3e08
commit cec0920f0c
9 changed files with 420 additions and 108 deletions
+30 -7
View File
@@ -17,7 +17,8 @@
- 支持从 `txt` 文件批量导入目录列表 - 支持从 `txt` 文件批量导入目录列表
- 使用 SQLite 落盘扫描状态,并基于 `path + size + mtime` 复用旧结果 - 使用 SQLite 落盘扫描状态,并基于 `path + size + mtime` 复用旧结果
- stage1 使用 `ffprobe` 做容器级快速校验 - stage1 使用 `ffprobe` 做容器级快速校验
- stage2 只对可疑文件使用 `ffmpeg` 做定向采样或完整确认 - `scan` 默认只执行 stage1,把需要详细复核的文件标记为 `pending_stage2`
- `verify-stage2` 只对 `pending_stage2` 文件使用 `ffmpeg` 做定向采样或完整确认
- 扫描过程中输出 discovery、stage1、stage2 的阶段进度;长耗时阶段会持续输出 heartbeat - 扫描过程中输出 discovery、stage1、stage2 的阶段进度;长耗时阶段会持续输出 heartbeat
- 输出 JSONL / CSV 报告 - 输出 JSONL / CSV 报告
- 支持按扫描结果做 `confirmed_broken` 文件删除预览和真实删除 - 支持按扫描结果做 `confirmed_broken` 文件删除预览和真实删除
@@ -93,10 +94,11 @@ uv run tgvideo-check --help
不要一开始就扫正式目录,先用一个小目录做最小验收: 不要一开始就扫正式目录,先用一个小目录做最小验收:
```powershell ```powershell
.\.venv\Scripts\tgvideo-check.exe scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\small-test.jsonl" --export-csv ".\small-test.csv" .\.venv\Scripts\tgvideo-check.exe scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\small-test-stage1.jsonl" --export-csv ".\small-test-stage1.csv"
.\.venv\Scripts\tgvideo-check.exe verify-stage2 --db ".\small-test.db" --export-jsonl ".\small-test-final.jsonl" --export-csv ".\small-test-final.csv"
``` ```
如果能正常看到 discovery/stage1/stage2 进度以及最终扫描摘要,并生成 `db/jsonl/csv` 三个文件,说明目标机器环境基本没问题。 如果能先正常看到 `scan` 的 discovery/stage1 进度和摘要,再正常执行 `verify-stage2` 并生成最终 `db/jsonl/csv` 文件,说明目标机器环境基本没问题。
### 基本扫描用法 ### 基本扫描用法
@@ -106,12 +108,24 @@ uv run tgvideo-check --help
tgvideo-check scan "D:\Exports\ChatA" "D:\Exports\ChatB" --profile tg-hdd tgvideo-check scan "D:\Exports\ChatA" "D:\Exports\ChatB" --profile tg-hdd
``` ```
第一轮完成后,如果要继续做第二轮详细复核:
```powershell
tgvideo-check verify-stage2 --db .\scan_state.db
```
#### 用 txt 批量导入目录 #### 用 txt 批量导入目录
```powershell ```powershell
tgvideo-check scan --paths-file .\dirs.txt --db .\scan_state.db --export-jsonl .\results.jsonl --export-csv .\results.csv tgvideo-check scan --paths-file .\dirs.txt --db .\scan_state.db --export-jsonl .\results.jsonl --export-csv .\results.csv
``` ```
完成第一轮后输出第二轮最终结果:
```powershell
tgvideo-check verify-stage2 --db .\scan_state.db --export-jsonl .\results.final.jsonl --export-csv .\results.final.csv
```
#### 混合使用 #### 混合使用
```powershell ```powershell
@@ -123,11 +137,17 @@ tgvideo-check scan "D:\extra-folder" --paths-file .\dirs.txt --db .\scan_state.d
- `--db`:指定 SQLite 状态库 - `--db`:指定 SQLite 状态库
- `--paths-file`:从 txt 文件读取目录列表,一行一个目录 - `--paths-file`:从 txt 文件读取目录列表,一行一个目录
- `--export-jsonl` / `--export-csv`:导出结果报告 - `--export-jsonl` / `--export-csv`:导出结果报告
- `--stage1-workers` / `--stage2-workers`:覆盖 profile 默认 worker 数 - `scan --stage1-workers`:覆盖第一轮 worker 数
- `verify-stage2 --stage2-workers`:覆盖第二轮 worker 数
- `--extensions`:覆盖默认视频扩展名列表 - `--extensions`:覆盖默认视频扩展名列表
- `--queue-limit`:限制单阶段批次大小,避免把待处理文件一次性堆进内存 - `--queue-limit`:限制单阶段批次大小,避免把待处理文件一次性堆进内存
- `--full-decode-confirm`:对 stage2 强制使用完整解码确认 - `verify-stage2 --full-decode-confirm`:对第二轮强制使用完整解码确认
- `--resume / --no-resume`:控制是否复用旧结果 - `scan --resume / --no-resume`:控制是否复用旧的第一轮结果
命令边界:
- `scan`:只做第一轮快速筛查
- `verify-stage2`:只做第二轮详细复核
默认扩展名列表: 默认扩展名列表:
@@ -167,12 +187,14 @@ D:\普通目录
推荐测试命令: 推荐测试命令:
```powershell ```powershell
tgvideo-check scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\small-test.jsonl" --export-csv ".\small-test.csv" tgvideo-check scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\small-test-stage1.jsonl" --export-csv ".\small-test-stage1.csv"
tgvideo-check verify-stage2 --db ".\small-test.db" --export-jsonl ".\small-test-final.jsonl" --export-csv ".\small-test-final.csv"
``` ```
重点看: 重点看:
- `confirmed_broken` 是否命中了你故意造坏的样本 - `confirmed_broken` 是否命中了你故意造坏的样本
- `pending_stage2` 数量是否符合你对高风险格式/可疑样本的预期
- `passed_stage2` 里是否混进了你明知已坏的文件 - `passed_stage2` 里是否混进了你明知已坏的文件
- 是否误伤了你确认正常的视频 - 是否误伤了你确认正常的视频
@@ -184,6 +206,7 @@ tgvideo-check scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\smal
|---|---|---| |---|---|---|
| `confirmed_broken` | 明确损坏 | 高置信度确认文件损坏,适合进入删除候选 | | `confirmed_broken` | 明确损坏 | 高置信度确认文件损坏,适合进入删除候选 |
| `suspicious` | 可疑 | 有异常迹象,但证据还不足以直接当作明确损坏 | | `suspicious` | 可疑 | 有异常迹象,但证据还不足以直接当作明确损坏 |
| `pending_stage2` | 待第二轮复核 | 第一轮认为值得进一步做详细验证 |
| `passed_stage1` | 第一轮通过 | 容器级快速校验通过,未进入第二轮 | | `passed_stage1` | 第一轮通过 | 容器级快速校验通过,未进入第二轮 |
| `passed_stage2` | 第二轮通过 | 进入第二轮后仍通过采样或确认校验 | | `passed_stage2` | 第二轮通过 | 进入第二轮后仍通过采样或确认校验 |
| `skipped` | 已跳过 | 预留状态 | | `skipped` | 已跳过 | 预留状态 |
+131 -64
View File
@@ -153,14 +153,11 @@ def scan(
db_path: Path = typer.Option(Path("scan_state.db"), "--db"), db_path: Path = typer.Option(Path("scan_state.db"), "--db"),
resume: bool = typer.Option(True, "--resume/--no-resume"), resume: bool = typer.Option(True, "--resume/--no-resume"),
stage1_workers: int | None = typer.Option(None, "--stage1-workers"), stage1_workers: int | None = typer.Option(None, "--stage1-workers"),
stage2_workers: int | None = typer.Option(None, "--stage2-workers"),
queue_limit: int = typer.Option(64, "--queue-limit"), queue_limit: int = typer.Option(64, "--queue-limit"),
sample_duration: int | None = typer.Option(None, "--sample-duration"),
full_decode_confirm: bool = typer.Option(False, "--full-decode-confirm"),
export_jsonl_path: Path | None = typer.Option(None, "--export-jsonl"), export_jsonl_path: Path | None = typer.Option(None, "--export-jsonl"),
export_csv_path: Path | None = typer.Option(None, "--export-csv"), export_csv_path: Path | None = typer.Option(None, "--export-csv"),
) -> None: ) -> None:
"""Scan directories for suspicious video files.""" """Run the first-pass ffprobe scan and leave stage2 pending."""
config = resolve_config( config = resolve_config(
ScanConfig( ScanConfig(
@@ -171,6 +168,53 @@ def scan(
db_path=db_path, db_path=db_path,
resume=resume, resume=resume,
stage1_workers=stage1_workers, stage1_workers=stage1_workers,
queue_limit=queue_limit,
export_jsonl=export_jsonl_path,
export_csv=export_csv_path,
)
)
rows = _execute_stage1_scan(config)
typer.echo(f"Scanned {len(rows)} file(s). {_format_summary_text(rows)}".strip())
@app.command("verify-stage2")
def verify_stage2(
db_path: Path = typer.Option(Path("scan_state.db"), "--db"),
run_id: int | None = typer.Option(None, "--run-id"),
profile: str | None = typer.Option(
None,
"--profile",
help="Override the stored profile when resolving stage2 defaults.",
),
stage2_workers: int | None = typer.Option(None, "--stage2-workers"),
queue_limit: int = typer.Option(64, "--queue-limit"),
sample_duration: int | None = typer.Option(None, "--sample-duration"),
full_decode_confirm: bool = typer.Option(False, "--full-decode-confirm"),
export_jsonl_path: Path | None = typer.Option(None, "--export-jsonl"),
export_csv_path: Path | None = typer.Option(None, "--export-csv"),
) -> None:
"""Run the second-pass ffmpeg verification for pending_stage2 files."""
store = ScanStore(db_path)
store.initialize()
selected_run_id = run_id or store.get_latest_run_id()
if selected_run_id is None:
raise typer.BadParameter(f"No scan runs found in database: {db_path}")
scan_run = store.get_run(selected_run_id)
if scan_run is None:
raise typer.BadParameter(f"Scan run does not exist: {selected_run_id}")
typer.echo(
f"Stage2 source: run_id={selected_run_id} profile={scan_run.profile}"
)
resolved_profile = profile or scan_run.profile
config = resolve_config(
ScanConfig(
roots=(),
profile=resolved_profile,
db_path=db_path,
stage2_workers=stage2_workers, stage2_workers=stage2_workers,
queue_limit=queue_limit, queue_limit=queue_limit,
sample_duration=sample_duration, sample_duration=sample_duration,
@@ -180,12 +224,14 @@ def scan(
) )
) )
rows = _execute_scan(config) rows = _execute_stage2_verification(
summary = Counter(row.final_status for row in rows) store,
summary_text = " ".join( run_id=selected_run_id,
f"{status}={summary[status]}" for status in sorted(summary.keys()) config=config,
)
typer.echo(
f"Verified stage2 for run {selected_run_id}. {_format_summary_text(rows)}".strip()
) )
typer.echo(f"Scanned {len(rows)} file(s). {summary_text}".strip())
@app.command() @app.command()
@@ -265,12 +311,11 @@ def main() -> None:
app() app()
def _execute_scan(config: ScanConfig) -> list[ReportRow]: def _execute_stage1_scan(config: ScanConfig) -> list[ReportRow]:
store = ScanStore(config.db_path) store = ScanStore(config.db_path)
store.initialize() store.initialize()
run_id = store.create_run(profile=config.profile, root_paths=config.roots) run_id = store.create_run(profile=config.profile, root_paths=config.roots)
reasons: dict[int, str] = {} reasons: dict[int, str] = {}
stage1_payloads: dict[int, dict[str, object]] = {}
progress = ScanProgressReporter() progress = ScanProgressReporter()
try: try:
@@ -287,6 +332,7 @@ def _execute_scan(config: ScanConfig) -> list[ReportRow]:
run_id, run_id,
candidates, candidates,
reuse_previous_results=config.resume, reuse_previous_results=config.resume,
reuse_stage2_results=False,
) )
progress.discovery_completed(stored_files) progress.discovery_completed(stored_files)
@@ -368,62 +414,9 @@ def _execute_scan(config: ScanConfig) -> list[ReportRow]:
reason_code=classification.reason_code, reason_code=classification.reason_code,
) )
reasons[stored_file.file_id] = classification.reason_code reasons[stored_file.file_id] = classification.reason_code
if payload is not None:
stage1_payloads[stored_file.file_id] = payload
if stage1_progress is not None: if stage1_progress is not None:
stage1_progress.record(classification.status) stage1_progress.record(classification.status)
stored_files = store.list_files(run_id)
stage2_files = select_pending_stage2_files(stored_files)
stage2_batches = build_stage_plan(
stage1_files=[],
stage2_files=stage2_files,
stage1_workers=config.stage1_workers or 1,
stage2_workers=config.stage2_workers or 1,
queue_limit=config.queue_limit,
)
stage2_progress = progress.stage_started(
stage_name="stage2",
total=len(stage2_files),
workers=config.stage2_workers or 1,
)
for batch in stage2_batches:
for stored_file, (stage2_status, reason_code, stage2_result) in iter_stage_batch_results(
batch,
lambda item: _run_stage2_for_file(
item.path,
payload=stage1_payloads.get(item.file_id)
or _parse_probe_payload(store.get_stage1_stdout_text(item.file_id) or "")
or _reload_probe_payload(item.path),
sample_duration=config.sample_duration or 3,
full_decode_confirm=config.full_decode_confirm,
),
heartbeat_seconds=PROGRESS_HEARTBEAT_SECONDS,
on_heartbeat=(
(lambda unfinished: stage2_progress.heartbeat(unfinished))
if stage2_progress is not None
else None
),
):
store.record_stage2_result(
stored_file.file_id,
mode=stage2_result["mode"],
sample_points_json=stage2_result["sample_points_json"],
exit_code=stage2_result["exit_code"],
stderr_text=stage2_result["stderr_text"],
reason_code=reason_code,
elapsed_ms=stage2_result["elapsed_ms"],
)
store.record_stage2_status(
stored_file.file_id,
stage2_status=stage2_status,
final_status=stage2_status,
reason_code=reason_code,
)
reasons[stored_file.file_id] = reason_code
if stage2_progress is not None:
stage2_progress.record(stage2_status)
rows = _build_report_rows(store.list_files(run_id), reasons) rows = _build_report_rows(store.list_files(run_id), reasons)
if config.export_jsonl is not None: if config.export_jsonl is not None:
export_jsonl(rows, config.export_jsonl) export_jsonl(rows, config.export_jsonl)
@@ -437,6 +430,73 @@ def _execute_scan(config: ScanConfig) -> list[ReportRow]:
raise raise
def _execute_stage2_verification(
store: ScanStore,
*,
run_id: int,
config: ScanConfig,
) -> list[ReportRow]:
reasons: dict[int, str] = {}
progress = ScanProgressReporter()
stored_files = store.list_files(run_id)
stage2_files = select_pending_stage2_files(stored_files)
stage2_batches = build_stage_plan(
stage1_files=[],
stage2_files=stage2_files,
stage1_workers=config.stage1_workers or 1,
stage2_workers=config.stage2_workers or 1,
queue_limit=config.queue_limit,
)
stage2_progress = progress.stage_started(
stage_name="stage2",
total=len(stage2_files),
workers=config.stage2_workers or 1,
)
for batch in stage2_batches:
for stored_file, (stage2_status, reason_code, stage2_result) in iter_stage_batch_results(
batch,
lambda item: _run_stage2_for_file(
item.path,
payload=_parse_probe_payload(store.get_stage1_stdout_text(item.file_id) or "")
or _reload_probe_payload(item.path),
sample_duration=config.sample_duration or 3,
full_decode_confirm=config.full_decode_confirm,
),
heartbeat_seconds=PROGRESS_HEARTBEAT_SECONDS,
on_heartbeat=(
(lambda unfinished: stage2_progress.heartbeat(unfinished))
if stage2_progress is not None
else None
),
):
store.record_stage2_result(
stored_file.file_id,
mode=stage2_result["mode"],
sample_points_json=stage2_result["sample_points_json"],
exit_code=stage2_result["exit_code"],
stderr_text=stage2_result["stderr_text"],
reason_code=reason_code,
elapsed_ms=stage2_result["elapsed_ms"],
)
store.record_stage2_status(
stored_file.file_id,
stage2_status=stage2_status,
final_status=stage2_status,
reason_code=reason_code,
)
reasons[stored_file.file_id] = reason_code
if stage2_progress is not None:
stage2_progress.record(stage2_status)
rows = _build_report_rows(store.list_files(run_id), reasons)
if config.export_jsonl is not None:
export_jsonl(rows, config.export_jsonl)
if config.export_csv is not None:
export_csv_summary(rows, config.export_csv)
return rows
def _build_report_rows( def _build_report_rows(
stored_files: list, stored_files: list,
reasons: dict[int, str], reasons: dict[int, str],
@@ -456,6 +516,13 @@ def _build_report_rows(
return rows return rows
def _format_summary_text(rows: list[ReportRow]) -> str:
summary = Counter(row.final_status for row in rows)
return " ".join(
f"{status}={summary[status]}" for status in sorted(summary.keys())
)
def _run_stage2_for_file( def _run_stage2_for_file(
path: Path, path: Path,
*, *,
+1
View File
@@ -18,6 +18,7 @@ class ReportRow:
FINAL_STATUS_ZH: dict[str, str] = { FINAL_STATUS_ZH: dict[str, str] = {
"confirmed_broken": "明确损坏", "confirmed_broken": "明确损坏",
"suspicious": "可疑", "suspicious": "可疑",
"pending_stage2": "待第二轮复核",
"passed_stage1": "第一轮通过", "passed_stage1": "第一轮通过",
"passed_stage2": "第二轮通过", "passed_stage2": "第二轮通过",
"skipped": "已跳过", "skipped": "已跳过",
+46 -6
View File
@@ -124,11 +124,17 @@ class ScanStore:
candidates: list[FileCandidate], candidates: list[FileCandidate],
*, *,
reuse_previous_results: bool = True, reuse_previous_results: bool = True,
reuse_stage2_results: bool = True,
) -> list[StoredFile]: ) -> list[StoredFile]:
with self._connect() as connection: with self._connect() as connection:
for candidate in candidates: for candidate in candidates:
previous = ( previous = (
self._find_previous_match(connection, run_id, candidate) self._find_previous_match(
connection,
run_id,
candidate,
allow_stage2_match=reuse_stage2_results,
)
if reuse_previous_results if reuse_previous_results
else None else None
) )
@@ -192,6 +198,7 @@ class ScanStore:
connection, connection,
previous_file_id=previous["file_id"], previous_file_id=previous["file_id"],
current_file_id=current_file_id, current_file_id=current_file_id,
copy_stage2_results=reuse_stage2_results,
) )
rows = connection.execute( rows = connection.execute(
@@ -218,6 +225,26 @@ class ScanStore:
return self._rows_to_stored_files(rows) return self._rows_to_stored_files(rows)
def get_run(self, run_id: int) -> ScanRun | None:
with self._connect() as connection:
row = connection.execute(
"""
SELECT id, profile, root_paths_json, status
FROM scan_runs
WHERE id = ?
""",
(run_id,),
).fetchone()
if row is None:
return None
return ScanRun(
run_id=int(row["id"]),
profile=row["profile"],
root_paths_json=row["root_paths_json"],
status=row["status"],
)
def record_stage1_status( def record_stage1_status(
self, self,
file_id: int, file_id: int,
@@ -400,17 +427,27 @@ class ScanStore:
connection: sqlite3.Connection, connection: sqlite3.Connection,
run_id: int, run_id: int,
candidate: FileCandidate, candidate: FileCandidate,
*,
allow_stage2_match: bool,
) -> sqlite3.Row | None: ) -> sqlite3.Row | None:
return connection.execute( query = """
"""
SELECT id AS file_id, run_id, stage1_status, stage2_status, final_status, reason_code SELECT id AS file_id, run_id, stage1_status, stage2_status, final_status, reason_code
FROM files FROM files
WHERE path = ? AND size = ? AND mtime_ns = ? AND run_id <> ? WHERE path = ? AND size = ? AND mtime_ns = ? AND run_id <> ?
"""
params: tuple[object, ...] = (
str(candidate.path),
candidate.size,
candidate.mtime_ns,
run_id,
)
if not allow_stage2_match:
query += " AND stage2_status IS NULL"
query += """
ORDER BY run_id DESC ORDER BY run_id DESC
LIMIT 1 LIMIT 1
""", """
(str(candidate.path), candidate.size, candidate.mtime_ns, run_id), return connection.execute(query, params).fetchone()
).fetchone()
def _copy_previous_results( def _copy_previous_results(
self, self,
@@ -418,6 +455,7 @@ class ScanStore:
*, *,
previous_file_id: int, previous_file_id: int,
current_file_id: int, current_file_id: int,
copy_stage2_results: bool,
) -> None: ) -> None:
connection.execute( connection.execute(
""" """
@@ -433,6 +471,8 @@ class ScanStore:
""", """,
(current_file_id, previous_file_id), (current_file_id, previous_file_id),
) )
if not copy_stage2_results:
return
connection.execute( connection.execute(
""" """
INSERT INTO stage2_results( INSERT INTO stage2_results(
+160 -19
View File
@@ -15,6 +15,7 @@ def test_help_shows_scan_command() -> None:
assert result.exit_code == 0 assert result.exit_code == 0
assert "scan" in result.stdout assert "scan" in result.stdout
assert "verify-stage2" in result.stdout
def test_scan_command_processes_stage1_results_and_exports_reports( def test_scan_command_processes_stage1_results_and_exports_reports(
@@ -74,7 +75,7 @@ def test_scan_command_processes_stage1_results_and_exports_reports(
assert csv_path.exists() assert csv_path.exists()
def test_scan_command_runs_stage2_for_pending_files(tmp_path, monkeypatch) -> None: def test_scan_command_leaves_stage2_pending_by_default(tmp_path, monkeypatch) -> None:
root = tmp_path / "videos" root = tmp_path / "videos"
root.mkdir() root.mkdir()
risky = root / "risky.mp4" risky = root / "risky.mp4"
@@ -90,19 +91,7 @@ def test_scan_command_runs_stage2_for_pending_files(tmp_path, monkeypatch) -> No
elapsed_ms=1, elapsed_ms=1,
) )
def fake_run_ffmpeg(path, **kwargs):
return FfmpegExecutionResult(
status="completed",
command=("ffmpeg", str(path)),
exit_code=0,
stdout_text="frame=10",
stderr_text="",
elapsed_ms=2,
reason_code=None,
)
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe) monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
result = runner.invoke( result = runner.invoke(
app, app,
@@ -116,14 +105,15 @@ def test_scan_command_runs_stage2_for_pending_files(tmp_path, monkeypatch) -> No
assert result.exit_code == 0 assert result.exit_code == 0
assert "Scanned 1 file(s)" in result.stdout assert "Scanned 1 file(s)" in result.stdout
assert "passed_stage2=1" in result.stdout assert "pending_stage2=1" in result.stdout
def test_scan_command_reports_discovery_and_stage_progress(tmp_path, monkeypatch) -> None: def test_verify_stage2_command_runs_pending_stage2_files(tmp_path, monkeypatch) -> None:
root = tmp_path / "videos" root = tmp_path / "videos"
root.mkdir() root.mkdir()
risky = root / "risky.mp4" risky = root / "risky.mp4"
risky.write_bytes(b"risky") risky.write_bytes(b"risky")
db_path = tmp_path / "scan_state.db"
def fake_run_ffprobe(path, **kwargs): def fake_run_ffprobe(path, **kwargs):
return ProbeExecutionResult( return ProbeExecutionResult(
@@ -149,6 +139,52 @@ def test_scan_command_reports_discovery_and_stage_progress(tmp_path, monkeypatch
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe) monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg) monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
scan_result = runner.invoke(
app,
[
"scan",
str(root),
"--db",
str(db_path),
],
)
assert scan_result.exit_code == 0
assert "pending_stage2=1" in scan_result.stdout
verify_result = runner.invoke(
app,
[
"verify-stage2",
"--db",
str(db_path),
],
)
assert verify_result.exit_code == 0
assert "Stage2 source: run_id=1 profile=tg-hdd" in verify_result.stdout
assert "Stage2 progress: 1/1 (100%) passed_stage2=1" in verify_result.stdout
assert "Verified stage2 for run 1. passed_stage2=1" in verify_result.stdout
def test_scan_command_reports_discovery_and_stage1_progress(tmp_path, monkeypatch) -> None:
root = tmp_path / "videos"
root.mkdir()
risky = root / "risky.mp4"
risky.write_bytes(b"risky")
def fake_run_ffprobe(path, **kwargs):
return ProbeExecutionResult(
status="completed",
command=("ffprobe", str(path)),
exit_code=0,
stdout_text='{"format":{"format_name":"mov,mp4,m4a,3gp,3g2,mj2","duration":"20.0"},"streams":[{"codec_type":"video"}]}',
stderr_text="",
elapsed_ms=1,
)
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
result = runner.invoke( result = runner.invoke(
app, app,
[ [
@@ -166,14 +202,16 @@ def test_scan_command_reports_discovery_and_stage_progress(tmp_path, monkeypatch
in result.stdout in result.stdout
) )
assert "Stage1 progress: 1/1 (100%) pending_stage2=1" in result.stdout assert "Stage1 progress: 1/1 (100%) pending_stage2=1" in result.stdout
assert "Stage2 progress: 1/1 (100%) passed_stage2=1" in result.stdout assert "Stage2 progress:" not in result.stdout
assert "Stage2: no pending files" not in result.stdout
def test_scan_command_reports_stage2_heartbeat_while_waiting(tmp_path, monkeypatch) -> None: def test_verify_stage2_command_reports_heartbeat_while_waiting(tmp_path, monkeypatch) -> None:
root = tmp_path / "videos" root = tmp_path / "videos"
root.mkdir() root.mkdir()
for index in range(60): for index in range(60):
(root / f"risky-{index:03d}.mp4").write_bytes(b"risky") (root / f"risky-{index:03d}.mp4").write_bytes(b"risky")
db_path = tmp_path / "scan_state.db"
def fake_run_ffprobe(path, **kwargs): def fake_run_ffprobe(path, **kwargs):
return ProbeExecutionResult( return ProbeExecutionResult(
@@ -201,13 +239,25 @@ def test_scan_command_reports_stage2_heartbeat_while_waiting(tmp_path, monkeypat
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe) monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg) monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
result = runner.invoke( scan_result = runner.invoke(
app, app,
[ [
"scan", "scan",
str(root), str(root),
"--db", "--db",
str(tmp_path / "scan_state.db"), str(db_path),
],
)
assert scan_result.exit_code == 0
assert "pending_stage2=60" in scan_result.stdout
result = runner.invoke(
app,
[
"verify-stage2",
"--db",
str(db_path),
], ],
) )
@@ -272,6 +322,97 @@ def test_scan_resume_preserves_reason_code_in_exported_report(tmp_path, monkeypa
assert '"reason_code":"ffprobe_exit_nonzero"' in second_jsonl.read_text(encoding="utf-8") assert '"reason_code":"ffprobe_exit_nonzero"' in second_jsonl.read_text(encoding="utf-8")
def test_scan_resume_does_not_reuse_previous_stage2_results(tmp_path, monkeypatch) -> None:
root = tmp_path / "videos"
root.mkdir()
risky = root / "risky.mp4"
risky.write_bytes(b"risky")
db_path = tmp_path / "scan_state.db"
def fake_run_ffprobe(path, **kwargs):
return ProbeExecutionResult(
status="completed",
command=("ffprobe", str(path)),
exit_code=0,
stdout_text='{"format":{"format_name":"mov,mp4,m4a,3gp,3g2,mj2","duration":"20.0"},"streams":[{"codec_type":"video"}]}',
stderr_text="",
elapsed_ms=1,
)
def fake_run_ffmpeg(path, **kwargs):
return FfmpegExecutionResult(
status="completed",
command=("ffmpeg", str(path)),
exit_code=0,
stdout_text="frame=10",
stderr_text="",
elapsed_ms=2,
reason_code=None,
)
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
first_result = runner.invoke(
app,
[
"scan",
str(root),
"--db",
str(db_path),
],
)
assert first_result.exit_code == 0
assert "pending_stage2=1" in first_result.stdout
verify_result = runner.invoke(
app,
[
"verify-stage2",
"--db",
str(db_path),
],
)
assert verify_result.exit_code == 0
assert "passed_stage2=1" in verify_result.stdout
def fail_ffmpeg(*args, **kwargs):
raise AssertionError("stage2 should not run during default scan resume")
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fail_ffmpeg)
probe_calls = 0
def second_run_ffprobe(path, **kwargs):
nonlocal probe_calls
probe_calls += 1
return ProbeExecutionResult(
status="completed",
command=("ffprobe", str(path)),
exit_code=0,
stdout_text='{"format":{"format_name":"mov,mp4,m4a,3gp,3g2,mj2","duration":"20.0"},"streams":[{"codec_type":"video"}]}',
stderr_text="",
elapsed_ms=1,
)
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", second_run_ffprobe)
second_result = runner.invoke(
app,
[
"scan",
str(root),
"--db",
str(db_path),
],
)
assert second_result.exit_code == 0
assert "pending_stage2=1" in second_result.stdout
assert "passed_stage2=1" not in second_result.stdout
assert probe_calls == 1
def test_scan_no_resume_forces_probe_execution(tmp_path, monkeypatch) -> None: def test_scan_no_resume_forces_probe_execution(tmp_path, monkeypatch) -> None:
root = tmp_path / "videos" root = tmp_path / "videos"
root.mkdir() root.mkdir()
+2 -6
View File
@@ -32,16 +32,12 @@ def test_scan_reuses_previous_results_without_rerunning_probes(
], ],
) )
assert first_result.exit_code == 0 assert first_result.exit_code == 0
assert "passed_stage2=1" in first_result.stdout assert "pending_stage2=1" in first_result.stdout
def fail_ffprobe(*args, **kwargs): def fail_ffprobe(*args, **kwargs):
raise AssertionError("run_ffprobe should not be called during resume") raise AssertionError("run_ffprobe should not be called during resume")
def fail_ffmpeg(*args, **kwargs):
raise AssertionError("run_ffmpeg_verification should not be called during resume")
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fail_ffprobe) monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fail_ffprobe)
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fail_ffmpeg)
second_result = runner.invoke( second_result = runner.invoke(
app, app,
@@ -54,7 +50,7 @@ def test_scan_reuses_previous_results_without_rerunning_probes(
) )
assert second_result.exit_code == 0 assert second_result.exit_code == 0
assert "passed_stage2=1" in second_result.stdout assert "pending_stage2=1" in second_result.stdout
def _generate_test_video(path: Path, *, duration: int) -> None: def _generate_test_video(path: Path, *, duration: int) -> None:
+15 -3
View File
@@ -14,7 +14,7 @@ HAS_FFMPEG = shutil.which("ffmpeg") is not None and shutil.which("ffprobe") is n
@pytest.mark.skipif(not HAS_FFMPEG, reason="ffmpeg/ffprobe not installed") @pytest.mark.skipif(not HAS_FFMPEG, reason="ffmpeg/ffprobe not installed")
def test_scan_detects_truncated_mkv_during_stage2(tmp_path: Path) -> None: def test_verify_stage2_detects_truncated_mkv(tmp_path: Path) -> None:
root = tmp_path / "videos" root = tmp_path / "videos"
root.mkdir() root.mkdir()
valid_path = root / "valid.mkv" valid_path = root / "valid.mkv"
@@ -27,7 +27,7 @@ def test_scan_detects_truncated_mkv_during_stage2(tmp_path: Path) -> None:
) )
db_path = tmp_path / "scan_state.db" db_path = tmp_path / "scan_state.db"
result = runner.invoke( scan_result = runner.invoke(
app, app,
[ [
"scan", "scan",
@@ -37,7 +37,19 @@ def test_scan_detects_truncated_mkv_during_stage2(tmp_path: Path) -> None:
], ],
) )
assert result.exit_code == 0 assert scan_result.exit_code == 0
assert "pending_stage2=2" in scan_result.stdout
verify_result = runner.invoke(
app,
[
"verify-stage2",
"--db",
str(db_path),
],
)
assert verify_result.exit_code == 0
store = ScanStore(db_path) store = ScanStore(db_path)
rows = store.list_files(1) rows = store.list_files(1)
-2
View File
@@ -43,8 +43,6 @@ def _run_scan(root: Path, db_path: Path, *, stage1_workers: int) -> float:
str(db_path), str(db_path),
"--stage1-workers", "--stage1-workers",
str(stage1_workers), str(stage1_workers),
"--stage2-workers",
"1",
], ],
) )
elapsed = time.perf_counter() - start elapsed = time.perf_counter() - start
+35 -1
View File
@@ -70,6 +70,41 @@ def test_upsert_files_reuses_previous_status_when_identity_matches(tmp_path: Pat
assert resumed.resumed_from_run_id == first_run assert resumed.resumed_from_run_id == first_run
def test_upsert_files_can_skip_reusing_previous_stage2_results(tmp_path: Path) -> None:
store = ScanStore(tmp_path / "scan_state.db")
store.initialize()
first_run = store.create_run(profile="tg-hdd", root_paths=(tmp_path,))
file_path = tmp_path / "clip.mp4"
candidate = make_candidate(file_path, order=0, size=12, mtime_ns=999)
[stored] = store.upsert_files(first_run, [candidate])
store.record_stage1_status(
stored.file_id,
stage1_status="pending_stage2",
final_status=None,
reason_code="high_risk_extension",
)
store.record_stage2_status(
stored.file_id,
stage2_status="passed_stage2",
final_status="passed_stage2",
reason_code="passed_stage2",
)
second_run = store.create_run(profile="tg-hdd", root_paths=(tmp_path,))
[resumed] = store.upsert_files(
second_run,
[candidate],
reuse_stage2_results=False,
)
assert resumed.stage1_status == "pending_stage1"
assert resumed.stage2_status is None
assert resumed.final_status is None
assert resumed.reason_code is None
assert resumed.resumed_from_run_id is None
def test_upsert_files_does_not_reuse_changed_file(tmp_path: Path) -> None: def test_upsert_files_does_not_reuse_changed_file(tmp_path: Path) -> None:
store = ScanStore(tmp_path / "scan_state.db") store = ScanStore(tmp_path / "scan_state.db")
store.initialize() store.initialize()
@@ -97,4 +132,3 @@ def test_upsert_files_does_not_reuse_changed_file(tmp_path: Path) -> None:
assert fresh.stage1_status == "pending_stage1" assert fresh.stage1_status == "pending_stage1"
assert fresh.final_status is None assert fresh.final_status is None
assert fresh.resumed_from_run_id is None assert fresh.resumed_from_run_id is None