Separate stage1 scan from stage2 verification
This commit is contained in:
@@ -17,7 +17,8 @@
|
|||||||
- 支持从 `txt` 文件批量导入目录列表
|
- 支持从 `txt` 文件批量导入目录列表
|
||||||
- 使用 SQLite 落盘扫描状态,并基于 `path + size + mtime` 复用旧结果
|
- 使用 SQLite 落盘扫描状态,并基于 `path + size + mtime` 复用旧结果
|
||||||
- stage1 使用 `ffprobe` 做容器级快速校验
|
- stage1 使用 `ffprobe` 做容器级快速校验
|
||||||
- stage2 只对可疑文件使用 `ffmpeg` 做定向采样或完整确认
|
- `scan` 默认只执行 stage1,把需要详细复核的文件标记为 `pending_stage2`
|
||||||
|
- `verify-stage2` 只对 `pending_stage2` 文件使用 `ffmpeg` 做定向采样或完整确认
|
||||||
- 扫描过程中输出 discovery、stage1、stage2 的阶段进度;长耗时阶段会持续输出 heartbeat
|
- 扫描过程中输出 discovery、stage1、stage2 的阶段进度;长耗时阶段会持续输出 heartbeat
|
||||||
- 输出 JSONL / CSV 报告
|
- 输出 JSONL / CSV 报告
|
||||||
- 支持按扫描结果做 `confirmed_broken` 文件删除预览和真实删除
|
- 支持按扫描结果做 `confirmed_broken` 文件删除预览和真实删除
|
||||||
@@ -93,10 +94,11 @@ uv run tgvideo-check --help
|
|||||||
不要一开始就扫正式目录,先用一个小目录做最小验收:
|
不要一开始就扫正式目录,先用一个小目录做最小验收:
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
.\.venv\Scripts\tgvideo-check.exe scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\small-test.jsonl" --export-csv ".\small-test.csv"
|
.\.venv\Scripts\tgvideo-check.exe scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\small-test-stage1.jsonl" --export-csv ".\small-test-stage1.csv"
|
||||||
|
.\.venv\Scripts\tgvideo-check.exe verify-stage2 --db ".\small-test.db" --export-jsonl ".\small-test-final.jsonl" --export-csv ".\small-test-final.csv"
|
||||||
```
|
```
|
||||||
|
|
||||||
如果能正常看到 discovery/stage1/stage2 进度以及最终扫描摘要,并生成 `db/jsonl/csv` 三个文件,说明目标机器环境基本没问题。
|
如果能先正常看到 `scan` 的 discovery/stage1 进度和摘要,再正常执行 `verify-stage2` 并生成最终 `db/jsonl/csv` 文件,说明目标机器环境基本没问题。
|
||||||
|
|
||||||
### 基本扫描用法
|
### 基本扫描用法
|
||||||
|
|
||||||
@@ -106,12 +108,24 @@ uv run tgvideo-check --help
|
|||||||
tgvideo-check scan "D:\Exports\ChatA" "D:\Exports\ChatB" --profile tg-hdd
|
tgvideo-check scan "D:\Exports\ChatA" "D:\Exports\ChatB" --profile tg-hdd
|
||||||
```
|
```
|
||||||
|
|
||||||
|
第一轮完成后,如果要继续做第二轮详细复核:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
tgvideo-check verify-stage2 --db .\scan_state.db
|
||||||
|
```
|
||||||
|
|
||||||
#### 用 txt 批量导入目录
|
#### 用 txt 批量导入目录
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
tgvideo-check scan --paths-file .\dirs.txt --db .\scan_state.db --export-jsonl .\results.jsonl --export-csv .\results.csv
|
tgvideo-check scan --paths-file .\dirs.txt --db .\scan_state.db --export-jsonl .\results.jsonl --export-csv .\results.csv
|
||||||
```
|
```
|
||||||
|
|
||||||
|
完成第一轮后输出第二轮最终结果:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
tgvideo-check verify-stage2 --db .\scan_state.db --export-jsonl .\results.final.jsonl --export-csv .\results.final.csv
|
||||||
|
```
|
||||||
|
|
||||||
#### 混合使用
|
#### 混合使用
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
@@ -123,11 +137,17 @@ tgvideo-check scan "D:\extra-folder" --paths-file .\dirs.txt --db .\scan_state.d
|
|||||||
- `--db`:指定 SQLite 状态库
|
- `--db`:指定 SQLite 状态库
|
||||||
- `--paths-file`:从 txt 文件读取目录列表,一行一个目录
|
- `--paths-file`:从 txt 文件读取目录列表,一行一个目录
|
||||||
- `--export-jsonl` / `--export-csv`:导出结果报告
|
- `--export-jsonl` / `--export-csv`:导出结果报告
|
||||||
- `--stage1-workers` / `--stage2-workers`:覆盖 profile 默认 worker 数
|
- `scan --stage1-workers`:覆盖第一轮 worker 数
|
||||||
|
- `verify-stage2 --stage2-workers`:覆盖第二轮 worker 数
|
||||||
- `--extensions`:覆盖默认视频扩展名列表
|
- `--extensions`:覆盖默认视频扩展名列表
|
||||||
- `--queue-limit`:限制单阶段批次大小,避免把待处理文件一次性堆进内存
|
- `--queue-limit`:限制单阶段批次大小,避免把待处理文件一次性堆进内存
|
||||||
- `--full-decode-confirm`:对 stage2 强制使用完整解码确认
|
- `verify-stage2 --full-decode-confirm`:对第二轮强制使用完整解码确认
|
||||||
- `--resume / --no-resume`:控制是否复用旧结果
|
- `scan --resume / --no-resume`:控制是否复用旧的第一轮结果
|
||||||
|
|
||||||
|
命令边界:
|
||||||
|
|
||||||
|
- `scan`:只做第一轮快速筛查
|
||||||
|
- `verify-stage2`:只做第二轮详细复核
|
||||||
|
|
||||||
默认扩展名列表:
|
默认扩展名列表:
|
||||||
|
|
||||||
@@ -167,12 +187,14 @@ D:\普通目录
|
|||||||
推荐测试命令:
|
推荐测试命令:
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
tgvideo-check scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\small-test.jsonl" --export-csv ".\small-test.csv"
|
tgvideo-check scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\small-test-stage1.jsonl" --export-csv ".\small-test-stage1.csv"
|
||||||
|
tgvideo-check verify-stage2 --db ".\small-test.db" --export-jsonl ".\small-test-final.jsonl" --export-csv ".\small-test-final.csv"
|
||||||
```
|
```
|
||||||
|
|
||||||
重点看:
|
重点看:
|
||||||
|
|
||||||
- `confirmed_broken` 是否命中了你故意造坏的样本
|
- `confirmed_broken` 是否命中了你故意造坏的样本
|
||||||
|
- `pending_stage2` 数量是否符合你对高风险格式/可疑样本的预期
|
||||||
- `passed_stage2` 里是否混进了你明知已坏的文件
|
- `passed_stage2` 里是否混进了你明知已坏的文件
|
||||||
- 是否误伤了你确认正常的视频
|
- 是否误伤了你确认正常的视频
|
||||||
|
|
||||||
@@ -184,6 +206,7 @@ tgvideo-check scan "D:\small-test" --db ".\small-test.db" --export-jsonl ".\smal
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `confirmed_broken` | 明确损坏 | 高置信度确认文件损坏,适合进入删除候选 |
|
| `confirmed_broken` | 明确损坏 | 高置信度确认文件损坏,适合进入删除候选 |
|
||||||
| `suspicious` | 可疑 | 有异常迹象,但证据还不足以直接当作明确损坏 |
|
| `suspicious` | 可疑 | 有异常迹象,但证据还不足以直接当作明确损坏 |
|
||||||
|
| `pending_stage2` | 待第二轮复核 | 第一轮认为值得进一步做详细验证 |
|
||||||
| `passed_stage1` | 第一轮通过 | 容器级快速校验通过,未进入第二轮 |
|
| `passed_stage1` | 第一轮通过 | 容器级快速校验通过,未进入第二轮 |
|
||||||
| `passed_stage2` | 第二轮通过 | 进入第二轮后仍通过采样或确认校验 |
|
| `passed_stage2` | 第二轮通过 | 进入第二轮后仍通过采样或确认校验 |
|
||||||
| `skipped` | 已跳过 | 预留状态 |
|
| `skipped` | 已跳过 | 预留状态 |
|
||||||
|
|||||||
+131
-64
@@ -153,14 +153,11 @@ def scan(
|
|||||||
db_path: Path = typer.Option(Path("scan_state.db"), "--db"),
|
db_path: Path = typer.Option(Path("scan_state.db"), "--db"),
|
||||||
resume: bool = typer.Option(True, "--resume/--no-resume"),
|
resume: bool = typer.Option(True, "--resume/--no-resume"),
|
||||||
stage1_workers: int | None = typer.Option(None, "--stage1-workers"),
|
stage1_workers: int | None = typer.Option(None, "--stage1-workers"),
|
||||||
stage2_workers: int | None = typer.Option(None, "--stage2-workers"),
|
|
||||||
queue_limit: int = typer.Option(64, "--queue-limit"),
|
queue_limit: int = typer.Option(64, "--queue-limit"),
|
||||||
sample_duration: int | None = typer.Option(None, "--sample-duration"),
|
|
||||||
full_decode_confirm: bool = typer.Option(False, "--full-decode-confirm"),
|
|
||||||
export_jsonl_path: Path | None = typer.Option(None, "--export-jsonl"),
|
export_jsonl_path: Path | None = typer.Option(None, "--export-jsonl"),
|
||||||
export_csv_path: Path | None = typer.Option(None, "--export-csv"),
|
export_csv_path: Path | None = typer.Option(None, "--export-csv"),
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Scan directories for suspicious video files."""
|
"""Run the first-pass ffprobe scan and leave stage2 pending."""
|
||||||
|
|
||||||
config = resolve_config(
|
config = resolve_config(
|
||||||
ScanConfig(
|
ScanConfig(
|
||||||
@@ -171,6 +168,53 @@ def scan(
|
|||||||
db_path=db_path,
|
db_path=db_path,
|
||||||
resume=resume,
|
resume=resume,
|
||||||
stage1_workers=stage1_workers,
|
stage1_workers=stage1_workers,
|
||||||
|
queue_limit=queue_limit,
|
||||||
|
export_jsonl=export_jsonl_path,
|
||||||
|
export_csv=export_csv_path,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = _execute_stage1_scan(config)
|
||||||
|
typer.echo(f"Scanned {len(rows)} file(s). {_format_summary_text(rows)}".strip())
|
||||||
|
|
||||||
|
|
||||||
|
@app.command("verify-stage2")
|
||||||
|
def verify_stage2(
|
||||||
|
db_path: Path = typer.Option(Path("scan_state.db"), "--db"),
|
||||||
|
run_id: int | None = typer.Option(None, "--run-id"),
|
||||||
|
profile: str | None = typer.Option(
|
||||||
|
None,
|
||||||
|
"--profile",
|
||||||
|
help="Override the stored profile when resolving stage2 defaults.",
|
||||||
|
),
|
||||||
|
stage2_workers: int | None = typer.Option(None, "--stage2-workers"),
|
||||||
|
queue_limit: int = typer.Option(64, "--queue-limit"),
|
||||||
|
sample_duration: int | None = typer.Option(None, "--sample-duration"),
|
||||||
|
full_decode_confirm: bool = typer.Option(False, "--full-decode-confirm"),
|
||||||
|
export_jsonl_path: Path | None = typer.Option(None, "--export-jsonl"),
|
||||||
|
export_csv_path: Path | None = typer.Option(None, "--export-csv"),
|
||||||
|
) -> None:
|
||||||
|
"""Run the second-pass ffmpeg verification for pending_stage2 files."""
|
||||||
|
|
||||||
|
store = ScanStore(db_path)
|
||||||
|
store.initialize()
|
||||||
|
selected_run_id = run_id or store.get_latest_run_id()
|
||||||
|
if selected_run_id is None:
|
||||||
|
raise typer.BadParameter(f"No scan runs found in database: {db_path}")
|
||||||
|
|
||||||
|
scan_run = store.get_run(selected_run_id)
|
||||||
|
if scan_run is None:
|
||||||
|
raise typer.BadParameter(f"Scan run does not exist: {selected_run_id}")
|
||||||
|
|
||||||
|
typer.echo(
|
||||||
|
f"Stage2 source: run_id={selected_run_id} profile={scan_run.profile}"
|
||||||
|
)
|
||||||
|
resolved_profile = profile or scan_run.profile
|
||||||
|
config = resolve_config(
|
||||||
|
ScanConfig(
|
||||||
|
roots=(),
|
||||||
|
profile=resolved_profile,
|
||||||
|
db_path=db_path,
|
||||||
stage2_workers=stage2_workers,
|
stage2_workers=stage2_workers,
|
||||||
queue_limit=queue_limit,
|
queue_limit=queue_limit,
|
||||||
sample_duration=sample_duration,
|
sample_duration=sample_duration,
|
||||||
@@ -180,12 +224,14 @@ def scan(
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
rows = _execute_scan(config)
|
rows = _execute_stage2_verification(
|
||||||
summary = Counter(row.final_status for row in rows)
|
store,
|
||||||
summary_text = " ".join(
|
run_id=selected_run_id,
|
||||||
f"{status}={summary[status]}" for status in sorted(summary.keys())
|
config=config,
|
||||||
|
)
|
||||||
|
typer.echo(
|
||||||
|
f"Verified stage2 for run {selected_run_id}. {_format_summary_text(rows)}".strip()
|
||||||
)
|
)
|
||||||
typer.echo(f"Scanned {len(rows)} file(s). {summary_text}".strip())
|
|
||||||
|
|
||||||
|
|
||||||
@app.command()
|
@app.command()
|
||||||
@@ -265,12 +311,11 @@ def main() -> None:
|
|||||||
app()
|
app()
|
||||||
|
|
||||||
|
|
||||||
def _execute_scan(config: ScanConfig) -> list[ReportRow]:
|
def _execute_stage1_scan(config: ScanConfig) -> list[ReportRow]:
|
||||||
store = ScanStore(config.db_path)
|
store = ScanStore(config.db_path)
|
||||||
store.initialize()
|
store.initialize()
|
||||||
run_id = store.create_run(profile=config.profile, root_paths=config.roots)
|
run_id = store.create_run(profile=config.profile, root_paths=config.roots)
|
||||||
reasons: dict[int, str] = {}
|
reasons: dict[int, str] = {}
|
||||||
stage1_payloads: dict[int, dict[str, object]] = {}
|
|
||||||
progress = ScanProgressReporter()
|
progress = ScanProgressReporter()
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -287,6 +332,7 @@ def _execute_scan(config: ScanConfig) -> list[ReportRow]:
|
|||||||
run_id,
|
run_id,
|
||||||
candidates,
|
candidates,
|
||||||
reuse_previous_results=config.resume,
|
reuse_previous_results=config.resume,
|
||||||
|
reuse_stage2_results=False,
|
||||||
)
|
)
|
||||||
progress.discovery_completed(stored_files)
|
progress.discovery_completed(stored_files)
|
||||||
|
|
||||||
@@ -368,62 +414,9 @@ def _execute_scan(config: ScanConfig) -> list[ReportRow]:
|
|||||||
reason_code=classification.reason_code,
|
reason_code=classification.reason_code,
|
||||||
)
|
)
|
||||||
reasons[stored_file.file_id] = classification.reason_code
|
reasons[stored_file.file_id] = classification.reason_code
|
||||||
if payload is not None:
|
|
||||||
stage1_payloads[stored_file.file_id] = payload
|
|
||||||
if stage1_progress is not None:
|
if stage1_progress is not None:
|
||||||
stage1_progress.record(classification.status)
|
stage1_progress.record(classification.status)
|
||||||
|
|
||||||
stored_files = store.list_files(run_id)
|
|
||||||
stage2_files = select_pending_stage2_files(stored_files)
|
|
||||||
stage2_batches = build_stage_plan(
|
|
||||||
stage1_files=[],
|
|
||||||
stage2_files=stage2_files,
|
|
||||||
stage1_workers=config.stage1_workers or 1,
|
|
||||||
stage2_workers=config.stage2_workers or 1,
|
|
||||||
queue_limit=config.queue_limit,
|
|
||||||
)
|
|
||||||
stage2_progress = progress.stage_started(
|
|
||||||
stage_name="stage2",
|
|
||||||
total=len(stage2_files),
|
|
||||||
workers=config.stage2_workers or 1,
|
|
||||||
)
|
|
||||||
for batch in stage2_batches:
|
|
||||||
for stored_file, (stage2_status, reason_code, stage2_result) in iter_stage_batch_results(
|
|
||||||
batch,
|
|
||||||
lambda item: _run_stage2_for_file(
|
|
||||||
item.path,
|
|
||||||
payload=stage1_payloads.get(item.file_id)
|
|
||||||
or _parse_probe_payload(store.get_stage1_stdout_text(item.file_id) or "")
|
|
||||||
or _reload_probe_payload(item.path),
|
|
||||||
sample_duration=config.sample_duration or 3,
|
|
||||||
full_decode_confirm=config.full_decode_confirm,
|
|
||||||
),
|
|
||||||
heartbeat_seconds=PROGRESS_HEARTBEAT_SECONDS,
|
|
||||||
on_heartbeat=(
|
|
||||||
(lambda unfinished: stage2_progress.heartbeat(unfinished))
|
|
||||||
if stage2_progress is not None
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
):
|
|
||||||
store.record_stage2_result(
|
|
||||||
stored_file.file_id,
|
|
||||||
mode=stage2_result["mode"],
|
|
||||||
sample_points_json=stage2_result["sample_points_json"],
|
|
||||||
exit_code=stage2_result["exit_code"],
|
|
||||||
stderr_text=stage2_result["stderr_text"],
|
|
||||||
reason_code=reason_code,
|
|
||||||
elapsed_ms=stage2_result["elapsed_ms"],
|
|
||||||
)
|
|
||||||
store.record_stage2_status(
|
|
||||||
stored_file.file_id,
|
|
||||||
stage2_status=stage2_status,
|
|
||||||
final_status=stage2_status,
|
|
||||||
reason_code=reason_code,
|
|
||||||
)
|
|
||||||
reasons[stored_file.file_id] = reason_code
|
|
||||||
if stage2_progress is not None:
|
|
||||||
stage2_progress.record(stage2_status)
|
|
||||||
|
|
||||||
rows = _build_report_rows(store.list_files(run_id), reasons)
|
rows = _build_report_rows(store.list_files(run_id), reasons)
|
||||||
if config.export_jsonl is not None:
|
if config.export_jsonl is not None:
|
||||||
export_jsonl(rows, config.export_jsonl)
|
export_jsonl(rows, config.export_jsonl)
|
||||||
@@ -437,6 +430,73 @@ def _execute_scan(config: ScanConfig) -> list[ReportRow]:
|
|||||||
raise
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
def _execute_stage2_verification(
|
||||||
|
store: ScanStore,
|
||||||
|
*,
|
||||||
|
run_id: int,
|
||||||
|
config: ScanConfig,
|
||||||
|
) -> list[ReportRow]:
|
||||||
|
reasons: dict[int, str] = {}
|
||||||
|
progress = ScanProgressReporter()
|
||||||
|
|
||||||
|
stored_files = store.list_files(run_id)
|
||||||
|
stage2_files = select_pending_stage2_files(stored_files)
|
||||||
|
stage2_batches = build_stage_plan(
|
||||||
|
stage1_files=[],
|
||||||
|
stage2_files=stage2_files,
|
||||||
|
stage1_workers=config.stage1_workers or 1,
|
||||||
|
stage2_workers=config.stage2_workers or 1,
|
||||||
|
queue_limit=config.queue_limit,
|
||||||
|
)
|
||||||
|
stage2_progress = progress.stage_started(
|
||||||
|
stage_name="stage2",
|
||||||
|
total=len(stage2_files),
|
||||||
|
workers=config.stage2_workers or 1,
|
||||||
|
)
|
||||||
|
for batch in stage2_batches:
|
||||||
|
for stored_file, (stage2_status, reason_code, stage2_result) in iter_stage_batch_results(
|
||||||
|
batch,
|
||||||
|
lambda item: _run_stage2_for_file(
|
||||||
|
item.path,
|
||||||
|
payload=_parse_probe_payload(store.get_stage1_stdout_text(item.file_id) or "")
|
||||||
|
or _reload_probe_payload(item.path),
|
||||||
|
sample_duration=config.sample_duration or 3,
|
||||||
|
full_decode_confirm=config.full_decode_confirm,
|
||||||
|
),
|
||||||
|
heartbeat_seconds=PROGRESS_HEARTBEAT_SECONDS,
|
||||||
|
on_heartbeat=(
|
||||||
|
(lambda unfinished: stage2_progress.heartbeat(unfinished))
|
||||||
|
if stage2_progress is not None
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
):
|
||||||
|
store.record_stage2_result(
|
||||||
|
stored_file.file_id,
|
||||||
|
mode=stage2_result["mode"],
|
||||||
|
sample_points_json=stage2_result["sample_points_json"],
|
||||||
|
exit_code=stage2_result["exit_code"],
|
||||||
|
stderr_text=stage2_result["stderr_text"],
|
||||||
|
reason_code=reason_code,
|
||||||
|
elapsed_ms=stage2_result["elapsed_ms"],
|
||||||
|
)
|
||||||
|
store.record_stage2_status(
|
||||||
|
stored_file.file_id,
|
||||||
|
stage2_status=stage2_status,
|
||||||
|
final_status=stage2_status,
|
||||||
|
reason_code=reason_code,
|
||||||
|
)
|
||||||
|
reasons[stored_file.file_id] = reason_code
|
||||||
|
if stage2_progress is not None:
|
||||||
|
stage2_progress.record(stage2_status)
|
||||||
|
|
||||||
|
rows = _build_report_rows(store.list_files(run_id), reasons)
|
||||||
|
if config.export_jsonl is not None:
|
||||||
|
export_jsonl(rows, config.export_jsonl)
|
||||||
|
if config.export_csv is not None:
|
||||||
|
export_csv_summary(rows, config.export_csv)
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
def _build_report_rows(
|
def _build_report_rows(
|
||||||
stored_files: list,
|
stored_files: list,
|
||||||
reasons: dict[int, str],
|
reasons: dict[int, str],
|
||||||
@@ -456,6 +516,13 @@ def _build_report_rows(
|
|||||||
return rows
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _format_summary_text(rows: list[ReportRow]) -> str:
|
||||||
|
summary = Counter(row.final_status for row in rows)
|
||||||
|
return " ".join(
|
||||||
|
f"{status}={summary[status]}" for status in sorted(summary.keys())
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _run_stage2_for_file(
|
def _run_stage2_for_file(
|
||||||
path: Path,
|
path: Path,
|
||||||
*,
|
*,
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ class ReportRow:
|
|||||||
FINAL_STATUS_ZH: dict[str, str] = {
|
FINAL_STATUS_ZH: dict[str, str] = {
|
||||||
"confirmed_broken": "明确损坏",
|
"confirmed_broken": "明确损坏",
|
||||||
"suspicious": "可疑",
|
"suspicious": "可疑",
|
||||||
|
"pending_stage2": "待第二轮复核",
|
||||||
"passed_stage1": "第一轮通过",
|
"passed_stage1": "第一轮通过",
|
||||||
"passed_stage2": "第二轮通过",
|
"passed_stage2": "第二轮通过",
|
||||||
"skipped": "已跳过",
|
"skipped": "已跳过",
|
||||||
|
|||||||
@@ -124,11 +124,17 @@ class ScanStore:
|
|||||||
candidates: list[FileCandidate],
|
candidates: list[FileCandidate],
|
||||||
*,
|
*,
|
||||||
reuse_previous_results: bool = True,
|
reuse_previous_results: bool = True,
|
||||||
|
reuse_stage2_results: bool = True,
|
||||||
) -> list[StoredFile]:
|
) -> list[StoredFile]:
|
||||||
with self._connect() as connection:
|
with self._connect() as connection:
|
||||||
for candidate in candidates:
|
for candidate in candidates:
|
||||||
previous = (
|
previous = (
|
||||||
self._find_previous_match(connection, run_id, candidate)
|
self._find_previous_match(
|
||||||
|
connection,
|
||||||
|
run_id,
|
||||||
|
candidate,
|
||||||
|
allow_stage2_match=reuse_stage2_results,
|
||||||
|
)
|
||||||
if reuse_previous_results
|
if reuse_previous_results
|
||||||
else None
|
else None
|
||||||
)
|
)
|
||||||
@@ -192,6 +198,7 @@ class ScanStore:
|
|||||||
connection,
|
connection,
|
||||||
previous_file_id=previous["file_id"],
|
previous_file_id=previous["file_id"],
|
||||||
current_file_id=current_file_id,
|
current_file_id=current_file_id,
|
||||||
|
copy_stage2_results=reuse_stage2_results,
|
||||||
)
|
)
|
||||||
|
|
||||||
rows = connection.execute(
|
rows = connection.execute(
|
||||||
@@ -218,6 +225,26 @@ class ScanStore:
|
|||||||
|
|
||||||
return self._rows_to_stored_files(rows)
|
return self._rows_to_stored_files(rows)
|
||||||
|
|
||||||
|
def get_run(self, run_id: int) -> ScanRun | None:
|
||||||
|
with self._connect() as connection:
|
||||||
|
row = connection.execute(
|
||||||
|
"""
|
||||||
|
SELECT id, profile, root_paths_json, status
|
||||||
|
FROM scan_runs
|
||||||
|
WHERE id = ?
|
||||||
|
""",
|
||||||
|
(run_id,),
|
||||||
|
).fetchone()
|
||||||
|
|
||||||
|
if row is None:
|
||||||
|
return None
|
||||||
|
return ScanRun(
|
||||||
|
run_id=int(row["id"]),
|
||||||
|
profile=row["profile"],
|
||||||
|
root_paths_json=row["root_paths_json"],
|
||||||
|
status=row["status"],
|
||||||
|
)
|
||||||
|
|
||||||
def record_stage1_status(
|
def record_stage1_status(
|
||||||
self,
|
self,
|
||||||
file_id: int,
|
file_id: int,
|
||||||
@@ -400,17 +427,27 @@ class ScanStore:
|
|||||||
connection: sqlite3.Connection,
|
connection: sqlite3.Connection,
|
||||||
run_id: int,
|
run_id: int,
|
||||||
candidate: FileCandidate,
|
candidate: FileCandidate,
|
||||||
|
*,
|
||||||
|
allow_stage2_match: bool,
|
||||||
) -> sqlite3.Row | None:
|
) -> sqlite3.Row | None:
|
||||||
return connection.execute(
|
query = """
|
||||||
"""
|
|
||||||
SELECT id AS file_id, run_id, stage1_status, stage2_status, final_status, reason_code
|
SELECT id AS file_id, run_id, stage1_status, stage2_status, final_status, reason_code
|
||||||
FROM files
|
FROM files
|
||||||
WHERE path = ? AND size = ? AND mtime_ns = ? AND run_id <> ?
|
WHERE path = ? AND size = ? AND mtime_ns = ? AND run_id <> ?
|
||||||
|
"""
|
||||||
|
params: tuple[object, ...] = (
|
||||||
|
str(candidate.path),
|
||||||
|
candidate.size,
|
||||||
|
candidate.mtime_ns,
|
||||||
|
run_id,
|
||||||
|
)
|
||||||
|
if not allow_stage2_match:
|
||||||
|
query += " AND stage2_status IS NULL"
|
||||||
|
query += """
|
||||||
ORDER BY run_id DESC
|
ORDER BY run_id DESC
|
||||||
LIMIT 1
|
LIMIT 1
|
||||||
""",
|
"""
|
||||||
(str(candidate.path), candidate.size, candidate.mtime_ns, run_id),
|
return connection.execute(query, params).fetchone()
|
||||||
).fetchone()
|
|
||||||
|
|
||||||
def _copy_previous_results(
|
def _copy_previous_results(
|
||||||
self,
|
self,
|
||||||
@@ -418,6 +455,7 @@ class ScanStore:
|
|||||||
*,
|
*,
|
||||||
previous_file_id: int,
|
previous_file_id: int,
|
||||||
current_file_id: int,
|
current_file_id: int,
|
||||||
|
copy_stage2_results: bool,
|
||||||
) -> None:
|
) -> None:
|
||||||
connection.execute(
|
connection.execute(
|
||||||
"""
|
"""
|
||||||
@@ -433,6 +471,8 @@ class ScanStore:
|
|||||||
""",
|
""",
|
||||||
(current_file_id, previous_file_id),
|
(current_file_id, previous_file_id),
|
||||||
)
|
)
|
||||||
|
if not copy_stage2_results:
|
||||||
|
return
|
||||||
connection.execute(
|
connection.execute(
|
||||||
"""
|
"""
|
||||||
INSERT INTO stage2_results(
|
INSERT INTO stage2_results(
|
||||||
|
|||||||
+160
-19
@@ -15,6 +15,7 @@ def test_help_shows_scan_command() -> None:
|
|||||||
|
|
||||||
assert result.exit_code == 0
|
assert result.exit_code == 0
|
||||||
assert "scan" in result.stdout
|
assert "scan" in result.stdout
|
||||||
|
assert "verify-stage2" in result.stdout
|
||||||
|
|
||||||
|
|
||||||
def test_scan_command_processes_stage1_results_and_exports_reports(
|
def test_scan_command_processes_stage1_results_and_exports_reports(
|
||||||
@@ -74,7 +75,7 @@ def test_scan_command_processes_stage1_results_and_exports_reports(
|
|||||||
assert csv_path.exists()
|
assert csv_path.exists()
|
||||||
|
|
||||||
|
|
||||||
def test_scan_command_runs_stage2_for_pending_files(tmp_path, monkeypatch) -> None:
|
def test_scan_command_leaves_stage2_pending_by_default(tmp_path, monkeypatch) -> None:
|
||||||
root = tmp_path / "videos"
|
root = tmp_path / "videos"
|
||||||
root.mkdir()
|
root.mkdir()
|
||||||
risky = root / "risky.mp4"
|
risky = root / "risky.mp4"
|
||||||
@@ -90,19 +91,7 @@ def test_scan_command_runs_stage2_for_pending_files(tmp_path, monkeypatch) -> No
|
|||||||
elapsed_ms=1,
|
elapsed_ms=1,
|
||||||
)
|
)
|
||||||
|
|
||||||
def fake_run_ffmpeg(path, **kwargs):
|
|
||||||
return FfmpegExecutionResult(
|
|
||||||
status="completed",
|
|
||||||
command=("ffmpeg", str(path)),
|
|
||||||
exit_code=0,
|
|
||||||
stdout_text="frame=10",
|
|
||||||
stderr_text="",
|
|
||||||
elapsed_ms=2,
|
|
||||||
reason_code=None,
|
|
||||||
)
|
|
||||||
|
|
||||||
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
|
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
|
||||||
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
|
|
||||||
|
|
||||||
result = runner.invoke(
|
result = runner.invoke(
|
||||||
app,
|
app,
|
||||||
@@ -116,14 +105,15 @@ def test_scan_command_runs_stage2_for_pending_files(tmp_path, monkeypatch) -> No
|
|||||||
|
|
||||||
assert result.exit_code == 0
|
assert result.exit_code == 0
|
||||||
assert "Scanned 1 file(s)" in result.stdout
|
assert "Scanned 1 file(s)" in result.stdout
|
||||||
assert "passed_stage2=1" in result.stdout
|
assert "pending_stage2=1" in result.stdout
|
||||||
|
|
||||||
|
|
||||||
def test_scan_command_reports_discovery_and_stage_progress(tmp_path, monkeypatch) -> None:
|
def test_verify_stage2_command_runs_pending_stage2_files(tmp_path, monkeypatch) -> None:
|
||||||
root = tmp_path / "videos"
|
root = tmp_path / "videos"
|
||||||
root.mkdir()
|
root.mkdir()
|
||||||
risky = root / "risky.mp4"
|
risky = root / "risky.mp4"
|
||||||
risky.write_bytes(b"risky")
|
risky.write_bytes(b"risky")
|
||||||
|
db_path = tmp_path / "scan_state.db"
|
||||||
|
|
||||||
def fake_run_ffprobe(path, **kwargs):
|
def fake_run_ffprobe(path, **kwargs):
|
||||||
return ProbeExecutionResult(
|
return ProbeExecutionResult(
|
||||||
@@ -149,6 +139,52 @@ def test_scan_command_reports_discovery_and_stage_progress(tmp_path, monkeypatch
|
|||||||
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
|
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
|
||||||
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
|
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
|
||||||
|
|
||||||
|
scan_result = runner.invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"scan",
|
||||||
|
str(root),
|
||||||
|
"--db",
|
||||||
|
str(db_path),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert scan_result.exit_code == 0
|
||||||
|
assert "pending_stage2=1" in scan_result.stdout
|
||||||
|
|
||||||
|
verify_result = runner.invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"verify-stage2",
|
||||||
|
"--db",
|
||||||
|
str(db_path),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert verify_result.exit_code == 0
|
||||||
|
assert "Stage2 source: run_id=1 profile=tg-hdd" in verify_result.stdout
|
||||||
|
assert "Stage2 progress: 1/1 (100%) passed_stage2=1" in verify_result.stdout
|
||||||
|
assert "Verified stage2 for run 1. passed_stage2=1" in verify_result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def test_scan_command_reports_discovery_and_stage1_progress(tmp_path, monkeypatch) -> None:
|
||||||
|
root = tmp_path / "videos"
|
||||||
|
root.mkdir()
|
||||||
|
risky = root / "risky.mp4"
|
||||||
|
risky.write_bytes(b"risky")
|
||||||
|
|
||||||
|
def fake_run_ffprobe(path, **kwargs):
|
||||||
|
return ProbeExecutionResult(
|
||||||
|
status="completed",
|
||||||
|
command=("ffprobe", str(path)),
|
||||||
|
exit_code=0,
|
||||||
|
stdout_text='{"format":{"format_name":"mov,mp4,m4a,3gp,3g2,mj2","duration":"20.0"},"streams":[{"codec_type":"video"}]}',
|
||||||
|
stderr_text="",
|
||||||
|
elapsed_ms=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
|
||||||
|
|
||||||
result = runner.invoke(
|
result = runner.invoke(
|
||||||
app,
|
app,
|
||||||
[
|
[
|
||||||
@@ -166,14 +202,16 @@ def test_scan_command_reports_discovery_and_stage_progress(tmp_path, monkeypatch
|
|||||||
in result.stdout
|
in result.stdout
|
||||||
)
|
)
|
||||||
assert "Stage1 progress: 1/1 (100%) pending_stage2=1" in result.stdout
|
assert "Stage1 progress: 1/1 (100%) pending_stage2=1" in result.stdout
|
||||||
assert "Stage2 progress: 1/1 (100%) passed_stage2=1" in result.stdout
|
assert "Stage2 progress:" not in result.stdout
|
||||||
|
assert "Stage2: no pending files" not in result.stdout
|
||||||
|
|
||||||
|
|
||||||
def test_scan_command_reports_stage2_heartbeat_while_waiting(tmp_path, monkeypatch) -> None:
|
def test_verify_stage2_command_reports_heartbeat_while_waiting(tmp_path, monkeypatch) -> None:
|
||||||
root = tmp_path / "videos"
|
root = tmp_path / "videos"
|
||||||
root.mkdir()
|
root.mkdir()
|
||||||
for index in range(60):
|
for index in range(60):
|
||||||
(root / f"risky-{index:03d}.mp4").write_bytes(b"risky")
|
(root / f"risky-{index:03d}.mp4").write_bytes(b"risky")
|
||||||
|
db_path = tmp_path / "scan_state.db"
|
||||||
|
|
||||||
def fake_run_ffprobe(path, **kwargs):
|
def fake_run_ffprobe(path, **kwargs):
|
||||||
return ProbeExecutionResult(
|
return ProbeExecutionResult(
|
||||||
@@ -201,13 +239,25 @@ def test_scan_command_reports_stage2_heartbeat_while_waiting(tmp_path, monkeypat
|
|||||||
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
|
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
|
||||||
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
|
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
|
||||||
|
|
||||||
result = runner.invoke(
|
scan_result = runner.invoke(
|
||||||
app,
|
app,
|
||||||
[
|
[
|
||||||
"scan",
|
"scan",
|
||||||
str(root),
|
str(root),
|
||||||
"--db",
|
"--db",
|
||||||
str(tmp_path / "scan_state.db"),
|
str(db_path),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert scan_result.exit_code == 0
|
||||||
|
assert "pending_stage2=60" in scan_result.stdout
|
||||||
|
|
||||||
|
result = runner.invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"verify-stage2",
|
||||||
|
"--db",
|
||||||
|
str(db_path),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -272,6 +322,97 @@ def test_scan_resume_preserves_reason_code_in_exported_report(tmp_path, monkeypa
|
|||||||
assert '"reason_code":"ffprobe_exit_nonzero"' in second_jsonl.read_text(encoding="utf-8")
|
assert '"reason_code":"ffprobe_exit_nonzero"' in second_jsonl.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def test_scan_resume_does_not_reuse_previous_stage2_results(tmp_path, monkeypatch) -> None:
|
||||||
|
root = tmp_path / "videos"
|
||||||
|
root.mkdir()
|
||||||
|
risky = root / "risky.mp4"
|
||||||
|
risky.write_bytes(b"risky")
|
||||||
|
db_path = tmp_path / "scan_state.db"
|
||||||
|
|
||||||
|
def fake_run_ffprobe(path, **kwargs):
|
||||||
|
return ProbeExecutionResult(
|
||||||
|
status="completed",
|
||||||
|
command=("ffprobe", str(path)),
|
||||||
|
exit_code=0,
|
||||||
|
stdout_text='{"format":{"format_name":"mov,mp4,m4a,3gp,3g2,mj2","duration":"20.0"},"streams":[{"codec_type":"video"}]}',
|
||||||
|
stderr_text="",
|
||||||
|
elapsed_ms=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_run_ffmpeg(path, **kwargs):
|
||||||
|
return FfmpegExecutionResult(
|
||||||
|
status="completed",
|
||||||
|
command=("ffmpeg", str(path)),
|
||||||
|
exit_code=0,
|
||||||
|
stdout_text="frame=10",
|
||||||
|
stderr_text="",
|
||||||
|
elapsed_ms=2,
|
||||||
|
reason_code=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fake_run_ffprobe)
|
||||||
|
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fake_run_ffmpeg)
|
||||||
|
|
||||||
|
first_result = runner.invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"scan",
|
||||||
|
str(root),
|
||||||
|
"--db",
|
||||||
|
str(db_path),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
assert first_result.exit_code == 0
|
||||||
|
assert "pending_stage2=1" in first_result.stdout
|
||||||
|
|
||||||
|
verify_result = runner.invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"verify-stage2",
|
||||||
|
"--db",
|
||||||
|
str(db_path),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
assert verify_result.exit_code == 0
|
||||||
|
assert "passed_stage2=1" in verify_result.stdout
|
||||||
|
|
||||||
|
def fail_ffmpeg(*args, **kwargs):
|
||||||
|
raise AssertionError("stage2 should not run during default scan resume")
|
||||||
|
|
||||||
|
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fail_ffmpeg)
|
||||||
|
|
||||||
|
probe_calls = 0
|
||||||
|
|
||||||
|
def second_run_ffprobe(path, **kwargs):
|
||||||
|
nonlocal probe_calls
|
||||||
|
probe_calls += 1
|
||||||
|
return ProbeExecutionResult(
|
||||||
|
status="completed",
|
||||||
|
command=("ffprobe", str(path)),
|
||||||
|
exit_code=0,
|
||||||
|
stdout_text='{"format":{"format_name":"mov,mp4,m4a,3gp,3g2,mj2","duration":"20.0"},"streams":[{"codec_type":"video"}]}',
|
||||||
|
stderr_text="",
|
||||||
|
elapsed_ms=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", second_run_ffprobe)
|
||||||
|
|
||||||
|
second_result = runner.invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"scan",
|
||||||
|
str(root),
|
||||||
|
"--db",
|
||||||
|
str(db_path),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert second_result.exit_code == 0
|
||||||
|
assert "pending_stage2=1" in second_result.stdout
|
||||||
|
assert "passed_stage2=1" not in second_result.stdout
|
||||||
|
assert probe_calls == 1
|
||||||
|
|
||||||
|
|
||||||
def test_scan_no_resume_forces_probe_execution(tmp_path, monkeypatch) -> None:
|
def test_scan_no_resume_forces_probe_execution(tmp_path, monkeypatch) -> None:
|
||||||
root = tmp_path / "videos"
|
root = tmp_path / "videos"
|
||||||
root.mkdir()
|
root.mkdir()
|
||||||
|
|||||||
@@ -32,16 +32,12 @@ def test_scan_reuses_previous_results_without_rerunning_probes(
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
assert first_result.exit_code == 0
|
assert first_result.exit_code == 0
|
||||||
assert "passed_stage2=1" in first_result.stdout
|
assert "pending_stage2=1" in first_result.stdout
|
||||||
|
|
||||||
def fail_ffprobe(*args, **kwargs):
|
def fail_ffprobe(*args, **kwargs):
|
||||||
raise AssertionError("run_ffprobe should not be called during resume")
|
raise AssertionError("run_ffprobe should not be called during resume")
|
||||||
|
|
||||||
def fail_ffmpeg(*args, **kwargs):
|
|
||||||
raise AssertionError("run_ffmpeg_verification should not be called during resume")
|
|
||||||
|
|
||||||
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fail_ffprobe)
|
monkeypatch.setattr("tgvideo_check.cli.run_ffprobe", fail_ffprobe)
|
||||||
monkeypatch.setattr("tgvideo_check.cli.run_ffmpeg_verification", fail_ffmpeg)
|
|
||||||
|
|
||||||
second_result = runner.invoke(
|
second_result = runner.invoke(
|
||||||
app,
|
app,
|
||||||
@@ -54,7 +50,7 @@ def test_scan_reuses_previous_results_without_rerunning_probes(
|
|||||||
)
|
)
|
||||||
|
|
||||||
assert second_result.exit_code == 0
|
assert second_result.exit_code == 0
|
||||||
assert "passed_stage2=1" in second_result.stdout
|
assert "pending_stage2=1" in second_result.stdout
|
||||||
|
|
||||||
|
|
||||||
def _generate_test_video(path: Path, *, duration: int) -> None:
|
def _generate_test_video(path: Path, *, duration: int) -> None:
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ HAS_FFMPEG = shutil.which("ffmpeg") is not None and shutil.which("ffprobe") is n
|
|||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(not HAS_FFMPEG, reason="ffmpeg/ffprobe not installed")
|
@pytest.mark.skipif(not HAS_FFMPEG, reason="ffmpeg/ffprobe not installed")
|
||||||
def test_scan_detects_truncated_mkv_during_stage2(tmp_path: Path) -> None:
|
def test_verify_stage2_detects_truncated_mkv(tmp_path: Path) -> None:
|
||||||
root = tmp_path / "videos"
|
root = tmp_path / "videos"
|
||||||
root.mkdir()
|
root.mkdir()
|
||||||
valid_path = root / "valid.mkv"
|
valid_path = root / "valid.mkv"
|
||||||
@@ -27,7 +27,7 @@ def test_scan_detects_truncated_mkv_during_stage2(tmp_path: Path) -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
db_path = tmp_path / "scan_state.db"
|
db_path = tmp_path / "scan_state.db"
|
||||||
result = runner.invoke(
|
scan_result = runner.invoke(
|
||||||
app,
|
app,
|
||||||
[
|
[
|
||||||
"scan",
|
"scan",
|
||||||
@@ -37,7 +37,19 @@ def test_scan_detects_truncated_mkv_during_stage2(tmp_path: Path) -> None:
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
assert result.exit_code == 0
|
assert scan_result.exit_code == 0
|
||||||
|
assert "pending_stage2=2" in scan_result.stdout
|
||||||
|
|
||||||
|
verify_result = runner.invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"verify-stage2",
|
||||||
|
"--db",
|
||||||
|
str(db_path),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert verify_result.exit_code == 0
|
||||||
|
|
||||||
store = ScanStore(db_path)
|
store = ScanStore(db_path)
|
||||||
rows = store.list_files(1)
|
rows = store.list_files(1)
|
||||||
|
|||||||
@@ -43,8 +43,6 @@ def _run_scan(root: Path, db_path: Path, *, stage1_workers: int) -> float:
|
|||||||
str(db_path),
|
str(db_path),
|
||||||
"--stage1-workers",
|
"--stage1-workers",
|
||||||
str(stage1_workers),
|
str(stage1_workers),
|
||||||
"--stage2-workers",
|
|
||||||
"1",
|
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
elapsed = time.perf_counter() - start
|
elapsed = time.perf_counter() - start
|
||||||
|
|||||||
@@ -70,6 +70,41 @@ def test_upsert_files_reuses_previous_status_when_identity_matches(tmp_path: Pat
|
|||||||
assert resumed.resumed_from_run_id == first_run
|
assert resumed.resumed_from_run_id == first_run
|
||||||
|
|
||||||
|
|
||||||
|
def test_upsert_files_can_skip_reusing_previous_stage2_results(tmp_path: Path) -> None:
|
||||||
|
store = ScanStore(tmp_path / "scan_state.db")
|
||||||
|
store.initialize()
|
||||||
|
|
||||||
|
first_run = store.create_run(profile="tg-hdd", root_paths=(tmp_path,))
|
||||||
|
file_path = tmp_path / "clip.mp4"
|
||||||
|
candidate = make_candidate(file_path, order=0, size=12, mtime_ns=999)
|
||||||
|
[stored] = store.upsert_files(first_run, [candidate])
|
||||||
|
store.record_stage1_status(
|
||||||
|
stored.file_id,
|
||||||
|
stage1_status="pending_stage2",
|
||||||
|
final_status=None,
|
||||||
|
reason_code="high_risk_extension",
|
||||||
|
)
|
||||||
|
store.record_stage2_status(
|
||||||
|
stored.file_id,
|
||||||
|
stage2_status="passed_stage2",
|
||||||
|
final_status="passed_stage2",
|
||||||
|
reason_code="passed_stage2",
|
||||||
|
)
|
||||||
|
|
||||||
|
second_run = store.create_run(profile="tg-hdd", root_paths=(tmp_path,))
|
||||||
|
[resumed] = store.upsert_files(
|
||||||
|
second_run,
|
||||||
|
[candidate],
|
||||||
|
reuse_stage2_results=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert resumed.stage1_status == "pending_stage1"
|
||||||
|
assert resumed.stage2_status is None
|
||||||
|
assert resumed.final_status is None
|
||||||
|
assert resumed.reason_code is None
|
||||||
|
assert resumed.resumed_from_run_id is None
|
||||||
|
|
||||||
|
|
||||||
def test_upsert_files_does_not_reuse_changed_file(tmp_path: Path) -> None:
|
def test_upsert_files_does_not_reuse_changed_file(tmp_path: Path) -> None:
|
||||||
store = ScanStore(tmp_path / "scan_state.db")
|
store = ScanStore(tmp_path / "scan_state.db")
|
||||||
store.initialize()
|
store.initialize()
|
||||||
@@ -97,4 +132,3 @@ def test_upsert_files_does_not_reuse_changed_file(tmp_path: Path) -> None:
|
|||||||
assert fresh.stage1_status == "pending_stage1"
|
assert fresh.stage1_status == "pending_stage1"
|
||||||
assert fresh.final_status is None
|
assert fresh.final_status is None
|
||||||
assert fresh.resumed_from_run_id is None
|
assert fresh.resumed_from_run_id is None
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user