rgonsal/comunidadhll
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Isolate runner read model refresh from legacy snapshot failures

Browse Source
This commit is contained in:
devRaGonSa
2026-06-09 12:34:34 +02:00
parent f341b5510c
commit d9371d6912
7 changed files with 412 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

196
ai/tasks/done/TASK-202-isolate-runner-read-model-refresh-from-legacy-snapshot-failures.md Normal file
View File

@@ -0,0 +1,196 @@
---
id: TASK-202-isolate-runner-read-model-refresh-from-legacy-snapshot-failures
title: Isolate runner read model refresh from legacy snapshot failures
status: done
type: backend
team: Backend Senior
supporting_teams:
- Arquitecto de Base de Datos
- Arquitecto Python
roadmap_item: foundation
priority: high
---
# TASK-202 - Isolate runner read model refresh from legacy snapshot failures
## Goal
Prevent legacy historical snapshot failures from blocking the operational PostgreSQL read-model refreshes in the periodic historical runner.
## Context
After `TASK-201`, the periodic runner starts correctly and announces the expected hourly scope, but the first cycle aborts before reaching the PostgreSQL operational read models when legacy historical snapshot generation fails with `sqlite3.OperationalError: no such table: player_event_raw_ledger`.
The current failure path happens inside `generate_historical_snapshots(...)`, which was still executed inside the runner's broader refresh attempt. That legacy failure prevented the cycle from refreshing:
- `player_search_index`
- `player_period_stats`
- `ranking_snapshots`
This task isolates the cycle steps so the legacy error stays visible in logs and results, while the operational PostgreSQL refreshes are still attempted in the same cycle. Preserve the current product identity: Spanish-speaking HLL Vietnam community, military/Vietnam/tactical/sober visual direction and controlled repository evolution.
## Steps
1. Read the listed files first.
2. Confirm the current runner failure path and where legacy snapshot generation aborts the cycle.
3. Refactor the periodic runner so each refresh step is handled as an isolated unit with a separately reported result.
4. Preserve retries for full-attempt failures where they still apply, but avoid retry-only behavior for a legacy step that should no longer block operational refreshes.
5. Keep the legacy snapshot error visible in logs and cycle results.
6. Ensure the runner still attempts:
- `refresh_player_search_index(...)`
- `refresh_player_period_stats(...)`
- `refresh_ranking_snapshots(...)`
even when `generate_historical_snapshots(...)` fails.
7. Return a global cycle status of `partial` or equivalent when the legacy snapshot step fails but operational steps continue.
8. Keep manual CLIs and public API contracts unchanged.
9. Update operational documentation with the new partial-cycle behavior and production validation guidance.
## Files to Read First
- `AGENTS.md`
- `ai/repo-context.md`
- `ai/architecture-index.md`
- `backend/app/historical_runner.py`
- `backend/app/historical_snapshots.py`
- `backend/app/historical_storage.py`
- `backend/app/rcon_historical_player_stats.py`
- `backend/app/rcon_historical_leaderboards.py`
- `scripts/run-stats-validation.ps1`
- `docs/player-search-read-model-plan.md`
- `docs/player-period-stats-read-model-plan.md`
- `docs/ranking-snapshot-read-model-plan.md`
- `ai/tasks/done/TASK-201-automate-player-read-model-refresh.md`
## Expected Files to Modify
- `backend/app/historical_runner.py`
- `backend/tests/test_historical_snapshot_refresh.py`
- `scripts/run-stats-validation.ps1`
- `docs/player-search-read-model-plan.md`
- `docs/player-period-stats-read-model-plan.md`
- `docs/ranking-snapshot-read-model-plan.md`
- `ai/tasks/done/TASK-202-isolate-runner-read-model-refresh-from-legacy-snapshot-failures.md`
## Constraints
- Keep the change minimal and backend/documentation only.
- Do not execute `ai-platform run`.
- Do not modify frontend.
- Do not change design.
- Do not touch images or assets.
- Do not reactivate Elo/MMR.
- Do not reintroduce Comunidad Hispana `#03`.
- Do not fix or migrate `player_event_raw_ledger` in this task unless analysis proves it is strictly necessary.
- Do not create or modify legacy tables unless it becomes strictly necessary and is explicitly justified in the outcome.
- Do not hide legacy errors; they must remain visible in logs and result payloads.
- Do not change manual CLI behavior.
- Do not change public API contracts.
## Validation
Before completing the task ensure:
- `python -m py_compile backend/app/historical_runner.py`
- `powershell -ExecutionPolicy Bypass -File scripts/run-stats-validation.ps1`
- `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1`
- local/import validation proves that when `generate_historical_snapshots(...)` fails:
- `refresh_player_search_index(...)` is still attempted
- `refresh_player_period_stats(...)` is still attempted
- `refresh_ranking_snapshots(...)` is still attempted
- the global result is `partial` or equivalent
- the legacy error remains reported
- local/import validation proves that when all steps succeed, the cycle still returns `ok`
- local/import validation proves separate result payloads still exist:
- `historical_snapshot_result`
- `player_search_index_result`
- `player_period_stats_result`
- `ranking_snapshot_result`
- `git diff --name-only` matches the expected scope
## Outcome
Root cause:
- `generate_historical_snapshots(...)` still ran inside the runner attempt's broad failure boundary.
- When the legacy monthly MVP V2 snapshot path touched missing `player_event_raw_ledger` state, it raised `OperationalError`.
- That exception aborted the attempt before the PostgreSQL operational read-model refreshes were called.
Previous behavior:
- a legacy snapshot failure stopped the cycle before:
- `player_search_index`
- `player_period_stats`
- `ranking_snapshots`
- the cycle returned attempt-level failure instead of a partial operational result
- operators could see the legacy error, but the operational read models stayed stale
New behavior:
- `backend/app/historical_runner.py`
- wraps legacy snapshot refresh in an isolated step via `refresh_periodic_historical_snapshots(...)`
- preserves the legacy error in logs and result payloads with:
- `historical_snapshot_result`
- compatibility alias `snapshot_result`
- event `historical-snapshot-refresh-failed`
- continues attempting:
- `player_search_index`
- `player_period_stats`
- `ranking_snapshots`
after a legacy snapshot failure
- resolves the overall cycle to `partial` when the isolated legacy step fails but operational refreshes continue
- `backend/tests/test_historical_snapshot_refresh.py`
- adds coverage for legacy snapshot failure isolation
- adds coverage for the all-success path still returning `ok`
- `scripts/run-stats-validation.ps1`
- validates that a legacy snapshot failure still leaves the operational read-model refresh sequence running
- validates:
- `historical_snapshot_result`
- `player_search_index_result`
- `player_period_stats_result`
- `ranking_snapshot_result`
- validates the global `partial` result and preserved legacy error reporting
- operational docs now explain that the runner can finish `partial` if the legacy block fails while PostgreSQL read-model refreshes still run
Validations executed:
- `python -m py_compile backend/app/historical_runner.py`
- `python -m unittest tests.test_historical_snapshot_refresh`
- `powershell -ExecutionPolicy Bypass -File scripts/run-stats-validation.ps1`
- `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1`
Production validation:
1. Run one controlled cycle:
- `python -m app.historical_runner --max-runs 1`
2. Inspect the cycle payload and logs for:
- `status` equal to `ok` or `partial`
- `historical_snapshot_result`
- `player_search_index_result`
- `player_period_stats_result`
- `ranking_snapshot_result`
3. If `status=partial`, confirm the legacy failure remains visible under:
- `historical_snapshot_result.error_type`
- `historical_snapshot_result.error`
- `historical_snapshot_result.traceback`
4. Confirm operational timestamps still advance in PostgreSQL for:
- `player_search_index`
- `player_period_stats`
- `ranking_snapshots`
5. Confirm public reads still use the refreshed read models:
- `/api/stats/players/search`
- `/api/stats/players/{player_id}`
- `/api/ranking`
Scope notes:
- no frontend files were modified
- no images/assets were touched
- `player_event_raw_ledger` was not fixed or migrated in this task
- manual CLIs were preserved unchanged
- pre-existing unrelated changes in `ai/system-metrics.md` and frontend assets were intentionally left untouched
## Change Budget
- Prefer fewer than 5 modified files.
- Prefer changes under 200 lines when feasible.
- Split the work into follow-up tasks if limits are exceeded.

43
backend/app/historical_runner.py
View File

@@ -127,7 +127,7 @@ def _run_refresh_with_retries(
page_size=page_size,
rebuild_snapshots=False,
)
snapshot_result = generate_historical_snapshots(
historical_snapshot_result = refresh_periodic_historical_snapshots(
server_slug=server_slug,
run_number=run_number,
)
@@ -141,12 +141,12 @@ def _run_refresh_with_retries(
"reason": "rcon-primary-cycle-no-classic-fallback-needed",
}
if should_generate_snapshots:
snapshot_result = generate_historical_snapshots(
historical_snapshot_result = refresh_periodic_historical_snapshots(
server_slug=server_slug,
run_number=run_number,
)
snapshot_result = {
**snapshot_result,
historical_snapshot_result = {
**historical_snapshot_result,
"generation_policy": "rcon-primary-useful-cycle",
"reason": "rcon-primary-cycle-produced-new-useful-coverage",
}
@@ -168,7 +168,7 @@ def _run_refresh_with_retries(
**elo_policy,
}
else:
snapshot_result = {
historical_snapshot_result = {
"status": "skipped",
"reason": "rcon-primary-cycle-had-no-new-useful-data",
"generation_policy": "rcon-primary-no-new-useful-data",
@@ -197,7 +197,7 @@ def _run_refresh_with_retries(
return {
"status": _resolve_refresh_cycle_status(
refresh_result=refresh_result,
snapshot_result=snapshot_result,
historical_snapshot_result=historical_snapshot_result,
player_search_index_result=player_search_index_result,
player_period_stats_result=player_period_stats_result,
ranking_snapshot_result=ranking_snapshot_result,
@@ -210,7 +210,8 @@ def _run_refresh_with_retries(
"classic_fallback_used": should_run_classic_fallback,
"classic_fallback_reason": classic_fallback_reason,
"refresh_result": refresh_result,
"snapshot_result": snapshot_result,
"historical_snapshot_result": historical_snapshot_result,
"snapshot_result": historical_snapshot_result,
"player_search_index_result": player_search_index_result,
"player_period_stats_result": player_period_stats_result,
"ranking_snapshot_result": ranking_snapshot_result,
@@ -279,6 +280,34 @@ def generate_historical_snapshots(
}
def refresh_periodic_historical_snapshots(
*,
server_slug: str | None = None,
run_number: int = 1,
) -> dict[str, Any]:
"""Refresh legacy historical snapshots without aborting operational read-model refreshes."""
try:
return generate_historical_snapshots(
server_slug=server_slug,
run_number=run_number,
)
except Exception as exc: # noqa: BLE001 - legacy failures must remain visible but isolated
failure_result = {
"status": "error",
"error_type": type(exc).__name__,
"error": str(exc),
"traceback": traceback.format_exc(),
"run_number": run_number,
"refresh_interval_seconds": get_historical_refresh_interval_seconds(),
"server_slug": server_slug,
"generation_policy": "periodic-historical-refresh-cycle",
"snapshot_scope": _describe_snapshot_scope(server_slug),
"legacy_block": "historical-snapshots",
}
_emit_json_log({"event": "historical-snapshot-refresh-failed", **failure_result})
return failure_result
def refresh_periodic_ranking_snapshots(
*,
server_slug: str | None = None,

101
backend/tests/test_historical_snapshot_refresh.py
View File

@@ -121,6 +121,107 @@ class HistoricalSnapshotRefreshTests(unittest.TestCase):
self.assertIn('"status": "ok"', stream.getvalue())
self.assertIn('"captured_at": "2026-05-22 00:00:00+00:00"', stream.getvalue())
def test_runner_continues_when_legacy_snapshot_refresh_fails(self) -> None:
with (
patch("app.historical_runner.backend_writer_lock", return_value=nullcontext()),
patch("app.historical_runner._run_primary_rcon_capture", return_value={"status": "ok", "targets": []}),
patch(
"app.historical_runner._resolve_classic_fallback_policy",
return_value=(False, "validation-rcon-primary-cycle"),
),
patch("app.historical_runner._rcon_capture_has_new_useful_data", return_value=True),
patch(
"app.historical_runner.generate_historical_snapshots",
side_effect=RuntimeError("legacy snapshot failure"),
),
patch(
"app.historical_runner._build_elo_mmr_rebuild_policy",
return_value={
"due": False,
"policy": "validation-policy",
"last_generated_at": None,
"samples_since_last_rebuild": 1,
"minutes_since_last_rebuild": None,
"rebuild_interval_minutes": 60,
"min_new_samples": 10,
},
),
patch("app.historical_runner.refresh_player_search_index", return_value={"status": "ok"}) as search_refresh,
patch("app.historical_runner.refresh_player_period_stats", return_value={"status": "ok"}) as period_refresh,
patch("app.historical_runner.refresh_ranking_snapshots", return_value={"status": "ok"}) as ranking_refresh,
patch(
"app.historical_runner._maybe_run_database_maintenance",
return_value={"status": "skipped", "reason": "disabled"},
),
):
result = _run_refresh_with_retries(
max_retries=0,
retry_delay_seconds=0,
server_slug=None,
max_pages=None,
page_size=None,
run_number=1,
)
self.assertEqual(result["status"], "partial")
self.assertEqual(result["historical_snapshot_result"]["status"], "error")
self.assertEqual(result["snapshot_result"]["status"], "error")
self.assertEqual(
result["historical_snapshot_result"]["error"],
"legacy snapshot failure",
)
search_refresh.assert_called_once()
period_refresh.assert_called_once()
ranking_refresh.assert_called_once()
def test_runner_returns_ok_when_legacy_snapshot_and_read_models_succeed(self) -> None:
with (
patch("app.historical_runner.backend_writer_lock", return_value=nullcontext()),
patch("app.historical_runner._run_primary_rcon_capture", return_value={"status": "ok", "targets": []}),
patch(
"app.historical_runner._resolve_classic_fallback_policy",
return_value=(False, "validation-rcon-primary-cycle"),
),
patch("app.historical_runner._rcon_capture_has_new_useful_data", return_value=True),
patch(
"app.historical_runner.generate_historical_snapshots",
return_value={"status": "ok", "generated_at": "2026-06-09T08:00:00Z"},
),
patch(
"app.historical_runner._build_elo_mmr_rebuild_policy",
return_value={
"due": False,
"policy": "validation-policy",
"last_generated_at": None,
"samples_since_last_rebuild": 1,
"minutes_since_last_rebuild": None,
"rebuild_interval_minutes": 60,
"min_new_samples": 10,
},
),
patch("app.historical_runner.refresh_player_search_index", return_value={"status": "ok"}),
patch("app.historical_runner.refresh_player_period_stats", return_value={"status": "ok"}),
patch("app.historical_runner.refresh_ranking_snapshots", return_value={"status": "ok"}),
patch(
"app.historical_runner._maybe_run_database_maintenance",
return_value={"status": "skipped", "reason": "disabled"},
),
):
result = _run_refresh_with_retries(
max_retries=0,
retry_delay_seconds=0,
server_slug=None,
max_pages=None,
page_size=None,
run_number=1,
)
self.assertEqual(result["status"], "ok")
self.assertEqual(result["historical_snapshot_result"]["status"], "ok")
self.assertIn("player_search_index_result", result)
self.assertIn("player_period_stats_result", result)
self.assertIn("ranking_snapshot_result", result)
if __name__ == "__main__":
unittest.main()

6
docs/player-period-stats-read-model-plan.md
View File

@@ -107,6 +107,8 @@ Automatic runner refresh:
- `backend/app/historical_runner.py` refreshes `player_period_stats` automatically
- it inherits the periodic cadence of the historical runner via `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS`
- the runner executes this step after `player_search_index` and before `ranking_snapshots`
- the runner still attempts this step even if the legacy historical snapshot block fails earlier in the same cycle
- the overall runner result may end as `partial` when that legacy block fails but operational PostgreSQL refreshes continue
- the public profile path keeps runtime fallback preserved if the read model is empty, incomplete or unavailable
Emergency manual command:
@@ -160,6 +162,10 @@ Indexes kept for the public profile flow:
Recommended checks after refresh:
- confirm the historical runner output reports either `status=ok` or `status=partial`
- confirm `historical_snapshot_result`, `player_search_index_result`, `player_period_stats_result` and `ranking_snapshot_result` are present in the cycle payload
- if the cycle is `partial`, inspect `historical_snapshot_result.error_type`, `historical_snapshot_result.error` and the runner logs for the legacy failure
- confirm `player_period_stats.updated_at` advanced even when a legacy snapshot error was reported
- confirm the historical runner output reports `player_period_stats_result`
- if an emergency rebuild is needed, run `python -m app.rcon_historical_player_stats refresh-player-period-stats`
- confirm the command reports rows for:

6
docs/player-search-read-model-plan.md
View File

@@ -76,6 +76,8 @@ Automatic runner refresh:
- `backend/app/historical_runner.py` refreshes `player_search_index` automatically
- it inherits the periodic cadence of the historical runner via `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS`
- the runner executes this step after the existing RCON ingestion/materialization cycle
- the runner still attempts this step even if the legacy historical snapshot block fails earlier in the same cycle
- the overall runner result may end as `partial` when that legacy block fails but operational PostgreSQL refreshes continue
- the runner keeps runtime fallback preserved for the public endpoint
Emergency manual command:
@@ -133,6 +135,10 @@ Search tolerance is implemented with:
Recommended checks after refresh:
- confirm the historical runner output reports either `status=ok` or `status=partial`
- confirm `historical_snapshot_result`, `player_search_index_result`, `player_period_stats_result` and `ranking_snapshot_result` are present in the cycle payload
- if the cycle is `partial`, inspect `historical_snapshot_result.error_type`, `historical_snapshot_result.error` and the runner logs for the legacy failure
- confirm `player_search_index.updated_at` advanced even when a legacy snapshot error was reported
- confirm the historical runner output reports `player_search_index_result`
- if an emergency rebuild is needed, run `python -m app.rcon_historical_player_stats refresh-player-search-index`
- confirm the command reports rows for `all-servers`, `comunidad-hispana-01` and `comunidad-hispana-02`

4
docs/ranking-snapshot-read-model-plan.md
View File

@@ -295,10 +295,12 @@ Current implementation note:
- the periodic historical runner in `backend/app/historical_runner.py` invokes that bulk refresh as part of the normal backend refresh cycle
- current cycle order is:
- existing RCON ingestion/materialization cycle
- legacy `historical_snapshot_result`
- `player_search_index`
- `player_period_stats`
- `ranking_snapshots`
- per-combination failures should be reported without aborting the entire matrix refresh
- a legacy historical snapshot failure can leave the full cycle in `partial`, but the runner should still attempt the operational PostgreSQL refreshes and report `ranking_snapshot_result`
## Recommended Combinations
@@ -329,6 +331,8 @@ Operational guidance:
- regenerate after metric SQL changes that affect ranking totals or ordering
- when using the periodic backend runner, keep ranking refresh attached to the same recurring cycle rather than a separate scheduler unless operational load proves otherwise
- the runner inherits its cadence from `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS`
- if the runner reports `partial`, review `historical_snapshot_result` and backend logs before assuming `ranking_snapshots` was skipped
- confirm `ranking_snapshots.generated_at` or the latest `updated_at`/window timestamp advanced in PostgreSQL even when a legacy snapshot error was logged
- fallback runtime remains preserved for `/api/ranking` if a requested snapshot is missing or unavailable
## Ready Vs Fallback

63
scripts/run-stats-validation.ps1
View File

@@ -1309,6 +1309,14 @@ def validate_historical_runner_read_model_refresh_cycle():
call_order == ["player-search-index", "player-period-stats", "ranking-snapshots"],
"Historical runner should refresh player_search_index, then player_period_stats, then ranking_snapshots.",
)
require(
(result.get("historical_snapshot_result") or {}).get("status") == "ok",
"Historical runner should report historical_snapshot_result separately.",
)
require(
(result.get("snapshot_result") or {}).get("status") == "ok",
"Historical runner should preserve snapshot_result compatibility.",
)
require(
(result.get("player_search_index_result") or {}).get("status") == "ok",
"Historical runner should report player_search_index_result separately.",
@@ -1382,6 +1390,61 @@ def validate_historical_runner_read_model_refresh_cycle():
),
"Historical runner should emit a dedicated failure log for player_search_index refresh failures.",
)
call_order.clear()
captured_logs.clear()
historical_runner.refresh_player_search_index = fake_refresh_player_search_index
historical_runner.generate_historical_snapshots = lambda **kwargs: (_ for _ in ()).throw(
RuntimeError("legacy snapshot failure")
)
result = historical_runner._run_refresh_with_retries(
max_retries=0,
retry_delay_seconds=0,
server_slug=None,
max_pages=None,
page_size=None,
run_number=1,
)
require(
result.get("status") == "partial",
"Historical runner should return partial when legacy historical snapshot refresh fails but operational read-model refreshes continue.",
)
require(
call_order == ["player-search-index", "player-period-stats", "ranking-snapshots"],
"Historical runner should still attempt operational read-model refreshes after a legacy historical snapshot failure.",
)
require(
(result.get("historical_snapshot_result") or {}).get("status") == "error",
"Historical runner should expose the legacy historical snapshot failure in historical_snapshot_result.",
)
require(
(result.get("snapshot_result") or {}).get("status") == "error",
"Historical runner should preserve snapshot_result compatibility for the legacy failure path.",
)
require(
(result.get("historical_snapshot_result") or {}).get("error_type") == "RuntimeError",
"Historical runner should preserve the legacy snapshot error type.",
)
require(
(result.get("player_search_index_result") or {}).get("status") == "ok",
"Historical runner should still refresh player_search_index after a legacy snapshot failure.",
)
require(
(result.get("player_period_stats_result") or {}).get("status") == "ok",
"Historical runner should still refresh player_period_stats after a legacy snapshot failure.",
)
require(
(result.get("ranking_snapshot_result") or {}).get("status") == "ok",
"Historical runner should still refresh ranking snapshots after a legacy snapshot failure.",
)
require(
any(
log.get("event") == "historical-snapshot-refresh-failed"
for log in captured_logs
if isinstance(log, dict)
),
"Historical runner should emit a dedicated failure log for legacy historical snapshot refresh failures.",
)
finally:
historical_runner.backend_writer_lock = original_backend_writer_lock
historical_runner.build_writer_lock_holder = original_build_writer_lock_holder