diff --git a/ai/tasks/done/TASK-201-automate-player-read-model-refresh.md b/ai/tasks/done/TASK-201-automate-player-read-model-refresh.md new file mode 100644 index 0000000..765a6cf --- /dev/null +++ b/ai/tasks/done/TASK-201-automate-player-read-model-refresh.md @@ -0,0 +1,223 @@ +--- +id: TASK-201-automate-player-read-model-refresh +title: Automate player read model refresh +status: done +type: backend +team: Backend Senior +supporting_teams: + - Arquitecto de Base de Datos + - Arquitecto Python +roadmap_item: foundation +priority: high +--- + +# TASK-201 - Automate player read model refresh + +## Goal + +Automate the periodic refresh of `player_search_index` and `player_period_stats` inside the existing backend historical runner so the public stats endpoints can keep using their read models without depending on manual operator refreshes. + +## Context + +Production already validates the full read-model chain required by the public player stats flows: + +- `player_search_index` exists and `/api/stats/players/search` already uses `read_model=player-search-index` with `fallback_used=false` +- `player_period_stats` exists and `/api/stats/players/{player_id}` already uses `read_model=player-period-stats` with `fallback_used=false` +- `ranking_snapshots` / `ranking_snapshot_items` already exist +- `refresh-ranking-snapshots --limit 30` already runs in the periodic historical runner +- PostgreSQL is the operational storage target +- runtime fallback must remain preserved for player stats and ranking + +The remaining gap is operational automation for the two player read models. The existing periodic runner in `backend/app/historical_runner.py` is the intended scheduler surface, so this task must extend that cycle instead of inventing a new scheduler. The cycle should keep `refresh-ranking-snapshots` and add dedicated, separately reported refresh steps for player search and player period stats. + +Preserve the current product identity: Spanish-speaking HLL Vietnam community, military/Vietnam/tactical/sober visual direction and controlled repository evolution. + +## Steps + +1. Read the listed files first. +2. Inspect the current historical runner flow and the existing manual read-model refresh entrypoints. +3. Integrate `refresh_player_search_index(...)` into the periodic backend cycle. +4. Integrate `refresh_player_period_stats(...)` into the periodic backend cycle. +5. Keep `refresh-ranking-snapshots` in the same cycle and preserve its current behavior. +6. Run the cycle in this order unless the current runner architecture safely requires a different invariant: + - existing RCON ingestion/materialization cycle + - `player_search_index` + - `player_period_stats` + - `ranking_snapshots` +7. Report `player_search_index_result`, `player_period_stats_result` and `ranking_snapshot_result` separately. +8. Preserve visibility of failures and avoid hiding one read-model failure behind another. One failure should not necessarily prevent the remaining refresh attempts unless the runner already enforces a stricter global policy. +9. Preserve the manual CLIs and the public contracts of: + - `/api/stats/players/search` + - `/api/stats/players/{player_id}` + - `/api/ranking` +10. Document the automatic runner refresh, inherited cadence, emergency manual commands, fallback preservation and final cycle order. + +## Files to Read First + +- `AGENTS.md` +- `ai/repo-context.md` +- `ai/architecture-index.md` +- `backend/app/historical_runner.py` +- `backend/app/rcon_historical_player_stats.py` +- `backend/app/rcon_historical_leaderboards.py` +- `backend/app/postgres_rcon_storage.py` +- `backend/app/payloads.py` +- `docs/player-search-read-model-plan.md` +- `docs/player-period-stats-read-model-plan.md` +- `docs/ranking-snapshot-read-model-plan.md` +- `scripts/run-stats-validation.ps1` +- `ai/tasks/done/TASK-197-automate-ranking-snapshot-refresh.md` +- `ai/tasks/done/TASK-198-add-player-search-read-model.md` +- `ai/tasks/done/TASK-199-add-player-period-stats-read-model.md` +- `ai/tasks/done/TASK-200-fix-player-profile-runtime-postgres-grouping.md` + +## Expected Files to Modify + +- `backend/app/historical_runner.py` +- `docs/player-search-read-model-plan.md` +- `docs/player-period-stats-read-model-plan.md` +- `docs/ranking-snapshot-read-model-plan.md` +- `scripts/run-stats-validation.ps1` +- `ai/tasks/in-progress/TASK-201-automate-player-read-model-refresh.md` + +## Constraints + +- Keep the change minimal and backend/documentation only. +- Do not execute `ai-platform run`. +- Do not modify frontend. +- Do not change design. +- Do not touch images or assets. +- Do not reactivate Elo/MMR. +- Do not reintroduce Comunidad Hispana `#03`. +- Do not mix visual corrections into this task. +- Preserve PostgreSQL as the operational storage target. +- Preserve runtime fallback for player stats and ranking. +- Do not break the manual CLIs: + - `refresh-player-search-index` + - `refresh-player-period-stats` + - `refresh-ranking-snapshots` + - `generate-ranking-snapshot` +- Do not change public contracts of: + - `/api/stats/players/search` + - `/api/stats/players/{player_id}` + - `/api/ranking` + +## Validation + +Before completing the task ensure: + +- `python -m py_compile backend/app/historical_runner.py backend/app/rcon_historical_player_stats.py` +- `powershell -ExecutionPolicy Bypass -File scripts/run-stats-validation.ps1` +- `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1` +- local/import validation proves the runner calls: + - `refresh_player_search_index(...)` + - `refresh_player_period_stats(...)` + - `refresh_ranking_snapshots(...)` +- validation proves the periodic cycle reports these results separately: + - `player_search_index_result` + - `player_period_stats_result` + - `ranking_snapshot_result` +- validation proves the manual CLIs still work: + - `refresh-player-search-index` + - `refresh-player-period-stats` + - `refresh-ranking-snapshots` +- validation proves `/api/stats/players/search` can still use `player-search-index` +- validation proves `/api/stats/players/{player_id}` can still use `player-period-stats` +- validation proves ranking snapshots are not broken +- `git diff --name-only` matches the expected scope + +## Outcome + +Implemented: + +- `backend/app/historical_runner.py` + - integrated automatic periodic refresh of `player_search_index` + - integrated automatic periodic refresh of `player_period_stats` + - preserved the existing `ranking_snapshots` refresh + - the runner now reports: + - `player_search_index_result` + - `player_period_stats_result` + - `ranking_snapshot_result` + - one read-model failure is now reported explicitly and does not prevent the runner from attempting the remaining refresh steps in the same cycle + - the runner returns `status=partial` when one of those periodic read-model refresh steps fails but the cycle continues +- `docs/player-search-read-model-plan.md` + - documents automatic runner refresh + - documents inherited cadence from `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS` + - documents the emergency manual command + - documents fallback preservation +- `docs/player-period-stats-read-model-plan.md` + - documents automatic runner refresh + - documents inherited cadence from `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS` + - documents the emergency manual command + - documents fallback preservation +- `docs/ranking-snapshot-read-model-plan.md` + - documents the final runner order + - documents that ranking refresh remains part of the same periodic cycle + - documents cadence inheritance and fallback preservation +- `scripts/run-stats-validation.ps1` + - validates that the runner calls the player read-model refreshes before ranking snapshots + - validates separate result reporting + - validates that a `player_search_index` refresh failure remains visible and does not stop the remaining periodic refresh attempts + +Modified files: + +- `backend/app/historical_runner.py` +- `docs/player-search-read-model-plan.md` +- `docs/player-period-stats-read-model-plan.md` +- `docs/ranking-snapshot-read-model-plan.md` +- `scripts/run-stats-validation.ps1` +- `ai/tasks/done/TASK-201-automate-player-read-model-refresh.md` + +Final runner order: + +1. existing RCON ingestion/materialization cycle +2. `player_search_index` +3. `player_period_stats` +4. `ranking_snapshots` + +Added cycle results/logs: + +- `player_search_index_result` +- `player_period_stats_result` +- `ranking_snapshot_result` +- start events: + - `player-search-index-refresh-started` + - `player-period-stats-refresh-started` +- failure events: + - `player-search-index-refresh-failed` + - `player-period-stats-refresh-failed` + +Validations executed: + +- `python -m py_compile backend/app/historical_runner.py backend/app/rcon_historical_player_stats.py` +- `powershell -ExecutionPolicy Bypass -File scripts/run-stats-validation.ps1` +- `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1` + +Production validation path: + +- run one controlled backend cycle: + - `python -m app.historical_runner --max-runs 1` +- verify the cycle output includes: + - `player_search_index_result` + - `player_period_stats_result` + - `ranking_snapshot_result` +- confirm the player stats endpoints still report: + - `/api/stats/players/search` -> `source.read_model=player-search-index` + - `/api/stats/players/{player_id}` -> `source.read_model=player-period-stats` +- confirm `/api/ranking` still serves ranking snapshots when ready and preserves fallback behavior when a snapshot is missing +- if emergency rebuild is needed, operators can still run: + - `python -m app.rcon_historical_player_stats refresh-player-search-index` + - `python -m app.rcon_historical_player_stats refresh-player-period-stats` + - `python -m app.rcon_historical_leaderboards refresh-ranking-snapshots --limit 30` + +Pending limitations kept out of scope: + +- the runner still refreshes the player read models for all supported public scopes even when a manual runner execution is limited with `--server` +- this task does not introduce a separate external scheduler or deployment-specific cron wiring +- live HTTP validation at `http://127.0.0.1:8000` was not available in this environment; route-contract validation passed via local Python imports instead + +## Change Budget + +- Prefer fewer than 5 modified files. +- Prefer changes under 200 lines when feasible. +- Split the work into follow-up tasks if limits are exceeded. diff --git a/backend/app/historical_runner.py b/backend/app/historical_runner.py index 74b72ee..ebfea7e 100644 --- a/backend/app/historical_runner.py +++ b/backend/app/historical_runner.py @@ -30,6 +30,10 @@ from .historical_snapshots import ( generate_and_persist_priority_historical_snapshots, ) from .rcon_historical_leaderboards import refresh_ranking_snapshots +from .rcon_historical_player_stats import ( + refresh_player_period_stats, + refresh_player_search_index, +) from .rcon_historical_storage import count_rcon_historical_samples_since from .rcon_historical_worker import run_rcon_historical_capture from .writer_lock import backend_writer_lock, build_writer_lock_holder @@ -177,13 +181,29 @@ def _run_refresh_with_retries( rcon_capture_result=rcon_capture_result ), } + player_search_index_result = refresh_periodic_player_search_index( + server_slug=server_slug, + run_number=run_number, + ) + player_period_stats_result = refresh_periodic_player_period_stats( + server_slug=server_slug, + run_number=run_number, + ) ranking_snapshot_result = refresh_periodic_ranking_snapshots( server_slug=server_slug, run_number=run_number, ) maintenance_result = _maybe_run_database_maintenance() return { - "status": "ok", + "status": _resolve_refresh_cycle_status( + refresh_result=refresh_result, + snapshot_result=snapshot_result, + player_search_index_result=player_search_index_result, + player_period_stats_result=player_period_stats_result, + ranking_snapshot_result=ranking_snapshot_result, + elo_mmr_result=elo_mmr_result, + database_maintenance_result=maintenance_result, + ), "attempts_used": attempt, "max_retries": max_retries, "rcon_capture_result": rcon_capture_result, @@ -191,6 +211,8 @@ def _run_refresh_with_retries( "classic_fallback_reason": classic_fallback_reason, "refresh_result": refresh_result, "snapshot_result": snapshot_result, + "player_search_index_result": player_search_index_result, + "player_period_stats_result": player_period_stats_result, "ranking_snapshot_result": ranking_snapshot_result, "elo_mmr_result": elo_mmr_result, "database_maintenance_result": maintenance_result, @@ -286,6 +308,102 @@ def refresh_periodic_ranking_snapshots( } +def refresh_periodic_player_search_index( + *, + server_slug: str | None = None, + run_number: int = 1, +) -> dict[str, Any]: + """Refresh the player search read model without aborting the remaining cycle on failure.""" + _emit_json_log( + { + "event": "player-search-index-refresh-started", + "run_number": run_number, + "server_slug": server_slug, + "refresh_scope": "supported-public-player-search-scopes", + } + ) + try: + result = refresh_player_search_index() + except Exception as exc: # noqa: BLE001 - one read-model failure must stay visible + failure_result = { + "status": "error", + "error_type": type(exc).__name__, + "error": str(exc), + "run_number": run_number, + "refresh_interval_seconds": get_historical_refresh_interval_seconds(), + "server_slug": server_slug, + "generation_policy": "periodic-historical-refresh-cycle", + "scope_policy": "always-refresh-supported-public-player-search-scopes", + } + _emit_json_log({"event": "player-search-index-refresh-failed", **failure_result}) + return failure_result + return { + **result, + "run_number": run_number, + "refresh_interval_seconds": get_historical_refresh_interval_seconds(), + "server_slug": server_slug, + "generation_policy": "periodic-historical-refresh-cycle", + "scope_policy": "always-refresh-supported-public-player-search-scopes", + } + + +def refresh_periodic_player_period_stats( + *, + server_slug: str | None = None, + run_number: int = 1, +) -> dict[str, Any]: + """Refresh the player period stats read model without aborting the remaining cycle on failure.""" + _emit_json_log( + { + "event": "player-period-stats-refresh-started", + "run_number": run_number, + "server_slug": server_slug, + "refresh_scope": "supported-public-player-period-scopes", + } + ) + try: + result = refresh_player_period_stats() + except Exception as exc: # noqa: BLE001 - one read-model failure must stay visible + failure_result = { + "status": "error", + "error_type": type(exc).__name__, + "error": str(exc), + "run_number": run_number, + "refresh_interval_seconds": get_historical_refresh_interval_seconds(), + "server_slug": server_slug, + "generation_policy": "periodic-historical-refresh-cycle", + "scope_policy": "always-refresh-supported-public-player-period-scopes", + } + _emit_json_log({"event": "player-period-stats-refresh-failed", **failure_result}) + return failure_result + return { + **result, + "run_number": run_number, + "refresh_interval_seconds": get_historical_refresh_interval_seconds(), + "server_slug": server_slug, + "generation_policy": "periodic-historical-refresh-cycle", + "scope_policy": "always-refresh-supported-public-player-period-scopes", + } + + +def _resolve_refresh_cycle_status(**results: dict[str, Any]) -> str: + statuses = [ + str(result.get("status") or "").strip().lower() + for result in results.values() + if isinstance(result, dict) and result.get("status") is not None + ] + if not statuses: + return "ok" + if any(status == "error" for status in statuses): + ok_like_statuses = {"ok", "skipped"} + if all(status not in ok_like_statuses for status in statuses): + return "error" + return "partial" + if any(status == "partial" for status in statuses): + return "partial" + return "ok" + + def _emit_json_log(payload: dict[str, Any]) -> None: """Print JSON logs that remain safe for Compose and log collectors.""" print(json.dumps(payload, ensure_ascii=True, default=str), flush=True) diff --git a/docs/player-period-stats-read-model-plan.md b/docs/player-period-stats-read-model-plan.md index 7a69bc6..d9cfd5a 100644 --- a/docs/player-period-stats-read-model-plan.md +++ b/docs/player-period-stats-read-model-plan.md @@ -102,6 +102,19 @@ Refresh policy: - keep the latest player name seen inside the selected period window - persist ranking position by kills inside each generated period window +Automatic runner refresh: + +- `backend/app/historical_runner.py` refreshes `player_period_stats` automatically +- it inherits the periodic cadence of the historical runner via `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS` +- the runner executes this step after `player_search_index` and before `ranking_snapshots` +- the public profile path keeps runtime fallback preserved if the read model is empty, incomplete or unavailable + +Emergency manual command: + +```bash +python -m app.rcon_historical_player_stats refresh-player-period-stats +``` + ## Public Read Path Priority for `/api/stats/players/{player_id}`: @@ -147,7 +160,8 @@ Indexes kept for the public profile flow: Recommended checks after refresh: -- run `python -m app.rcon_historical_player_stats refresh-player-period-stats` +- confirm the historical runner output reports `player_period_stats_result` +- if an emergency rebuild is needed, run `python -m app.rcon_historical_player_stats refresh-player-period-stats` - confirm the command reports rows for: - `all-servers` - `comunidad-hispana-01` @@ -162,7 +176,7 @@ Recommended checks after refresh: ## Current Limitations -- periodic refresh is still manual; there is no scheduled operational refresh yet - the public route still exposes weekly/monthly only; yearly is prepared internally for future use -- runtime fallback remains necessary until production refresh automation is in place +- the runner refreshes all supported public scopes and periods on each cycle even when a manual runner execution is limited with `--server` +- runtime fallback remains necessary as a safety net even with periodic automation in place - canonical historical truth remains in materialized RCON tables, not in the read model diff --git a/docs/player-search-read-model-plan.md b/docs/player-search-read-model-plan.md index 2f518a8..064bbb8 100644 --- a/docs/player-search-read-model-plan.md +++ b/docs/player-search-read-model-plan.md @@ -71,6 +71,19 @@ Refresh policy: - keep the latest current-year player name - store accent-insensitive normalized names in Python +Automatic runner refresh: + +- `backend/app/historical_runner.py` refreshes `player_search_index` automatically +- it inherits the periodic cadence of the historical runner via `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS` +- the runner executes this step after the existing RCON ingestion/materialization cycle +- the runner keeps runtime fallback preserved for the public endpoint + +Emergency manual command: + +```bash +python -m app.rcon_historical_player_stats refresh-player-search-index +``` + ## Public Read Path Priority for `/api/stats/players/search`: @@ -113,13 +126,15 @@ Search tolerance is implemented with: - this read model is focused on player search only, not personal profile totals - counts are current-year only by design - historical players with no activity in the current UTC year are not intentionally prioritized in this first model -- profile and personal stats still use runtime aggregation over materialized tables +- the runner refreshes all supported public scopes on each cycle even when a manual runner execution is limited with `--server` +- profile and personal stats use their own dedicated read model and still preserve runtime fallback when needed ## Production Validation Recommended checks after refresh: -- run `python -m app.rcon_historical_player_stats refresh-player-search-index` +- confirm the historical runner output reports `player_search_index_result` +- if an emergency rebuild is needed, run `python -m app.rcon_historical_player_stats refresh-player-search-index` - confirm the command reports rows for `all-servers`, `comunidad-hispana-01` and `comunidad-hispana-02` - call `/api/stats/players/search?q=&limit=5` - verify response metadata reports `read_model=player-search-index` diff --git a/docs/ranking-snapshot-read-model-plan.md b/docs/ranking-snapshot-read-model-plan.md index 65e31e4..1fef3cc 100644 --- a/docs/ranking-snapshot-read-model-plan.md +++ b/docs/ranking-snapshot-read-model-plan.md @@ -292,7 +292,12 @@ Supported manual parameters: Current implementation note: - `generate-ranking-snapshot` remains unitary per command invocation for explicit manual control - `refresh-ranking-snapshots --limit 30` generates the full weekly/monthly public matrix in one run -- the periodic historical runner in `backend/app/historical_runner.py` should invoke that bulk refresh as part of the normal backend refresh cycle +- the periodic historical runner in `backend/app/historical_runner.py` invokes that bulk refresh as part of the normal backend refresh cycle +- current cycle order is: + - existing RCON ingestion/materialization cycle + - `player_search_index` + - `player_period_stats` + - `ranking_snapshots` - per-combination failures should be reported without aborting the entire matrix refresh ## Recommended Combinations @@ -323,6 +328,8 @@ Operational guidance: - regenerate after manual backfill - regenerate after metric SQL changes that affect ranking totals or ordering - when using the periodic backend runner, keep ranking refresh attached to the same recurring cycle rather than a separate scheduler unless operational load proves otherwise +- the runner inherits its cadence from `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS` +- fallback runtime remains preserved for `/api/ranking` if a requested snapshot is missing or unavailable ## Ready Vs Fallback diff --git a/scripts/run-stats-validation.ps1 b/scripts/run-stats-validation.ps1 index 8fd3a90..b5db5fe 100644 --- a/scripts/run-stats-validation.ps1 +++ b/scripts/run-stats-validation.ps1 @@ -127,6 +127,7 @@ sys.path.insert(0, "backend") from app.routes import resolve_get_payload from app.config import use_postgres_rcon_storage import app.postgres_rcon_storage as postgres_rcon_storage +import app.historical_runner as historical_runner import app.rcon_historical_leaderboards as ranking_leaderboards import app.rcon_historical_player_stats as player_search_stats @@ -1217,6 +1218,186 @@ def validate_player_period_stats_fallbacks(): pass +def validate_historical_runner_read_model_refresh_cycle(): + original_backend_writer_lock = historical_runner.backend_writer_lock + original_build_writer_lock_holder = historical_runner.build_writer_lock_holder + original_run_primary_rcon_capture = historical_runner._run_primary_rcon_capture + original_resolve_classic_fallback_policy = historical_runner._resolve_classic_fallback_policy + original_rcon_capture_has_new_useful_data = historical_runner._rcon_capture_has_new_useful_data + original_generate_historical_snapshots = historical_runner.generate_historical_snapshots + original_build_elo_mmr_rebuild_policy = historical_runner._build_elo_mmr_rebuild_policy + original_rebuild_elo_mmr_models = historical_runner.rebuild_elo_mmr_models + original_refresh_player_search_index = historical_runner.refresh_player_search_index + original_refresh_player_period_stats = historical_runner.refresh_player_period_stats + original_refresh_ranking_snapshots = historical_runner.refresh_ranking_snapshots + original_maybe_run_database_maintenance = historical_runner._maybe_run_database_maintenance + original_emit_json_log = historical_runner._emit_json_log + + @contextmanager + def fake_backend_writer_lock(holder): + yield + + captured_logs = [] + call_order = [] + + historical_runner.backend_writer_lock = fake_backend_writer_lock + historical_runner.build_writer_lock_holder = lambda label: label + historical_runner._run_primary_rcon_capture = lambda: { + "status": "ok", + "targets": [{"target_key": "comunidad-hispana-01", "sample_inserted": True}], + "totals": { + "samples_inserted": 1, + "admin_log_events_inserted": 0, + "materialized_matches_inserted": 0, + }, + } + historical_runner._resolve_classic_fallback_policy = lambda **kwargs: ( + False, + "validation-rcon-primary-cycle", + ) + historical_runner._rcon_capture_has_new_useful_data = lambda payload: True + historical_runner.generate_historical_snapshots = lambda **kwargs: { + "status": "ok", + "generated_at": "2026-06-09T08:00:00Z", + } + historical_runner._build_elo_mmr_rebuild_policy = lambda **kwargs: { + "due": False, + "policy": "validation-policy", + "last_generated_at": None, + "samples_since_last_rebuild": 1, + "minutes_since_last_rebuild": None, + "rebuild_interval_minutes": 60, + "min_new_samples": 10, + } + historical_runner.rebuild_elo_mmr_models = lambda: {"status": "ok"} + historical_runner._maybe_run_database_maintenance = lambda: { + "status": "skipped", + "reason": "validation-disabled", + } + historical_runner._emit_json_log = lambda payload: captured_logs.append(payload) + + def fake_refresh_player_search_index(**kwargs): + call_order.append("player-search-index") + return {"status": "ok", "total_rows": 3} + + def fake_refresh_player_period_stats(**kwargs): + call_order.append("player-period-stats") + return {"status": "ok", "total_rows": 9} + + def fake_refresh_ranking_snapshots(**kwargs): + call_order.append("ranking-snapshots") + return { + "status": "ok", + "totals": {"succeeded": 36, "failed": 0, "skipped_regeneration": 0}, + } + + historical_runner.refresh_player_search_index = fake_refresh_player_search_index + historical_runner.refresh_player_period_stats = fake_refresh_player_period_stats + historical_runner.refresh_ranking_snapshots = fake_refresh_ranking_snapshots + + try: + result = historical_runner._run_refresh_with_retries( + max_retries=0, + retry_delay_seconds=0, + server_slug=None, + max_pages=None, + page_size=None, + run_number=1, + ) + require(result.get("status") == "ok", "Historical runner should return ok when all periodic refresh steps succeed.") + require( + call_order == ["player-search-index", "player-period-stats", "ranking-snapshots"], + "Historical runner should refresh player_search_index, then player_period_stats, then ranking_snapshots.", + ) + require( + (result.get("player_search_index_result") or {}).get("status") == "ok", + "Historical runner should report player_search_index_result separately.", + ) + require( + (result.get("player_period_stats_result") or {}).get("status") == "ok", + "Historical runner should report player_period_stats_result separately.", + ) + require( + (result.get("ranking_snapshot_result") or {}).get("status") == "ok", + "Historical runner should keep reporting ranking_snapshot_result separately.", + ) + require( + any( + log.get("event") == "player-search-index-refresh-started" + for log in captured_logs + if isinstance(log, dict) + ), + "Historical runner should log the player_search_index refresh start event.", + ) + require( + any( + log.get("event") == "player-period-stats-refresh-started" + for log in captured_logs + if isinstance(log, dict) + ), + "Historical runner should log the player_period_stats refresh start event.", + ) + + call_order.clear() + captured_logs.clear() + + def fake_refresh_player_search_index_failure(**kwargs): + call_order.append("player-search-index") + raise RuntimeError("forced player search refresh failure") + + historical_runner.refresh_player_search_index = fake_refresh_player_search_index_failure + result = historical_runner._run_refresh_with_retries( + max_retries=0, + retry_delay_seconds=0, + server_slug=None, + max_pages=None, + page_size=None, + run_number=1, + ) + require( + result.get("status") == "partial", + "Historical runner should return partial when one read-model refresh fails but the cycle continues.", + ) + require( + call_order == ["player-search-index", "player-period-stats", "ranking-snapshots"], + "Historical runner should still attempt the remaining refresh steps after a player search read-model failure.", + ) + require( + (result.get("player_search_index_result") or {}).get("status") == "error", + "Historical runner should expose the player_search_index refresh failure explicitly.", + ) + require( + (result.get("player_period_stats_result") or {}).get("status") == "ok", + "Historical runner should continue refreshing player_period_stats after a player_search_index failure.", + ) + require( + (result.get("ranking_snapshot_result") or {}).get("status") == "ok", + "Historical runner should continue refreshing ranking snapshots after a player_search_index failure.", + ) + require( + any( + log.get("event") == "player-search-index-refresh-failed" + for log in captured_logs + if isinstance(log, dict) + ), + "Historical runner should emit a dedicated failure log for player_search_index refresh failures.", + ) + finally: + historical_runner.backend_writer_lock = original_backend_writer_lock + historical_runner.build_writer_lock_holder = original_build_writer_lock_holder + historical_runner._run_primary_rcon_capture = original_run_primary_rcon_capture + historical_runner._resolve_classic_fallback_policy = original_resolve_classic_fallback_policy + historical_runner._rcon_capture_has_new_useful_data = original_rcon_capture_has_new_useful_data + historical_runner.generate_historical_snapshots = original_generate_historical_snapshots + historical_runner._build_elo_mmr_rebuild_policy = original_build_elo_mmr_rebuild_policy + historical_runner.rebuild_elo_mmr_models = original_rebuild_elo_mmr_models + historical_runner.refresh_player_search_index = original_refresh_player_search_index + historical_runner.refresh_player_period_stats = original_refresh_player_period_stats + historical_runner.refresh_ranking_snapshots = original_refresh_ranking_snapshots + historical_runner._maybe_run_database_maintenance = original_maybe_run_database_maintenance + historical_runner._emit_json_log = original_emit_json_log + + def cleanup_snapshot_fixture(db_path): connection = sqlite3.connect(db_path) with connection: @@ -1271,6 +1452,7 @@ validate_player_period_stats_cli_defaults() validate_fetch_player_stats_sql_contract() validate_player_period_stats_refresh_and_read_model() validate_player_period_stats_fallbacks() +validate_historical_runner_read_model_refresh_cycle() kd_metric_sql, _, _ = ranking_leaderboards._resolve_metric_sql("kd_ratio") require( @@ -1654,6 +1836,7 @@ print(json.dumps({ "ranking-snapshot-missing", "player-search-index", "player-period-stats", + "historical-runner-player-read-model-refresh", ], "annual_snapshot_status": annual_data.get("snapshot_status"), "global_ranking_annual_snapshot_status": annual_ranking_data.get("snapshot_status"),