diff --git a/ai/system-metrics.md b/ai/system-metrics.md index 6e1b458..9ea39f4 100644 --- a/ai/system-metrics.md +++ b/ai/system-metrics.md @@ -17,3 +17,4 @@ Date | Task | Duration | Result | Notes 2026-03-19T13:59:33 | worker-cycle | 641.24 sec | success | codex-runner 2026-05-19T08:39:12 | worker-cycle | 663.71 sec | success | codex-runner 2026-05-19T17:20:44 | worker-cycle | 2240.22 sec | success | codex-runner +2026-05-21T14:43:44 | worker-cycle | 5580.9 sec | success | codex-runner diff --git a/ai/tasks/pending/TASK-147-persist-scoreboard-list-candidates.md b/ai/tasks/done/TASK-147-persist-scoreboard-list-candidates.md similarity index 74% rename from ai/tasks/pending/TASK-147-persist-scoreboard-list-candidates.md rename to ai/tasks/done/TASK-147-persist-scoreboard-list-candidates.md index 52733e5..9b10103 100644 --- a/ai/tasks/pending/TASK-147-persist-scoreboard-list-candidates.md +++ b/ai/tasks/done/TASK-147-persist-scoreboard-list-candidates.md @@ -1,7 +1,7 @@ --- id: TASK-147 title: Persist public scoreboard list matches as RCON candidates -status: pending +status: done type: backend team: Backend Senior supporting_teams: @@ -157,6 +157,42 @@ outcome. Document validation, candidate upsert decisions, counters added to the JSON report, and any follow-up task instead of widening this task. +- Persisted list payload candidates before detail fetch through a focused + PostgreSQL upsert for `rcon_scoreboard_match_candidates`. +- Built list candidate URLs only from the trusted scoreboard origin catalog and + numeric public match IDs. List rows are accepted only when the selected + historical server slug/base URL/server number and payload `server_number` + agree with the trusted origin. +- Kept detail fetching for the existing `historical_*` match/player enrichment + path. The backfill report now exposes `list_candidates_inserted`, + `list_candidates_updated` and `list_candidates_skipped` independently from + the detail match counters. +- Added focused regression coverage for the Foy list row `1562115`; detail + fetch failure still leaves the list candidate present, and a second run + updates the same candidate key instead of creating a duplicate. +- Validation: + - `python -m compileall backend/app` + - `python -m unittest discover -s tests -p "*scoreboard*"` from `backend/` + passed with pre-existing SQLite `ResourceWarning` output in the scoreboard + regression file. + - `python -m app.storage_diagnostics` from `backend/` + - `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1` + passed; the script reports no product integration tests configured for its + platform-only scope. + - `docker compose --profile advanced up -d --build backend frontend postgres` + - `docker compose exec backend python -m app.scoreboard_candidate_backfill + --server comunidad-hispana-02 --from 2026-05-20T00:00:00Z --to + 2026-05-21T23:59:59Z --max-pages 5 --page-size 100` + reported list candidate inserts/updates with no errors. + - Queried PostgreSQL and verified external match id `1562115` exists for + `comunidad-hispana-02` with URL + `https://scoreboard.comunidadhll.es:5443/games/1562115`. + - Invoked `/api/historical/matches/detail` for + `comunidad-hispana-02:1779310451:1779315851:foywarfare`; payload `found` + was true and `match_url` used the expected trusted URL. +- Scope review used `git diff --name-only`; changed implementation files stay + within this task plus this workflow task file. + ## Change Budget - Prefer fewer than 5 modified files. diff --git a/ai/tasks/pending/TASK-148-relink-existing-rcon-matches-to-scoreboard-candidates.md b/ai/tasks/done/TASK-148-relink-existing-rcon-matches-to-scoreboard-candidates.md similarity index 71% rename from ai/tasks/pending/TASK-148-relink-existing-rcon-matches-to-scoreboard-candidates.md rename to ai/tasks/done/TASK-148-relink-existing-rcon-matches-to-scoreboard-candidates.md index 98490e9..a32fec4 100644 --- a/ai/tasks/pending/TASK-148-relink-existing-rcon-matches-to-scoreboard-candidates.md +++ b/ai/tasks/done/TASK-148-relink-existing-rcon-matches-to-scoreboard-candidates.md @@ -1,7 +1,7 @@ --- id: TASK-148 title: Relink existing RCON matches to scoreboard candidates -status: pending +status: done type: backend team: Backend Senior supporting_teams: @@ -155,6 +155,47 @@ Document correlation scoring choices, time-window derivation, ambiguity handling, relink command output, validation, and any follow-up task instead of expanding scope. +- Added `python -m app.rcon_scoreboard_relink` to scan already-materialized + ended RCON matches against the same trusted candidate resolution used by the + detail read model. The command reports `matches_scanned`, + `candidates_scanned`, `matches_linked`, `matches_skipped_no_candidate`, + `matches_skipped_ambiguous` and `errors` as JSON. +- Kept URL availability dynamic in the read model instead of adding a second + persisted URL column to RCON matches. The new correlation resolution summary + preserves the existing deterministic map/time/duration/score scoring path, + returns explicit low-confidence/no-candidate/ambiguous reasons, and exposes + the selected external candidate only to command callers. +- Reused the existing materialized window derivation for closed-at plus + server-time duration cases so relink and detail payloads evaluate the same + effective correlation window. +- Added a Foy regression for materialized match key + `comunidad-hispana-02:1779310451:1779315851:foywarfare`: relink reports a + safe link and detail returns + `https://scoreboard.comunidadhll.es:5443/games/1562115`. +- Validation: + - `python -m compileall backend/app` + - `python -m unittest discover -s tests -p "*scoreboard*"` from `backend/` + passed with existing SQLite `ResourceWarning` output in the scoreboard + regression file. + - `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1` + passed; the script reports no product integration tests configured for its + platform-only scope. + - `docker compose --profile advanced up -d --build backend frontend postgres` + - Re-ran `app.scoreboard_candidate_backfill` for the requested May 20-21 + window. Candidate refresh succeeded (`list_candidates_updated: 11`), while + detail enrichment returned a partial status because SQLite reported + `database is locked` for detail match `1562104`. + - `docker compose exec backend python -m app.rcon_scoreboard_relink --server + comunidad-hispana-02` reported 19 scanned matches, 13 linked matches, 6 + no-candidate skips, 0 ambiguous skips and no errors. + - HTTP detail verification returned `found: true` and the expected Foy URL. + Existing Carentan detail key + `comunidad-hispana-02:1779178461:1779183861:carentanwarfare` also kept a + trusted `match_url`. +- Scope review used `git diff --name-only`; relink changes stay within the + correlation/read-model command path, focused scoreboard tests and this task + file. + ## Change Budget - Prefer fewer than 5 modified files. diff --git a/ai/tasks/pending/TASK-149-scoreboard-correlation-diagnostics.md b/ai/tasks/done/TASK-149-scoreboard-correlation-diagnostics.md similarity index 69% rename from ai/tasks/pending/TASK-149-scoreboard-correlation-diagnostics.md rename to ai/tasks/done/TASK-149-scoreboard-correlation-diagnostics.md index 8adde63..4cd172c 100644 --- a/ai/tasks/pending/TASK-149-scoreboard-correlation-diagnostics.md +++ b/ai/tasks/done/TASK-149-scoreboard-correlation-diagnostics.md @@ -1,7 +1,7 @@ --- id: TASK-149 title: Add scoreboard correlation diagnostics -status: pending +status: done type: backend team: Backend Senior supporting_teams: @@ -139,6 +139,42 @@ Document diagnostic fields, known Foy output behavior, documentation added, validation performed, and any follow-up task instead of expanding normal UI payloads. +- Added `python -m app.scoreboard_correlation_diagnostics` for one + materialized RCON match. JSON output includes match key, server, map, + timestamps, duration, score, candidate search window, safe top candidates, + selected candidate and `final_reason`. +- Candidate summaries expose trusted public `match_url` values only. Untrusted + URLs are omitted and receive `unsafe-url`; non-scoring candidates report + `map-or-window-mismatch` where applicable. +- Kept diagnostics out of normal detail and frontend payloads. The command + reuses the materialized correlation window and safe candidate scoring that + relink/detail use. +- Added `docs/scoreboard-correlation-debugging.md` with the missing-button + sequence: candidate backfill, relink scan, diagnostics command and detail + endpoint check. +- Extended focused Foy coverage so diagnostics select external match id + `1562115` with the expected Foy candidate summary. +- Validation: + - `python -m compileall backend/app` + - `python -m unittest discover -s tests -p "*scoreboard*"` from `backend/` + passed with existing SQLite `ResourceWarning` output in the scoreboard + regression file. + - `python -m app.scoreboard_correlation_diagnostics --server + comunidad-hispana-02 --match + comunidad-hispana-02:1779310451:1779315851:foywarfare` from `backend/` + returned `final_reason: linked` and selected candidate `1562115`. + - `python -m app.storage_diagnostics` + - `node --check frontend/assets/js/historico-partida.js` + - `node --check frontend/assets/js/historico.js` + - `node --check frontend/assets/js/historico-recent-live.js` + - `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1` + exited successfully after reporting its platform checks. Its later local + route probe printed a SQLite `database disk image is malformed` traceback + from `historical_storage`; this task did not alter or repair runtime DB + files. +- Scope review used `git diff --name-only`; diagnostics changes stay in the + correlation command/test/doc path and this task file. + ## Change Budget - Prefer fewer than 5 modified files. diff --git a/backend/app/postgres_rcon_storage.py b/backend/app/postgres_rcon_storage.py index 75f307c..4fad140 100644 --- a/backend/app/postgres_rcon_storage.py +++ b/backend/app/postgres_rcon_storage.py @@ -668,6 +668,61 @@ def upsert_scoreboard_candidates( return inserted_or_updated +def upsert_scoreboard_candidate( + *, + server_slug: str, + candidate: Mapping[str, object], +) -> str: + """Persist one trusted scoreboard correlation candidate and report the upsert path.""" + external_match_id = str(candidate.get("external_match_id") or "").strip() + match_url = str(candidate.get("match_url") or "").strip() + if not external_match_id or not match_url: + return "skipped" + + initialize_postgres_rcon_storage() + with connect_postgres() as connection: + existing = connection.execute( + """ + SELECT id + FROM rcon_scoreboard_match_candidates + WHERE server_slug = %s AND external_match_id = %s + LIMIT 1 + """, + (server_slug, external_match_id), + ).fetchone() + connection.execute( + """ + INSERT INTO rcon_scoreboard_match_candidates ( + server_slug, external_match_id, started_at, ended_at, map_name, + map_pretty_name, allied_score, axis_score, player_count, match_url + ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s) + ON CONFLICT(server_slug, external_match_id) DO UPDATE SET + started_at = EXCLUDED.started_at, + ended_at = EXCLUDED.ended_at, + map_name = EXCLUDED.map_name, + map_pretty_name = EXCLUDED.map_pretty_name, + allied_score = EXCLUDED.allied_score, + axis_score = EXCLUDED.axis_score, + player_count = EXCLUDED.player_count, + match_url = EXCLUDED.match_url, + updated_at = CURRENT_TIMESTAMP + """, + ( + server_slug, + external_match_id, + candidate.get("started_at"), + candidate.get("ended_at"), + candidate.get("map_name"), + candidate.get("map_pretty_name"), + candidate.get("allied_score"), + candidate.get("axis_score"), + candidate.get("player_count"), + match_url, + ), + ) + return "updated" if existing else "inserted" + + def count_migrated_tables() -> dict[str, int]: table_names = ( "rcon_admin_log_events", diff --git a/backend/app/rcon_historical_read_model.py b/backend/app/rcon_historical_read_model.py index 20ba791..299ff5d 100644 --- a/backend/app/rcon_historical_read_model.py +++ b/backend/app/rcon_historical_read_model.py @@ -206,10 +206,9 @@ def get_rcon_historical_match_detail( def _build_materialized_recent_item(item: dict[str, object]) -> dict[str, object]: - server_slug = item.get("external_server_id") or item.get("target_key") timestamps = _build_materialized_timestamp_payload(item) - correlation_window = _build_materialized_scoreboard_correlation_window(item, timestamps) player_count = _resolve_materialized_player_count(item) + scoreboard_correlation = build_materialized_scoreboard_correlation_input(item) return { "server": { "slug": item.get("target_key"), @@ -247,13 +246,7 @@ def _build_materialized_recent_item(item: dict[str, object]) -> dict[str, object else SESSION_RESULT_SOURCE ), "match_url": resolve_rcon_scoreboard_match_url( - server_slug=server_slug, - map_name=item.get("map_pretty_name") or item.get("map_name"), - started_at=correlation_window["started_at"], - ended_at=correlation_window["ended_at"], - duration_seconds=_calculate_match_duration_seconds(item), - allied_score=item.get("allied_score"), - axis_score=item.get("axis_score"), + **scoreboard_correlation, ), "capabilities": describe_rcon_historical_read_model()["capabilities"], } @@ -387,6 +380,23 @@ def _build_materialized_scoreboard_correlation_window( } +def build_materialized_scoreboard_correlation_input( + item: dict[str, object], +) -> dict[str, object]: + """Build safe candidate correlation inputs for one materialized RCON match.""" + timestamps = _build_materialized_timestamp_payload(item) + correlation_window = _build_materialized_scoreboard_correlation_window(item, timestamps) + return { + "server_slug": item.get("external_server_id") or item.get("target_key"), + "map_name": item.get("map_pretty_name") or item.get("map_name"), + "started_at": correlation_window["started_at"], + "ended_at": correlation_window["ended_at"], + "duration_seconds": _calculate_match_duration_seconds(item), + "allied_score": item.get("allied_score"), + "axis_score": item.get("axis_score"), + } + + def _merge_recent_items( primary_items: list[dict[str, object]], fallback_items: list[dict[str, object]], diff --git a/backend/app/rcon_scoreboard_correlation.py b/backend/app/rcon_scoreboard_correlation.py index 2f9a5f2..4465fca 100644 --- a/backend/app/rcon_scoreboard_correlation.py +++ b/backend/app/rcon_scoreboard_correlation.py @@ -30,12 +30,42 @@ def resolve_rcon_scoreboard_match_url( db_path: Path | None = None, ) -> str | None: """Return a trusted scoreboard URL for an RCON window only on strong evidence.""" + resolution = resolve_rcon_scoreboard_correlation( + server_slug=server_slug, + map_name=map_name, + started_at=started_at, + ended_at=ended_at, + duration_seconds=duration_seconds, + player_count=player_count, + peak_players=peak_players, + allied_score=allied_score, + axis_score=axis_score, + db_path=db_path, + ) + match_url = resolution.get("match_url") + return str(match_url) if match_url else None + + +def resolve_rcon_scoreboard_correlation( + *, + server_slug: object, + map_name: object, + started_at: object, + ended_at: object, + duration_seconds: object = None, + player_count: object = None, + peak_players: object = None, + allied_score: object = None, + axis_score: object = None, + db_path: Path | None = None, +) -> dict[str, object]: + """Return a safe candidate selection summary for one RCON match window.""" normalized_server_slug = str(server_slug or "").strip() normalized_map = normalize_map_name(map_name) rcon_start = _parse_timestamp(started_at) rcon_end = _parse_timestamp(ended_at) if not normalized_server_slug or not normalized_map or not rcon_start or not rcon_end: - return None + return {"match_url": None, "candidate_count": 0, "reason": "invalid-rcon-window"} if rcon_end < rcon_start: rcon_start, rcon_end = rcon_end, rcon_start @@ -60,15 +90,127 @@ def resolve_rcon_scoreboard_match_url( is not None ] if not scored_candidates: - return None + return { + "match_url": None, + "candidate_count": len(candidates), + "reason": "no-safe-candidate", + } scored_candidates.sort(key=lambda item: item["score"], reverse=True) best = scored_candidates[0] if int(best["score"]) < MIN_CONFIDENCE_SCORE: - return None + return { + "match_url": None, + "candidate_count": len(candidates), + "reason": "low-confidence", + } if len(scored_candidates) > 1 and int(scored_candidates[1]["score"]) >= int(best["score"]): - return None - return str(best["match_url"]) + return { + "match_url": None, + "candidate_count": len(candidates), + "reason": "ambiguous-candidate", + } + return { + "match_url": str(best["match_url"]), + "candidate_count": len(candidates), + "reason": "linked", + "selected_candidate": { + "external_match_id": best.get("external_match_id"), + "correlation_score": int(best["score"]), + }, + } + + +def diagnose_rcon_scoreboard_correlation( + *, + server_slug: object, + map_name: object, + started_at: object, + ended_at: object, + duration_seconds: object = None, + player_count: object = None, + peak_players: object = None, + allied_score: object = None, + axis_score: object = None, + db_path: Path | None = None, +) -> dict[str, object]: + """Describe safe candidate scoring for a single RCON correlation window.""" + normalized_server_slug = str(server_slug or "").strip() + normalized_map = normalize_map_name(map_name) + rcon_start = _parse_timestamp(started_at) + rcon_end = _parse_timestamp(ended_at) + if not normalized_server_slug or not normalized_map or not rcon_start or not rcon_end: + return { + "candidate_search_window": { + "started_at": started_at, + "ended_at": ended_at, + "candidate_limit": MAX_CANDIDATES, + }, + "candidate_count": 0, + "top_candidates": [], + "selected_candidate": None, + "final_reason": "invalid-rcon-window", + } + if rcon_end < rcon_start: + rcon_start, rcon_end = rcon_end, rcon_start + + candidates = _list_persisted_scoreboard_candidates( + server_slug=normalized_server_slug, + db_path=db_path or get_storage_path(), + ) + resolution = resolve_rcon_scoreboard_correlation( + server_slug=server_slug, + map_name=map_name, + started_at=started_at, + ended_at=ended_at, + duration_seconds=duration_seconds, + player_count=player_count, + peak_players=peak_players, + allied_score=allied_score, + axis_score=axis_score, + db_path=db_path, + ) + summaries = [ + _diagnostic_candidate_summary( + candidate, + server_slug=normalized_server_slug, + normalized_map=normalized_map, + rcon_start=rcon_start, + rcon_end=rcon_end, + duration_seconds=_coerce_int(duration_seconds), + player_count=_coerce_int(player_count), + peak_players=_coerce_int(peak_players), + allied_score=_coerce_int(allied_score), + axis_score=_coerce_int(axis_score), + ) + for candidate in candidates + ] + summaries.sort( + key=lambda item: ( + -int(item["correlation_score"] or -1), + str(item.get("external_match_id") or ""), + ) + ) + selected_id = ( + resolution.get("selected_candidate", {}).get("external_match_id") + if isinstance(resolution.get("selected_candidate"), dict) + else None + ) + selected_candidate = next( + (item for item in summaries if item.get("external_match_id") == selected_id), + None, + ) + return { + "candidate_search_window": { + "started_at": rcon_start.isoformat().replace("+00:00", "Z"), + "ended_at": rcon_end.isoformat().replace("+00:00", "Z"), + "candidate_limit": MAX_CANDIDATES, + }, + "candidate_count": len(candidates), + "top_candidates": summaries[:5], + "selected_candidate": selected_candidate, + "final_reason": resolution["reason"], + } def _list_persisted_scoreboard_candidates( @@ -221,10 +363,61 @@ def _score_candidate( return None return { "score": score, + "external_match_id": candidate.get("external_match_id"), "match_url": candidate["match_url"], } +def _diagnostic_candidate_summary( + candidate: dict[str, object], + *, + server_slug: str, + normalized_map: str, + rcon_start: datetime, + rcon_end: datetime, + duration_seconds: int | None, + player_count: int | None, + peak_players: int | None, + allied_score: int | None, + axis_score: int | None, +) -> dict[str, object]: + match_url = resolve_trusted_scoreboard_match_url(candidate.get("match_url"), server_slug) + safe_candidate = {**candidate, "match_url": match_url} if match_url else None + scored = ( + _score_candidate( + safe_candidate, + normalized_map=normalized_map, + rcon_start=rcon_start, + rcon_end=rcon_end, + duration_seconds=duration_seconds, + player_count=player_count, + peak_players=peak_players, + allied_score=allied_score, + axis_score=axis_score, + ) + if safe_candidate + else None + ) + map_label = candidate.get("map_pretty_name") or candidate.get("map_name") + summary = { + "external_match_id": candidate.get("external_match_id"), + "started_at": candidate.get("started_at"), + "ended_at": candidate.get("ended_at"), + "map": map_label, + "score": { + "allied_score": _coerce_int(candidate.get("allied_score")), + "axis_score": _coerce_int(candidate.get("axis_score")), + }, + "match_url": match_url, + "correlation_score": int(scored["score"]) if scored else None, + } + if not match_url: + summary["rejection_reason"] = "unsafe-url" + elif scored is None: + summary["rejection_reason"] = "map-or-window-mismatch" + return summary + + def _overlap_seconds( first_start: datetime, first_end: datetime, diff --git a/backend/app/rcon_scoreboard_relink.py b/backend/app/rcon_scoreboard_relink.py new file mode 100644 index 0000000..3835cb2 --- /dev/null +++ b/backend/app/rcon_scoreboard_relink.py @@ -0,0 +1,78 @@ +"""Report safe scoreboard links for existing materialized RCON matches.""" + +from __future__ import annotations + +import argparse +import json +from collections.abc import Iterable +from pathlib import Path + +from .rcon_admin_log_materialization import list_materialized_rcon_matches +from .rcon_historical_read_model import build_materialized_scoreboard_correlation_input +from .rcon_scoreboard_correlation import resolve_rcon_scoreboard_correlation + + +DEFAULT_LIMIT = 500 + + +def relink_materialized_matches( + *, + server_key: str | None = None, + limit: int = DEFAULT_LIMIT, + db_path: Path | None = None, +) -> dict[str, object]: + """Scan existing matches against trusted candidates used by the detail read model.""" + matches = list_materialized_rcon_matches( + target_key=server_key, + only_ended=True, + limit=limit, + db_path=db_path, + ) + report: dict[str, object] = { + "matches_scanned": len(matches), + "candidates_scanned": 0, + "matches_linked": 0, + "matches_skipped_no_candidate": 0, + "matches_skipped_ambiguous": 0, + "errors": [], + } + for match in matches: + try: + resolution = resolve_rcon_scoreboard_correlation( + **build_materialized_scoreboard_correlation_input(match), + db_path=db_path, + ) + except Exception as exc: + report["errors"].append( + {"match_key": match.get("match_key"), "message": str(exc)} + ) + continue + report["candidates_scanned"] += int(resolution.get("candidate_count") or 0) + if resolution.get("match_url"): + report["matches_linked"] += 1 + elif resolution.get("reason") == "ambiguous-candidate": + report["matches_skipped_ambiguous"] += 1 + else: + report["matches_skipped_no_candidate"] += 1 + return report + + +def main(argv: Iterable[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Resolve trusted scoreboard links for materialized RCON matches." + ) + parser.add_argument("--server", dest="server_key") + parser.add_argument("--limit", type=int, default=DEFAULT_LIMIT) + parser.add_argument("--db-path", type=Path, default=None) + args = parser.parse_args(list(argv) if argv is not None else None) + report = relink_materialized_matches( + server_key=args.server_key, + limit=max(1, args.limit), + db_path=args.db_path, + ) + print(json.dumps(report, ensure_ascii=False, indent=2)) + return 0 if not report["errors"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/app/scoreboard_candidate_backfill.py b/backend/app/scoreboard_candidate_backfill.py index 03b72ff..5db8a08 100644 --- a/backend/app/scoreboard_candidate_backfill.py +++ b/backend/app/scoreboard_candidate_backfill.py @@ -5,11 +5,17 @@ from __future__ import annotations import argparse import json from datetime import datetime, timezone +from collections.abc import Mapping from typing import Iterable from .historical_storage import initialize_historical_storage, list_historical_servers, upsert_historical_match +from .postgres_rcon_storage import upsert_scoreboard_candidate from .providers.public_scoreboard_provider import PublicScoreboardHistoricalDataSource -from .scoreboard_origins import list_trusted_public_scoreboard_origins +from .scoreboard_origins import ( + build_trusted_scoreboard_match_url, + get_trusted_public_scoreboard_origin, + list_trusted_public_scoreboard_origins, +) DEFAULT_MAX_PAGES = 20 DEFAULT_PAGE_SIZE = 100 @@ -34,8 +40,19 @@ def run_backfill(*, server: dict[str, object], start_at: datetime, end_at: datet provider = PublicScoreboardHistoricalDataSource() server_slug = str(server["slug"]) base_url = str(server["scoreboard_base_url"]) - counters = {"pages_processed": 0, "candidates_seen": 0, "candidates_inserted": 0, "candidates_updated": 0, "player_rows_inserted": 0, "player_rows_updated": 0} + counters = { + "pages_processed": 0, + "candidates_seen": 0, + "list_candidates_inserted": 0, + "list_candidates_updated": 0, + "list_candidates_skipped": 0, + "candidates_inserted": 0, + "candidates_updated": 0, + "player_rows_inserted": 0, + "player_rows_updated": 0, + } errors: list[dict[str, object]] = [] + skipped_unsafe_urls = 0 stopped_after_window = False for page in range(1, max_pages + 1): try: @@ -56,6 +73,27 @@ def run_backfill(*, server: dict[str, object], start_at: datetime, end_at: datet continue if ref_time and ref_time >= end_at: continue + candidate = _build_list_candidate(server=server, match=match) + if candidate is None: + counters["list_candidates_skipped"] += 1 + skipped_unsafe_urls += int(_list_candidate_url_is_unsafe(server=server, match=match)) + else: + try: + outcome = upsert_scoreboard_candidate( + server_slug=server_slug, + candidate=candidate, + ) + except Exception as exc: + counters["list_candidates_skipped"] += 1 + errors.append( + { + "stage": "upsert_list_scoreboard_candidate", + "match_id": candidate["external_match_id"], + "message": str(exc), + } + ) + else: + counters[f"list_candidates_{outcome}"] += 1 match_id = _stringify(match.get("id")) if match_id: ids.append(match_id) @@ -77,7 +115,7 @@ def run_backfill(*, server: dict[str, object], start_at: datetime, end_at: datet counters["player_rows_updated"] += _coerce_int(delta.get("player_rows_updated")) if stopped_after_window: break - return {"status": "ok" if not errors else "partial", "server": server_slug, "scoreboard_base_url": base_url, "requested_window": {"from": _format_timestamp(start_at), "to": _format_timestamp(end_at)}, "stopped_after_window": stopped_after_window, "skipped_unsafe_urls": 0, "errors": errors, **counters} + return {"status": "ok" if not errors else "partial", "server": server_slug, "scoreboard_base_url": base_url, "requested_window": {"from": _format_timestamp(start_at), "to": _format_timestamp(end_at)}, "stopped_after_window": stopped_after_window, "skipped_unsafe_urls": skipped_unsafe_urls, "errors": errors, **counters} def build_arg_parser() -> argparse.ArgumentParser: @@ -137,6 +175,65 @@ def _pick_match_timestamp(match: dict[str, object]) -> object: return None +def _build_list_candidate( + *, + server: Mapping[str, object], + match: Mapping[str, object], +) -> dict[str, object] | None: + server_slug = _stringify(server.get("slug")) + external_match_id = _stringify(match.get("id")) + origin = get_trusted_public_scoreboard_origin(server_slug) + map_payload = match.get("map") + result_payload = match.get("result") + if ( + origin is None + or not external_match_id + or not external_match_id.isdigit() + or str(server.get("scoreboard_base_url") or "").strip() != origin.base_url + or _coerce_optional_int(server.get("server_number")) != origin.server_number + or _coerce_optional_int(match.get("server_number")) != origin.server_number + or not isinstance(map_payload, Mapping) + or not isinstance(result_payload, Mapping) + ): + return None + + started_at = _stringify(match.get("start")) + ended_at = _stringify(match.get("end")) + match_url = build_trusted_scoreboard_match_url( + server_slug=server_slug, + external_match_id=external_match_id, + ) + if not started_at or not ended_at or not match_url: + return None + return { + "external_match_id": external_match_id, + "started_at": started_at, + "ended_at": ended_at, + "map_name": _stringify(map_payload.get("id") or map_payload.get("name")), + "map_pretty_name": _stringify(map_payload.get("pretty_name")), + "allied_score": _coerce_optional_int(result_payload.get("allied")), + "axis_score": _coerce_optional_int(result_payload.get("axis")), + "player_count": _coerce_optional_int(match.get("player_count")), + "match_url": match_url, + } + + +def _list_candidate_url_is_unsafe( + *, + server: Mapping[str, object], + match: Mapping[str, object], +) -> bool: + external_match_id = _stringify(match.get("id")) + return bool( + external_match_id + and build_trusted_scoreboard_match_url( + server_slug=server.get("slug"), + external_match_id=external_match_id, + ) + is None + ) + + def _stringify(value: object) -> str | None: if value is None: return None @@ -151,5 +248,12 @@ def _coerce_int(value: object) -> int: return 0 +def _coerce_optional_int(value: object) -> int | None: + try: + return None if value is None else int(value) + except (TypeError, ValueError): + return None + + if __name__ == "__main__": raise SystemExit(main()) diff --git a/backend/app/scoreboard_correlation_diagnostics.py b/backend/app/scoreboard_correlation_diagnostics.py new file mode 100644 index 0000000..0b88d8f --- /dev/null +++ b/backend/app/scoreboard_correlation_diagnostics.py @@ -0,0 +1,83 @@ +"""JSON diagnostics for missing materialized RCON scoreboard links.""" + +from __future__ import annotations + +import argparse +import json +from collections.abc import Iterable +from pathlib import Path + +from .rcon_admin_log_materialization import get_materialized_rcon_match_detail +from .rcon_historical_read_model import build_materialized_scoreboard_correlation_input +from .rcon_scoreboard_correlation import diagnose_rcon_scoreboard_correlation + + +def inspect_materialized_match_correlation( + *, + server_key: str, + match_key: str, + db_path: Path | None = None, +) -> dict[str, object]: + """Return safe scoreboard correlation diagnostics for one materialized match.""" + materialized = get_materialized_rcon_match_detail( + server_key=server_key, + match_key=match_key, + db_path=db_path, + ) + if materialized is None: + return { + "rcon_match_key": match_key, + "server": server_key, + "candidate_count": 0, + "top_candidates": [], + "selected_candidate": None, + "final_reason": "rcon-match-not-found", + } + + match = materialized["match"] + correlation_input = build_materialized_scoreboard_correlation_input(match) + correlation = diagnose_rcon_scoreboard_correlation( + **correlation_input, + db_path=db_path, + ) + return { + "rcon_match_key": match.get("match_key"), + "server": match.get("external_server_id") or match.get("target_key"), + "map": match.get("map_pretty_name") or match.get("map_name"), + "started_at": match.get("started_at"), + "ended_at": match.get("ended_at"), + "closed_at": match.get("ended_at") or match.get("started_at"), + "duration_seconds": correlation_input.get("duration_seconds"), + "score": { + "allied_score": match.get("allied_score"), + "axis_score": match.get("axis_score"), + "winner": match.get("winner"), + }, + **correlation, + } + + +def main(argv: Iterable[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Explain scoreboard candidate correlation for one RCON match." + ) + parser.add_argument("--server", required=True) + parser.add_argument("--match", dest="match_key", required=True) + parser.add_argument("--db-path", type=Path, default=None) + args = parser.parse_args(list(argv) if argv is not None else None) + print( + json.dumps( + inspect_materialized_match_correlation( + server_key=args.server, + match_key=args.match_key, + db_path=args.db_path, + ), + ensure_ascii=False, + indent=2, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/app/scoreboard_origins.py b/backend/app/scoreboard_origins.py index a58854e..1aaf1b3 100644 --- a/backend/app/scoreboard_origins.py +++ b/backend/app/scoreboard_origins.py @@ -79,3 +79,19 @@ def resolve_trusted_scoreboard_match_url( if candidate_parts.params or candidate_parts.query or candidate_parts.fragment: return None return candidate + + +def build_trusted_scoreboard_match_url( + *, + server_slug: object, + external_match_id: object, +) -> str | None: + """Build a trusted scoreboard match URL from one numeric public match id.""" + origin = get_trusted_public_scoreboard_origin(server_slug) + match_id = str(external_match_id or "").strip() + if origin is None or not match_id.isdigit(): + return None + return resolve_trusted_scoreboard_match_url( + f"{origin.base_url}/games/{match_id}", + origin.slug, + ) diff --git a/backend/tests/test_scoreboard_match_links.py b/backend/tests/test_scoreboard_match_links.py index 210e00e..2d30d9a 100644 --- a/backend/tests/test_scoreboard_match_links.py +++ b/backend/tests/test_scoreboard_match_links.py @@ -8,7 +8,9 @@ import sqlite3 import tempfile import unittest from pathlib import Path +from unittest.mock import patch +from app.scoreboard_candidate_backfill import run_backfill from app.historical_storage import ( get_historical_match_detail, initialize_historical_storage, @@ -19,9 +21,80 @@ from app.rcon_historical_storage import initialize_rcon_historical_storage from app.rcon_historical_storage import persist_rcon_historical_sample from app.rcon_historical_storage import start_rcon_historical_capture_run from app.rcon_historical_read_model import get_rcon_historical_match_detail +from app.rcon_admin_log_materialization import materialize_rcon_admin_log +from app.rcon_admin_log_storage import persist_rcon_admin_log_entries +from app.rcon_scoreboard_relink import relink_materialized_matches +from app.scoreboard_correlation_diagnostics import inspect_materialized_match_correlation class PersistedScoreboardMatchLinkTests(unittest.TestCase): + def test_list_backfill_persists_foy_candidate_before_detail_fetch_failure(self) -> None: + stored: dict[tuple[str, str], dict[str, object]] = {} + + class FoyListProvider: + def fetch_match_page(self, *, base_url: str, page: int, limit: int) -> dict[str, object]: + return {"maps": [_foy_list_match()]} if page == 1 else {"maps": []} + + def fetch_match_details( + self, + *, + base_url: str, + match_ids: list[str], + max_workers: int, + ) -> list[dict[str, object]]: + raise RuntimeError("detail endpoint unavailable") + + def fake_upsert(*, server_slug: str, candidate: dict[str, object]) -> str: + key = (server_slug, str(candidate["external_match_id"])) + outcome = "updated" if key in stored else "inserted" + stored[key] = dict(candidate) + return outcome + + server = { + "slug": "comunidad-hispana-02", + "scoreboard_base_url": "https://scoreboard.comunidadhll.es:5443", + "server_number": 2, + } + with ( + patch("app.scoreboard_candidate_backfill.initialize_historical_storage"), + patch( + "app.scoreboard_candidate_backfill.PublicScoreboardHistoricalDataSource", + return_value=FoyListProvider(), + ), + patch( + "app.scoreboard_candidate_backfill.upsert_scoreboard_candidate", + side_effect=fake_upsert, + ), + ): + first = run_backfill( + server=server, + start_at=_backfill_timestamp("2026-05-20T00:00:00Z"), + end_at=_backfill_timestamp("2026-05-21T23:59:59Z"), + max_pages=2, + page_size=100, + detail_workers=1, + ) + second = run_backfill( + server=server, + start_at=_backfill_timestamp("2026-05-20T00:00:00Z"), + end_at=_backfill_timestamp("2026-05-21T23:59:59Z"), + max_pages=2, + page_size=100, + detail_workers=1, + ) + + candidate = stored[("comunidad-hispana-02", "1562115")] + self.assertEqual( + candidate["match_url"], + "https://scoreboard.comunidadhll.es:5443/games/1562115", + ) + self.assertEqual(first["list_candidates_inserted"], 1) + self.assertEqual(first["list_candidates_updated"], 0) + self.assertEqual(first["errors"][0]["stage"], "fetch_match_details") + self.assertEqual(second["list_candidates_inserted"], 0) + self.assertEqual(second["list_candidates_updated"], 1) + self.assertEqual(len(stored), 1) + def test_recent_and_detail_payloads_expose_safe_persisted_match_url(self) -> None: with tempfile.TemporaryDirectory() as tmpdir: db_path = Path(tmpdir) / "historical.sqlite3" @@ -201,6 +274,70 @@ class PersistedScoreboardMatchLinkTests(unittest.TestCase): self.assertIsNone(detail["match_url"]) gc.collect() + def test_foy_relink_reports_existing_materialized_match_url(self) -> None: + with tempfile.TemporaryDirectory() as tmpdir: + db_path = Path(tmpdir) / "historical.sqlite3" + previous_storage_path = os.environ.get("HLL_BACKEND_STORAGE_PATH") + os.environ["HLL_BACKEND_STORAGE_PATH"] = str(db_path) + try: + _persist_match( + db_path, + server_slug="comunidad-hispana-02", + match_id="1562115", + map_name="Foy Warfare", + started_at="2026-05-20T20:54:11Z", + ended_at="2026-05-20T22:24:11Z", + ) + persist_rcon_admin_log_entries( + target={ + "target_key": "comunidad-hispana-02", + "external_server_id": "comunidad-hispana-02", + }, + entries=[ + { + "timestamp": "2026-05-20T20:54:11Z", + "message": "[1 min (1779310451)] MATCH START Foy Warfare", + }, + { + "timestamp": "2026-05-20T22:24:11Z", + "message": "[91 min (1779315851)] MATCH ENDED `Foy Warfare` ALLIED (4 - 1) AXIS", + }, + ], + db_path=db_path, + ) + materialize_rcon_admin_log(db_path=db_path) + report = relink_materialized_matches( + server_key="comunidad-hispana-02", + db_path=db_path, + ) + detail = get_rcon_historical_match_detail( + server_key="comunidad-hispana-02", + match_id="comunidad-hispana-02:1779310451:1779315851:foywarfare", + ) + diagnostics = inspect_materialized_match_correlation( + server_key="comunidad-hispana-02", + match_key="comunidad-hispana-02:1779310451:1779315851:foywarfare", + db_path=db_path, + ) + finally: + if previous_storage_path is None: + os.environ.pop("HLL_BACKEND_STORAGE_PATH", None) + else: + os.environ["HLL_BACKEND_STORAGE_PATH"] = previous_storage_path + + self.assertEqual(report["matches_scanned"], 1) + self.assertEqual(report["matches_linked"], 1) + self.assertGreaterEqual(report["candidates_scanned"], 1) + self.assertIsNotNone(detail) + self.assertEqual( + detail["match_url"], + "https://scoreboard.comunidadhll.es:5443/games/1562115", + ) + self.assertEqual(diagnostics["final_reason"], "linked") + self.assertEqual(diagnostics["selected_candidate"]["external_match_id"], "1562115") + self.assertEqual(diagnostics["top_candidates"][0]["map"], "Foy Warfare") + gc.collect() + def _persist_match( db_path: Path, @@ -227,6 +364,23 @@ def _persist_match( ) +def _foy_list_match() -> dict[str, object]: + return { + "id": 1562115, + "server_number": 2, + "start": "2026-05-20T20:54:11+00:00", + "end": "2026-05-20T22:24:11+00:00", + "map": {"id": "foywarfare", "pretty_name": "Foy Warfare"}, + "result": {"allied": 4, "axis": 1}, + } + + +def _backfill_timestamp(raw_value: str): + from app.scoreboard_candidate_backfill import _parse_timestamp + + return _parse_timestamp(raw_value, option_name="test") + + def _persist_rcon_window( db_path: Path, *, diff --git a/docs/scoreboard-correlation-debugging.md b/docs/scoreboard-correlation-debugging.md new file mode 100644 index 0000000..0fee09e --- /dev/null +++ b/docs/scoreboard-correlation-debugging.md @@ -0,0 +1,45 @@ +# Scoreboard Correlation Debugging + +Use backend commands to debug a missing public scoreboard button on an RCON +historical match. Normal frontend payloads and pages should stay free of +correlation diagnostics. + +## Sequence + +1. Refresh trusted public scoreboard candidates for the relevant server: + + ```powershell + docker compose exec backend python -m app.scoreboard_candidate_backfill --server comunidad-hispana-02 --from 2026-05-20T00:00:00Z --to 2026-05-21T23:59:59Z --max-pages 5 --page-size 100 + ``` + +2. Scan existing materialized RCON matches against those candidates: + + ```powershell + docker compose exec backend python -m app.rcon_scoreboard_relink --server comunidad-hispana-02 + ``` + +3. Inspect one match correlation: + + ```powershell + docker compose exec backend python -m app.scoreboard_correlation_diagnostics --server comunidad-hispana-02 --match comunidad-hispana-02:1779310451:1779315851:foywarfare + ``` + +4. Verify the detail endpoint used by the match page: + + ```powershell + Invoke-WebRequest 'http://localhost:8000/api/historical/matches/detail?server=comunidad-hispana-02&match=comunidad-hispana-02%3A1779310451%3A1779315851%3Afoywarfare' | Select-Object -ExpandProperty Content + ``` + +## Reading Output + +The diagnostic JSON includes the RCON match window, score, candidate search +window, safe top candidate summaries, the selected candidate when one is strong +enough, and `final_reason`. + +- `linked` means the detail read model can expose the trusted `match_url`. +- `no-safe-candidate` means candidate persistence or map/window matching needs + inspection. +- `low-confidence` means candidates exist but evidence is insufficient. +- `ambiguous-candidate` means two candidates tie and no public URL is selected. +- `unsafe-url` in a candidate summary means the raw candidate URL is not emitted + or selected.