rgonsal/comunidadhll
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add rcon historical backfill for leaderboard windows

Browse Source
This commit is contained in:
Raúl González
2026-05-23 13:51:16 +02:00
parent a215b382db
commit 0d061803a0
9 changed files with 1017 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files

78
ai/tasks/done/TASK-rcon-admin-log-monthly-backfill.md Normal file
View File

@@ -0,0 +1,78 @@
---
id: TASK-rcon-admin-log-monthly-backfill
title: Add RCON AdminLog historical backfill
status: in-progress
type: backend
team: Backend Senior
supporting_teams: [Arquitecto Python]
roadmap_item: historical-rcon
priority: high
---
# TASK-rcon-admin-log-monthly-backfill - Add RCON AdminLog historical backfill
## Goal
Add an explicit RCON/AdminLog backfill CLI that can populate materialized closed matches for recent-match and leaderboard windows, and align monthly RCON leaderboard snapshots with the previous-month day 1-7 policy.
## Context
HLL Vietnam runs historical data in RCON-first mode. Prospective AdminLog capture exists, but fresh databases need an operator-run backfill path to recover recent closed matches and produce reliable weekly/monthly snapshots without changing web request startup behavior.
## Steps
1. Inspect the listed files first.
2. Add the scoped backfill CLI and window policy helpers.
3. Integrate the monthly fallback policy into RCON-backed snapshots.
4. Add focused tests and documentation.
5. Validate with the documented Python checks and Docker checks where available.
## Files to Read First
- `ai/architecture-index.md`
- `ai/repo-context.md`
- `backend/README.md`
- `backend/app/rcon_admin_log_ingestion.py`
- `backend/app/rcon_admin_log_materialization.py`
- `backend/app/historical_runner.py`
- `backend/app/rcon_historical_leaderboards.py`
## Expected Files to Modify
- `backend/app/rcon_historical_backfill.py`
- `backend/app/rcon_historical_leaderboards.py`
- `backend/app/config.py`
- `backend/README.md` or `docs/historical-rcon-backfill.md`
- `docker-compose.yml`
- focused tests under `backend/tests/`
## Constraints
- Do not touch unrelated UI layout or current-match live page.
- Keep normal backend startup free from long blocking backfill work.
- Keep inserts idempotent/deduplicated and do not remove existing data.
- Keep PostgreSQL compatibility and SQLite compatibility for tests.
- Do not reintroduce `comunidad-hispana-03` by default.
- Avoid public scoreboard fallback for RCON leaderboards when materialized RCON data exists.
## Validation
- `python -m compileall backend/app`
- `python -m unittest discover -s backend/tests -p "*historical*"`
- `python -m unittest discover -s backend/tests -p "*rcon*"`
- `git diff --check`
- Docker Compose build/run checks when available in the environment.
## Outcome
Implemented.
- Added `app.rcon_historical_backfill` as an explicit operator CLI.
- Added RCON monthly day 1-7 previous-month policy and weekly sufficient-sample fallback metadata.
- Routed persisted RCON leaderboard snapshot generation through the materialized RCON read model.
- Added Docker/README documentation and focused unittest coverage.
- Docker dry-run passed. A real backfill run was started after stopping writer services; the first attempt correctly reported a busy writer lock, and the second run inserted additional materialized data before the command timeout required stopping the one-off container. Advanced services were restarted afterwards.
## Change Budget
- This task is expected to exceed the default line budget because it introduces a new operator CLI plus tests and docs, but changes should remain limited to backend historical RCON surfaces.

13
backend/README.md
View File

@@ -71,6 +71,10 @@ Variables opcionales:
- `HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS`
- `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES`
- `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS`
- `HLL_RCON_BACKFILL_CHUNK_HOURS`
- `HLL_RCON_BACKFILL_SLEEP_SECONDS`
- `HLL_RCON_BACKFILL_MAX_DAYS_BACK`
- `HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES`
- `HLL_BACKEND_SQLITE_WRITER_TIMEOUT_SECONDS`
- `HLL_BACKEND_SQLITE_BUSY_TIMEOUT_MS`
- `HLL_BACKEND_WRITER_LOCK_TIMEOUT_SECONDS`
@@ -466,6 +470,15 @@ docker compose exec backend python -m app.rcon_admin_log_ingestion --minutes 144
docker compose exec backend python -m app.rcon_historical_worker capture
```
Backfill historico RCON/AdminLog:
- runbook: `docs/historical-rcon-backfill.md`
- ejemplo seco:
```powershell
docker compose run --rm rcon-historical-worker python -m app.rcon_historical_backfill --ensure-recent-matches 100 --servers comunidad-hispana-01,comunidad-hispana-02 --dry-run
```
Comandos manuales desde `backend/`:
```powershell

30
backend/app/config.py
View File

@@ -35,6 +35,9 @@ DEFAULT_PLAYER_EVENT_REFRESH_RETRY_DELAY_SECONDS = 30
DEFAULT_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS = 600
DEFAULT_RCON_HISTORICAL_CAPTURE_MAX_RETRIES = 2
DEFAULT_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS = 15
DEFAULT_RCON_BACKFILL_CHUNK_HOURS = 6
DEFAULT_RCON_BACKFILL_SLEEP_SECONDS = 1.0
DEFAULT_RCON_BACKFILL_MAX_DAYS_BACK = 45
DEFAULT_SQLITE_WRITER_TIMEOUT_SECONDS = 30.0
DEFAULT_SQLITE_BUSY_TIMEOUT_MS = 30000
DEFAULT_WRITER_LOCK_TIMEOUT_SECONDS = 120.0
@@ -484,6 +487,33 @@ def get_rcon_historical_capture_retry_delay_seconds() -> int:
return retry_delay_seconds
def get_rcon_backfill_chunk_hours() -> int:
"""Return the AdminLog backfill chunk size in hours."""
return _read_int_env(
"HLL_RCON_BACKFILL_CHUNK_HOURS",
str(DEFAULT_RCON_BACKFILL_CHUNK_HOURS),
minimum=1,
)
def get_rcon_backfill_sleep_seconds() -> float:
"""Return the delay between AdminLog backfill RCON requests."""
return _read_float_env(
"HLL_RCON_BACKFILL_SLEEP_SECONDS",
str(DEFAULT_RCON_BACKFILL_SLEEP_SECONDS),
minimum=0,
)
def get_rcon_backfill_max_days_back() -> int:
"""Return the maximum AdminLog backfill lookback horizon in days."""
return _read_int_env(
"HLL_RCON_BACKFILL_MAX_DAYS_BACK",
str(DEFAULT_RCON_BACKFILL_MAX_DAYS_BACK),
minimum=1,
)
def get_a2s_targets_payload() -> str | None:
"""Return the optional JSON payload that overrides local A2S targets."""
raw_payload = os.getenv(DEFAULT_A2S_TARGETS_ENV_VAR)

25
backend/app/historical_snapshots.py
View File

@@ -67,7 +67,6 @@ SUPPORTED_LEADERBOARD_METRICS = frozenset(
PREWARM_SNAPSHOT_SERVER_KEYS = (
"comunidad-hispana-01",
"comunidad-hispana-02",
"comunidad-hispana-03",
ALL_SERVERS_SLUG,
)
PREWARM_LEADERBOARD_METRICS = ("kills",)
@@ -435,6 +434,18 @@ def _build_weekly_leaderboard_snapshot(
limit: int,
db_path: Path | None = None,
) -> dict[str, object]:
if get_historical_data_source_kind() == SOURCE_KIND_RCON:
from .rcon_historical_leaderboards import list_rcon_materialized_leaderboard
leaderboard_result = list_rcon_materialized_leaderboard(
limit=limit,
server_key=server_key,
metric=metric,
timeframe="weekly",
db_path=db_path,
now=generated_at,
)
else:
leaderboard_result = list_weekly_leaderboard(
limit=limit,
server_id=server_key,
@@ -468,6 +479,18 @@ def _build_monthly_leaderboard_snapshot(
limit: int,
db_path: Path | None = None,
) -> dict[str, object]:
if get_historical_data_source_kind() == SOURCE_KIND_RCON:
from .rcon_historical_leaderboards import list_rcon_materialized_leaderboard
leaderboard_result = list_rcon_materialized_leaderboard(
limit=limit,
server_key=server_key,
metric=metric,
timeframe="monthly",
db_path=db_path,
now=generated_at,
)
else:
leaderboard_result = list_monthly_leaderboard(
limit=limit,
server_id=server_key,

484
backend/app/rcon_historical_backfill.py Normal file
View File

@@ -0,0 +1,484 @@
"""Explicit RCON/AdminLog historical backfill command."""
from __future__ import annotations
import argparse
import json
import time
from dataclasses import dataclass
from contextlib import closing
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Iterable
from .config import (
get_rcon_backfill_chunk_hours,
get_rcon_backfill_max_days_back,
get_rcon_backfill_sleep_seconds,
get_rcon_request_timeout_seconds,
use_postgres_rcon_storage,
)
from .historical_runner import generate_historical_snapshots
from .historical_storage import ALL_SERVERS_SLUG
from .rcon_admin_log_materialization import (
MATCH_RESULT_SOURCE,
initialize_rcon_materialized_storage,
materialize_rcon_admin_log,
)
from .rcon_admin_log_storage import persist_rcon_admin_log_entries
from .rcon_client import HllRconConnection, RconServerTarget, build_rcon_target_key, load_rcon_targets
from .rcon_historical_leaderboards import list_rcon_materialized_leaderboard
from .sqlite_utils import connect_sqlite_readonly
from .writer_lock import backend_writer_lock, build_writer_lock_holder
DEFAULT_ALLOWED_SERVER_KEYS = frozenset({"comunidad-hispana-01", "comunidad-hispana-02"})
EXCLUDED_BY_DEFAULT_SERVER_KEYS = frozenset({"comunidad-hispana-03"})
@dataclass(frozen=True, slots=True)
class BackfillWindow:
start: datetime
end: datetime
@property
def lookback_seconds(self) -> int:
now = datetime.now(timezone.utc)
return max(1, int((now - self.start).total_seconds()))
def run_rcon_historical_backfill(
*,
servers: str | None = None,
from_value: str | None = None,
to_value: str | None = None,
ensure_recent_matches: int | None = None,
ensure_current_month: bool = False,
ensure_leaderboard_windows: bool = False,
chunk_hours: int | None = None,
sleep_seconds: float | None = None,
max_days_back: int | None = None,
dry_run: bool = False,
regenerate_snapshots: bool = False,
db_path: Path | None = None,
) -> dict[str, object]:
"""Backfill AdminLog events and materialized RCON matches on explicit operator command."""
anchor = datetime.now(timezone.utc)
resolved_chunk_hours = chunk_hours or get_rcon_backfill_chunk_hours()
resolved_sleep_seconds = (
get_rcon_backfill_sleep_seconds() if sleep_seconds is None else sleep_seconds
)
resolved_max_days_back = max_days_back or get_rcon_backfill_max_days_back()
selected_targets = select_backfill_targets(servers)
recent_before = count_recent_materialized_closed_matches(db_path=db_path)
monthly_before = _window_diagnostic("monthly", db_path=db_path, now=anchor)
weekly_before = _window_diagnostic("weekly", db_path=db_path, now=anchor)
requested_range = _resolve_requested_range(
anchor=anchor,
from_value=from_value,
to_value=to_value,
ensure_recent_matches=ensure_recent_matches,
ensure_current_month=ensure_current_month,
ensure_leaderboard_windows=ensure_leaderboard_windows,
max_days_back=resolved_max_days_back,
)
windows = _build_backfill_windows(
start=requested_range["start"],
end=requested_range["end"],
chunk_hours=resolved_chunk_hours,
)
result: dict[str, object] = {
"status": "dry-run" if dry_run else "ok",
"dry_run": dry_run,
"servers_processed": [build_rcon_target_key(target) for target in selected_targets],
"requested_range": {
"from": _to_iso(requested_range["start"]),
"to": _to_iso(requested_range["end"]),
"reason": requested_range["reason"],
"admin_log_api": "lookback-only",
},
"actual_windows_scanned": [],
"events_seen": 0,
"events_inserted": 0,
"duplicate_events": 0,
"matches_materialized": 0,
"matches_updated": 0,
"player_stats_materialized": 0,
"player_stats_updated": 0,
"recent_materialized_closed_match_count_before": recent_before,
"recent_materialized_closed_match_count_after": recent_before,
"monthly_selected_window_before": monthly_before,
"monthly_selected_window": monthly_before,
"weekly_selected_window_before": weekly_before,
"weekly_selected_window": weekly_before,
"snapshot_regeneration_result": None,
"errors": [],
}
if dry_run:
result["actual_windows_scanned"] = [
_serialize_window(window) for window in _limit_windows_for_recent_need(
windows,
ensure_recent_matches=ensure_recent_matches,
db_path=db_path,
)
]
return result
try:
with backend_writer_lock(
holder=build_writer_lock_holder("app.rcon_historical_backfill")
):
windows_to_scan = _limit_windows_for_recent_need(
windows,
ensure_recent_matches=ensure_recent_matches,
db_path=db_path,
)
for window in windows_to_scan:
for target in selected_targets:
window_result = _scan_target_window(target, window)
result["actual_windows_scanned"].append(window_result["window"])
result["events_seen"] = int(result["events_seen"]) + int(
window_result["events_seen"]
)
result["events_inserted"] = int(result["events_inserted"]) + int(
window_result["events_inserted"]
)
result["duplicate_events"] = int(result["duplicate_events"]) + int(
window_result["duplicate_events"]
)
if window_result.get("error"):
result["errors"].append(window_result["error"])
if resolved_sleep_seconds > 0:
time.sleep(resolved_sleep_seconds)
materialized = materialize_rcon_admin_log(db_path=db_path)
result["matches_materialized"] = int(result["matches_materialized"]) + int(
materialized.get("matches_materialized") or 0
)
result["matches_updated"] = int(result["matches_updated"]) + int(
materialized.get("matches_updated") or 0
)
result["player_stats_materialized"] = int(
result["player_stats_materialized"]
) + int(materialized.get("player_stats_materialized") or 0)
result["player_stats_updated"] = int(result["player_stats_updated"]) + int(
materialized.get("player_stats_updated") or 0
)
if ensure_recent_matches and count_recent_materialized_closed_matches(
db_path=db_path
) >= ensure_recent_matches:
break
if regenerate_snapshots:
result["snapshot_regeneration_result"] = generate_historical_snapshots(
server_slug=None,
run_number=1,
)
except Exception as exc: # noqa: BLE001 - CLI reports structured operator diagnostics
result["status"] = "error"
result["errors"].append({"error_type": type(exc).__name__, "message": str(exc)})
recent_after = count_recent_materialized_closed_matches(db_path=db_path)
result["recent_materialized_closed_match_count_after"] = recent_after
result["monthly_selected_window"] = _window_diagnostic("monthly", db_path=db_path, now=anchor)
result["weekly_selected_window"] = _window_diagnostic("weekly", db_path=db_path, now=anchor)
if result["errors"] and result["status"] == "ok":
result["status"] = "partial"
return result
def select_backfill_targets(servers: str | None) -> list[RconServerTarget]:
"""Load configured RCON targets and apply safe server selection rules."""
configured_targets = list(load_rcon_targets())
if not configured_targets:
raise RuntimeError("No RCON targets configured in HLL_BACKEND_RCON_TARGETS.")
by_key = {build_rcon_target_key(target): target for target in configured_targets}
requested_keys = _parse_server_keys(servers)
if requested_keys:
unknown = sorted(key for key in requested_keys if key not in by_key)
if unknown:
raise ValueError(f"Unknown RCON server key(s): {', '.join(unknown)}")
return [by_key[key] for key in requested_keys]
selected = [
target
for key, target in by_key.items()
if key in DEFAULT_ALLOWED_SERVER_KEYS and key not in EXCLUDED_BY_DEFAULT_SERVER_KEYS
]
if not selected:
raise RuntimeError(
"No default backfill targets selected. Pass --servers with configured keys explicitly."
)
return selected
def count_recent_materialized_closed_matches(
*,
server_key: str | None = None,
db_path: Path | None = None,
) -> int:
"""Count materialized closed AdminLog matches available for recent-match UI."""
resolved_path = initialize_rcon_materialized_storage(db_path=db_path)
scope_sql = ""
params: list[object] = [MATCH_RESULT_SOURCE]
if server_key and server_key != ALL_SERVERS_SLUG:
scope_sql = "AND (target_key = ? OR external_server_id = ?)"
params.extend([server_key, server_key])
if use_postgres_rcon_storage(explicit_sqlite_path=db_path):
from .postgres_rcon_storage import connect_postgres_compat
connection_scope = connect_postgres_compat()
else:
connection_scope = closing(connect_sqlite_readonly(resolved_path))
with connection_scope as connection:
row = connection.execute(
f"""
SELECT COUNT(*) AS count
FROM rcon_materialized_matches
WHERE source_basis = ?
AND ended_at IS NOT NULL
{scope_sql}
""",
params,
).fetchone()
return int(row["count"] or 0) if row else 0
def _scan_target_window(target: RconServerTarget, window: BackfillWindow) -> dict[str, object]:
target_metadata = _serialize_target(target)
serialized_window = _serialize_window(window)
try:
with HllRconConnection(timeout_seconds=get_rcon_request_timeout_seconds()) as connection:
connection.connect(host=target.host, port=target.port, password=target.password)
payload = connection.execute_json(
"GetAdminLog",
{
"LogBackTrackTime": window.lookback_seconds,
"Filters": [],
},
)
entries = payload.get("entries")
if not isinstance(entries, list):
entries = []
normalized_entries = [entry for entry in entries if isinstance(entry, dict)]
delta = persist_rcon_admin_log_entries(
target=target_metadata,
entries=normalized_entries,
)
return {"window": serialized_window, "error": None, **delta}
except Exception as exc: # noqa: BLE001 - per-window errors must not hide neighboring windows
return {
"window": serialized_window,
"events_seen": 0,
"events_inserted": 0,
"duplicate_events": 0,
"error": {
**target_metadata,
**serialized_window,
"error_type": type(exc).__name__,
"message": str(exc),
},
}
def _resolve_requested_range(
*,
anchor: datetime,
from_value: str | None,
to_value: str | None,
ensure_recent_matches: int | None,
ensure_current_month: bool,
ensure_leaderboard_windows: bool,
max_days_back: int,
) -> dict[str, object]:
end = _parse_datetime_argument(to_value, default=anchor)
starts = []
reasons = []
if from_value:
starts.append(_parse_datetime_argument(from_value, default=anchor))
reasons.append("explicit-range")
if ensure_current_month:
starts.append(_month_start(anchor))
reasons.append("ensure-current-month")
if ensure_leaderboard_windows:
starts.append(_previous_month_start(_month_start(anchor)))
starts.append(_week_start(anchor) - timedelta(days=7))
reasons.append("ensure-leaderboard-windows")
if ensure_recent_matches:
starts.append(anchor - timedelta(days=max_days_back))
reasons.append(f"ensure-recent-matches-{ensure_recent_matches}")
if not starts:
starts.append(anchor - timedelta(days=max_days_back))
reasons.append("default-max-days-back")
start = max(min(starts), anchor - timedelta(days=max_days_back))
return {"start": start, "end": end, "reason": ",".join(reasons)}
def _build_backfill_windows(
*,
start: datetime,
end: datetime,
chunk_hours: int,
) -> list[BackfillWindow]:
windows: list[BackfillWindow] = []
cursor = _as_utc(end)
lower = _as_utc(start)
chunk = timedelta(hours=chunk_hours)
while cursor > lower:
window_start = max(lower, cursor - chunk)
windows.append(BackfillWindow(start=window_start, end=cursor))
cursor = window_start
return windows
def _limit_windows_for_recent_need(
windows: list[BackfillWindow],
*,
ensure_recent_matches: int | None,
db_path: Path | None,
) -> list[BackfillWindow]:
if not ensure_recent_matches:
return windows
if count_recent_materialized_closed_matches(db_path=db_path) >= ensure_recent_matches:
return []
return windows
def _window_diagnostic(
timeframe: str,
*,
db_path: Path | None,
now: datetime,
) -> dict[str, object]:
payload = list_rcon_materialized_leaderboard(
server_key=ALL_SERVERS_SLUG,
timeframe=timeframe,
metric="kills",
limit=1,
db_path=db_path,
now=now,
)
return {
"window_kind": payload.get("window_kind"),
"window_label": payload.get("window_label"),
"window_start": payload.get("window_start"),
"window_end": payload.get("window_end"),
"selection_reason": payload.get("selection_reason"),
"current_week_closed_matches": payload.get("current_week_closed_matches"),
"previous_week_closed_matches": payload.get("previous_week_closed_matches"),
"selected_month_start": payload.get("selected_month_start"),
"selected_month_end": payload.get("selected_month_end"),
"current_month_closed_matches": payload.get("current_month_closed_matches"),
"previous_month_closed_matches": payload.get("previous_month_closed_matches"),
"sufficient_sample": payload.get("sufficient_sample"),
}
def _parse_server_keys(value: str | None) -> list[str]:
return [part.strip() for part in str(value or "").split(",") if part.strip()]
def _parse_datetime_argument(value: str | None, *, default: datetime) -> datetime:
if value is None or str(value).strip().lower() == "now":
return default
raw = str(value).strip()
if len(raw) == 10:
raw = f"{raw}T00:00:00+00:00"
parsed = datetime.fromisoformat(raw.replace("Z", "+00:00"))
return _as_utc(parsed)
def _month_start(value: datetime) -> datetime:
point = _as_utc(value)
return point.replace(day=1, hour=0, minute=0, second=0, microsecond=0)
def _previous_month_start(current_month_start: datetime) -> datetime:
return _month_start(current_month_start - timedelta(days=1))
def _week_start(value: datetime) -> datetime:
point = _as_utc(value)
return (point - timedelta(days=point.weekday())).replace(
hour=0,
minute=0,
second=0,
microsecond=0,
)
def _as_utc(value: datetime) -> datetime:
if value.tzinfo is None:
return value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc)
def _serialize_target(target: RconServerTarget) -> dict[str, object]:
return {
"target_key": build_rcon_target_key(target),
"external_server_id": target.external_server_id,
"name": target.name,
"host": target.host,
"port": target.port,
"source_name": target.source_name,
}
def _serialize_window(window: BackfillWindow) -> dict[str, object]:
return {
"start": _to_iso(window.start),
"end": _to_iso(window.end),
"requested_log_backtrack_seconds": window.lookback_seconds,
}
def _to_iso(value: datetime) -> str:
return _as_utc(value).isoformat().replace("+00:00", "Z")
def _main(argv: Iterable[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Backfill RCON AdminLog historical materialized matches.")
parser.add_argument("--from", dest="from_value", default=None)
parser.add_argument("--to", dest="to_value", default=None)
parser.add_argument("--servers", default=None)
parser.add_argument("--ensure-recent-matches", type=int, default=None)
parser.add_argument("--ensure-current-month", action="store_true")
parser.add_argument("--ensure-leaderboard-windows", action="store_true")
parser.add_argument("--chunk-hours", type=int, default=get_rcon_backfill_chunk_hours())
parser.add_argument("--sleep-seconds", type=float, default=get_rcon_backfill_sleep_seconds())
parser.add_argument("--max-days-back", type=int, default=get_rcon_backfill_max_days_back())
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--regenerate-snapshots", action="store_true")
parser.add_argument("--db-path", type=Path, default=None)
args = parser.parse_args(list(argv) if argv is not None else None)
if args.ensure_recent_matches is not None and args.ensure_recent_matches <= 0:
raise ValueError("--ensure-recent-matches must be positive.")
if args.chunk_hours <= 0:
raise ValueError("--chunk-hours must be positive.")
if args.sleep_seconds < 0:
raise ValueError("--sleep-seconds must be zero or positive.")
if args.max_days_back <= 0:
raise ValueError("--max-days-back must be positive.")
payload = run_rcon_historical_backfill(
servers=args.servers,
from_value=args.from_value,
to_value=args.to_value,
ensure_recent_matches=args.ensure_recent_matches,
ensure_current_month=args.ensure_current_month,
ensure_leaderboard_windows=args.ensure_leaderboard_windows,
chunk_hours=args.chunk_hours,
sleep_seconds=args.sleep_seconds,
max_days_back=args.max_days_back,
dry_run=args.dry_run,
regenerate_snapshots=args.regenerate_snapshots,
db_path=args.db_path,
)
print(json.dumps(payload, ensure_ascii=False, indent=2, default=str))
return 0 if payload.get("status") != "error" else 1
if __name__ == "__main__":
raise SystemExit(_main())

180
backend/app/rcon_historical_leaderboards.py
View File

@@ -8,6 +8,7 @@ from pathlib import Path
from typing import Literal
from .config import get_storage_path, use_postgres_rcon_storage
from .config import get_historical_weekly_fallback_min_matches
from .historical_storage import ALL_SERVERS_SLUG
from .rcon_admin_log_materialization import (
MATCH_RESULT_SOURCE,
@@ -75,6 +76,8 @@ def build_rcon_materialized_leaderboard_snapshot_payload(
"current_week_closed_matches": result.get("current_week_closed_matches"),
"previous_week_closed_matches": result.get("previous_week_closed_matches"),
"current_month_start": result.get("current_month_start"),
"selected_month_start": result.get("selected_month_start"),
"selected_month_end": result.get("selected_month_end"),
"current_month_closed_matches": result.get("current_month_closed_matches"),
"previous_month_closed_matches": result.get("previous_month_closed_matches"),
"sufficient_sample": result.get("sufficient_sample"),
@@ -109,6 +112,7 @@ def list_rcon_materialized_leaderboard(
metric: str = "kills",
limit: int = 10,
db_path: Path | None = None,
now: datetime | None = None,
) -> dict[str, object]:
"""Return a leaderboard built from materialized RCON/AdminLog player stats.
@@ -120,8 +124,17 @@ def list_rcon_materialized_leaderboard(
normalized_timeframe = _normalize_timeframe(timeframe)
normalized_metric = _normalize_metric(metric)
normalized_limit = max(1, int(limit or 10))
window = _build_window(normalized_timeframe)
anchor = _as_utc(now or datetime.now(timezone.utc))
resolved_path = initialize_rcon_materialized_storage(db_path=db_path)
connection_scope = _connect_scope(resolved_path, db_path=db_path)
with connection_scope as connection:
window = select_leaderboard_window(
connection=connection,
server_key=server_key,
timeframe=normalized_timeframe,
now=anchor,
)
if normalized_metric == "support":
return _empty_payload(
server_key=server_key,
@@ -131,10 +144,6 @@ def list_rcon_materialized_leaderboard(
window=window,
reason="rcon-materialized-stats-do-not-include-support-score",
)
resolved_path = initialize_rcon_materialized_storage(db_path=db_path)
connection_scope = _connect_scope(resolved_path, db_path=db_path)
with connection_scope as connection:
rows = _fetch_leaderboard_rows(
connection,
server_key=server_key,
@@ -143,13 +152,6 @@ def list_rcon_materialized_leaderboard(
window_start=window["start"],
window_end=window["end"],
)
counts = _fetch_match_counts(
connection,
server_key=server_key,
timeframe=normalized_timeframe,
window_start=window["start"],
window_end=window["end"],
)
source_range = _fetch_source_range(
connection,
server_key=server_key,
@@ -169,20 +171,16 @@ def list_rcon_materialized_leaderboard(
"window_kind": window["kind"],
"window_label": window["label"],
"uses_fallback": False,
"selection_reason": "rcon-materialized-current-window",
"current_week_start": _to_iso(_week_start(window["end"])),
"current_week_closed_matches": counts["current_week_closed_matches"],
"previous_week_closed_matches": counts["previous_week_closed_matches"],
"current_month_start": _to_iso(_month_start(window["end"])),
"current_month_closed_matches": counts["current_month_closed_matches"],
"previous_month_closed_matches": counts["previous_month_closed_matches"],
"sufficient_sample": {
"minimum_closed_matches": 1,
"current_week_closed_matches": counts["current_week_closed_matches"],
"current_week_has_sufficient_sample": counts["current_week_closed_matches"] >= 1,
"is_early_week": False,
"fallback_max_weekday": 2,
},
"selection_reason": window["selection_reason"],
"current_week_start": _to_iso(window["current_week_start"]),
"current_week_closed_matches": window["current_week_closed_matches"],
"previous_week_closed_matches": window["previous_week_closed_matches"],
"current_month_start": _to_iso(window["current_month_start"]),
"selected_month_start": _to_iso(window["selected_month_start"]),
"selected_month_end": _to_iso(window["selected_month_end"]),
"current_month_closed_matches": window["current_month_closed_matches"],
"previous_month_closed_matches": window["previous_month_closed_matches"],
"sufficient_sample": window["sufficient_sample"],
"source_range_start": _to_iso(source_range[0]) if source_range[0] else None,
"source_range_end": _to_iso(source_range[1]) if source_range[1] else None,
"items": items,
@@ -280,6 +278,108 @@ def _fetch_match_counts(
}
def select_leaderboard_window(
*,
connection: object,
server_key: str | None,
timeframe: str,
now: datetime | None = None,
) -> dict[str, object]:
"""Select the RCON leaderboard window using weekly/monthly fallback policy."""
anchor = _as_utc(now or datetime.now(timezone.utc))
current_week_start = _week_start(anchor)
previous_week_start = current_week_start - timedelta(days=7)
current_month_start = _month_start(anchor)
previous_month_start = _previous_month_start(current_month_start)
minimum_week_matches = get_historical_weekly_fallback_min_matches()
current_week_count = _count_matches(
connection,
server_key=server_key,
start=current_week_start,
end=anchor,
)
previous_week_count = _count_matches(
connection,
server_key=server_key,
start=previous_week_start,
end=current_week_start,
)
current_month_count = _count_matches(
connection,
server_key=server_key,
start=current_month_start,
end=anchor,
)
previous_month_count = _count_matches(
connection,
server_key=server_key,
start=previous_month_start,
end=current_month_start,
)
if timeframe == "monthly":
use_previous_month = anchor.day <= 7
start = previous_month_start if use_previous_month else current_month_start
end = current_month_start if use_previous_month else anchor
return {
"start": start,
"end": end,
"days": max(1, (end.date() - start.date()).days),
"kind": "previous-month" if use_previous_month else "current-month",
"label": "Mes anterior" if use_previous_month else "Mes actual",
"selection_reason": (
"monthly-uses-previous-month-until-day-8"
if use_previous_month
else "monthly-uses-current-month-after-day-7"
),
"current_week_start": current_week_start,
"current_week_closed_matches": current_week_count,
"previous_week_closed_matches": previous_week_count,
"current_month_start": current_month_start,
"selected_month_start": start,
"selected_month_end": end,
"current_month_closed_matches": current_month_count,
"previous_month_closed_matches": previous_month_count,
"sufficient_sample": {
"minimum_closed_matches": 1,
"current_month_closed_matches": current_month_count,
"previous_month_closed_matches": previous_month_count,
"current_month_has_sufficient_sample": current_month_count >= 1,
"uses_previous_month_until_day": 7,
},
}
current_week_has_sample = current_week_count >= minimum_week_matches
start = current_week_start if current_week_has_sample else previous_week_start
end = anchor if current_week_has_sample else current_week_start
return {
"start": start,
"end": end,
"days": max(1, (end.date() - start.date()).days),
"kind": "current-week" if current_week_has_sample else "previous-week",
"label": "Semana actual" if current_week_has_sample else "Semana anterior",
"selection_reason": (
"weekly-current-week-has-sufficient-closed-matches"
if current_week_has_sample
else "weekly-fallback-previous-week-insufficient-current-week-data"
),
"current_week_start": current_week_start,
"current_week_closed_matches": current_week_count,
"previous_week_closed_matches": previous_week_count,
"current_month_start": current_month_start,
"selected_month_start": current_month_start,
"selected_month_end": anchor,
"current_month_closed_matches": current_month_count,
"previous_month_closed_matches": previous_month_count,
"sufficient_sample": {
"minimum_closed_matches": minimum_week_matches,
"current_week_closed_matches": current_week_count,
"current_week_has_sufficient_sample": current_week_has_sample,
"previous_week_closed_matches": previous_week_count,
},
}
def _fetch_source_range(
connection: object,
*,
@@ -390,19 +490,15 @@ def _empty_payload(
"window_label": window["label"],
"uses_fallback": False,
"selection_reason": reason,
"current_week_start": _to_iso(_week_start(window["end"])),
"current_week_closed_matches": 0,
"previous_week_closed_matches": 0,
"current_month_start": _to_iso(_month_start(window["end"])),
"current_month_closed_matches": 0,
"previous_month_closed_matches": 0,
"sufficient_sample": {
"minimum_closed_matches": 1,
"current_week_closed_matches": 0,
"current_week_has_sufficient_sample": False,
"is_early_week": False,
"fallback_max_weekday": 2,
},
"current_week_start": _to_iso(window["current_week_start"]),
"current_week_closed_matches": window["current_week_closed_matches"],
"previous_week_closed_matches": window["previous_week_closed_matches"],
"current_month_start": _to_iso(window["current_month_start"]),
"selected_month_start": _to_iso(window["selected_month_start"]),
"selected_month_end": _to_iso(window["selected_month_end"]),
"current_month_closed_matches": window["current_month_closed_matches"],
"previous_month_closed_matches": window["previous_month_closed_matches"],
"sufficient_sample": window["sufficient_sample"],
"source_range_start": None,
"source_range_end": None,
"items": [],
@@ -430,6 +526,12 @@ def _build_window(timeframe: str) -> dict[str, object]:
}
def _as_utc(value: datetime) -> datetime:
if value.tzinfo is None:
return value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc)
def _week_start(value: datetime) -> datetime:
point = value.astimezone(timezone.utc)
start = point - timedelta(days=point.weekday())

171
backend/tests/test_rcon_historical_backfill.py Normal file
View File

@@ -0,0 +1,171 @@
from __future__ import annotations
import json
import os
import sqlite3
import tempfile
import unittest
from datetime import datetime, timezone
from pathlib import Path
from contextlib import closing
from unittest.mock import patch
from app.rcon_admin_log_materialization import (
MATCH_RESULT_SOURCE,
initialize_rcon_materialized_storage,
)
from app.rcon_historical_backfill import (
count_recent_materialized_closed_matches,
run_rcon_historical_backfill,
select_backfill_targets,
)
from app.rcon_historical_leaderboards import list_rcon_materialized_leaderboard
TARGETS_JSON = json.dumps(
[
{
"name": "Comunidad Hispana #01",
"slug": "comunidad-hispana-01",
"external_server_id": "comunidad-hispana-01",
"host": "127.0.0.1",
"port": 7779,
"password": "secret",
},
{
"name": "Comunidad Hispana #02",
"slug": "comunidad-hispana-02",
"external_server_id": "comunidad-hispana-02",
"host": "127.0.0.1",
"port": 7879,
"password": "secret",
},
{
"name": "Comunidad Hispana #03",
"slug": "comunidad-hispana-03",
"external_server_id": "comunidad-hispana-03",
"host": "127.0.0.1",
"port": 7979,
"password": "secret",
},
]
)
class RconHistoricalBackfillTests(unittest.TestCase):
def test_monthly_window_selects_previous_month_on_days_1_to_7(self) -> None:
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
payload = list_rcon_materialized_leaderboard(
server_key="all-servers",
timeframe="monthly",
metric="kills",
db_path=Path(tmpdir) / "historical.sqlite3",
now=datetime(2026, 5, 7, 12, tzinfo=timezone.utc),
)
self.assertEqual(payload["window_kind"], "previous-month")
self.assertEqual(payload["selected_month_start"], "2026-04-01T00:00:00Z")
self.assertEqual(payload["selected_month_end"], "2026-05-01T00:00:00Z")
def test_monthly_window_selects_current_month_on_day_8_plus(self) -> None:
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
payload = list_rcon_materialized_leaderboard(
server_key="all-servers",
timeframe="monthly",
metric="kills",
db_path=Path(tmpdir) / "historical.sqlite3",
now=datetime(2026, 5, 8, 12, tzinfo=timezone.utc),
)
self.assertEqual(payload["window_kind"], "current-month")
self.assertEqual(payload["selected_month_start"], "2026-05-01T00:00:00Z")
def test_recent_match_ensure_stops_when_count_is_already_satisfied(self) -> None:
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir, _patched_targets():
db_path = Path(tmpdir) / "historical.sqlite3"
_insert_closed_matches(db_path, 100)
payload = run_rcon_historical_backfill(
servers="comunidad-hispana-01,comunidad-hispana-02",
ensure_recent_matches=100,
dry_run=True,
db_path=db_path,
)
self.assertEqual(payload["recent_materialized_closed_match_count_before"], 100)
self.assertEqual(payload["actual_windows_scanned"], [])
def test_unknown_server_is_rejected(self) -> None:
with _patched_targets():
with self.assertRaises(ValueError):
select_backfill_targets("unknown-server")
def test_comunidad_hispana_03_is_not_included_by_default(self) -> None:
with _patched_targets():
selected = select_backfill_targets(None)
self.assertEqual(
[target.external_server_id for target in selected],
["comunidad-hispana-01", "comunidad-hispana-02"],
)
def test_dry_run_does_not_insert_data(self) -> None:
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir, _patched_targets():
db_path = Path(tmpdir) / "historical.sqlite3"
payload = run_rcon_historical_backfill(
servers="comunidad-hispana-01",
ensure_current_month=True,
dry_run=True,
db_path=db_path,
)
count_after = count_recent_materialized_closed_matches(db_path=db_path)
self.assertEqual(payload["status"], "dry-run")
self.assertEqual(payload["events_inserted"], 0)
self.assertEqual(count_after, 0)
def test_backfill_output_is_json_serializable(self) -> None:
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir, _patched_targets():
payload = run_rcon_historical_backfill(
servers="comunidad-hispana-01",
ensure_current_month=True,
dry_run=True,
db_path=Path(tmpdir) / "historical.sqlite3",
)
json.dumps(payload, ensure_ascii=True)
def _insert_closed_matches(db_path: Path, count: int) -> None:
initialize_rcon_materialized_storage(db_path=db_path)
with closing(sqlite3.connect(db_path)) as connection:
for index in range(count):
connection.execute(
"""
INSERT INTO rcon_materialized_matches (
target_key, external_server_id, match_key, map_name, map_pretty_name,
started_at, ended_at, confidence_mode, source_basis
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
"comunidad-hispana-01",
"comunidad-hispana-01",
f"match-{index}",
"stmariedumont",
"ST MARIE DU MONT",
"2026-05-01T10:00:00Z",
f"2026-05-{(index % 28) + 1:02d}T12:00:00Z",
"exact",
MATCH_RESULT_SOURCE,
),
)
connection.commit()
def _patched_targets():
return patch.dict(os.environ, {"HLL_BACKEND_RCON_TARGETS": TARGETS_JSON})
if __name__ == "__main__":
unittest.main()

1
docker-compose.yml
View File

@@ -83,6 +83,7 @@ services:
HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS: ${HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS:-600}
HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES: ${HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES:-2}
HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS: ${HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS:-15}
HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES: ${HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES:-10}
depends_on:
postgres:
condition: service_healthy

57
docs/historical-rcon-backfill.md Normal file
View File

@@ -0,0 +1,57 @@
# Historical RCON AdminLog Backfill
The RCON/AdminLog backfill is an explicit operator command. It does not run on
backend startup or on web requests.
Run it through the advanced worker image:
```powershell
docker compose run --rm rcon-historical-worker python -m app.rcon_historical_backfill --ensure-recent-matches 100 --servers comunidad-hispana-01,comunidad-hispana-02 --dry-run
```
Before a real manual backfill, stop the writer services to avoid waiting on the
shared writer lock:
```powershell
docker compose --profile advanced stop historical-runner rcon-historical-worker
```
Restart them afterwards:
```powershell
docker compose --profile advanced up -d historical-runner rcon-historical-worker
```
Examples:
```powershell
docker compose run --rm rcon-historical-worker python -m app.rcon_historical_backfill --ensure-recent-matches 100 --servers comunidad-hispana-01,comunidad-hispana-02
docker compose run --rm rcon-historical-worker python -m app.rcon_historical_backfill --ensure-current-month --servers comunidad-hispana-01,comunidad-hispana-02
docker compose run --rm rcon-historical-worker python -m app.rcon_historical_backfill --ensure-leaderboard-windows --servers comunidad-hispana-01,comunidad-hispana-02
docker compose run --rm rcon-historical-worker python -m app.rcon_historical_backfill --ensure-recent-matches 100 --servers comunidad-hispana-01,comunidad-hispana-02 --chunk-hours 6 --sleep-seconds 1 --max-days-back 45 --regenerate-snapshots
```
Direct module examples:
```powershell
python -m app.rcon_historical_backfill --from 2026-05-01 --to now --servers comunidad-hispana-01,comunidad-hispana-02
python -m app.rcon_historical_backfill --ensure-recent-matches 100 --servers comunidad-hispana-01,comunidad-hispana-02
python -m app.rcon_historical_backfill --ensure-current-month --servers comunidad-hispana-01,comunidad-hispana-02
python -m app.rcon_historical_backfill --ensure-leaderboard-windows --servers comunidad-hispana-01,comunidad-hispana-02
```
Useful configuration:
- `HLL_RCON_BACKFILL_CHUNK_HOURS`, default `6`
- `HLL_RCON_BACKFILL_SLEEP_SECONDS`, default `1`
- `HLL_RCON_BACKFILL_MAX_DAYS_BACK`, default `45`
- `HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES`, for normal prospective worker capture only
The command only selects `comunidad-hispana-01` and `comunidad-hispana-02` by
default. `comunidad-hispana-03` is not included unless it is configured in
`HLL_BACKEND_RCON_TARGETS` and explicitly passed with `--servers`.
Monthly RCON leaderboards use the previous calendar month on days 1 through 7.
From day 8 onward they use the current calendar month. Weekly RCON leaderboards
use the current week only when the current week has enough closed materialized
matches; otherwise they fall back to the previous week.