Add weekly monthly ranking snapshot generator

This commit is contained in:
devRaGonSa
2026-06-09 09:02:09 +02:00
parent 86b1660490
commit 07a79b3294
4 changed files with 680 additions and 4 deletions

View File

@@ -0,0 +1,149 @@
---
id: TASK-194-add-weekly-monthly-ranking-snapshot-generator
title: Add weekly monthly ranking snapshot generator
status: done
type: backend
team: Backend Senior
supporting_teams:
- Arquitecto de Base de Datos
- Arquitecto Python
roadmap_item: foundation
priority: high
---
# TASK-194 - Add weekly monthly ranking snapshot generator
## Goal
Implement a manual weekly/monthly ranking snapshot generator plus CLI so PostgreSQL-backed `/api/ranking` can serve persisted global ranking snapshots instead of runtime fallback when operators generate the required combinations.
## Context
`TASK-191` introduced the snapshot-first read path for weekly/monthly global ranking, and `TASK-192` / `TASK-193` stabilized the PostgreSQL path and runtime fallback metrics. The missing piece is still out-of-band generation for `ranking_snapshots` and `ranking_snapshot_items`, so production ranking requests continue to return `snapshot_status=missing`, `freshness=runtime` and `fallback_used=true`.
Preserve the current product identity: Spanish-speaking HLL Vietnam community, military/Vietnam/tactical/sober visual direction and controlled repository evolution.
## Steps
1. Read the listed files first.
2. Add a manual generator for weekly/monthly ranking snapshots using the existing materialized RCON leaderboard logic and snapshot tables.
3. Expose a CLI command for one explicit combination; only add matrix generation if it remains small and justified.
4. Validate generated snapshot reads, preserved runtime fallback behavior and production-oriented run commands.
5. Document the manual generation workflow and recommended combinations.
## Files to Read First
- `AGENTS.md`
- `ai/repo-context.md`
- `ai/architecture-index.md`
- `backend/app/rcon_historical_leaderboards.py`
- `backend/app/rcon_annual_rankings.py`
- `backend/app/routes.py`
- `backend/app/payloads.py`
- `backend/app/postgres_rcon_storage.py`
- `scripts/run-stats-validation.ps1`
- `docs/ranking-snapshot-read-model-plan.md`
- `docs/ranking-stats-performance-audit.md`
- `ai/tasks/done/TASK-191-serve-ranking-from-snapshots-with-runtime-fallback.md`
- `ai/tasks/done/TASK-192-fix-postgres-ranking-snapshot-schema.md`
- `ai/tasks/done/TASK-193-fix-postgres-ranking-derived-metrics.md`
## Expected Files to Modify
- `backend/app/rcon_historical_leaderboards.py`
- `scripts/run-stats-validation.ps1`
- `docs/ranking-snapshot-read-model-plan.md`
- `ai/tasks/done/TASK-194-add-weekly-monthly-ranking-snapshot-generator.md`
## Constraints
- Keep the change minimal.
- Do not modify frontend, assets or design.
- Do not reactivate Elo/MMR.
- Do not reintroduce Comunidad Hispana #03.
- Keep annual ranking behavior unchanged unless a helper can be safely shared without changing annual contracts.
- Prefer the existing module pattern already used by `backend/app/rcon_annual_rankings.py`.
- Do not change `/api/ranking` unless strictly necessary for validation-safe compatibility.
- Keep PostgreSQL as the operational target and preserve SQLite compatibility already present in the shared backend path.
## Validation
Before completing the task ensure:
- `powershell -ExecutionPolicy Bypass -File scripts/run-stats-validation.ps1`
- `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1`
- local import validation proves:
- `generate_ranking_snapshot(timeframe="weekly", server_key="all", metric="kills", limit=20)` creates a snapshot
- `get_latest_ranking_snapshot(...)` can read that snapshot
- `/api/ranking?timeframe=weekly&server_id=all&metric=kills&limit=20` returns `snapshot_status=ready`
- the same response returns `fallback_used=false`
- requesting a combination without a snapshot still preserves runtime fallback when enabled
- document the manual CLI command for local and Docker execution
- `git diff --name-only` matches the expected scope
## Outcome
Implemented:
- `backend/app/rcon_historical_leaderboards.py`
- `generate_ranking_snapshot(...)`
- persistence helpers for `ranking_snapshots` and `ranking_snapshot_items`
- manual CLI entrypoint:
- `python -m app.rcon_historical_leaderboards generate-ranking-snapshot --timeframe weekly --server-key all --metric kills --limit 20`
- `scripts/run-stats-validation.ps1`
- now validates a real generated weekly snapshot through:
- direct function import
- `get_latest_ranking_snapshot(...)`
- `/api/ranking`
- confirms generated snapshot reads return:
- `snapshot_status=ready`
- `fallback_used=false`
- confirms runtime fallback still works for a missing combination
- `docs/ranking-snapshot-read-model-plan.md`
- documents the manual command
- documents the recommended weekly/monthly/server/metric matrix
- documents suggested refresh cadence
- documents the expected `ready` vs fallback response states
Supported generation scope:
- `timeframe`
- `weekly`
- `monthly`
- `server-key`
- `all`
- `all-servers`
- `comunidad-hispana-01`
- `comunidad-hispana-02`
- `metric`
- `kills`
- `deaths`
- `teamkills`
- `matches_considered`
- `kd_ratio`
- `kills_per_match`
Validations executed:
- `python -m compileall backend/app/rcon_historical_leaderboards.py backend/app/payloads.py backend/app/routes.py backend/app/postgres_rcon_storage.py`
- direct local import probe for `generate_ranking_snapshot(...)`
- `powershell -ExecutionPolicy Bypass -File scripts/run-stats-validation.ps1`
- `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1`
- local CLI probe:
- `python -m app.rcon_historical_leaderboards generate-ranking-snapshot --timeframe weekly --server-key all --metric kills --limit 20`
Validation notes:
- Live backend HTTP at `http://127.0.0.1:8000` was not available, so route validation completed through local imports and repository scripts.
- The manual CLI is unitary per `(timeframe, server-key, metric)` combination.
- A broader matrix helper was intentionally left out to keep the task small and verifiable.
Recommended Docker command for production:
- `docker compose exec backend python -m app.rcon_historical_leaderboards generate-ranking-snapshot --timeframe weekly --server-key all --metric kills --limit 20`
## Change Budget
- Prefer fewer than 5 modified files.
- Prefer changes under 200 lines when feasible.
- Split the work into follow-up tasks if limits are exceeded.

View File

@@ -2,6 +2,8 @@
from __future__ import annotations
import argparse
import json
import os
from contextlib import closing
from datetime import datetime, timedelta, timezone
@@ -29,6 +31,21 @@ LeaderboardMetric = Literal[
"support",
]
SNAPSHOT_GENERATOR_TIMEFRAMES = ("weekly", "monthly")
SNAPSHOT_GENERATOR_SERVER_KEYS = (
ALL_SERVERS_SLUG,
"comunidad-hispana-01",
"comunidad-hispana-02",
)
SNAPSHOT_GENERATOR_METRICS = (
"kills",
"deaths",
"teamkills",
"matches_considered",
"kd_ratio",
"kills_per_match",
)
RANKING_SNAPSHOT_SQLITE_SCHEMA_SQL = """
CREATE TABLE IF NOT EXISTS ranking_snapshots (
id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -102,6 +119,107 @@ def is_ranking_runtime_fallback_enabled() -> bool:
return normalized in {"1", "true", "yes", "on"}
def generate_ranking_snapshot(
*,
timeframe: str,
server_key: str | None,
metric: str,
limit: int,
replace_existing: bool = True,
now: datetime | None = None,
db_path: Path | None = None,
) -> dict[str, object]:
"""Generate and persist one weekly/monthly ranking snapshot."""
normalized_timeframe = _normalize_generator_timeframe(timeframe)
normalized_server_key = _normalize_generator_server_key(server_key)
normalized_metric = _normalize_generator_metric(metric)
normalized_limit = _normalize_limit(limit)
anchor = _as_utc(now or datetime.now(timezone.utc))
resolved_path = initialize_ranking_snapshot_storage(db_path=db_path)
connection_scope = _connect_write_scope(resolved_path, db_path=db_path)
with connection_scope as connection:
window = select_leaderboard_window(
connection=connection,
server_key=normalized_server_key,
timeframe=normalized_timeframe,
now=anchor,
)
window_start = _to_iso(window["start"])
window_end = _to_iso(window["end"])
existing_snapshot_id = _find_existing_snapshot_for_window(
connection=connection,
timeframe=normalized_timeframe,
server_key=normalized_server_key,
metric=normalized_metric,
window_start=window_start,
window_end=window_end,
)
if existing_snapshot_id is not None and not replace_existing:
snapshot = _get_snapshot_record(connection=connection, snapshot_id=existing_snapshot_id)
items = _list_snapshot_items(connection=connection, snapshot_id=existing_snapshot_id)
return {
"status": "ok",
"snapshot": snapshot,
"items": items,
"source_matches_count": int(snapshot.get("source_matches_count") or 0)
if snapshot
else 0,
"ranked_players": len(items),
"skipped_regeneration": True,
}
ranking_rows = _fetch_leaderboard_rows(
connection,
server_key=normalized_server_key,
metric=normalized_metric,
limit=normalized_limit,
window_start=window["start"],
window_end=window["end"],
)
source_matches_count = _count_matches(
connection,
server_key=normalized_server_key,
start=window["start"],
end=window["end"],
)
if existing_snapshot_id is not None:
_delete_snapshot(connection=connection, snapshot_id=existing_snapshot_id)
snapshot_id = _insert_snapshot_record(
connection=connection,
timeframe=normalized_timeframe,
server_key=normalized_server_key,
metric=normalized_metric,
limit=normalized_limit,
source_matches_count=source_matches_count,
window_start=window_start,
window_end=window_end,
generated_at=_to_iso(datetime.now(timezone.utc)),
window_kind=str(window["kind"]),
window_label=str(window["label"]),
item_count=len(ranking_rows),
)
_insert_snapshot_items(
connection=connection,
snapshot_id=snapshot_id,
rows=ranking_rows,
limit=normalized_limit,
)
snapshot = _get_snapshot_record(connection=connection, snapshot_id=snapshot_id)
items = _list_snapshot_items(connection=connection, snapshot_id=snapshot_id)
return {
"status": "ok",
"snapshot": snapshot,
"items": items,
"source_matches_count": source_matches_count,
"ranked_players": len(items),
"skipped_regeneration": False,
}
def get_latest_ranking_snapshot(
*,
server_key: str | None = None,
@@ -383,6 +501,64 @@ def _find_latest_snapshot(
return dict(row) if row else None
def _find_existing_snapshot_for_window(
*,
connection: object,
timeframe: str,
server_key: str,
metric: str,
window_start: str,
window_end: str,
) -> int | None:
row = connection.execute(
"""
SELECT id
FROM ranking_snapshots
WHERE timeframe = ?
AND server_id = ?
AND metric = ?
AND window_start = ?
AND window_end = ?
LIMIT 1
""",
[timeframe, server_key, metric, window_start, window_end],
).fetchone()
return int(row["id"]) if row else None
def _get_snapshot_record(
*,
connection: object,
snapshot_id: int,
) -> dict[str, object] | None:
row = connection.execute(
"""
SELECT
id,
timeframe,
server_id,
metric,
window_start,
window_end,
generated_at,
source,
snapshot_status,
item_count,
limit_size,
source_matches_count,
freshness,
window_kind,
window_label,
error_message
FROM ranking_snapshots
WHERE id = ?
LIMIT 1
""",
[snapshot_id],
).fetchone()
return dict(row) if row else None
def _list_snapshot_items(
*,
connection: object,
@@ -413,6 +589,115 @@ def _list_snapshot_items(
return [dict(row) for row in rows]
def _delete_snapshot(*, connection: object, snapshot_id: int) -> None:
connection.execute(
"DELETE FROM ranking_snapshot_items WHERE snapshot_id = ?",
[snapshot_id],
)
connection.execute(
"DELETE FROM ranking_snapshots WHERE id = ?",
[snapshot_id],
)
def _insert_snapshot_record(
*,
connection: object,
timeframe: str,
server_key: str,
metric: str,
limit: int,
source_matches_count: int,
window_start: str,
window_end: str,
generated_at: str,
window_kind: str,
window_label: str,
item_count: int,
) -> int:
cursor = connection.execute(
"""
INSERT INTO ranking_snapshots (
timeframe,
server_id,
metric,
window_start,
window_end,
generated_at,
source,
snapshot_status,
item_count,
limit_size,
source_matches_count,
freshness,
window_kind,
window_label,
error_message
) VALUES (?, ?, ?, ?, ?, ?, ?, 'ready', ?, ?, ?, 'fresh', ?, ?, NULL)
RETURNING id
""",
[
timeframe,
server_key,
metric,
window_start,
window_end,
generated_at,
"rcon-materialized-admin-log",
item_count,
limit,
source_matches_count,
window_kind,
window_label,
],
)
row = cursor.fetchone()
if row is None or row["id"] is None:
raise RuntimeError("Unable to resolve ranking snapshot id after insert.")
return int(row["id"])
def _insert_snapshot_items(
*,
connection: object,
snapshot_id: int,
rows: list[dict[str, object]],
limit: int,
) -> None:
for index, row in enumerate(rows[:limit], start=1):
item = _build_item(row, index=index)
connection.execute(
"""
INSERT INTO ranking_snapshot_items (
snapshot_id,
ranking_position,
player_id,
player_name,
metric_value,
matches_considered,
kills,
deaths,
teamkills,
kd_ratio,
kills_per_match
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
[
snapshot_id,
int(item["ranking_position"]),
str(item["player_id"]),
str(item["player_name"]),
float(item["metric_value"]),
int(item["matches_considered"]),
int(item["kills"]),
int(item["deaths"]),
int(item["teamkills"]),
float(item["kd_ratio"]),
float(item["kills_per_match"]),
],
)
def _fetch_leaderboard_rows(
connection: object,
*,
@@ -702,6 +987,14 @@ def _connect_scope(resolved_path: Path, *, db_path: Path | None):
return closing(connect_sqlite_readonly(resolved_path))
def _connect_write_scope(resolved_path: Path, *, db_path: Path | None):
if use_postgres_rcon_storage(explicit_sqlite_path=db_path):
from .postgres_rcon_storage import connect_postgres_compat
return connect_postgres_compat()
return connect_sqlite_writer(resolved_path)
def _empty_payload(
*,
server_key: str | None,
@@ -785,6 +1078,40 @@ def _normalize_timeframe(value: str) -> LeaderboardTimeframe:
return "monthly" if str(value or "").strip().lower() == "monthly" else "weekly"
def _normalize_generator_timeframe(value: str) -> str:
normalized = str(value or "").strip().lower()
if normalized not in SNAPSHOT_GENERATOR_TIMEFRAMES:
raise ValueError(
f"timeframe must be one of: {', '.join(SNAPSHOT_GENERATOR_TIMEFRAMES)}"
)
return normalized
def _normalize_generator_server_key(server_key: str | None) -> str:
normalized = _normalize_snapshot_server_key(server_key)
if normalized not in SNAPSHOT_GENERATOR_SERVER_KEYS:
raise ValueError(
"server_key must be one of: all, all-servers, comunidad-hispana-01, comunidad-hispana-02"
)
return normalized
def _normalize_generator_metric(metric: str) -> str:
normalized = str(metric or "").strip().lower()
if normalized not in SNAPSHOT_GENERATOR_METRICS:
raise ValueError(
f"metric must be one of: {', '.join(SNAPSHOT_GENERATOR_METRICS)}"
)
return normalized
def _normalize_limit(limit: object, *, maximum: int = 100) -> int:
normalized_limit = int(limit or 1)
if normalized_limit < 1:
raise ValueError("limit must be greater than zero")
return min(normalized_limit, maximum)
def _normalize_metric(value: str) -> LeaderboardMetric:
normalized = str(value or "kills").strip().lower()
if normalized in {
@@ -909,3 +1236,50 @@ def _to_iso(value: object) -> str:
if parsed is None:
parsed = datetime.now(timezone.utc)
return parsed.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
def _main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description="Read and generate weekly/monthly ranking snapshots.",
)
subparsers = parser.add_subparsers(dest="command")
generate_parser = subparsers.add_parser("generate-ranking-snapshot")
generate_parser.add_argument(
"--timeframe",
required=True,
choices=SNAPSHOT_GENERATOR_TIMEFRAMES,
)
generate_parser.add_argument("--server-key", default=None)
generate_parser.add_argument(
"--metric",
required=True,
choices=SNAPSHOT_GENERATOR_METRICS,
)
generate_parser.add_argument("--limit", type=int, default=20)
generate_parser.add_argument(
"--no-replace-existing",
action="store_false",
dest="replace_existing",
help="keep an existing snapshot for the selected exact window and metric scope",
)
parser.set_defaults(command="generate-ranking-snapshot", replace_existing=True)
args = parser.parse_args(argv)
if args.command == "generate-ranking-snapshot":
payload = generate_ranking_snapshot(
timeframe=args.timeframe,
server_key=args.server_key,
metric=args.metric,
limit=args.limit,
replace_existing=args.replace_existing,
db_path=get_storage_path(),
)
print(json.dumps({"status": "ok", "data": payload}, ensure_ascii=True, indent=2))
return 0
parser.print_help()
return 2
if __name__ == "__main__":
raise SystemExit(_main())

View File

@@ -245,6 +245,72 @@ Operational note after `TASK-191`:
- runtime fallback remains enabled by default for transition through `HLL_BACKEND_RANKING_RUNTIME_FALLBACK_ENABLED=true`
- operators can force controlled missing behavior by setting `HLL_BACKEND_RANKING_RUNTIME_FALLBACK_ENABLED=false`
## Manual Generation Workflow
Manual generator entrypoint:
```bash
python -m app.rcon_historical_leaderboards generate-ranking-snapshot --timeframe weekly --server-key all --metric kills --limit 20
```
Docker form:
```bash
docker compose exec backend python -m app.rcon_historical_leaderboards generate-ranking-snapshot --timeframe weekly --server-key all --metric kills --limit 20
```
Supported manual parameters:
- `timeframe`: `weekly`, `monthly`
- `server-key`: `all`, `all-servers`, `comunidad-hispana-01`, `comunidad-hispana-02`
- `metric`: `kills`, `deaths`, `teamkills`, `matches_considered`, `kd_ratio`, `kills_per_match`
- `limit`: positive integer, normally `20`
Current implementation note:
- V1 generator is unitary per command invocation
- operators should run one command per required `(timeframe, server-key, metric)` combination
- broad matrix generation can remain a later operational helper if scheduling needs justify it
## Recommended Combinations
Minimum production matrix for parity with the public Ranking filters:
- weekly + `all-servers` + all six supported metrics
- weekly + `comunidad-hispana-01` + all six supported metrics
- weekly + `comunidad-hispana-02` + all six supported metrics
- monthly + `all-servers` + all six supported metrics
- monthly + `comunidad-hispana-01` + all six supported metrics
- monthly + `comunidad-hispana-02` + all six supported metrics
That matrix requires `36` snapshot generations for each refresh cycle.
## Suggested Frequency
Suggested operator cadence:
- weekly current window: regenerate every `5` to `15` minutes while the active week is changing
- monthly current window: regenerate every `15` to `30` minutes while the active month is changing
- previous week and previous month windows: regenerate once after closure or after any historical backfill that changes source coverage
Operational guidance:
- regenerate after materialized RCON/AdminLog data grows
- regenerate after manual backfill
- regenerate after metric SQL changes that affect ranking totals or ordering
## Ready Vs Fallback
Expected API result after successful generation for an exact requested combination:
- `snapshot_status=ready`
- `fallback_used=false`
- `source.read_model=ranking-snapshot`
Expected API result when the requested combination has not been generated yet and fallback remains enabled:
- `snapshot_status=missing`
- `fallback_used=true`
- `freshness=runtime`
Expected API result when the requested combination has not been generated and fallback is disabled:
- `snapshot_status=missing`
- `fallback_used=false`
- `items=[]`
## Out Of Scope
- backend implementation of the snapshot generator

View File

@@ -127,6 +127,8 @@ import app.postgres_rcon_storage as postgres_rcon_storage
import app.rcon_historical_leaderboards as ranking_leaderboards
initialize_ranking_snapshot_storage = ranking_leaderboards.initialize_ranking_snapshot_storage
generate_ranking_snapshot = ranking_leaderboards.generate_ranking_snapshot
get_latest_ranking_snapshot = ranking_leaderboards.get_latest_ranking_snapshot
def require(condition, message):
@@ -301,6 +303,23 @@ def cleanup_snapshot_fixture(db_path):
connection.close()
def cleanup_generated_snapshot(snapshot_id):
if not snapshot_id:
return
db_path = Path("backend/data/hll_vietnam_dev.sqlite3")
connection = sqlite3.connect(db_path)
with connection:
connection.execute(
"DELETE FROM ranking_snapshot_items WHERE snapshot_id = ?",
(snapshot_id,),
)
connection.execute(
"DELETE FROM ranking_snapshots WHERE id = ?",
(snapshot_id,),
)
connection.close()
health_status, health_payload = read_payload("/health")
require(health_status == 200, "Route resolver /health should return 200.")
require(health_payload.get("status") == "ok", "/health payload should be ok.")
@@ -567,6 +586,73 @@ missing_year_ranking_status, _ = read_payload(
)
require(missing_year_ranking_status == 400, "Global ranking annual requests without year should return 400.")
generated_snapshot_id = None
try:
generated_snapshot = generate_ranking_snapshot(
timeframe="weekly",
server_key="all",
metric="kills",
limit=20,
now=datetime(2026, 5, 20, 12, 0, 0, tzinfo=timezone.utc),
db_path=Path("backend/data/hll_vietnam_dev.sqlite3"),
)
generated_snapshot_record = generated_snapshot.get("snapshot") or {}
generated_snapshot_id = generated_snapshot_record.get("id")
require(generated_snapshot.get("status") == "ok", "Weekly ranking snapshot generation should return ok.")
require(generated_snapshot_id, "Generated weekly ranking snapshot should expose snapshot id.")
require(
generated_snapshot_record.get("timeframe") == "weekly",
"Generated weekly ranking snapshot should preserve timeframe.",
)
require(
generated_snapshot_record.get("metric") == "kills",
"Generated weekly ranking snapshot should preserve metric.",
)
latest_generated_snapshot = get_latest_ranking_snapshot(
server_key="all",
timeframe="weekly",
metric="kills",
limit=20,
db_path=Path("backend/data/hll_vietnam_dev.sqlite3"),
)
require(
latest_generated_snapshot.get("snapshot_status") == "ready",
"Generated weekly ranking snapshot should be readable as ready.",
)
require(
latest_generated_snapshot.get("generated_at"),
"Generated weekly ranking snapshot should expose generated_at.",
)
generated_route_status, generated_route_payload = read_payload(
"/api/ranking?timeframe=weekly&server_id=all&metric=kills&limit=20"
)
require(generated_route_status == 200, "Generated weekly ranking route should return 200.")
generated_route_data = generated_route_payload.get("data") or {}
require(
generated_route_data.get("snapshot_status") == "ready",
"Generated weekly ranking route should return snapshot_status=ready.",
)
require(
generated_route_data.get("fallback_used") is False,
"Generated weekly ranking route should not use runtime fallback.",
)
require(
isinstance(generated_route_data.get("items"), list),
"Generated weekly ranking route should return items list.",
)
generated_fallback_status, generated_fallback_payload = read_payload(
"/api/ranking?timeframe=weekly&server_id=all&metric=deaths&limit=20"
)
require(generated_fallback_status == 200, "Generated validation fallback route should return 200.")
generated_fallback_data = generated_fallback_payload.get("data") or {}
require(
generated_fallback_data.get("fallback_used") is True,
"Missing weekly deaths snapshot should still use runtime fallback when enabled.",
)
finally:
cleanup_generated_snapshot(generated_snapshot_id)
fixture_db_path = build_snapshot_fixture()
try:
fixture_weekly_status, fixture_weekly_payload = read_payload(
@@ -610,6 +696,7 @@ print(json.dumps({
"global-ranking",
"postgres-ranking-derived-metric-sql",
"postgres-ranking-schema-path",
"ranking-snapshot-generator",
"ranking-snapshot-ready",
"ranking-snapshot-missing",
],