diff --git a/ai/tasks/done/TASK-087-configurable-historical-refresh-overlap.md b/ai/tasks/done/TASK-087-configurable-historical-refresh-overlap.md new file mode 100644 index 0000000..a9e4ab6 --- /dev/null +++ b/ai/tasks/done/TASK-087-configurable-historical-refresh-overlap.md @@ -0,0 +1,71 @@ +# TASK-087-configurable-historical-refresh-overlap + +## Goal +Hacer configurable la ventana de solape del refresh historico para permitir pasadas manuales de 24h, 48h o mas sobre todos los servidores sin recurrir a un bootstrap grande innecesario. + +## Context +La operativa actual del historico tiene dos limites practicos: +- el refresh incremental del historico base relee solo una ventana corta reciente +- el worker de player-events V2 tambien corta demasiado pronto para recuperar huecos recientes de forma controlada + +Esto complica recuperar faltantes reales de los ultimos 1-3 dias, por ejemplo cuando faltan datos desde el lunes en uno o varios servidores. + +La solucion inmediata no es RCON historico, sino exponer una ventana de solape configurable para: +- `historical_ingestion refresh` +- `player_event_worker refresh` + +## Steps +1. Revisar el flujo actual de cutoff y solape en: + - `backend/app/historical_ingestion.py` + - `backend/app/historical_storage.py` + - `backend/app/player_event_worker.py` + - `backend/app/player_event_storage.py` + - `backend/app/config.py` +2. Añadir configuracion explicita para el solape temporal del refresh, manteniendo compatibilidad hacia atras: + - `HLL_HISTORICAL_REFRESH_OVERLAP_HOURS` + - `HLL_PLAYER_EVENT_REFRESH_OVERLAP_HOURS` +3. Exponer tambien override por CLI en ambos comandos: + - `python -m app.historical_ingestion refresh --overlap-hours 48` + - `python -m app.player_event_worker refresh --overlap-hours 48` +4. Mantener el comportamiento por defecto actual o equivalente si el operador no pasa override. +5. Asegurar que el refresh global sin `--server` recorre los tres servidores historicos ya registrados. +6. Documentar en README el runbook operativo para: + - pasada manual de 48 horas + - pasada de validacion por un solo servidor + - recomposicion posterior de snapshots +7. Mantener el trabajo limitado al backend y runbook operativo. No tocar UI. + +## Files to Read First +- `backend/README.md` +- `backend/app/historical_ingestion.py` +- `backend/app/historical_storage.py` +- `backend/app/player_event_worker.py` +- `backend/app/player_event_storage.py` +- `backend/app/config.py` + +## Expected Files to Modify +- `backend/app/historical_ingestion.py` +- `backend/app/historical_storage.py` +- `backend/app/player_event_worker.py` +- `backend/app/player_event_storage.py` +- `backend/app/config.py` +- `backend/README.md` + +## Constraints +- No romper el refresh incremental actual. +- No cambiar el proveedor historico actual por defecto. +- No tocar frontend. +- No introducir dependencias nuevas. +- El override por CLI debe ser opcional. +- La ventana por defecto debe seguir siendo conservadora para no disparar coste innecesario. + +## Validation +- Existe soporte de configuracion/env para overlap del historico base. +- Existe soporte de configuracion/env para overlap de player-events. +- Existen flags CLI `--overlap-hours` en ambos comandos. +- Una pasada manual de 48 horas queda documentada para todos los servidores. +- El repositorio queda consistente. + +## Change Budget +- Preferir menos de 7 archivos modificados o creados. +- Preferir menos de 260 lineas cambiadas. diff --git a/ai/tasks/done/TASK-088-rcon-historical-ingestion-feasibility.md b/ai/tasks/done/TASK-088-rcon-historical-ingestion-feasibility.md new file mode 100644 index 0000000..6daf1d8 --- /dev/null +++ b/ai/tasks/done/TASK-088-rcon-historical-ingestion-feasibility.md @@ -0,0 +1,70 @@ +# TASK-088-rcon-historical-ingestion-feasibility + +## Goal +Aterrizar con precision si la repo puede soportar una ingesta historica por RCON y, en caso afirmativo, definir una arquitectura minima, incremental y defendible sin asumir capacidades que hoy no estan probadas. + +## Context +La repo ya tiene: +- proveedor live por RCON +- seleccion de `historical_data_source` +- placeholder `RconHistoricalDataSource` + +Pero todavia no existe una implementacion historica real por RCON. Antes de abrir trabajo de implementacion, hace falta una auditoria tecnica que determine: +- que datos puede dar realmente el cliente RCON actual +- si permiten reconstruccion historica, solo captura prospectiva o solo telemetria parcial +- que huecos deben mantenerse temporalmente en `public-scoreboard` +- que contrato minimo puede exponerse sin vender capacidades inexistentes + +## Steps +1. Revisar la capa actual de seleccion de proveedores y el adapter RCON existente. +2. Auditar el cliente RCON y documentar exactamente: + - comandos soportados hoy + - forma del payload disponible hoy + - frecuencia de captura razonable + - si hay o no base para historico real de partidas cerradas +3. Redactar una decision tecnica clara con una de estas salidas: + - no viable con el cliente actual + - viable solo para captura prospectiva + - viable para una capa historica parcial +4. Diseñar la arquitectura minima recomendada: + - almacenamiento + - workers + - checkpoints + - compatibilidad con `public-scoreboard` + - politica de degradacion si faltan metricas +5. Dejar una propuesta de fases realista: + - fase 1: captura prospectiva + - fase 2: lectura operativa minima + - fase 3: metricas competitivas si la senal lo permite +6. Actualizar README para reflejar el estado real y evitar ambiguedad sobre “historico por RCON”. + +## Files to Read First +- `backend/README.md` +- `backend/app/data_sources.py` +- `backend/app/providers/rcon_provider.py` +- `backend/app/rcon_client.py` +- `backend/app/historical_ingestion.py` +- `backend/app/historical_storage.py` +- `backend/app/player_event_worker.py` +- `backend/app/player_event_storage.py` + +## Expected Files to Modify +- `docs/rcon-historical-ingestion-design.md` +- `backend/README.md` + +## Constraints +- No implementar aun la ingesta historica por RCON. +- No cambiar runtime behavior del backend. +- No tocar frontend. +- No asumir que RCON resuelve backfill retroactivo si eso no esta demostrado. +- Mantener el documento muy concreto y util para una implementacion posterior. + +## Validation +- Existe un documento de diseno con conclusion clara. +- El README deja claro que parte esta implementada y cual no. +- No se introducen cambios de comportamiento en produccion o desarrollo. +- El repositorio queda consistente. + +## Change Budget +- Preferir menos de 3 archivos modificados o creados. +- Preferir menos de 220 lineas cambiadas. diff --git a/ai/tasks/done/TASK-089-rcon-prospective-historical-capture-foundation.md b/ai/tasks/done/TASK-089-rcon-prospective-historical-capture-foundation.md new file mode 100644 index 0000000..5188ad0 --- /dev/null +++ b/ai/tasks/done/TASK-089-rcon-prospective-historical-capture-foundation.md @@ -0,0 +1,72 @@ +# TASK-089-rcon-prospective-historical-capture-foundation + +## Goal +Implementar una base de captura historica prospectiva por RCON que empiece a persistir datos hacia delante sin sustituir todavia `public-scoreboard` y sin prometer recuperacion retroactiva de periodos ya perdidos. + +## Context +Esta task depende del diseno aprobado en la task anterior. + +El objetivo aqui no es “rehacer todo el historico por RCON” en un solo paso, sino dejar una primera capacidad operativa para recoger telemetria historica hacia delante desde los targets RCON configurados. + +La captura debe: +- vivir fuera del request path HTTP +- persistir datos con trazabilidad y checkpoints +- convivir con el historico actual basado en `public-scoreboard` +- ser util aunque al principio no cubra todas las metricas competitivas + +## Steps +1. Revisar el diseno de `docs/rcon-historical-ingestion-design.md`. +2. Crear una capa de almacenamiento propia para historico prospectivo RCON: + - tablas o estructuras separadas del historico `historical_*` actual + - trazabilidad por servidor, run y checkpoint +3. Extender el cliente/provider RCON solo en la medida aprobada por el diseno: + - sin asumir comandos no auditados + - sin mezclar live state puntual con historico persistido +4. Crear un worker o runner dedicado de captura prospectiva RCON. +5. Permitir ejecucion: + - manual de una pasada + - periodica por bucle local o Compose +6. Añadir configuracion explicita para: + - targets + - intervalo + - reintentos + - timeouts +7. Añadir metadata de estado minima y runbook en README. +8. Mantener `public-scoreboard` como fuente historica por defecto hasta que exista una capa de lectura historica RCON util. + +## Files to Read First +- `docs/rcon-historical-ingestion-design.md` +- `backend/README.md` +- `backend/app/data_sources.py` +- `backend/app/providers/rcon_provider.py` +- `backend/app/rcon_client.py` +- `backend/app/config.py` +- `docker-compose.yml` + +## Expected Files to Modify +- `backend/app/config.py` +- `backend/app/data_sources.py` +- `backend/app/providers/rcon_provider.py` +- `backend/app/rcon_client.py` +- uno o varios archivos nuevos bajo `backend/app/` para worker/storage/run tracking +- `backend/README.md` +- opcionalmente `docker-compose.yml` si conviene dejar un servicio dedicado + +## Constraints +- No reemplazar todavia `public-scoreboard` como fuente historica principal. +- No tocar la UI. +- No romper `/api/servers` live por RCON. +- No prometer backfill retroactivo. +- Mantener separada la telemetria prospectiva RCON del historico importado actual. +- No introducir dependencias externas innecesarias. + +## Validation +- Existe una pasada manual funcional de captura prospectiva RCON. +- Existen checkpoints o run tracking. +- La persistencia queda separada y consistente. +- README documenta la operativa minima. +- La repo sigue pudiendo arrancar sin obligar a usar RCON historico. + +## Change Budget +- Preferir menos de 9 archivos modificados o creados. +- Preferir menos de 420 lineas cambiadas. diff --git a/ai/tasks/done/TASK-090-rcon-historical-provider-minimal-read-model.md b/ai/tasks/done/TASK-090-rcon-historical-provider-minimal-read-model.md new file mode 100644 index 0000000..5185d52 --- /dev/null +++ b/ai/tasks/done/TASK-090-rcon-historical-provider-minimal-read-model.md @@ -0,0 +1,65 @@ +# TASK-090-rcon-historical-provider-minimal-read-model + +## Goal +Construir una primera capa de lectura historica minima sobre la persistencia prospectiva RCON, sin intentar aun paridad completa con todos los rankings competitivos del historico actual. + +## Context +Esta task depende de la task anterior. + +Una vez exista captura prospectiva RCON, hace falta una primera capa de lectura util que permita comprobar cobertura real y exponer algo operativo sin mentir sobre la profundidad disponible. + +La prioridad aqui no es clonar toda la salida de `public-scoreboard`, sino exponer: +- cobertura +- actividad reciente +- estado del historico RCON disponible +- una base compatible para evolucion posterior + +## Steps +1. Implementar una primera version funcional de `RconHistoricalDataSource` basada en datos persistidos, no en consultas RCON on-demand dentro del request path. +2. Definir un read model minimo util para: + - resumen/cobertura por servidor + - actividad o sesiones recientes + - metadata de disponibilidad y frescura +3. Exponer un camino de seleccion seguro por `HLL_BACKEND_HISTORICAL_DATA_SOURCE=rcon` sin romper el modo `public-scoreboard`. +4. Mantener la degradacion controlada cuando falten metricas: + - devolver payload coherente + - documentar que contratos quedan soportados y cuales no todavia +5. Actualizar README y runbook para aclarar: + - que endpoints funcionan con la lectura RCON minima + - que endpoints siguen dependiendo de `public-scoreboard` +6. No intentar aun: + - weekly/monthly leaderboards completos + - MVP V1/V2 completos + - equivalencia total con `historico.html` + +## Files to Read First +- `docs/rcon-historical-ingestion-design.md` +- `backend/README.md` +- `backend/app/data_sources.py` +- `backend/app/payloads.py` +- `backend/app/routes.py` +- los archivos creados en la task anterior para captura prospectiva RCON + +## Expected Files to Modify +- `backend/app/data_sources.py` +- `backend/app/payloads.py` +- `backend/app/routes.py` +- uno o varios archivos nuevos bajo `backend/app/` para read model RCON historico +- `backend/README.md` + +## Constraints +- No sustituir aun el historico actual completo. +- No tocar frontend. +- No exponer contratos falsos o semicompletos como si fueran paridad total. +- La lectura HTTP debe seguir siendo fast-path de solo lectura sobre persistencia local. +- Si un endpoint no queda soportado por la capa minima, debe quedar documentado y degradar de forma clara. + +## Validation +- `HLL_BACKEND_HISTORICAL_DATA_SOURCE=rcon` deja una lectura historica minima operativa y documentada. +- El backend no rompe el modo `public-scoreboard`. +- La documentacion deja claro el alcance real de esta primera capa de lectura. +- El repositorio queda consistente. + +## Change Budget +- Preferir menos de 7 archivos modificados o creados. +- Preferir menos de 320 lineas cambiadas. diff --git a/backend/README.md b/backend/README.md index 59a4ed8..59edbd8 100644 --- a/backend/README.md +++ b/backend/README.md @@ -68,12 +68,16 @@ Variables opcionales: - `HLL_BACKEND_HISTORICAL_DATA_SOURCE` - `HLL_BACKEND_RCON_TIMEOUT_SECONDS` - `HLL_BACKEND_RCON_TARGETS` +- `HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS` +- `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES` +- `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS` - `HLL_HISTORICAL_CRCON_PAGE_SIZE` - `HLL_HISTORICAL_CRCON_TIMEOUT_SECONDS` - `HLL_HISTORICAL_CRCON_DETAIL_WORKERS` - `HLL_HISTORICAL_CRCON_REQUEST_RETRIES` - `HLL_HISTORICAL_CRCON_RETRY_DELAY_SECONDS` - `HLL_HISTORICAL_REFRESH_INTERVAL_SECONDS` +- `HLL_HISTORICAL_REFRESH_OVERLAP_HOURS` - `HLL_HISTORICAL_SNAPSHOT_REFRESH_INTERVAL_SECONDS` - `HLL_HISTORICAL_FULL_SNAPSHOT_EVERY_RUNS` - `HLL_HISTORICAL_REFRESH_MAX_RETRIES` @@ -96,6 +100,7 @@ Variables especialmente relevantes para Docker y Compose: - `HLL_HISTORICAL_CRCON_DETAIL_WORKERS` - `HLL_HISTORICAL_CRCON_REQUEST_RETRIES` - `HLL_HISTORICAL_CRCON_RETRY_DELAY_SECONDS` +- `HLL_HISTORICAL_REFRESH_OVERLAP_HOURS` - `HLL_HISTORICAL_SNAPSHOT_REFRESH_INTERVAL_SECONDS` - `HLL_HISTORICAL_FULL_SNAPSHOT_EVERY_RUNS` - `HLL_HISTORICAL_REFRESH_MAX_RETRIES` @@ -285,11 +290,25 @@ Limitacion actual de `rcon`: `public-scoreboard`, porque la repo todavia no incluye una canalizacion persistente de eventos o logs RCON para reconstruir partidas cerradas +Estado real de "historico por RCON" en esta repo: + +- no existe backfill retroactivo por RCON con el cliente actual +- la viabilidad documentada hoy es solo para captura prospectiva separada +- `public-scoreboard` sigue siendo la fuente historica principal +- el diseno tecnico de esa linea prospectiva queda en + `docs/rcon-historical-ingestion-design.md` + Variables especificas de RCON live: - `HLL_BACKEND_RCON_TIMEOUT_SECONDS` - `HLL_BACKEND_RCON_TARGETS` +Variables especificas de captura historica prospectiva RCON: + +- `HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS` +- `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES` +- `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS` + `HLL_BACKEND_RCON_TARGETS` acepta un array JSON con: - `name` @@ -339,6 +358,80 @@ Invoke-WebRequest http://127.0.0.1:8000/health | Select-Object -Expand Content La respuesta incluye `live_data_source` y `historical_data_source`, util para confirmar si la instancia esta usando `a2s` o `rcon` para live. +Captura historica prospectiva por RCON: + +- se ejecuta fuera del request path HTTP +- persiste muestras live hacia delante en tablas `rcon_historical_*` +- no sustituye todavia el historico competitivo basado en `public-scoreboard` +- no promete backfill retroactivo de matches ya perdidos + +Comandos manuales desde `backend/`: + +```powershell +python -m app.rcon_historical_worker capture +python -m app.rcon_historical_worker capture --target comunidad-hispana-01 +python -m app.rcon_historical_worker loop --interval 120 +``` + +Runbook minimo: + +- una pasada manual sobre todos los targets RCON configurados: + + ```powershell + python -m app.rcon_historical_worker capture + ``` + +- una validacion acotada sobre un target concreto: + + ```powershell + python -m app.rcon_historical_worker capture --target comunidad-hispana-01 + ``` + +- un worker local en bucle: + + ```powershell + python -m app.rcon_historical_worker loop --interval 120 --max-runs 1 + ``` + +La salida del worker incluye: + +- `target_scope` +- `captured_at` +- `targets` +- `errors` +- `storage_status` + +La persistencia queda separada del historico `historical_*` actual y usa: + +- `rcon_historical_targets` +- `rcon_historical_capture_runs` +- `rcon_historical_samples` +- `rcon_historical_checkpoints` + +Lectura historica minima cuando `HLL_BACKEND_HISTORICAL_DATA_SOURCE=rcon`: + +- endpoints soportados hoy: + - `GET /api/historical/server-summary` + - `GET /api/historical/recent-matches` +- lo que devuelven: + - cobertura por target RCON configurado + - frescura del ultimo capture exitoso + - actividad reciente persistida +- endpoints que siguen dependiendo de `public-scoreboard` para el contrato + completo: + - `GET /api/historical/weekly-top-kills` + - `GET /api/historical/weekly-leaderboard` + - `GET /api/historical/leaderboard` + - `GET /api/historical/monthly-mvp` + - `GET /api/historical/monthly-mvp-v2` + - `GET /api/historical/player-events` + - `GET /api/historical/player-profile` + - `GET /api/historical/snapshots/*` + +Cuando esos endpoints se consultan con `historical_data_source=rcon`, el backend +devuelve un payload coherente con `supported: false` y deja claro que esa parte +del contrato todavia requiere `public-scoreboard`. + ## Criterio de estructura - `__init__.py` declara el paquete `app` y reexporta las utilidades publicas @@ -886,6 +979,7 @@ Flags utiles: - `--server comunidad-hispana-01` para limitar a un servidor - `--server comunidad-hispana-03` para validar solo el tercer scoreboard historico +- `--overlap-hours 48` para releer una ventana reciente mayor sin relanzar bootstrap - `--max-pages 2` para validacion local acotada - `--page-size 25` para ajustar paginacion - `--start-page 4` para forzar una pagina concreta en bootstraps largos @@ -929,6 +1023,19 @@ la sesion anterior se corta por tiempo disponible o por inestabilidad puntual del origen. `--start-page` queda como override manual cuando se quiera reprocesar o inspeccionar un tramo concreto. +Runbook operativo para overlap manual: + +```powershell +python -m app.historical_ingestion refresh --overlap-hours 48 +python -m app.historical_ingestion refresh --server comunidad-hispana-01 --overlap-hours 48 --max-pages 2 +python -m app.historical_runner --max-runs 1 +``` + +La primera pasada relee 48 horas sobre los tres servidores historicos ya +registrados. La segunda sirve para validar un solo servidor con alcance +acotado. La tercera recompone snapshots despues de una pasada manual cuando se +quiere confirmar que la capa precalculada vuelve a quedar alineada. + Los reintentos de cada request JSON pueden ajustarse sin tocar codigo con: - `HLL_HISTORICAL_CRCON_REQUEST_RETRIES` @@ -1018,6 +1125,14 @@ Comprobaciones utiles con Compose: - `docker compose logs -f historical-runner` - `docker compose exec backend python -m app.historical_runner --max-runs 1` +Compose para captura prospectiva RCON: + +```powershell +docker compose up -d rcon-historical-worker +docker compose logs -f rcon-historical-worker +docker compose exec backend python -m app.rcon_historical_worker capture +``` + Variables utiles del runner: - `HLL_HISTORICAL_SNAPSHOT_REFRESH_INTERVAL_SECONDS` @@ -1118,9 +1233,23 @@ python -m app.player_event_worker loop --interval 1800 Variables opcionales del worker: - `HLL_PLAYER_EVENT_REFRESH_INTERVAL_SECONDS` +- `HLL_PLAYER_EVENT_REFRESH_OVERLAP_HOURS` - `HLL_PLAYER_EVENT_REFRESH_MAX_RETRIES` - `HLL_PLAYER_EVENT_REFRESH_RETRY_DELAY_SECONDS` +Flags utiles del worker: + +- `--server comunidad-hispana-01` para validar un solo servidor +- `--overlap-hours 48` para releer una ventana reciente mayor +- `--max-pages 1` para una comprobacion acotada + +Ejemplos operativos: + +```powershell +python -m app.player_event_worker refresh --overlap-hours 48 +python -m app.player_event_worker refresh --server comunidad-hispana-01 --overlap-hours 48 --max-pages 1 +``` + Politica operativa minima: - el worker corre fuera del request path HTTP diff --git a/backend/app/config.py b/backend/app/config.py index b3803c0..2cb0455 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -19,6 +19,7 @@ DEFAULT_HISTORICAL_CRCON_DETAIL_WORKERS = 8 DEFAULT_HISTORICAL_CRCON_REQUEST_RETRIES = 3 DEFAULT_HISTORICAL_CRCON_RETRY_DELAY_SECONDS = 0.5 DEFAULT_HISTORICAL_REFRESH_INTERVAL_SECONDS = 1800 +DEFAULT_HISTORICAL_REFRESH_OVERLAP_HOURS = 12 DEFAULT_HISTORICAL_SNAPSHOT_REFRESH_INTERVAL_SECONDS = 900 DEFAULT_HISTORICAL_REFRESH_MAX_RETRIES = 2 DEFAULT_HISTORICAL_REFRESH_RETRY_DELAY_SECONDS = 30 @@ -26,8 +27,12 @@ DEFAULT_HISTORICAL_FULL_SNAPSHOT_EVERY_RUNS = 4 DEFAULT_HISTORICAL_WEEKLY_FALLBACK_MIN_MATCHES = 3 DEFAULT_HISTORICAL_WEEKLY_FALLBACK_MAX_WEEKDAY = 2 DEFAULT_PLAYER_EVENT_REFRESH_INTERVAL_SECONDS = 1800 +DEFAULT_PLAYER_EVENT_REFRESH_OVERLAP_HOURS = 12 DEFAULT_PLAYER_EVENT_REFRESH_MAX_RETRIES = 2 DEFAULT_PLAYER_EVENT_REFRESH_RETRY_DELAY_SECONDS = 30 +DEFAULT_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS = 120 +DEFAULT_RCON_HISTORICAL_CAPTURE_MAX_RETRIES = 2 +DEFAULT_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS = 15 DEFAULT_ALLOWED_ORIGINS = ( "null", "http://127.0.0.1:5500", @@ -172,6 +177,19 @@ def get_historical_refresh_interval_seconds() -> int: return interval_seconds +def get_historical_refresh_overlap_hours() -> int: + """Return the overlap window used by incremental historical refreshes.""" + configured_value = os.getenv( + "HLL_HISTORICAL_REFRESH_OVERLAP_HOURS", + str(DEFAULT_HISTORICAL_REFRESH_OVERLAP_HOURS), + ) + overlap_hours = int(configured_value) + if overlap_hours < 0: + raise ValueError("HLL_HISTORICAL_REFRESH_OVERLAP_HOURS must be zero or positive.") + + return overlap_hours + + def get_live_data_source_kind() -> str: """Return the live provider kind selected for the current environment.""" source_kind = os.getenv("HLL_BACKEND_LIVE_DATA_SOURCE", DEFAULT_LIVE_DATA_SOURCE).strip() @@ -284,6 +302,18 @@ def get_player_event_refresh_interval_seconds() -> int: return interval_seconds +def get_player_event_refresh_overlap_hours() -> int: + """Return the overlap window used by player event refresh runs.""" + configured_value = os.getenv( + "HLL_PLAYER_EVENT_REFRESH_OVERLAP_HOURS", + str(DEFAULT_PLAYER_EVENT_REFRESH_OVERLAP_HOURS), + ) + overlap_hours = int(configured_value) + if overlap_hours < 0: + raise ValueError("HLL_PLAYER_EVENT_REFRESH_OVERLAP_HOURS must be zero or positive.") + return overlap_hours + + def get_player_event_refresh_max_retries() -> int: """Return the retry count used by the player event refresh loop.""" configured_value = os.getenv( @@ -310,6 +340,44 @@ def get_player_event_refresh_retry_delay_seconds() -> int: return retry_delay_seconds +def get_rcon_historical_capture_interval_seconds() -> int: + """Return the default interval used by the prospective RCON capture loop.""" + configured_value = os.getenv( + "HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS", + str(DEFAULT_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS), + ) + interval_seconds = int(configured_value) + if interval_seconds <= 0: + raise ValueError("HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS must be positive.") + return interval_seconds + + +def get_rcon_historical_capture_max_retries() -> int: + """Return the retry count used by the prospective RCON capture loop.""" + configured_value = os.getenv( + "HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES", + str(DEFAULT_RCON_HISTORICAL_CAPTURE_MAX_RETRIES), + ) + max_retries = int(configured_value) + if max_retries < 0: + raise ValueError("HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES must be zero or positive.") + return max_retries + + +def get_rcon_historical_capture_retry_delay_seconds() -> int: + """Return the wait time between failed prospective RCON capture attempts.""" + configured_value = os.getenv( + "HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS", + str(DEFAULT_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS), + ) + retry_delay_seconds = int(configured_value) + if retry_delay_seconds < 0: + raise ValueError( + "HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS must be zero or positive." + ) + return retry_delay_seconds + + def get_a2s_targets_payload() -> str | None: """Return the optional JSON payload that overrides local A2S targets.""" raw_payload = os.getenv(DEFAULT_A2S_TARGETS_ENV_VAR) diff --git a/backend/app/data_sources.py b/backend/app/data_sources.py index 9469494..237f2d0 100644 --- a/backend/app/data_sources.py +++ b/backend/app/data_sources.py @@ -9,6 +9,11 @@ from .collector import collect_server_snapshots from .config import get_historical_data_source_kind, get_live_data_source_kind from .providers.public_scoreboard_provider import PublicScoreboardHistoricalDataSource from .providers.rcon_provider import RconLiveDataSource +from .rcon_historical_read_model import ( + describe_rcon_historical_read_model, + list_rcon_historical_recent_activity, + list_rcon_historical_server_summaries, +) from .server_targets import A2SServerTarget, load_a2s_targets @@ -72,15 +77,19 @@ class A2SLiveDataSource: @dataclass(frozen=True, slots=True) class RconHistoricalDataSource: - """Placeholder historical provider for future production RCON integration.""" + """Minimal persisted historical read model over prospective RCON capture.""" source_kind: str = SOURCE_KIND_RCON def fetch_public_info(self, *, base_url: str) -> dict[str, object]: - raise RuntimeError("Historical RCON provider is not implemented yet.") + raise RuntimeError( + "RCON historical read mode does not support CRCON ingestion operations." + ) def fetch_match_page(self, *, base_url: str, page: int, limit: int) -> dict[str, object]: - raise RuntimeError("Historical RCON provider is not implemented yet.") + raise RuntimeError( + "RCON historical read mode does not support CRCON ingestion operations." + ) def fetch_match_details( self, @@ -89,7 +98,26 @@ class RconHistoricalDataSource: match_ids: list[str], max_workers: int, ) -> list[dict[str, object]]: - raise RuntimeError("Historical RCON provider is not implemented yet.") + raise RuntimeError( + "RCON historical read mode does not support CRCON ingestion operations." + ) + + def list_server_summaries(self, *, server_key: str | None = None) -> list[dict[str, object]]: + """Return coverage and freshness from persisted prospective RCON samples.""" + return list_rcon_historical_server_summaries(server_key=server_key) + + def list_recent_activity( + self, + *, + server_key: str | None = None, + limit: int = 20, + ) -> list[dict[str, object]]: + """Return recent persisted RCON activity without on-demand network calls.""" + return list_rcon_historical_recent_activity(server_key=server_key, limit=limit) + + def describe_capabilities(self) -> dict[str, object]: + """Describe the supported RCON historical read surface.""" + return describe_rcon_historical_read_model() def get_historical_data_source() -> HistoricalDataSource: diff --git a/backend/app/historical_ingestion.py b/backend/app/historical_ingestion.py index 83a37d4..9c841fc 100644 --- a/backend/app/historical_ingestion.py +++ b/backend/app/historical_ingestion.py @@ -10,6 +10,7 @@ from typing import Iterable from .config import ( get_historical_crcon_detail_workers, get_historical_crcon_page_size, + get_historical_refresh_overlap_hours, ) from .data_sources import HistoricalDataSource, get_historical_data_source from .historical_snapshots import generate_and_persist_historical_snapshots @@ -75,6 +76,7 @@ def run_incremental_refresh( page_size: int | None = None, start_page: int | None = None, detail_workers: int | None = None, + overlap_hours: int | None = None, rebuild_snapshots: bool = True, ) -> dict[str, object]: """Refresh recent historical pages without replaying the whole archive.""" @@ -85,6 +87,7 @@ def run_incremental_refresh( page_size=page_size, start_page=start_page, detail_workers=detail_workers, + overlap_hours=overlap_hours, incremental=True, rebuild_snapshots=rebuild_snapshots, ) @@ -98,6 +101,7 @@ def _run_ingestion( page_size: int | None, start_page: int | None, detail_workers: int | None, + overlap_hours: int | None, incremental: bool, rebuild_snapshots: bool, ) -> dict[str, object]: @@ -107,6 +111,13 @@ def _run_ingestion( selected_servers = _select_servers(server_slug) processed_servers: list[dict[str, object]] = [] active_runs: dict[str, int] = {} + resolved_overlap_hours = ( + get_historical_refresh_overlap_hours() + if overlap_hours is None + else overlap_hours + ) + if resolved_overlap_hours < 0: + raise ValueError("--overlap-hours must be zero or positive.") try: for server in selected_servers: @@ -118,7 +129,10 @@ def _run_ingestion( run_id=run_id, ) cutoff = ( - get_refresh_cutoff_for_server(str(server["slug"])) + get_refresh_cutoff_for_server( + str(server["slug"]), + overlap_hours=resolved_overlap_hours, + ) if incremental else None ) @@ -196,6 +210,7 @@ def _run_ingestion( "page_size": page_size or get_historical_crcon_page_size(), "start_page": start_page, "detail_workers": detail_workers or get_historical_crcon_detail_workers(), + "overlap_hours": resolved_overlap_hours if incremental else None, "servers": processed_servers, "coverage": list_historical_coverage_report(server_slug=server_slug), "snapshot_result": snapshot_result, @@ -408,6 +423,11 @@ def build_arg_parser() -> argparse.ArgumentParser: type=int, help="parallel worker count for per-match detail requests", ) + parser.add_argument( + "--overlap-hours", + type=int, + help="override the incremental overlap window in hours", + ) return parser @@ -431,6 +451,7 @@ def main(argv: Iterable[str] | None = None) -> int: page_size=args.page_size, start_page=args.start_page, detail_workers=args.detail_workers, + overlap_hours=args.overlap_hours, ) print(json.dumps(result, indent=2)) diff --git a/backend/app/historical_storage.py b/backend/app/historical_storage.py index 781dbf0..65692f7 100644 --- a/backend/app/historical_storage.py +++ b/backend/app/historical_storage.py @@ -8,6 +8,7 @@ from pathlib import Path from typing import Mapping from .config import ( + get_historical_refresh_overlap_hours, get_historical_weekly_fallback_max_weekday, get_historical_weekly_fallback_min_matches, get_storage_path, @@ -40,7 +41,6 @@ DEFAULT_HISTORICAL_SERVERS = ( ALL_SERVERS_SLUG = "all-servers" ALL_SERVERS_DISPLAY_NAME = "Todos" DEFAULT_WEEKLY_WINDOW_DAYS = 7 -DEFAULT_REFRESH_OVERLAP_HOURS = 12 SUPPORTED_WEEKLY_LEADERBOARD_METRICS = frozenset( { "kills", @@ -707,10 +707,17 @@ def upsert_historical_match( def get_refresh_cutoff_for_server( server_slug: str, *, - overlap_hours: int = DEFAULT_REFRESH_OVERLAP_HOURS, + overlap_hours: int | None = None, db_path: Path | None = None, ) -> str | None: """Return the timestamp used to stop incremental scans once older pages appear.""" + resolved_overlap_hours = ( + get_historical_refresh_overlap_hours() + if overlap_hours is None + else overlap_hours + ) + if resolved_overlap_hours < 0: + raise ValueError("overlap_hours must be zero or positive.") resolved_path = initialize_historical_storage(db_path=db_path) with _connect(resolved_path) as connection: server_row = _resolve_historical_server(connection, server_slug) @@ -726,7 +733,7 @@ def get_refresh_cutoff_for_server( if not latest_seen_at: return None - cutoff = _parse_timestamp(latest_seen_at) - timedelta(hours=overlap_hours) + cutoff = _parse_timestamp(latest_seen_at) - timedelta(hours=resolved_overlap_hours) return cutoff.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") diff --git a/backend/app/payloads.py b/backend/app/payloads.py index 14e80e0..91965c0 100644 --- a/backend/app/payloads.py +++ b/backend/app/payloads.py @@ -9,7 +9,7 @@ from .config import ( get_live_data_source_kind, get_refresh_interval_seconds, ) -from .data_sources import get_live_data_source +from .data_sources import RconHistoricalDataSource, get_historical_data_source, get_live_data_source from .historical_snapshot_storage import get_historical_snapshot from .historical_snapshots import ( DEFAULT_MONTHLY_SNAPSHOT_WINDOW, @@ -219,6 +219,14 @@ def build_weekly_top_kills_payload( server_id: str | None = None, ) -> dict[str, object]: """Return weekly top kills grouped by real community server.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Top kills semanales por servidor", + context="historical-top-kills", + server_key=server_id, + limit=limit, + ) + if rcon_payload is not None: + return rcon_payload result = list_weekly_top_kills(limit=limit, server_id=server_id) return { "status": "ok", @@ -244,6 +252,16 @@ def build_historical_leaderboard_payload( timeframe: str = "weekly", ) -> dict[str, object]: """Return one historical leaderboard for the requested timeframe and metric.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Ranking historico", + context="historical-leaderboard", + server_key=server_id, + limit=limit, + metric=metric, + timeframe=timeframe, + ) + if rcon_payload is not None: + return rcon_payload normalized_timeframe = timeframe.strip().lower() if isinstance(timeframe, str) else "weekly" if normalized_timeframe == "monthly": result = list_monthly_leaderboard(limit=limit, server_id=server_id, metric=metric) @@ -324,6 +342,26 @@ def build_recent_historical_matches_payload( server_slug: str | None = None, ) -> dict[str, object]: """Return recent historical matches from persisted CRCON data.""" + if get_historical_data_source_kind() == "rcon": + data_source = get_historical_data_source() + if isinstance(data_source, RconHistoricalDataSource): + items = data_source.list_recent_activity(server_key=server_slug, limit=limit) + capabilities = data_source.describe_capabilities() + return { + "status": "ok", + "data": { + "title": "Actividad reciente capturada por RCON", + "context": "historical-recent-matches", + "source": "rcon-historical-read-model", + "historical_data_source": "rcon", + "supported": True, + "coverage_basis": "prospective-rcon-samples", + "limit": limit, + "server_slug": server_slug, + "items": items, + "capabilities": capabilities, + }, + } items = list_recent_historical_matches(limit=limit, server_slug=server_slug) return { "status": "ok", @@ -344,6 +382,14 @@ def build_monthly_mvp_payload( server_id: str | None = None, ) -> dict[str, object]: """Return the precomputed monthly MVP payload through the stable API surface.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Top MVP mensual", + context="historical-monthly-mvp", + server_key=server_id, + limit=limit, + ) + if rcon_payload is not None: + return rcon_payload snapshot_payload = build_monthly_mvp_snapshot_payload( limit=limit, server_id=server_id, @@ -370,6 +416,15 @@ def build_player_event_payload( view: str = "most-killed", ) -> dict[str, object]: """Return one V2 player-event payload through the stable API surface.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Metricas V2 mensuales", + context="historical-player-events", + server_key=server_id, + limit=limit, + metric=view, + ) + if rcon_payload is not None: + return rcon_payload snapshot_payload = build_player_event_snapshot_payload( limit=limit, server_id=server_id, @@ -397,6 +452,14 @@ def build_monthly_mvp_v2_payload( server_id: str | None = None, ) -> dict[str, object]: """Return the precomputed monthly MVP V2 payload through the stable API surface.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Top MVP mensual V2", + context="historical-monthly-mvp-v2", + server_key=server_id, + limit=limit, + ) + if rcon_payload is not None: + return rcon_payload snapshot_payload = build_monthly_mvp_v2_snapshot_payload( limit=limit, server_id=server_id, @@ -421,6 +484,13 @@ def build_historical_server_summary_snapshot_payload( server_slug: str | None = None, ) -> dict[str, object]: """Return one precomputed summary snapshot without recalculating aggregates.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Snapshot historico de resumen por servidor", + context="historical-server-summary-snapshot", + server_key=server_slug, + ) + if rcon_payload is not None: + return rcon_payload snapshot = _get_historical_snapshot_record( server_key=server_slug, snapshot_type=SNAPSHOT_TYPE_SERVER_SUMMARY, @@ -450,6 +520,16 @@ def build_leaderboard_snapshot_payload( timeframe: str = "weekly", ) -> dict[str, object]: """Return one precomputed leaderboard snapshot for the requested timeframe.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Snapshot historico de leaderboard", + context="historical-leaderboard-snapshot", + server_key=server_id, + limit=limit, + metric=metric, + timeframe=timeframe, + ) + if rcon_payload is not None: + return rcon_payload normalized_timeframe = timeframe.strip().lower() if isinstance(timeframe, str) else "weekly" if normalized_timeframe == "monthly": snapshot_type = SNAPSHOT_TYPE_MONTHLY_LEADERBOARD @@ -552,6 +632,14 @@ def build_recent_historical_matches_snapshot_payload( server_slug: str | None = None, ) -> dict[str, object]: """Return one precomputed recent-matches snapshot.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Snapshot historico de actividad reciente por servidor", + context="historical-recent-matches-snapshot", + server_key=server_slug, + limit=limit, + ) + if rcon_payload is not None: + return rcon_payload snapshot = _get_historical_snapshot_record( server_key=server_slug, snapshot_type=SNAPSHOT_TYPE_RECENT_MATCHES, @@ -582,6 +670,14 @@ def build_monthly_mvp_snapshot_payload( server_id: str | None = None, ) -> dict[str, object]: """Return one precomputed monthly MVP snapshot.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Snapshot top MVP mensual", + context="historical-monthly-mvp-snapshot", + server_key=server_id, + limit=limit, + ) + if rcon_payload is not None: + return rcon_payload snapshot = _get_historical_snapshot_record( server_key=server_id, snapshot_type=SNAPSHOT_TYPE_MONTHLY_MVP, @@ -638,6 +734,14 @@ def build_monthly_mvp_v2_snapshot_payload( server_id: str | None = None, ) -> dict[str, object]: """Return one precomputed monthly MVP V2 snapshot.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Snapshot top MVP mensual V2", + context="historical-monthly-mvp-v2-snapshot", + server_key=server_id, + limit=limit, + ) + if rcon_payload is not None: + return rcon_payload snapshot = _get_historical_snapshot_record( server_key=server_id, snapshot_type=SNAPSHOT_TYPE_MONTHLY_MVP_V2, @@ -697,6 +801,15 @@ def build_player_event_snapshot_payload( view: str = "most-killed", ) -> dict[str, object]: """Return one precomputed V2 player-event snapshot.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Snapshot metricas V2 mensuales", + context="historical-player-events-snapshot", + server_key=server_id, + limit=limit, + metric=view, + ) + if rcon_payload is not None: + return rcon_payload snapshot_type = _resolve_player_event_snapshot_type(view) snapshot = _get_historical_snapshot_record( server_key=server_id, @@ -736,6 +849,29 @@ def build_historical_server_summary_payload( server_slug: str | None = None, ) -> dict[str, object]: """Return aggregated historical metrics per server.""" + if get_historical_data_source_kind() == "rcon": + data_source = get_historical_data_source() + if isinstance(data_source, RconHistoricalDataSource): + items = data_source.list_server_summaries(server_key=server_slug) + capabilities = data_source.describe_capabilities() + return { + "status": "ok", + "data": { + "title": ( + "Cobertura historica minima por RCON" + if server_slug != ALL_SERVERS_SLUG + else "Cobertura historica minima RCON agregada" + ), + "context": "historical-server-summary", + "source": "rcon-historical-read-model", + "historical_data_source": "rcon", + "summary_basis": "prospective-rcon-samples", + "server_slug": server_slug, + "supported": True, + "items": items, + "capabilities": capabilities, + }, + } items = list_historical_server_summaries(server_slug=server_slug) return { "status": "ok", @@ -757,6 +893,13 @@ def build_historical_server_summary_payload( def build_historical_player_profile_payload(player_id: str) -> dict[str, object]: """Return aggregate historical metrics for one player identity.""" + rcon_payload = _build_rcon_historical_unsupported_payload( + title="Perfil historico de jugador", + context="historical-player-profile", + server_key=player_id, + ) + if rcon_payload is not None: + return rcon_payload profile = get_historical_player_profile(player_id) return { "status": "ok", @@ -788,6 +931,45 @@ def _get_historical_snapshot_record( ) +def _build_rcon_historical_unsupported_payload( + *, + title: str, + context: str, + server_key: str | None = None, + limit: int | None = None, + metric: str | None = None, + timeframe: str | None = None, +) -> dict[str, object] | None: + if get_historical_data_source_kind() != "rcon": + return None + + data_source = get_historical_data_source() + if not isinstance(data_source, RconHistoricalDataSource): + return None + + return { + "status": "ok", + "data": { + "title": title, + "context": context, + "source": "rcon-historical-read-model", + "historical_data_source": "rcon", + "supported": False, + "server_slug": server_key, + "limit": limit, + "metric": metric, + "timeframe": timeframe, + "items": [], + "item": None, + "profile": None, + "found": False, + "missing_reason": "rcon-minimal-read-model-does-not-support-this-endpoint-yet", + "supported_historical_source_for_full_contract": "public-scoreboard", + "capabilities": data_source.describe_capabilities(), + }, + } + + def _build_historical_snapshot_metadata(snapshot: dict[str, object] | None) -> dict[str, object]: if snapshot is None: return { diff --git a/backend/app/player_event_storage.py b/backend/app/player_event_storage.py index 9074b9a..f1ed692 100644 --- a/backend/app/player_event_storage.py +++ b/backend/app/player_event_storage.py @@ -4,10 +4,10 @@ from __future__ import annotations import sqlite3 from collections.abc import Iterable -from datetime import datetime, timezone +from datetime import datetime, timedelta, timezone from pathlib import Path -from .config import get_storage_path +from .config import get_player_event_refresh_overlap_hours, get_storage_path from .player_event_models import PlayerEventRecord @@ -386,9 +386,17 @@ def get_player_event_resume_page( def get_player_event_refresh_cutoff_for_server( server_slug: str, *, + overlap_hours: int | None = None, db_path: Path | None = None, ) -> str | None: """Return the latest occurred_at already persisted for one server.""" + resolved_overlap_hours = ( + get_player_event_refresh_overlap_hours() + if overlap_hours is None + else overlap_hours + ) + if resolved_overlap_hours < 0: + raise ValueError("overlap_hours must be zero or positive.") resolved_path = initialize_player_event_storage(db_path=db_path) with _connect(resolved_path) as connection: row = connection.execute( @@ -399,7 +407,12 @@ def get_player_event_refresh_cutoff_for_server( """, (server_slug,), ).fetchone() - return str(row["latest_occurred_at"]) if row and row["latest_occurred_at"] else None + latest_occurred_at = str(row["latest_occurred_at"]) if row and row["latest_occurred_at"] else None + if not latest_occurred_at: + return None + + cutoff = _parse_timestamp(latest_occurred_at) - timedelta(hours=resolved_overlap_hours) + return cutoff.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") def _connect(db_path: Path) -> sqlite3.Connection: @@ -411,3 +424,11 @@ def _connect(db_path: Path) -> sqlite3.Connection: def _utc_now_iso() -> str: return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") + + +def _parse_timestamp(value: str) -> datetime: + normalized = value.strip().replace("Z", "+00:00") + parsed = datetime.fromisoformat(normalized) + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return parsed diff --git a/backend/app/player_event_worker.py b/backend/app/player_event_worker.py index d53eea3..56ddb0a 100644 --- a/backend/app/player_event_worker.py +++ b/backend/app/player_event_worker.py @@ -13,6 +13,7 @@ from .config import ( get_historical_crcon_page_size, get_player_event_refresh_interval_seconds, get_player_event_refresh_max_retries, + get_player_event_refresh_overlap_hours, get_player_event_refresh_retry_delay_seconds, ) from .data_sources import get_historical_data_source @@ -51,6 +52,7 @@ def run_player_event_refresh( page_size: int | None = None, start_page: int | None = None, detail_workers: int | None = None, + overlap_hours: int | None = None, ) -> dict[str, object]: """Refresh recent player event summaries from the configured historical source.""" initialize_player_event_storage() @@ -58,6 +60,13 @@ def run_player_event_refresh( event_source = get_player_event_source() resolved_page_size = page_size or get_historical_crcon_page_size() resolved_detail_workers = detail_workers or get_historical_crcon_detail_workers() + resolved_overlap_hours = ( + get_player_event_refresh_overlap_hours() + if overlap_hours is None + else overlap_hours + ) + if resolved_overlap_hours < 0: + raise ValueError("--overlap-hours must be zero or positive.") selected_servers = _select_servers(server_slug) processed_servers: list[dict[str, object]] = [] active_runs: dict[str, int] = {} @@ -70,7 +79,10 @@ def run_player_event_refresh( target_server_slug=current_server_slug, ) active_runs[current_server_slug] = run_id - cutoff = get_player_event_refresh_cutoff_for_server(current_server_slug) + cutoff = get_player_event_refresh_cutoff_for_server( + current_server_slug, + overlap_hours=resolved_overlap_hours, + ) mark_player_event_progress_started( server_slug=current_server_slug, mode="refresh", @@ -138,6 +150,7 @@ def run_player_event_refresh( "event_adapter": event_source.source_kind, "page_size": resolved_page_size, "detail_workers": resolved_detail_workers, + "overlap_hours": resolved_overlap_hours, "scope": event_source.describe_scope(), "servers": processed_servers, } @@ -394,6 +407,11 @@ def build_arg_parser() -> argparse.ArgumentParser: type=int, help="parallel worker count for per-match detail requests", ) + parser.add_argument( + "--overlap-hours", + type=int, + help="override the incremental overlap window in hours", + ) parser.add_argument( "--interval", type=int, @@ -432,6 +450,7 @@ def main(argv: Iterable[str] | None = None) -> int: page_size=args.page_size, start_page=args.start_page, detail_workers=args.detail_workers, + overlap_hours=args.overlap_hours, ) print(json.dumps(result, indent=2)) return 0 diff --git a/backend/app/providers/rcon_provider.py b/backend/app/providers/rcon_provider.py index f3b0f5f..61e7eef 100644 --- a/backend/app/providers/rcon_provider.py +++ b/backend/app/providers/rcon_provider.py @@ -4,7 +4,11 @@ from __future__ import annotations from dataclasses import dataclass -from ..rcon_client import RconServerTarget, load_rcon_targets, query_live_server_state +from ..rcon_client import ( + RconServerTarget, + load_rcon_targets, + query_live_server_sample, +) from ..snapshots import build_snapshot_batch, utc_now from ..storage import persist_snapshot_batch @@ -26,7 +30,7 @@ class RconLiveDataSource: for target in configured_targets: try: - normalized_records.append(query_live_server_state(target)) + normalized_records.append(query_live_server_sample(target)["normalized"]) except Exception as error: # noqa: BLE001 - keep provider failures controlled errors.append( { diff --git a/backend/app/rcon_client.py b/backend/app/rcon_client.py index 86ebaa8..00feeef 100644 --- a/backend/app/rcon_client.py +++ b/backend/app/rcon_client.py @@ -192,6 +192,16 @@ def query_live_server_state( timeout_seconds: float | None = None, ) -> dict[str, object]: """Query one HLL server via RCON and normalize it to the live snapshot shape.""" + sample = query_live_server_sample(target, timeout_seconds=timeout_seconds) + return dict(sample["normalized"]) + + +def query_live_server_sample( + target: RconServerTarget, + *, + timeout_seconds: float | None = None, +) -> dict[str, object]: + """Query one HLL server and return both normalized and raw session data.""" resolved_timeout = timeout_seconds or get_rcon_request_timeout_seconds() with HllRconConnection(timeout_seconds=resolved_timeout) as connection: connection.connect(host=target.host, port=target.port, password=target.password) @@ -202,19 +212,43 @@ def query_live_server_state( resolved_external_id = target.external_server_id or f"rcon:{target.host}:{target.port}" return { - "external_server_id": resolved_external_id, - "server_name": _string_or_none(session.get("serverName")) or target.name, - "status": "online", - "players": _coerce_optional_int(session.get("playerCount")), - "max_players": _coerce_optional_int(session.get("maxPlayerCount")), - "current_map": _string_or_none(session.get("mapId")) or _string_or_none(session.get("mapName")), - "region": target.region, - "source_name": target.source_name, - "snapshot_origin": "real-rcon", - "source_ref": f"rcon://{target.host}:{target.port}", + "target": { + "target_key": build_rcon_target_key(target), + "name": target.name, + "host": target.host, + "port": target.port, + "external_server_id": target.external_server_id, + "region": target.region, + "game_port": target.game_port, + "query_port": target.query_port, + "source_name": target.source_name, + }, + "normalized": { + "external_server_id": resolved_external_id, + "server_name": _string_or_none(session.get("serverName")) or target.name, + "status": "online", + "players": _coerce_optional_int(session.get("playerCount")), + "max_players": _coerce_optional_int(session.get("maxPlayerCount")), + "current_map": ( + _string_or_none(session.get("mapId")) or _string_or_none(session.get("mapName")) + ), + "region": target.region, + "source_name": target.source_name, + "snapshot_origin": "real-rcon", + "source_ref": f"rcon://{target.host}:{target.port}", + }, + "raw_session": session, } +def build_rcon_target_key(target: RconServerTarget) -> str: + """Build a stable local key for one configured RCON target.""" + external_server_id = _string_or_none(target.external_server_id) + if external_server_id: + return external_server_id + return f"rcon:{target.host}:{target.port}" + + def _coerce_rcon_target(raw_target: dict[str, object]) -> RconServerTarget: name = str(raw_target.get("name") or "Unnamed RCON target").strip() host = str(raw_target.get("host") or "").strip() diff --git a/backend/app/rcon_historical_read_model.py b/backend/app/rcon_historical_read_model.py new file mode 100644 index 0000000..25af4d6 --- /dev/null +++ b/backend/app/rcon_historical_read_model.py @@ -0,0 +1,198 @@ +"""Read-only minimal HTTP model over prospective RCON historical persistence.""" + +from __future__ import annotations + +from datetime import datetime, timezone + +from .historical_storage import ALL_SERVERS_SLUG +from .normalizers import normalize_map_name +from .rcon_historical_storage import ( + list_rcon_historical_target_statuses, + list_recent_rcon_historical_samples, +) + + +def list_rcon_historical_server_summaries( + *, + server_key: str | None = None, +) -> list[dict[str, object]]: + """Return per-target coverage and freshness from prospective RCON storage.""" + items = list_rcon_historical_target_statuses() + if server_key and server_key != ALL_SERVERS_SLUG: + normalized = server_key.strip() + items = [ + item + for item in items + if item["target_key"] == normalized or item["external_server_id"] == normalized + ] + + summaries = [_build_server_summary(item) for item in items] + if server_key == ALL_SERVERS_SLUG: + return [_build_all_servers_summary(summaries)] + return summaries + + +def list_rcon_historical_recent_activity( + *, + server_key: str | None = None, + limit: int = 20, +) -> list[dict[str, object]]: + """Return recent persisted RCON activity samples for one or all targets.""" + normalized_server_key = None if server_key == ALL_SERVERS_SLUG else server_key + items = list_recent_rcon_historical_samples(target_key=normalized_server_key, limit=limit) + return [ + { + **item, + "current_map": normalize_map_name(item.get("current_map")), + "minutes_since_capture": _minutes_since_timestamp(item.get("captured_at")), + } + for item in items + ] + + +def describe_rcon_historical_read_model() -> dict[str, object]: + """Describe what the minimal RCON historical read model currently supports.""" + return { + "source": "rcon-historical-read-model", + "supported_endpoints": [ + "/api/historical/server-summary", + "/api/historical/recent-matches", + ], + "unsupported_endpoints": [ + "/api/historical/weekly-top-kills", + "/api/historical/weekly-leaderboard", + "/api/historical/leaderboard", + "/api/historical/monthly-mvp", + "/api/historical/monthly-mvp-v2", + "/api/historical/player-events", + "/api/historical/player-profile", + "/api/historical/snapshots/*", + ], + "capabilities": [ + "coverage by configured RCON target", + "recent persisted live activity", + "freshness and last successful capture metadata", + ], + "limitations": [ + "No retroactive backfill of closed matches.", + "No weekly or monthly competitive leaderboards.", + "No MVP or player-event parity with public-scoreboard.", + "No precomputed historical snapshots for the RCON read model yet.", + ], + } + + +def _build_server_summary(item: dict[str, object]) -> dict[str, object]: + sample_count = int(item.get("sample_count") or 0) + first_last_points = list_rcon_historical_recent_activity( + server_key=str(item["target_key"]), + limit=1, + ) + last_sample_at = item.get("last_sample_at") + latest_activity = first_last_points[0] if first_last_points else None + + return { + "server": { + "slug": item["target_key"], + "name": item["display_name"], + "external_server_id": item["external_server_id"], + "region": item["region"], + }, + "coverage": { + "basis": "prospective-rcon-samples", + "status": "available" if sample_count > 0 else "empty", + "sample_count": sample_count, + "first_sample_at": item.get("first_sample_at"), + "last_sample_at": last_sample_at, + "coverage_hours": _calculate_coverage_hours(item.get("first_sample_at"), last_sample_at), + }, + "freshness": { + "last_successful_capture_at": item.get("last_successful_capture_at"), + "minutes_since_last_capture": _minutes_since_timestamp(last_sample_at), + "last_run_id": item.get("last_run_id"), + "last_run_status": item.get("last_run_status"), + "last_error": item.get("last_error"), + "last_error_at": item.get("last_error_at"), + }, + "activity": { + "latest_players": latest_activity.get("players") if latest_activity else None, + "latest_max_players": latest_activity.get("max_players") if latest_activity else None, + "latest_map": latest_activity.get("current_map") if latest_activity else None, + "latest_status": latest_activity.get("status") if latest_activity else None, + }, + "time_range": { + "start": None, + "end": last_sample_at, + }, + } + + +def _build_all_servers_summary(items: list[dict[str, object]]) -> dict[str, object]: + total_samples = sum(int(item["coverage"].get("sample_count") or 0) for item in items) + last_points = [ + item["time_range"].get("end") + for item in items + if item["time_range"].get("end") + ] + last_capture_at = max(last_points) if last_points else None + return { + "server": { + "slug": ALL_SERVERS_SLUG, + "name": "Todos", + "external_server_id": None, + "region": None, + }, + "coverage": { + "basis": "prospective-rcon-samples-aggregate", + "status": "available" if total_samples > 0 else "empty", + "sample_count": total_samples, + "first_sample_at": None, + "last_sample_at": last_capture_at, + "coverage_hours": None, + }, + "freshness": { + "last_successful_capture_at": last_capture_at, + "minutes_since_last_capture": _minutes_since_timestamp(last_capture_at), + "last_run_id": None, + "last_run_status": None, + "last_error": None, + "last_error_at": None, + }, + "activity": { + "latest_players": None, + "latest_max_players": None, + "latest_map": None, + "latest_status": None, + }, + "time_range": { + "start": None, + "end": last_capture_at, + }, + "server_count": len(items), + } + + +def _minutes_since_timestamp(timestamp: str | None) -> int | None: + if not timestamp: + return None + captured_at = datetime.fromisoformat(timestamp.replace("Z", "+00:00")) + if captured_at.tzinfo is None: + captured_at = captured_at.replace(tzinfo=timezone.utc) + delta = datetime.now(timezone.utc) - captured_at.astimezone(timezone.utc) + return max(0, int(delta.total_seconds() // 60)) + + +def _calculate_coverage_hours( + first_sample_at: str | None, + last_sample_at: str | None, +) -> float | None: + if not first_sample_at or not last_sample_at: + return None + first_point = datetime.fromisoformat(first_sample_at.replace("Z", "+00:00")) + last_point = datetime.fromisoformat(last_sample_at.replace("Z", "+00:00")) + if first_point.tzinfo is None: + first_point = first_point.replace(tzinfo=timezone.utc) + if last_point.tzinfo is None: + last_point = last_point.replace(tzinfo=timezone.utc) + delta = last_point.astimezone(timezone.utc) - first_point.astimezone(timezone.utc) + return round(delta.total_seconds() / 3600, 2) diff --git a/backend/app/rcon_historical_storage.py b/backend/app/rcon_historical_storage.py new file mode 100644 index 0000000..67fb731 --- /dev/null +++ b/backend/app/rcon_historical_storage.py @@ -0,0 +1,453 @@ +"""Separate storage and run tracking for prospective RCON historical capture.""" + +from __future__ import annotations + +import json +import sqlite3 +from collections.abc import Mapping +from datetime import datetime, timezone +from pathlib import Path + +from .config import get_storage_path + + +def initialize_rcon_historical_storage(*, db_path: Path | None = None) -> Path: + """Create the SQLite structures used by prospective RCON capture.""" + resolved_path = db_path or get_storage_path() + resolved_path.parent.mkdir(parents=True, exist_ok=True) + + with _connect(resolved_path) as connection: + connection.executescript( + """ + CREATE TABLE IF NOT EXISTS rcon_historical_targets ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + target_key TEXT NOT NULL UNIQUE, + external_server_id TEXT, + display_name TEXT NOT NULL, + host TEXT NOT NULL, + port INTEGER NOT NULL, + region TEXT, + game_port INTEGER, + query_port INTEGER, + source_name TEXT NOT NULL, + last_configured_at TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP + ); + + CREATE TABLE IF NOT EXISTS rcon_historical_capture_runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + mode TEXT NOT NULL, + status TEXT NOT NULL, + target_scope TEXT, + started_at TEXT NOT NULL, + completed_at TEXT, + targets_seen INTEGER NOT NULL DEFAULT 0, + samples_inserted INTEGER NOT NULL DEFAULT 0, + duplicate_samples INTEGER NOT NULL DEFAULT 0, + failed_targets INTEGER NOT NULL DEFAULT 0, + notes TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP + ); + + CREATE TABLE IF NOT EXISTS rcon_historical_samples ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + target_id INTEGER NOT NULL, + capture_run_id INTEGER, + captured_at TEXT NOT NULL, + source_kind TEXT NOT NULL, + status TEXT NOT NULL, + players INTEGER, + max_players INTEGER, + current_map TEXT, + normalized_payload_json TEXT NOT NULL, + raw_payload_json TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + UNIQUE(target_id, captured_at), + FOREIGN KEY (target_id) REFERENCES rcon_historical_targets(id), + FOREIGN KEY (capture_run_id) REFERENCES rcon_historical_capture_runs(id) + ); + + CREATE TABLE IF NOT EXISTS rcon_historical_checkpoints ( + target_id INTEGER PRIMARY KEY, + last_successful_capture_at TEXT, + last_sample_at TEXT, + last_run_id INTEGER, + last_run_status TEXT, + last_error TEXT, + last_error_at TEXT, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + FOREIGN KEY (target_id) REFERENCES rcon_historical_targets(id), + FOREIGN KEY (last_run_id) REFERENCES rcon_historical_capture_runs(id) + ); + + CREATE INDEX IF NOT EXISTS idx_rcon_historical_samples_target_time + ON rcon_historical_samples(target_id, captured_at DESC); + """ + ) + + return resolved_path + + +def start_rcon_historical_capture_run( + *, + mode: str, + target_scope: str, + db_path: Path | None = None, +) -> int: + """Create one run row for prospective RCON capture.""" + resolved_path = initialize_rcon_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + cursor = connection.execute( + """ + INSERT INTO rcon_historical_capture_runs ( + mode, + status, + target_scope, + started_at + ) VALUES (?, 'running', ?, ?) + """, + (mode, target_scope, _utc_now_iso()), + ) + return int(cursor.lastrowid) + + +def finalize_rcon_historical_capture_run( + run_id: int, + *, + status: str, + targets_seen: int, + samples_inserted: int, + duplicate_samples: int, + failed_targets: int, + notes: str | None = None, + db_path: Path | None = None, +) -> None: + """Finalize one prospective RCON capture run.""" + resolved_path = initialize_rcon_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + connection.execute( + """ + UPDATE rcon_historical_capture_runs + SET status = ?, + completed_at = ?, + targets_seen = ?, + samples_inserted = ?, + duplicate_samples = ?, + failed_targets = ?, + notes = ? + WHERE id = ? + """, + ( + status, + _utc_now_iso(), + targets_seen, + samples_inserted, + duplicate_samples, + failed_targets, + notes, + run_id, + ), + ) + + +def persist_rcon_historical_sample( + *, + run_id: int, + captured_at: str, + target: Mapping[str, object], + normalized_payload: Mapping[str, object], + raw_payload: Mapping[str, object] | None, + db_path: Path | None = None, +) -> dict[str, int]: + """Persist one prospective RCON sample and refresh its checkpoint.""" + resolved_path = initialize_rcon_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + target_id = _upsert_target(connection, target=target) + cursor = connection.execute( + """ + INSERT OR IGNORE INTO rcon_historical_samples ( + target_id, + capture_run_id, + captured_at, + source_kind, + status, + players, + max_players, + current_map, + normalized_payload_json, + raw_payload_json + ) VALUES (?, ?, ?, 'rcon-live-sample', ?, ?, ?, ?, ?, ?) + """, + ( + target_id, + run_id, + captured_at, + normalized_payload.get("status") or "unknown", + normalized_payload.get("players"), + normalized_payload.get("max_players"), + normalized_payload.get("current_map"), + json.dumps(dict(normalized_payload), separators=(",", ":")), + json.dumps(dict(raw_payload), separators=(",", ":")) if raw_payload else None, + ), + ) + inserted = int(cursor.rowcount or 0) + _upsert_checkpoint_success( + connection, + target_id=target_id, + run_id=run_id, + captured_at=captured_at, + ) + return { + "samples_inserted": inserted, + "duplicate_samples": 0 if inserted else 1, + } + + +def mark_rcon_historical_capture_failure( + *, + run_id: int, + target: Mapping[str, object], + error_message: str, + db_path: Path | None = None, +) -> None: + """Persist failure metadata for one target inside a capture run.""" + resolved_path = initialize_rcon_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + target_id = _upsert_target(connection, target=target) + connection.execute( + """ + INSERT INTO rcon_historical_checkpoints ( + target_id, + last_run_id, + last_run_status, + last_error, + last_error_at + ) VALUES (?, ?, 'failed', ?, ?) + ON CONFLICT(target_id) DO UPDATE SET + last_run_id = excluded.last_run_id, + last_run_status = excluded.last_run_status, + last_error = excluded.last_error, + last_error_at = excluded.last_error_at, + updated_at = CURRENT_TIMESTAMP + """, + (target_id, run_id, error_message, _utc_now_iso()), + ) + + +def list_rcon_historical_target_statuses( + *, + db_path: Path | None = None, +) -> list[dict[str, object]]: + """Return per-target coverage and freshness for prospective RCON capture.""" + resolved_path = initialize_rcon_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + rows = connection.execute( + """ + SELECT + targets.target_key, + targets.external_server_id, + targets.display_name, + targets.host, + targets.port, + targets.region, + targets.source_name, + checkpoints.last_successful_capture_at, + checkpoints.last_sample_at, + checkpoints.last_run_id, + checkpoints.last_run_status, + checkpoints.last_error, + checkpoints.last_error_at, + ( + SELECT MIN(samples.captured_at) + FROM rcon_historical_samples AS samples + WHERE samples.target_id = targets.id + ) AS first_sample_at, + ( + SELECT MAX(samples.captured_at) + FROM rcon_historical_samples AS samples + WHERE samples.target_id = targets.id + ) AS latest_sample_at, + ( + SELECT COUNT(*) + FROM rcon_historical_samples AS samples + WHERE samples.target_id = targets.id + ) AS sample_count + FROM rcon_historical_targets AS targets + LEFT JOIN rcon_historical_checkpoints AS checkpoints + ON checkpoints.target_id = targets.id + ORDER BY targets.display_name ASC, targets.target_key ASC + """ + ).fetchall() + return [ + { + "target_key": row["target_key"], + "external_server_id": row["external_server_id"], + "display_name": row["display_name"], + "host": row["host"], + "port": row["port"], + "region": row["region"], + "source_name": row["source_name"], + "sample_count": int(row["sample_count"] or 0), + "first_sample_at": row["first_sample_at"], + "last_successful_capture_at": row["last_successful_capture_at"], + "last_sample_at": row["latest_sample_at"] or row["last_sample_at"], + "last_run_id": row["last_run_id"], + "last_run_status": row["last_run_status"], + "last_error": row["last_error"], + "last_error_at": row["last_error_at"], + } + for row in rows + ] + + +def list_recent_rcon_historical_samples( + *, + target_key: str | None = None, + limit: int = 20, + db_path: Path | None = None, +) -> list[dict[str, object]]: + """Return recent prospective RCON samples for one or all configured targets.""" + resolved_path = initialize_rcon_historical_storage(db_path=db_path) + where_clause = "" + params: list[object] = [limit] + if target_key: + where_clause = "WHERE targets.target_key = ? OR targets.external_server_id = ?" + params = [target_key, target_key, limit] + + with _connect(resolved_path) as connection: + rows = connection.execute( + f""" + SELECT + targets.target_key, + targets.external_server_id, + targets.display_name, + targets.region, + samples.captured_at, + samples.status, + samples.players, + samples.max_players, + samples.current_map + FROM rcon_historical_samples AS samples + INNER JOIN rcon_historical_targets AS targets + ON targets.id = samples.target_id + {where_clause} + ORDER BY samples.captured_at DESC, targets.display_name ASC + LIMIT ? + """, + params, + ).fetchall() + return [ + { + "target_key": row["target_key"], + "external_server_id": row["external_server_id"], + "display_name": row["display_name"], + "region": row["region"], + "captured_at": row["captured_at"], + "status": row["status"], + "players": row["players"], + "max_players": row["max_players"], + "current_map": row["current_map"], + } + for row in rows + ] + + +def _connect(db_path: Path) -> sqlite3.Connection: + connection = sqlite3.connect(db_path) + connection.row_factory = sqlite3.Row + connection.execute("PRAGMA foreign_keys = ON") + return connection + + +def _upsert_target(connection: sqlite3.Connection, *, target: Mapping[str, object]) -> int: + target_key = str(target.get("target_key") or "").strip() + if not target_key: + raise ValueError("Prospective RCON targets require a non-empty target_key.") + display_name = str(target.get("name") or target.get("display_name") or target_key).strip() + host = str(target.get("host") or "").strip() + port = int(target.get("port") or 0) + if not host or port <= 0: + raise ValueError("Prospective RCON targets require host and port.") + + connection.execute( + """ + INSERT INTO rcon_historical_targets ( + target_key, + external_server_id, + display_name, + host, + port, + region, + game_port, + query_port, + source_name, + last_configured_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(target_key) DO UPDATE SET + external_server_id = excluded.external_server_id, + display_name = excluded.display_name, + host = excluded.host, + port = excluded.port, + region = excluded.region, + game_port = excluded.game_port, + query_port = excluded.query_port, + source_name = excluded.source_name, + last_configured_at = excluded.last_configured_at, + updated_at = CURRENT_TIMESTAMP + """, + ( + target_key, + target.get("external_server_id"), + display_name, + host, + port, + target.get("region"), + target.get("game_port"), + target.get("query_port"), + str(target.get("source_name") or "community-hispana-rcon"), + _utc_now_iso(), + ), + ) + row = connection.execute( + "SELECT id FROM rcon_historical_targets WHERE target_key = ?", + (target_key,), + ).fetchone() + if row is None: + raise RuntimeError("Failed to resolve prospective RCON target id.") + return int(row["id"]) + + +def _upsert_checkpoint_success( + connection: sqlite3.Connection, + *, + target_id: int, + run_id: int, + captured_at: str, +) -> None: + connection.execute( + """ + INSERT INTO rcon_historical_checkpoints ( + target_id, + last_successful_capture_at, + last_sample_at, + last_run_id, + last_run_status, + last_error, + last_error_at + ) VALUES (?, ?, ?, ?, 'success', NULL, NULL) + ON CONFLICT(target_id) DO UPDATE SET + last_successful_capture_at = excluded.last_successful_capture_at, + last_sample_at = excluded.last_sample_at, + last_run_id = excluded.last_run_id, + last_run_status = excluded.last_run_status, + last_error = NULL, + last_error_at = NULL, + updated_at = CURRENT_TIMESTAMP + """, + (target_id, captured_at, captured_at, run_id), + ) + + +def _utc_now_iso() -> str: + return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") diff --git a/backend/app/rcon_historical_worker.py b/backend/app/rcon_historical_worker.py new file mode 100644 index 0000000..34ff608 --- /dev/null +++ b/backend/app/rcon_historical_worker.py @@ -0,0 +1,299 @@ +"""Dedicated prospective RCON historical capture worker.""" + +from __future__ import annotations + +import argparse +import json +import time +from dataclasses import dataclass +from typing import Iterable + +from .config import ( + get_rcon_historical_capture_interval_seconds, + get_rcon_historical_capture_max_retries, + get_rcon_historical_capture_retry_delay_seconds, +) +from .rcon_client import build_rcon_target_key, load_rcon_targets, query_live_server_sample +from .rcon_historical_storage import ( + finalize_rcon_historical_capture_run, + initialize_rcon_historical_storage, + list_rcon_historical_target_statuses, + mark_rcon_historical_capture_failure, + persist_rcon_historical_sample, + start_rcon_historical_capture_run, +) +from .snapshots import utc_now + + +@dataclass(slots=True) +class RconHistoricalCaptureStats: + targets_seen: int = 0 + samples_inserted: int = 0 + duplicate_samples: int = 0 + failed_targets: int = 0 + + +def run_rcon_historical_capture( + *, + target_key: str | None = None, +) -> dict[str, object]: + """Capture one prospective RCON sample for one or all configured targets.""" + initialize_rcon_historical_storage() + selected_targets = _select_targets(target_key) + captured_at = utc_now().isoformat().replace("+00:00", "Z") + target_scope = target_key or "all-configured-rcon-targets" + run_id = start_rcon_historical_capture_run(mode="capture", target_scope=target_scope) + stats = RconHistoricalCaptureStats() + items: list[dict[str, object]] = [] + errors: list[dict[str, object]] = [] + + try: + for target in selected_targets: + target_metadata = _serialize_target(target) + stats.targets_seen += 1 + try: + sample = query_live_server_sample(target) + delta = persist_rcon_historical_sample( + run_id=run_id, + captured_at=captured_at, + target=target_metadata, + normalized_payload=sample["normalized"], + raw_payload=sample["raw_session"], + ) + stats.samples_inserted += int(delta["samples_inserted"]) + stats.duplicate_samples += int(delta["duplicate_samples"]) + items.append( + { + "target_key": target_metadata["target_key"], + "external_server_id": target.external_server_id, + "captured_at": captured_at, + "sample_inserted": bool(delta["samples_inserted"]), + "normalized": sample["normalized"], + } + ) + except Exception as exc: # noqa: BLE001 - controlled worker failures + stats.failed_targets += 1 + mark_rcon_historical_capture_failure( + run_id=run_id, + target=target_metadata, + error_message=str(exc), + ) + errors.append( + { + "target_key": target_metadata["target_key"], + "name": target.name, + "host": target.host, + "port": target.port, + "message": str(exc), + } + ) + + status = "success" if not errors else ("partial" if items else "failed") + finalize_rcon_historical_capture_run( + run_id, + status=status, + targets_seen=stats.targets_seen, + samples_inserted=stats.samples_inserted, + duplicate_samples=stats.duplicate_samples, + failed_targets=stats.failed_targets, + notes=None if not errors else json.dumps(errors, separators=(",", ":")), + ) + except Exception as exc: + finalize_rcon_historical_capture_run( + run_id, + status="failed", + targets_seen=stats.targets_seen, + samples_inserted=stats.samples_inserted, + duplicate_samples=stats.duplicate_samples, + failed_targets=max(1, stats.failed_targets), + notes=str(exc), + ) + raise + + return { + "status": "ok" if items else "error", + "run_status": status, + "captured_at": captured_at, + "target_scope": target_scope, + "targets": items, + "errors": errors, + "storage_status": list_rcon_historical_target_statuses(), + "totals": { + "targets_seen": stats.targets_seen, + "samples_inserted": stats.samples_inserted, + "duplicate_samples": stats.duplicate_samples, + "failed_targets": stats.failed_targets, + }, + } + + +def run_periodic_rcon_historical_capture( + *, + interval_seconds: int, + max_retries: int, + retry_delay_seconds: int, + target_key: str | None = None, + max_runs: int | None = None, +) -> None: + """Run prospective RCON capture in a local loop.""" + completed_runs = 0 + print( + json.dumps( + { + "event": "rcon-historical-capture-loop-started", + "interval_seconds": interval_seconds, + "max_retries": max_retries, + "retry_delay_seconds": retry_delay_seconds, + "target_scope": target_key or "all-configured-rcon-targets", + }, + indent=2, + ) + ) + print("Press Ctrl+C to stop.") + + try: + while max_runs is None or completed_runs < max_runs: + completed_runs += 1 + payload = _run_capture_with_retries( + max_retries=max_retries, + retry_delay_seconds=retry_delay_seconds, + target_key=target_key, + ) + print(json.dumps({"run": completed_runs, **payload}, indent=2)) + if max_runs is not None and completed_runs >= max_runs: + break + time.sleep(interval_seconds) + except KeyboardInterrupt: + print("\nRCON historical capture loop stopped by user.") + + +def _run_capture_with_retries( + *, + max_retries: int, + retry_delay_seconds: int, + target_key: str | None, +) -> dict[str, object]: + attempt = 0 + while True: + attempt += 1 + try: + return { + "status": "ok", + "attempts_used": attempt, + "capture_result": run_rcon_historical_capture(target_key=target_key), + } + except Exception as exc: + if attempt > max_retries: + return { + "status": "error", + "attempts_used": attempt, + "error": str(exc), + } + if retry_delay_seconds > 0: + time.sleep(retry_delay_seconds) + + +def _select_targets(target_key: str | None) -> list[object]: + configured_targets = list(load_rcon_targets()) + if not configured_targets: + raise RuntimeError("No RCON targets configured in HLL_BACKEND_RCON_TARGETS.") + if target_key is None: + return configured_targets + + normalized = target_key.strip() + selected = [ + target + for target in configured_targets + if build_rcon_target_key(target) == normalized + ] + if not selected: + raise ValueError(f"Unknown RCON target key: {target_key}") + return selected + + +def _serialize_target(target: object) -> dict[str, object]: + return { + "target_key": build_rcon_target_key(target), + "external_server_id": target.external_server_id, + "name": target.name, + "host": target.host, + "port": target.port, + "region": target.region, + "game_port": target.game_port, + "query_port": target.query_port, + "source_name": target.source_name, + } + + +def build_arg_parser() -> argparse.ArgumentParser: + """Create the CLI parser for manual or periodic prospective RCON capture.""" + parser = argparse.ArgumentParser( + description="Prospective RCON historical capture for HLL Vietnam.", + ) + parser.add_argument( + "mode", + choices=("capture", "loop"), + help="capture runs once; loop keeps collecting periodically", + ) + parser.add_argument( + "--target", + dest="target_key", + help="optional target key; defaults to all configured RCON targets", + ) + parser.add_argument( + "--interval", + type=int, + default=get_rcon_historical_capture_interval_seconds(), + help="seconds to wait between loop runs", + ) + parser.add_argument( + "--retries", + type=int, + default=get_rcon_historical_capture_max_retries(), + help="retry attempts after a failed capture", + ) + parser.add_argument( + "--retry-delay", + type=int, + default=get_rcon_historical_capture_retry_delay_seconds(), + help="seconds to wait between failed attempts", + ) + parser.add_argument( + "--max-runs", + type=int, + help="optional safety cap for loop mode", + ) + return parser + + +def main(argv: Iterable[str] | None = None) -> int: + """Run the prospective RCON historical capture CLI.""" + parser = build_arg_parser() + args = parser.parse_args(list(argv) if argv is not None else None) + + if args.mode == "capture": + result = run_rcon_historical_capture(target_key=args.target_key) + print(json.dumps(result, indent=2)) + return 0 + + if args.interval <= 0: + raise ValueError("--interval must be a positive integer.") + if args.retries < 0: + raise ValueError("--retries must be zero or positive.") + if args.retry_delay < 0: + raise ValueError("--retry-delay must be zero or positive.") + if args.max_runs is not None and args.max_runs <= 0: + raise ValueError("--max-runs must be positive when provided.") + + run_periodic_rcon_historical_capture( + interval_seconds=args.interval, + max_retries=args.retries, + retry_delay_seconds=args.retry_delay, + target_key=args.target_key, + max_runs=args.max_runs, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docker-compose.yml b/docker-compose.yml index 81da411..18aed61 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -34,6 +34,27 @@ services: - ./backend/data:/app/data restart: unless-stopped + rcon-historical-worker: + build: + context: ./backend + container_name: hll-vietnam-rcon-historical-worker + command: ["python", "-m", "app.rcon_historical_worker", "loop", "--interval", "120"] + env_file: + - ./backend/.env.example + environment: + HLL_BACKEND_LIVE_DATA_SOURCE: ${HLL_BACKEND_LIVE_DATA_SOURCE:-a2s} + HLL_BACKEND_HISTORICAL_DATA_SOURCE: ${HLL_BACKEND_HISTORICAL_DATA_SOURCE:-public-scoreboard} + HLL_BACKEND_RCON_TIMEOUT_SECONDS: ${HLL_BACKEND_RCON_TIMEOUT_SECONDS:-10} + HLL_BACKEND_RCON_TARGETS: ${HLL_BACKEND_RCON_TARGETS:-} + HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS: ${HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS:-120} + HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES: ${HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES:-2} + HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS: ${HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS:-15} + depends_on: + - backend + volumes: + - ./backend/data:/app/data + restart: unless-stopped + frontend: build: context: ./frontend diff --git a/docs/rcon-historical-ingestion-design.md b/docs/rcon-historical-ingestion-design.md new file mode 100644 index 0000000..9992c53 --- /dev/null +++ b/docs/rcon-historical-ingestion-design.md @@ -0,0 +1,271 @@ +# RCON Historical Ingestion Design + +## Validation Date + +- 2026-03-25 + +## Scope + +Definir si la repo puede soportar historico por RCON con la implementacion +actual y, si no puede hacerlo de forma retroactiva, dejar una arquitectura +minima y defendible para una primera captura prospectiva. + +Este documento se limita a la evidencia local de la repo. No asume comandos +RCON no integrados ni capacidades externas no demostradas aqui. + +## Evidence Reviewed + +- `backend/app/data_sources.py` +- `backend/app/providers/rcon_provider.py` +- `backend/app/rcon_client.py` +- `backend/app/historical_ingestion.py` +- `backend/app/historical_storage.py` +- `backend/app/player_event_worker.py` +- `backend/app/player_event_storage.py` +- `backend/README.md` +- `docs/rcon-data-capability-audit.md` +- `docs/crcon-advanced-metrics-origin-audit.md` + +## Current State In Code + +La separacion entre live e historico ya existe en la seleccion de proveedores: + +- `get_live_data_source()` puede resolver `rcon` +- `get_historical_data_source()` puede resolver `rcon`, pero hoy devuelve un + placeholder que falla + +La implementacion RCON real disponible en la repo es minima y esta concentrada +en `backend/app/rcon_client.py`. + +Comandos soportados hoy en codigo: + +- `ServerConnect` +- `Login` +- `GetServerInformation` + +No hay evidencia local de otros comandos ya integrados para: + +- scoreboards por jugador +- detalle de partida cerrada +- eventos kill por kill +- logs tacticos +- historico retroactivo de matches cerrados + +## Payload Available Today + +La salida efectiva que la repo consume desde RCON hoy es la normalizada por +`query_live_server_state()`: + +- `external_server_id` +- `server_name` +- `status` +- `players` +- `max_players` +- `current_map` +- `region` +- `source_name` +- `snapshot_origin` +- `source_ref` + +Inferencia basada en `rcon_client.py`: + +- el payload remoto de `GetServerInformation` contiene como minimo + `serverName`, `playerCount`, `maxPlayerCount` y `mapId` o `mapName` +- la repo no persiste hoy el payload crudo ni deriva entidades historicas de + match o jugador a partir de RCON + +## Operational Frequency Assessment + +Inferencia basada en la implementacion actual: + +- cada pasada por target abre una conexion TCP +- realiza handshake `ServerConnect` +- autentica con `Login` +- ejecuta una consulta `GetServerInformation` + +Con este alcance, una frecuencia inicial razonable para captura prospectiva es: + +- cada `60` a `300` segundos para operativa normal +- `30` segundos solo como validacion o monitoreo puntual + +No hay evidencia en la repo para defender un polling mas agresivo de forma +sostenida ni para asegurar que aportaria historico competitivo util. + +## Viability Decision + +Conclusion principal: + +- no viable hoy para historico real retroactivo de partidas cerradas con el + cliente actual +- viable solo para captura prospectiva + +Motivos: + +- el cliente actual solo consulta estado live puntual +- no existe base local para reconstruir partidas ya cerradas +- no existe feed raw de eventos ni logs persistidos +- `RconHistoricalDataSource` sigue siendo un placeholder y no puede sustituir a + `public-scoreboard` + +Conclusion secundaria: + +- una capa historica parcial por RCON si es defendible, pero solo si se define + como captura prospectiva de muestras live y no como backfill de matches ya + perdidos + +## Recommended Minimal Architecture + +### Storage + +Separar completamente la persistencia prospectiva RCON del historico actual +`historical_*`. + +Tablas minimas recomendadas: + +- `rcon_historical_targets` + - identidad estable del target configurado + - ultimo estado conocido de configuracion +- `rcon_historical_capture_runs` + - una fila por ejecucion del worker + - estado, inicio, fin, errores y target scope +- `rcon_historical_samples` + - una fila por muestra y target + - `captured_at` + - identidad de target + - payload normalizado + - payload crudo opcional de `GetServerInformation` + +Si se quiere checkpoint explicito desde la primera version: + +- `rcon_historical_checkpoints` + - `target_key` + - `last_successful_capture_at` + - `last_sample_at` + - `last_error` + +### Workers + +Worker dedicado fuera del request path HTTP: + +- `python -m app.rcon_historical_worker capture` +- `python -m app.rcon_historical_worker loop --interval 120` + +Responsabilidades: + +- cargar `HLL_BACKEND_RCON_TARGETS` +- consultar cada target con el cliente RCON actual +- persistir run tracking +- persistir muestras idempotentes por target y timestamp +- actualizar checkpoints + +### Checkpoints + +Como no existe backfill retroactivo real, el checkpoint no debe modelarse como +pagina o offset de archivo historico. Debe modelarse como tiempo de captura. + +Checkpoint minimo defendible: + +- ultimo `captured_at` exitoso por target +- ultimo error por target +- ultimo run exitoso global + +### Compatibility With `public-scoreboard` + +Politica recomendada: + +- `public-scoreboard` sigue siendo la fuente historica principal para: + - leaderboards semanales y mensuales + - MVP V1 y V2 + - recent matches cerrados + - player events derivados de la capa publica actual +- RCON prospectivo convive en una linea paralela para: + - cobertura temporal hacia delante + - disponibilidad del servidor + - actividad reciente + - trazabilidad de frescura por target + +## Recommended Degradation Policy + +Si en una fase posterior se habilita `HLL_BACKEND_HISTORICAL_DATA_SOURCE=rcon`, +la degradacion minima correcta es esta: + +- solo exponer endpoints o bloques claramente soportados por la persistencia + prospectiva RCON +- no simular leaderboards completos cuando no existan +- devolver metadata de cobertura y frescura antes que rankings vacios + +Contratos defendibles en una primera lectura minima: + +- resumen de cobertura por servidor +- actividad reciente por servidor +- estado de frescura +- rango temporal disponible + +Contratos que deben seguir dependiendo de `public-scoreboard` hasta nueva +evidencia: + +- weekly leaderboards completos +- monthly leaderboards completos +- monthly MVP V1 +- monthly MVP V2 +- player profiles competitivos +- equivalencia completa con `historico.html` + +## Recommended Phases + +### Phase 1: Prospective Capture + +Objetivo: + +- empezar a guardar muestras live RCON hacia delante + +Incluye: + +- storage separado +- worker dedicado +- run tracking +- checkpoints temporales +- ejecucion manual y en loop + +No incluye: + +- backfill retroactivo +- paridad con `public-scoreboard` +- endpoints competitivos nuevos + +### Phase 2: Minimal Operational Read Model + +Objetivo: + +- leer la persistencia prospectiva RCON sin consultar RCON on-demand en HTTP + +Incluye: + +- resumen por servidor +- ultima muestra +- cobertura disponible +- actividad reciente + +### Phase 3: Competitive Metrics Only If Signal Improves + +Objetivo: + +- evaluar si aparecen comandos, eventos o logs suficientes para enriquecer la + capa historica RCON + +Solo deberia abrirse si existe evidencia real de: + +- eventos reutilizables +- scoreboards historificables +- granularidad por jugador o por encounter + +## Final Recommendation + +La decision tecnica correcta para esta repo es: + +- mantener `public-scoreboard` como fuente historica por defecto +- tratar RCON historico como una linea prospectiva separada +- no prometer reconstruccion retroactiva con el cliente actual +- abrir implementacion incremental en dos tasks: + - captura prospectiva persistida + - lectura minima sobre persistencia local