From 5788c11ad764037a4626a4b5e295e0824acb33ad Mon Sep 17 00:00:00 2001 From: devRaGonSa Date: Fri, 20 Mar 2026 22:12:43 +0100 Subject: [PATCH] Add historical CRCON storage ingestion and weekly rankings --- ai/architecture-index.md | 1 + ...torical-domain-model-and-storage-schema.md | 102 ++ ...29-historical-crcon-ingestion-bootstrap.md | 97 ++ ...30-historical-crcon-incremental-refresh.md | 80 ++ .../done/TASK-031-weekly-top-kills-api.md | 81 ++ backend/README.md | 62 +- backend/app/config.py | 28 + backend/app/historical_ingestion.py | 371 ++++++ backend/app/historical_models.py | 93 ++ backend/app/historical_storage.py | 1032 +++++++++++++++++ backend/app/routes.py | 8 + docs/decisions.md | 27 + docs/historical-domain-model.md | 149 +++ 13 files changed, 2129 insertions(+), 2 deletions(-) create mode 100644 ai/tasks/done/TASK-028-historical-domain-model-and-storage-schema.md create mode 100644 ai/tasks/done/TASK-029-historical-crcon-ingestion-bootstrap.md create mode 100644 ai/tasks/done/TASK-030-historical-crcon-incremental-refresh.md create mode 100644 ai/tasks/done/TASK-031-weekly-top-kills-api.md create mode 100644 backend/app/historical_ingestion.py create mode 100644 backend/app/historical_models.py create mode 100644 backend/app/historical_storage.py create mode 100644 docs/historical-domain-model.md diff --git a/ai/architecture-index.md b/ai/architecture-index.md index e1500ae..a474af6 100644 --- a/ai/architecture-index.md +++ b/ai/architecture-index.md @@ -69,5 +69,6 @@ Community website repository with a static landing in the current phase and a pl - The logical storage foundation for persisting server snapshots is documented in `docs/stats-database-schema-foundation.md`. - Historical match and player statistics must come from the public CRCON scoreboard JSON layer, not from A2S or the `/games` HTML shell. - The validated discovery for those historical sources is documented in `docs/historical-crcon-source-discovery.md`. +- The persisted historical domain model for CRCON matches, players and ingestion runs is documented in `docs/historical-domain-model.md`. - Frontend data consumption should remain progressive, endpoint by endpoint, with static fallbacks preserved during migration. - The frontend integration strategy is documented in `docs/frontend-data-consumption-plan.md`. diff --git a/ai/tasks/done/TASK-028-historical-domain-model-and-storage-schema.md b/ai/tasks/done/TASK-028-historical-domain-model-and-storage-schema.md new file mode 100644 index 0000000..03896e8 --- /dev/null +++ b/ai/tasks/done/TASK-028-historical-domain-model-and-storage-schema.md @@ -0,0 +1,102 @@ +# TASK-028-historical-domain-model-and-storage-schema + +## Goal +Definir e implementar la base de modelo de dominio y almacenamiento para estadísticas históricas de los 2 servidores reales de la comunidad, preparada para ingerir datos desde la capa JSON pública del scoreboard CRCON y para soportar rankings semanales posteriores. + +## Context +La fase de discovery ya confirmó que la fuente histórica correcta para estos servidores no es A2S ni el HTML de `/games`, sino la capa JSON pública del scoreboard CRCON. El siguiente paso técnico es crear una base sólida de dominio y persistencia propia para poder ingerir datos históricos, deduplicarlos y consultarlos más adelante desde nuestra propia API. + +Esta task NO debe crear todavía páginas históricas en frontend ni depender de URLs públicas de la comunidad como solución de producto. La capa histórica debe vivir en backend, con persistencia propia y preparada para exponer endpoints internos del proyecto en fases posteriores. + +## Steps +1. Revisar la documentación de discovery histórica ya creada y confirmar qué datos están disponibles desde la capa JSON pública del scoreboard CRCON. +2. Definir el modelo de dominio histórico mínimo necesario. Incluir al menos: + - servidor + - partida + - mapa + - jugador + - estadísticas de jugador por partida + - ejecución de ingesta histórica +3. Diseñar el esquema de almacenamiento local inicial para esa información. +4. Definir claves e identidad estables para evitar duplicados. Incluir expresamente: + - identificación de partida + - identificación de servidor + - identificación de jugador + - estrategia de idempotencia +5. Incluir campos suficientes para soportar futuras consultas como: + - top kills de la última semana por servidor + - partidas recientes por servidor + - mapas jugados +6. Implementar la base de almacenamiento/esquema local de forma coherente con el backend actual del proyecto. +7. Mantener clara la separación entre: + - estado actual vía A2S + - histórico persistido vía CRCON scoreboard JSON +8. Documentar la estructura creada y cómo se usará en las siguientes tasks. +9. No implementar todavía: + - UI histórica + - páginas nuevas basadas en la URL de la comunidad + - redirecciones o vistas que repliquen la web de la comunidad + - rankings finales expuestos al frontend +10. Al completar la implementación: + - dejar el repositorio consistente + - hacer commit + - hacer push al remoto si el entorno lo permite + +## Files to Read First +- AGENTS.md +- ai/repo-context.md +- ai/architecture-index.md +- docs/decisions.md +- docs/historical-crcon-source-discovery.md +- backend/README.md +- backend/app/config.py +- backend/app/routes.py +- backend/app/payloads.py +- cualquier almacenamiento local o capa de persistencia ya existente en `backend/` +- cualquier módulo collector o snapshot ya existente que pueda influir en la estructura de datos + +## Expected Files to Modify +- ai/architecture-index.md +- docs/decisions.md +- backend/README.md +- uno o más archivos nuevos de backend para almacenamiento histórico, por ejemplo: + - backend/app/historical_models.py + - backend/app/historical_storage.py + - backend/app/historical_schema.py +- opcionalmente un documento nuevo, por ejemplo: + - docs/historical-domain-model.md + +## Constraints +- No basar esta capa histórica en A2S. +- No crear páginas frontend nuevas usando la URL de la comunidad. +- No incrustar ni replicar directamente páginas de `scoreboard.comunidadhll.es`. +- No implementar todavía ingesta completa ni UI histórica. +- No romper el flujo actual de estado en tiempo real. +- No introducir complejidad innecesaria. +- No hacer cambios destructivos. +- Mantener el trabajo centrado en dominio, persistencia e idempotencia. + +## Validation +- Existe un modelo de dominio histórico claro para los 2 servidores. +- Existe una base de almacenamiento/esquema local coherente con ese modelo. +- La estructura permite futuras consultas como top kills semanales por servidor. +- Queda clara la separación entre live status y histórico persistido. +- No se han creado páginas frontend nuevas ni soluciones basadas en URLs de la comunidad. +- Los cambios quedan committeados y se hace push al remoto si el entorno lo permite. + +## Change Budget +- Preferir menos de 7 archivos modificados o creados. +- Preferir menos de 260 líneas cambiadas. +## Outcome +- Se añadio `backend/app/historical_models.py` como capa minima de dominio para servidor, mapa, partida, jugador, estadisticas por partida y ejecucion de ingesta. +- Se implemento `backend/app/historical_storage.py` con tablas `historical_*` separadas del flujo live A2S, claves estables e `UPSERT` idempotente para partidas, jugadores y estadisticas por jugador. +- Se documento el modelo e identidad estable en `docs/historical-domain-model.md`, y se alinearon `docs/decisions.md`, `ai/architecture-index.md` y `backend/README.md`. +- La inicializacion de storage incluye migracion segura desde una version legacy ya existente en la base SQLite local, preservando los datos previos en la nueva estructura. + +## Validation Result +- Validado con `python -m compileall app` desde `backend/`. +- Validado con una comprobacion local de `initialize_historical_storage()`, `list_historical_servers()` y `list_recent_historical_matches(limit=3)`. +- Revisado `git diff --name-only` para confirmar que el cambio quedo centrado en `ai/`, `docs/`, `backend/` y archivos de task. + +## Decision Notes +- El historico CRCON comparte el mismo SQLite local de desarrollo que los snapshots A2S para no introducir infraestructura prematura, pero queda estrictamente separado por tablas `historical_*`. diff --git a/ai/tasks/done/TASK-029-historical-crcon-ingestion-bootstrap.md b/ai/tasks/done/TASK-029-historical-crcon-ingestion-bootstrap.md new file mode 100644 index 0000000..21e9916 --- /dev/null +++ b/ai/tasks/done/TASK-029-historical-crcon-ingestion-bootstrap.md @@ -0,0 +1,97 @@ +# TASK-029-historical-crcon-ingestion-bootstrap + +## Goal +Implementar una ingesta histórica inicial desde la capa JSON pública del scoreboard CRCON para los 2 servidores reales de la comunidad, persistiendo datos estructurados e idempotentes en el almacenamiento histórico propio del proyecto. + +## Context +La fuente histórica real ya está descubierta y el modelo/base de almacenamiento histórico ya debe estar definido en la task previa. El siguiente paso es construir una primera ingesta real que recorra los datos históricos disponibles, los transforme al modelo propio y los guarde de forma segura y reejecutable. + +La ingesta debe apoyarse en la capa JSON del scoreboard CRCON, no en A2S ni en scraping del HTML de `/games`, y no debe depender de crear páginas nuevas o de redirigir a la web de la comunidad. + +## Steps +1. Revisar la documentación y la estructura histórica definida en la task previa. +2. Implementar un cliente o adaptador para consultar la capa JSON pública del scoreboard CRCON de ambos servidores. +3. Resolver y documentar el mapeo de cada servidor real de la comunidad con su fuente histórica correspondiente. +4. Implementar una ingesta inicial que obtenga y persista, como mínimo: + - servidor + - partida + - fecha/hora de partida + - mapa + - jugador + - kills + - muertes si están disponibles + - otras métricas estables que la fuente ofrezca de forma consistente +5. Diseñar la ingesta para ser idempotente: + - evitar duplicados + - actualizar registros si una partida cambia o se completa más tarde +6. Registrar cada ejecución de ingesta con metadatos útiles: + - inicio + - fin + - estado + - número de partidas procesadas + - número de filas insertadas/actualizadas +7. Documentar cómo lanzar la ingesta manualmente en local. +8. Mantener intacto el flujo actual de estado en tiempo real. +9. No implementar todavía endpoints finales de ranking ni UI histórica. +10. Al completar la implementación: + - dejar el repositorio consistente + - hacer commit + - hacer push al remoto si el entorno lo permite + +## Files to Read First +- AGENTS.md +- ai/repo-context.md +- ai/architecture-index.md +- docs/historical-crcon-source-discovery.md +- docs/historical-domain-model.md +- backend/README.md +- backend/app/config.py +- backend/app/routes.py +- backend/app/payloads.py +- backend/app/historical_models.py +- backend/app/historical_storage.py +- cualquier collector o cliente HTTP ya existente en backend + +## Expected Files to Modify +- backend/README.md +- backend/app/config.py +- uno o más archivos nuevos o existentes para ingesta histórica, por ejemplo: + - backend/app/historical_ingestion.py + - backend/app/historical_crcon_client.py + - backend/app/historical_storage.py + - backend/app/historical_models.py +- opcionalmente documentación técnica adicional si hace falta aclarar ejecución y alcance + +## Constraints +- No basar la ingesta histórica en A2S. +- No scrapear el HTML de `/games` salvo fallback muy justificado y documentado. +- No crear páginas frontend nuevas usando la URL de la comunidad. +- No romper el flujo actual de live status. +- No introducir complejidad innecesaria. +- No hacer cambios destructivos. +- Mantener el trabajo centrado en ingesta, persistencia e idempotencia. + +## Validation +- Existe una ingesta histórica inicial real para los 2 servidores. +- La ingesta persiste datos estructurados en almacenamiento propio. +- La ingesta puede reejecutarse sin duplicados graves. +- Queda documentado cómo ejecutarla localmente. +- No se han creado páginas frontend nuevas ni acoplamientos al HTML público de la comunidad. +- Los cambios quedan committeados y se hace push al remoto si el entorno lo permite. + +## Change Budget +- Preferir menos de 8 archivos modificados o creados. +- Preferir menos de 320 líneas cambiadas. +## Outcome +- Se implemento `backend/app/historical_ingestion.py` como cliente y adaptador de la capa JSON publica CRCON usando solo libreria estandar. +- La ingesta bootstrap consulta `get_public_info`, `get_scoreboard_maps` y `get_map_scoreboard`, transforma los payloads reales envueltos en `result` y persiste partidas y estadisticas en `historical_*`. +- Cada ejecucion registra metadatos operativos en `historical_ingestion_runs`. +- `backend/README.md` documenta como lanzar el bootstrap manualmente en local. + +## Validation Result +- Validado con `python -m compileall app`. +- Validado indirectamente con `python -m app.historical_ingestion refresh --max-pages 1` tras ajustar el cliente al payload real de CRCON; el flujo ya inserta partidas y filas de jugadores en el almacenamiento historico propio. +- No se introdujeron cambios en frontend ni dependencias hacia HTML publico de la comunidad. + +## Decision Notes +- La API CRCON actual devuelve los datos historicos bajo una clave top-level `result`; la ingesta desempaqueta esa forma para evitar falsos payloads vacios. diff --git a/ai/tasks/done/TASK-030-historical-crcon-incremental-refresh.md b/ai/tasks/done/TASK-030-historical-crcon-incremental-refresh.md new file mode 100644 index 0000000..cfb1284 --- /dev/null +++ b/ai/tasks/done/TASK-030-historical-crcon-incremental-refresh.md @@ -0,0 +1,80 @@ +# TASK-030-historical-crcon-incremental-refresh + +## Goal +Añadir un mecanismo de refresco incremental para la ingesta histórica CRCON que permita mantener actualizados los datos de ambos servidores de la comunidad sin reimportar todo el histórico completo en cada ejecución. + +## Context +Tras disponer de una ingesta bootstrap, el sistema necesita una estrategia incremental para seguir incorporando partidas nuevas o actualizadas de forma eficiente. Esta task debe construir la capa de refresco histórico incremental sobre la base ya creada, manteniendo idempotencia y trazabilidad. + +## Steps +1. Revisar la ingesta histórica bootstrap y el esquema de persistencia existente. +2. Diseñar una estrategia incremental adecuada para la fuente CRCON descubierta: + - paginación + - match ids + - detección de nuevos registros + - actualización de partidas cambiantes +3. Implementar el refresco incremental para ambos servidores. +4. Reutilizar o ampliar el registro de ejecuciones de ingesta para diferenciar: + - bootstrap completo + - refresh incremental +5. Asegurar que el refresco incremental: + - no duplica datos + - no rompe datos ya persistidos + - puede ejecutarse repetidamente +6. Documentar cómo ejecutar este refresco incremental en local. +7. No implementar todavía UI histórica. +8. No acoplar el frontend a las URLs públicas de la comunidad. +9. Al completar la implementación: + - dejar el repositorio consistente + - hacer commit + - hacer push al remoto si el entorno lo permite + +## Files to Read First +- AGENTS.md +- ai/repo-context.md +- ai/architecture-index.md +- docs/historical-crcon-source-discovery.md +- docs/historical-domain-model.md +- backend/README.md +- backend/app/config.py +- backend/app/historical_ingestion.py +- backend/app/historical_storage.py +- backend/app/historical_models.py + +## Expected Files to Modify +- backend/README.md +- backend/app/config.py +- backend/app/historical_ingestion.py +- backend/app/historical_storage.py +- opcionalmente nuevos módulos auxiliares si mejoran claridad del refresco incremental + +## Constraints +- No basar el refresco histórico en A2S. +- No crear páginas frontend nuevas usando la URL de la comunidad. +- No introducir complejidad innecesaria. +- No romper el flujo actual de live status. +- No hacer cambios destructivos. +- Mantener el trabajo centrado en refresco incremental, coherencia y trazabilidad. + +## Validation +- Existe un refresco incremental funcional para ambos servidores. +- El sistema puede mantener el histórico sin reimportar todo en cada ejecución. +- La persistencia sigue siendo idempotente. +- La documentación explica el flujo incremental. +- Los cambios quedan committeados y se hace push al remoto si el entorno lo permite. + +## Change Budget +- Preferir menos de 6 archivos modificados o creados. +- Preferir menos de 240 líneas cambiadas. +## Outcome +- Se anadio `run_incremental_refresh()` en `backend/app/historical_ingestion.py` reutilizando la misma persistencia y el registro de ejecuciones. +- El refresco incremental calcula un cutoff por servidor desde la ultima partida persistida y aplica una ventana de solape de 12 horas para releer solo paginas recientes y absorber actualizaciones tardias. +- El resultado diferencia `mode: "incremental"` y conserva trazabilidad por servidor y por ejecucion. + +## Validation Result +- Validado con `python -m app.historical_ingestion refresh --max-pages 1`. +- La ejecucion proceso las dos fuentes configuradas y persistio nuevas partidas y estadisticas sin reimportar el historico completo. +- La consulta agregada posterior y el route resolver siguieron funcionando sobre la misma base tras el refresh. + +## Decision Notes +- El cutoff incremental se apoya en `MAX(ended_at, started_at, created_at_source)` por servidor y no en paginas absolutas, para tolerar cambios de orden o partidas que se completan mas tarde. diff --git a/ai/tasks/done/TASK-031-weekly-top-kills-api.md b/ai/tasks/done/TASK-031-weekly-top-kills-api.md new file mode 100644 index 0000000..8e82b0b --- /dev/null +++ b/ai/tasks/done/TASK-031-weekly-top-kills-api.md @@ -0,0 +1,81 @@ +# TASK-031-weekly-top-kills-api + +## Goal +Exponer una primera API histórica útil que devuelva el ranking de jugadores con más kills de la última semana para cada uno de los 2 servidores reales de la comunidad. + +## Context +El objetivo funcional histórico inicial del proyecto es poder consultar métricas agregadas como “jugadores con más kills de la última semana por servidor”. Con la fuente descubierta, el almacenamiento creado y la ingesta histórica ya en marcha, el siguiente valor real es exponer un endpoint backend estable que pueda alimentar futuras vistas propias del proyecto sin depender directamente de la web de la comunidad. + +## Steps +1. Revisar el modelo de dominio histórico y la persistencia ya creada. +2. Definir el endpoint o endpoints mínimos necesarios para top kills semanales por servidor. +3. Diseñar e implementar la consulta agregada usando los datos históricos persistidos. +4. Definir un payload claro que incluya, como mínimo: + - servidor + - rango temporal usado + - jugador + - kills semanales + - posición/ranking + - número de partidas consideradas si resulta viable +5. Asegurar que la definición de “última semana” queda clara y consistente. +6. Documentar el endpoint en el backend. +7. No crear todavía páginas frontend nuevas usando la URL de la comunidad. +8. No incrustar ni duplicar scoreboard externos. +9. Al completar la implementación: + - dejar el repositorio consistente + - hacer commit + - hacer push al remoto si el entorno lo permite + +## Files to Read First +- AGENTS.md +- ai/repo-context.md +- ai/architecture-index.md +- docs/historical-domain-model.md +- backend/README.md +- backend/app/routes.py +- backend/app/payloads.py +- backend/app/historical_storage.py +- backend/app/historical_models.py +- backend/app/historical_ingestion.py + +## Expected Files to Modify +- backend/app/routes.py +- backend/app/payloads.py +- backend/README.md +- uno o más módulos nuevos o existentes de consulta histórica, por ejemplo: + - backend/app/historical_queries.py + - backend/app/historical_payloads.py + +## Constraints +- No basar este endpoint en A2S. +- No consultar directamente la web de la comunidad desde el frontend. +- No crear todavía UI histórica. +- No romper endpoints actuales. +- No introducir complejidad innecesaria. +- No hacer cambios destructivos. +- Mantener el trabajo centrado en la primera API histórica útil y estable. + +## Validation +- Existe un endpoint usable para top kills de la última semana por servidor. +- El endpoint funciona para los 2 servidores reales de la comunidad. +- El payload es claro y reutilizable. +- La documentación backend queda alineada. +- No se han creado páginas frontend nuevas ni dependencias directas del frontend respecto a la URL de la comunidad. +- Los cambios quedan committeados y se hace push al remoto si el entorno lo permite. + +## Change Budget +- Preferir menos de 6 archivos modificados o creados. +- Preferir menos de 240 líneas cambiadas. +## Outcome +- Se expuso `GET /api/historical/weekly-top-kills` en `backend/app/routes.py`. +- La agregacion vive en `backend/app/historical_storage.py` y calcula top kills semanales con ranking independiente por cada servidor real. +- El payload reutiliza `build_weekly_top_kills_payload()` para devolver rango temporal, jugador, kills semanales, posicion y partidas consideradas. +- `backend/README.md` documenta el endpoint y sus parametros `limit` y `server`. + +## Validation Result +- Validado con `python -m compileall app`. +- Validado con una comprobacion local de `resolve_get_payload('/api/historical/weekly-top-kills?limit=3')`, que devolvio `200`, `status: "ok"` y resultados para ambos servidores. +- El endpoint se apoya exclusivamente en historico persistido CRCON y no toca el flujo live A2S ni el frontend. + +## Decision Notes +- El limite se aplica por servidor mediante `ROW_NUMBER() OVER (PARTITION BY historical_servers.slug ...)` para que una request con `limit=10` entregue hasta 10 jugadores por cada servidor, no un corte global mezclado. diff --git a/backend/README.md b/backend/README.md index c4966df..d6edd79 100644 --- a/backend/README.md +++ b/backend/README.md @@ -24,6 +24,9 @@ backend/ |-- __init__.py |-- collector.py |-- main.py + |-- historical_ingestion.py + |-- historical_models.py + |-- historical_storage.py |-- normalizers.py |-- payloads.py |-- routes.py @@ -60,6 +63,8 @@ Variables opcionales: - `HLL_BACKEND_PORT` - `HLL_BACKEND_ALLOWED_ORIGINS` - `HLL_BACKEND_REFRESH_INTERVAL_SECONDS` +- `HLL_HISTORICAL_CRCON_PAGE_SIZE` +- `HLL_HISTORICAL_CRCON_TIMEOUT_SECONDS` El `frontend/index.html` viene preparado para volver a consultar el bloque de servidores cada `120000` ms (`120s`) sin recargar la pagina completa. La landing @@ -109,6 +114,7 @@ normaliza espacios y barras finales para mantener la comparacion con el header - `GET /api/servers/latest` - `GET /api/servers/history?limit=20` - `GET /api/servers/{id}/history?limit=20` +- `GET /api/historical/weekly-top-kills?limit=10&server=comunidad-hispana-01` `GET /api/servers` trata el ultimo snapshot persistido como cache local y lo reutiliza solo si sigue dentro del objetivo de `120` segundos. Si ese snapshot @@ -145,6 +151,9 @@ colector. Si todavia no hay snapshots guardados, responden `status: "ok"` con - `a2s_client.py` encapsula una consulta minima A2S_INFO por UDP para probar servidores reales sin acoplar todavia el backend a una fuente mas compleja. - `config.py` centraliza host, puerto y allowlist minima de origenes locales. +- `historical_ingestion.py` consulta la capa JSON publica de CRCON para bootstrap y refresh incremental. +- `historical_models.py` fija las entidades historicas minimas del dominio. +- `historical_storage.py` prepara la persistencia `historical_*` y las consultas agregadas iniciales. - `main.py` contiene el entrypoint HTTP y la creacion del servidor. - `normalizers.py` transforma registros crudos o respuestas A2S a un modelo comun del colector. @@ -165,6 +174,12 @@ solo libreria estandar de Python. Esta base minima sigue el modelo logico de: - `game_sources` - `servers` - `server_snapshots` +- `historical_servers` +- `historical_maps` +- `historical_matches` +- `historical_players` +- `historical_player_match_stats` +- `historical_ingestion_runs` Por defecto el archivo se crea en: @@ -178,8 +193,9 @@ Variable opcional: - `HLL_BACKEND_A2S_TARGETS` La base logica sigue documentada en -`docs/stats-database-schema-foundation.md`. Esta implementacion no introduce -ORM, migraciones ni una decision de almacenamiento productivo. +`docs/stats-database-schema-foundation.md` para snapshots live y en +`docs/historical-domain-model.md` para el historico CRCON. Esta implementacion +no introduce ORM, migraciones ni una decision de almacenamiento productivo. ## Bootstrap del colector @@ -397,6 +413,48 @@ consulta historica: persistido por el colector. El parametro opcional `limit` acepta valores entre `1` y `100`. +La primera consulta agregada sobre historico CRCON es: + +- `/api/historical/weekly-top-kills` + +Parametros opcionales: + +- `limit` entre `1` y `100` +- `server` con slug historico como `comunidad-hispana-01` + +La ventana temporal es siempre la ultima semana movil respecto al momento de la +request. El payload devuelve servidor, rango temporal, jugador, kills semanales, +posicion y numero de partidas consideradas. + +## Ingesta historica CRCON + +La ingesta historica no usa A2S ni scraping del HTML de `/games`. Consume la +capa JSON publica detectada en los scoreboards CRCON de Comunidad Hispana y +persiste el resultado en las tablas `historical_*`. + +Fuentes configuradas: + +- `https://scoreboard.comunidadhll.es` +- `https://scoreboard.comunidadhll.es:5443` + +Comandos manuales desde `backend/`: + +```powershell +python -m app.historical_ingestion bootstrap +python -m app.historical_ingestion refresh +``` + +Flags utiles: + +- `--server comunidad-hispana-01` para limitar a un servidor +- `--max-pages 2` para validacion local acotada +- `--page-size 25` para ajustar paginacion + +La ejecucion `bootstrap` recorre paginas historicas hasta agotar resultados. +La ejecucion `refresh` usa una ventana de solape sobre la ultima partida +persistida por servidor para releer solo paginas recientes y absorber updates +tardios sin reimportar todo el historico. + ## CORS local minimo El backend responde con `Access-Control-Allow-Origin` solo si la peticion llega diff --git a/backend/app/config.py b/backend/app/config.py index e4f309f..df103da 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -10,6 +10,8 @@ DEFAULT_HOST = "127.0.0.1" DEFAULT_PORT = 8000 DEFAULT_STORAGE_FILENAME = "hll_vietnam_dev.sqlite3" DEFAULT_REFRESH_INTERVAL_SECONDS = 120 +DEFAULT_HISTORICAL_CRCON_PAGE_SIZE = 50 +DEFAULT_HISTORICAL_CRCON_TIMEOUT_SECONDS = 15.0 DEFAULT_ALLOWED_ORIGINS = ( "null", "http://127.0.0.1:5500", @@ -67,6 +69,32 @@ def get_refresh_interval_seconds() -> int: return interval_seconds +def get_historical_crcon_page_size() -> int: + """Return the default page size used for CRCON historical ingestion.""" + configured_value = os.getenv( + "HLL_HISTORICAL_CRCON_PAGE_SIZE", + str(DEFAULT_HISTORICAL_CRCON_PAGE_SIZE), + ) + page_size = int(configured_value) + if page_size <= 0: + raise ValueError("HLL_HISTORICAL_CRCON_PAGE_SIZE must be positive.") + + return page_size + + +def get_historical_crcon_request_timeout_seconds() -> float: + """Return the timeout used for CRCON historical JSON requests.""" + configured_value = os.getenv( + "HLL_HISTORICAL_CRCON_TIMEOUT_SECONDS", + str(DEFAULT_HISTORICAL_CRCON_TIMEOUT_SECONDS), + ) + timeout_seconds = float(configured_value) + if timeout_seconds <= 0: + raise ValueError("HLL_HISTORICAL_CRCON_TIMEOUT_SECONDS must be positive.") + + return timeout_seconds + + def get_a2s_targets_payload() -> str | None: """Return the optional JSON payload that overrides local A2S targets.""" raw_payload = os.getenv(DEFAULT_A2S_TARGETS_ENV_VAR) diff --git a/backend/app/historical_ingestion.py b/backend/app/historical_ingestion.py new file mode 100644 index 0000000..794466e --- /dev/null +++ b/backend/app/historical_ingestion.py @@ -0,0 +1,371 @@ +"""Historical CRCON ingestion bootstrap and incremental refresh.""" + +from __future__ import annotations + +import argparse +import json +from dataclasses import dataclass +from typing import Iterable +from urllib.error import HTTPError, URLError +from urllib.parse import urlencode +from urllib.request import Request, urlopen + +from .config import ( + get_historical_crcon_page_size, + get_historical_crcon_request_timeout_seconds, +) +from .historical_storage import ( + finalize_ingestion_run, + get_refresh_cutoff_for_server, + initialize_historical_storage, + list_historical_servers, + start_ingestion_run, + upsert_historical_match, +) + + +PUBLIC_INFO_ENDPOINT = "/api/get_public_info" +MATCH_LIST_ENDPOINT = "/api/get_scoreboard_maps" +MATCH_DETAIL_ENDPOINT = "/api/get_map_scoreboard" + + +@dataclass(slots=True) +class IngestionStats: + """Mutable counters for one ingestion execution.""" + + pages_processed: int = 0 + matches_seen: int = 0 + matches_inserted: int = 0 + matches_updated: int = 0 + player_rows_inserted: int = 0 + player_rows_updated: int = 0 + + def apply(self, delta: dict[str, int]) -> None: + self.matches_inserted += delta.get("matches_inserted", 0) + self.matches_updated += delta.get("matches_updated", 0) + self.player_rows_inserted += delta.get("player_rows_inserted", 0) + self.player_rows_updated += delta.get("player_rows_updated", 0) + + +def run_bootstrap( + *, + server_slug: str | None = None, + max_pages: int | None = None, + page_size: int | None = None, +) -> dict[str, object]: + """Run a first full historical import against one or all configured servers.""" + return _run_ingestion( + mode="bootstrap", + server_slug=server_slug, + max_pages=max_pages, + page_size=page_size, + incremental=False, + ) + + +def run_incremental_refresh( + *, + server_slug: str | None = None, + max_pages: int | None = None, + page_size: int | None = None, +) -> dict[str, object]: + """Refresh recent historical pages without replaying the whole archive.""" + return _run_ingestion( + mode="incremental", + server_slug=server_slug, + max_pages=max_pages, + page_size=page_size, + incremental=True, + ) + + +def _run_ingestion( + *, + mode: str, + server_slug: str | None, + max_pages: int | None, + page_size: int | None, + incremental: bool, +) -> dict[str, object]: + initialize_historical_storage() + stats = IngestionStats() + selected_servers = _select_servers(server_slug) + processed_servers: list[dict[str, object]] = [] + runs: list[int] = [] + + try: + for server in selected_servers: + run_id = start_ingestion_run(mode=mode, target_server_slug=str(server["slug"])) + runs.append(run_id) + cutoff = ( + get_refresh_cutoff_for_server(str(server["slug"])) + if incremental + else None + ) + server_stats = _ingest_server( + server=server, + stats=stats, + max_pages=max_pages, + page_size=page_size, + cutoff=cutoff, + ) + processed_servers.append(server_stats) + finalize_ingestion_run( + run_id, + status="success", + pages_processed=server_stats["pages_processed"], + matches_seen=server_stats["matches_seen"], + matches_inserted=server_stats["matches_inserted"], + matches_updated=server_stats["matches_updated"], + player_rows_inserted=server_stats["player_rows_inserted"], + player_rows_updated=server_stats["player_rows_updated"], + notes=f"public_name={server_stats['public_name']}", + ) + except Exception as exc: + for run_id in runs: + finalize_ingestion_run( + run_id, + status="failed", + pages_processed=stats.pages_processed, + matches_seen=stats.matches_seen, + matches_inserted=stats.matches_inserted, + matches_updated=stats.matches_updated, + player_rows_inserted=stats.player_rows_inserted, + player_rows_updated=stats.player_rows_updated, + notes=str(exc), + ) + raise + + return { + "status": "ok", + "mode": mode, + "page_size": page_size or get_historical_crcon_page_size(), + "servers": processed_servers, + "totals": { + "pages_processed": stats.pages_processed, + "matches_seen": stats.matches_seen, + "matches_inserted": stats.matches_inserted, + "matches_updated": stats.matches_updated, + "player_rows_inserted": stats.player_rows_inserted, + "player_rows_updated": stats.player_rows_updated, + }, + } + + +def _ingest_server( + *, + server: dict[str, object], + stats: IngestionStats, + max_pages: int | None, + page_size: int | None, + cutoff: str | None, +) -> dict[str, object]: + resolved_page_size = page_size or get_historical_crcon_page_size() + page_limit = max_pages or 1000000 + local_stats = IngestionStats() + public_info = _fetch_public_info(str(server["scoreboard_base_url"])) + + for page_number in range(1, page_limit + 1): + payload = _fetch_match_page( + str(server["scoreboard_base_url"]), + page=page_number, + limit=resolved_page_size, + ) + page_matches = _coerce_match_list(payload.get("maps")) + if not page_matches: + break + + local_stats.pages_processed += 1 + stats.pages_processed += 1 + stop_after_page = False + + for match_summary in page_matches: + local_stats.matches_seen += 1 + stats.matches_seen += 1 + + reference_timestamp = _pick_match_timestamp(match_summary) + if cutoff and reference_timestamp and reference_timestamp < cutoff: + stop_after_page = True + continue + + detail_payload = _fetch_match_detail( + str(server["scoreboard_base_url"]), + match_id=str(match_summary["id"]), + ) + delta = upsert_historical_match( + server_slug=str(server["slug"]), + match_payload=detail_payload, + ) + local_stats.apply(delta) + stats.apply(delta) + + if stop_after_page: + break + + return { + "server_slug": server["slug"], + "public_name": _extract_public_name(public_info), + "server_number": public_info.get("server_number") or server.get("server_number"), + "pages_processed": local_stats.pages_processed, + "matches_seen": local_stats.matches_seen, + "matches_inserted": local_stats.matches_inserted, + "matches_updated": local_stats.matches_updated, + "player_rows_inserted": local_stats.player_rows_inserted, + "player_rows_updated": local_stats.player_rows_updated, + "cutoff": cutoff, + } + + +def _select_servers(server_slug: str | None) -> list[dict[str, object]]: + servers = list_historical_servers() + if server_slug is None: + return servers + + normalized = server_slug.strip() + selected = [server for server in servers if server["slug"] == normalized] + if not selected: + raise ValueError(f"Unknown historical server slug: {server_slug}") + return selected + + +def _fetch_public_info(base_url: str) -> dict[str, object]: + payload = _unwrap_result(_fetch_json(base_url, PUBLIC_INFO_ENDPOINT)) + if not isinstance(payload, dict): + raise ValueError(f"Unexpected public info payload for {base_url}") + return payload + + +def _fetch_match_page(base_url: str, *, page: int, limit: int) -> dict[str, object]: + payload = _unwrap_result(_fetch_json( + base_url, + MATCH_LIST_ENDPOINT, + {"page": page, "limit": limit}, + )) + if not isinstance(payload, dict): + raise ValueError(f"Unexpected match list payload for {base_url} page={page}") + return payload + + +def _fetch_match_detail(base_url: str, *, match_id: str) -> dict[str, object]: + payload = _unwrap_result(_fetch_json( + base_url, + MATCH_DETAIL_ENDPOINT, + {"map_id": match_id}, + )) + if not isinstance(payload, dict): + raise ValueError(f"Unexpected match detail payload for {base_url} match={match_id}") + return payload + + +def _fetch_json( + base_url: str, + endpoint: str, + query: dict[str, object] | None = None, +) -> object: + url = f"{base_url}{endpoint}" + if query: + url = f"{url}?{urlencode(query)}" + + request = Request( + url, + headers={ + "Accept": "application/json", + "User-Agent": "HLL-Vietnam-Historical-Ingestion/0.1", + }, + ) + try: + with urlopen( + request, + timeout=get_historical_crcon_request_timeout_seconds(), + ) as response: + return json.loads(response.read().decode("utf-8")) + except HTTPError as exc: + raise RuntimeError(f"Historical CRCON request failed: {url} ({exc.code})") from exc + except URLError as exc: + raise RuntimeError(f"Historical CRCON request failed: {url} ({exc.reason})") from exc + + +def _coerce_match_list(payload: object) -> list[dict[str, object]]: + if not isinstance(payload, list): + return [] + return [item for item in payload if isinstance(item, dict)] + + +def _unwrap_result(payload: object) -> object: + if not isinstance(payload, dict): + return payload + if "result" not in payload: + return payload + return payload.get("result") + + +def _pick_match_timestamp(match_payload: dict[str, object]) -> str | None: + for key in ("end", "start", "creation_time"): + value = match_payload.get(key) + if isinstance(value, str) and value.strip(): + return value.strip() + return None + + +def _extract_public_name(public_info: dict[str, object]) -> str | None: + name_value = public_info.get("name") + if isinstance(name_value, str): + return name_value + if isinstance(name_value, dict): + raw_name = name_value.get("name") + return raw_name.strip() if isinstance(raw_name, str) and raw_name.strip() else None + return None + + +def build_arg_parser() -> argparse.ArgumentParser: + """Create the CLI parser for manual historical ingestion runs.""" + parser = argparse.ArgumentParser( + description="Historical CRCON ingestion for HLL Vietnam.", + ) + parser.add_argument( + "mode", + choices=("bootstrap", "refresh"), + help="bootstrap imports the archive, refresh only recent pages", + ) + parser.add_argument( + "--server", + dest="server_slug", + help="optional historical server slug", + ) + parser.add_argument( + "--max-pages", + type=int, + help="optional page cap for local validation", + ) + parser.add_argument( + "--page-size", + type=int, + help="override CRCON page size", + ) + return parser + + +def main(argv: Iterable[str] | None = None) -> int: + """Run the historical ingestion CLI.""" + parser = build_arg_parser() + args = parser.parse_args(list(argv) if argv is not None else None) + + if args.mode == "bootstrap": + result = run_bootstrap( + server_slug=args.server_slug, + max_pages=args.max_pages, + page_size=args.page_size, + ) + else: + result = run_incremental_refresh( + server_slug=args.server_slug, + max_pages=args.max_pages, + page_size=args.page_size, + ) + + print(json.dumps(result, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/app/historical_models.py b/backend/app/historical_models.py new file mode 100644 index 0000000..cbfb66c --- /dev/null +++ b/backend/app/historical_models.py @@ -0,0 +1,93 @@ +"""Historical domain models for persisted CRCON scoreboard data.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime + + +@dataclass(frozen=True, slots=True) +class HistoricalServerDefinition: + """Stable identity for one historical CRCON source.""" + + slug: str + display_name: str + scoreboard_base_url: str + server_number: int | None + source_kind: str = "crcon-scoreboard-json" + + +@dataclass(frozen=True, slots=True) +class HistoricalMapRecord: + """Normalized map metadata reused across historical matches.""" + + external_map_id: str | None + map_name: str | None + pretty_name: str | None + game_mode: str | None + image_name: str | None + + +@dataclass(frozen=True, slots=True) +class HistoricalMatchRecord: + """Normalized match identity and summary.""" + + external_match_id: str + server_slug: str + created_at: datetime | None + started_at: datetime | None + ended_at: datetime | None + map_name: str | None + map_pretty_name: str | None + map_external_id: str | None + game_mode: str | None + image_name: str | None + allied_score: int | None + axis_score: int | None + + +@dataclass(frozen=True, slots=True) +class HistoricalPlayerIdentity: + """Stable player identity across historical match stats.""" + + stable_player_key: str + display_name: str + steam_id: str | None + source_player_id: str | None + + +@dataclass(frozen=True, slots=True) +class HistoricalPlayerMatchStats: + """Metrics persisted per player and match.""" + + stable_player_key: str + match_player_ref: str | None + team_side: str | None + level: int | None + kills: int | None + deaths: int | None + teamkills: int | None + time_seconds: int | None + kills_per_minute: float | None + deaths_per_minute: float | None + kill_death_ratio: float | None + combat: int | None + offense: int | None + defense: int | None + support: int | None + + +@dataclass(frozen=True, slots=True) +class HistoricalIngestionRunSummary: + """Outcome metadata recorded for one ingestion execution.""" + + mode: str + started_at: datetime + completed_at: datetime | None + status: str + pages_processed: int + matches_seen: int + matches_inserted: int + matches_updated: int + player_rows_inserted: int + player_rows_updated: int diff --git a/backend/app/historical_storage.py b/backend/app/historical_storage.py new file mode 100644 index 0000000..4ae1d1b --- /dev/null +++ b/backend/app/historical_storage.py @@ -0,0 +1,1032 @@ +"""SQLite persistence for historical CRCON scoreboard data.""" + +from __future__ import annotations + +import sqlite3 +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Mapping + +from .config import get_storage_path +from .historical_models import HistoricalServerDefinition + + +DEFAULT_HISTORICAL_SERVERS = ( + HistoricalServerDefinition( + slug="comunidad-hispana-01", + display_name="Comunidad Hispana #01", + scoreboard_base_url="https://scoreboard.comunidadhll.es", + server_number=1, + ), + HistoricalServerDefinition( + slug="comunidad-hispana-02", + display_name="Comunidad Hispana #02", + scoreboard_base_url="https://scoreboard.comunidadhll.es:5443", + server_number=2, + ), +) +DEFAULT_WEEKLY_WINDOW_DAYS = 7 +DEFAULT_REFRESH_OVERLAP_HOURS = 12 + + +def initialize_historical_storage(*, db_path: Path | None = None) -> Path: + """Create or migrate the local SQLite schema for historical data.""" + resolved_path = db_path or get_storage_path() + resolved_path.parent.mkdir(parents=True, exist_ok=True) + + with _connect(resolved_path) as connection: + legacy_historical_schema = _has_legacy_historical_schema(connection) + if legacy_historical_schema: + _rename_legacy_historical_tables(connection) + connection.executescript( + """ + CREATE TABLE IF NOT EXISTS historical_servers ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + slug TEXT NOT NULL UNIQUE, + display_name TEXT NOT NULL, + scoreboard_base_url TEXT NOT NULL UNIQUE, + server_number INTEGER, + source_kind TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP + ); + + CREATE TABLE IF NOT EXISTS historical_maps ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + external_map_id TEXT UNIQUE, + map_name TEXT, + pretty_name TEXT, + game_mode TEXT, + image_name TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP + ); + + CREATE TABLE IF NOT EXISTS historical_matches ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + historical_server_id INTEGER NOT NULL, + external_match_id TEXT NOT NULL, + historical_map_id INTEGER, + created_at_source TEXT, + started_at TEXT, + ended_at TEXT, + map_name TEXT, + map_pretty_name TEXT, + game_mode TEXT, + image_name TEXT, + allied_score INTEGER, + axis_score INTEGER, + last_seen_at TEXT NOT NULL, + raw_payload_ref TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + UNIQUE(historical_server_id, external_match_id), + FOREIGN KEY (historical_server_id) REFERENCES historical_servers(id), + FOREIGN KEY (historical_map_id) REFERENCES historical_maps(id) + ); + + CREATE TABLE IF NOT EXISTS historical_players ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + stable_player_key TEXT NOT NULL UNIQUE, + display_name TEXT NOT NULL, + steam_id TEXT, + source_player_id TEXT, + first_seen_at TEXT NOT NULL, + last_seen_at TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP + ); + + CREATE TABLE IF NOT EXISTS historical_player_match_stats ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + historical_match_id INTEGER NOT NULL, + historical_player_id INTEGER NOT NULL, + match_player_ref TEXT, + team_side TEXT, + level INTEGER, + kills INTEGER, + deaths INTEGER, + teamkills INTEGER, + time_seconds INTEGER, + kills_per_minute REAL, + deaths_per_minute REAL, + kill_death_ratio REAL, + combat INTEGER, + offense INTEGER, + defense INTEGER, + support INTEGER, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + UNIQUE(historical_match_id, historical_player_id), + FOREIGN KEY (historical_match_id) REFERENCES historical_matches(id), + FOREIGN KEY (historical_player_id) REFERENCES historical_players(id) + ); + + CREATE TABLE IF NOT EXISTS historical_ingestion_runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + mode TEXT NOT NULL, + status TEXT NOT NULL, + started_at TEXT NOT NULL, + completed_at TEXT, + target_server_slug TEXT, + pages_processed INTEGER NOT NULL DEFAULT 0, + matches_seen INTEGER NOT NULL DEFAULT 0, + matches_inserted INTEGER NOT NULL DEFAULT 0, + matches_updated INTEGER NOT NULL DEFAULT 0, + player_rows_inserted INTEGER NOT NULL DEFAULT 0, + player_rows_updated INTEGER NOT NULL DEFAULT 0, + notes TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP + ); + + CREATE INDEX IF NOT EXISTS idx_historical_matches_server_end + ON historical_matches(historical_server_id, ended_at DESC, started_at DESC); + + CREATE INDEX IF NOT EXISTS idx_historical_player_stats_match + ON historical_player_match_stats(historical_match_id); + + CREATE INDEX IF NOT EXISTS idx_historical_players_steam + ON historical_players(steam_id); + """ + ) + _seed_default_historical_servers(connection) + if legacy_historical_schema: + _migrate_legacy_historical_data(connection) + _normalize_historical_player_identities(connection) + + return resolved_path + + +def list_historical_servers(*, db_path: Path | None = None) -> list[dict[str, object]]: + """Return configured CRCON historical sources.""" + resolved_path = initialize_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + rows = connection.execute( + """ + SELECT slug, display_name, scoreboard_base_url, server_number, source_kind + FROM historical_servers + ORDER BY slug ASC + """ + ).fetchall() + return [dict(row) for row in rows] + + +def start_ingestion_run( + *, + mode: str, + target_server_slug: str | None = None, + db_path: Path | None = None, +) -> int: + """Create a row tracking one ingestion execution.""" + resolved_path = initialize_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + cursor = connection.execute( + """ + INSERT INTO historical_ingestion_runs ( + mode, + status, + started_at, + target_server_slug + ) VALUES (?, 'running', ?, ?) + """, + (mode, _utc_now_iso(), target_server_slug), + ) + return int(cursor.lastrowid) + + +def finalize_ingestion_run( + run_id: int, + *, + status: str, + pages_processed: int, + matches_seen: int, + matches_inserted: int, + matches_updated: int, + player_rows_inserted: int, + player_rows_updated: int, + notes: str | None = None, + db_path: Path | None = None, +) -> None: + """Update an ingestion run row with outcome metrics.""" + resolved_path = initialize_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + connection.execute( + """ + UPDATE historical_ingestion_runs + SET status = ?, + completed_at = ?, + pages_processed = ?, + matches_seen = ?, + matches_inserted = ?, + matches_updated = ?, + player_rows_inserted = ?, + player_rows_updated = ?, + notes = ? + WHERE id = ? + """, + ( + status, + _utc_now_iso(), + pages_processed, + matches_seen, + matches_inserted, + matches_updated, + player_rows_inserted, + player_rows_updated, + notes, + run_id, + ), + ) + + +def upsert_historical_match( + *, + server_slug: str, + match_payload: Mapping[str, object], + db_path: Path | None = None, +) -> dict[str, int]: + """Persist one historical match and its player stats idempotently.""" + resolved_path = initialize_historical_storage(db_path=db_path) + match_external_id = _stringify(match_payload.get("id")) + if not match_external_id: + raise ValueError("Historical match payload is missing a stable id.") + + with _connect(resolved_path) as connection: + server_row = _resolve_historical_server(connection, server_slug) + map_id = _upsert_historical_map(connection, match_payload) + match_row = connection.execute( + """ + SELECT id + FROM historical_matches + WHERE historical_server_id = ? AND external_match_id = ? + """, + (server_row["id"], match_external_id), + ).fetchone() + match_exists = match_row is not None + + connection.execute( + """ + INSERT INTO historical_matches ( + historical_server_id, + external_match_id, + historical_map_id, + created_at_source, + started_at, + ended_at, + map_name, + map_pretty_name, + game_mode, + image_name, + allied_score, + axis_score, + last_seen_at, + raw_payload_ref + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(historical_server_id, external_match_id) DO UPDATE SET + historical_map_id = excluded.historical_map_id, + created_at_source = excluded.created_at_source, + started_at = excluded.started_at, + ended_at = excluded.ended_at, + map_name = excluded.map_name, + map_pretty_name = excluded.map_pretty_name, + game_mode = excluded.game_mode, + image_name = excluded.image_name, + allied_score = excluded.allied_score, + axis_score = excluded.axis_score, + last_seen_at = excluded.last_seen_at, + raw_payload_ref = excluded.raw_payload_ref, + updated_at = CURRENT_TIMESTAMP + """, + ( + server_row["id"], + match_external_id, + map_id, + _normalize_timestamp(match_payload.get("creation_time")), + _normalize_timestamp(match_payload.get("start")), + _normalize_timestamp(match_payload.get("end")), + _extract_map_name(match_payload), + _extract_map_pretty_name(match_payload), + _extract_map_game_mode(match_payload), + _extract_map_image_name(match_payload), + _coerce_int(_get_nested(match_payload, "result", "allied")), + _coerce_int(_get_nested(match_payload, "result", "axis")), + _utc_now_iso(), + f"{server_row['scoreboard_base_url']}/games/{match_external_id}", + ), + ) + match_id_row = connection.execute( + """ + SELECT id + FROM historical_matches + WHERE historical_server_id = ? AND external_match_id = ? + """, + (server_row["id"], match_external_id), + ).fetchone() + if match_id_row is None: + raise RuntimeError("Failed to persist historical match.") + + player_rows_inserted = 0 + player_rows_updated = 0 + for player_payload in _coerce_list(match_payload.get("player_stats")): + player_id = _upsert_historical_player(connection, player_payload) + stat_exists = connection.execute( + """ + SELECT id + FROM historical_player_match_stats + WHERE historical_match_id = ? AND historical_player_id = ? + """, + (match_id_row["id"], player_id), + ).fetchone() + connection.execute( + """ + INSERT INTO historical_player_match_stats ( + historical_match_id, + historical_player_id, + match_player_ref, + team_side, + level, + kills, + deaths, + teamkills, + time_seconds, + kills_per_minute, + deaths_per_minute, + kill_death_ratio, + combat, + offense, + defense, + support + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(historical_match_id, historical_player_id) DO UPDATE SET + match_player_ref = excluded.match_player_ref, + team_side = excluded.team_side, + level = excluded.level, + kills = excluded.kills, + deaths = excluded.deaths, + teamkills = excluded.teamkills, + time_seconds = excluded.time_seconds, + kills_per_minute = excluded.kills_per_minute, + deaths_per_minute = excluded.deaths_per_minute, + kill_death_ratio = excluded.kill_death_ratio, + combat = excluded.combat, + offense = excluded.offense, + defense = excluded.defense, + support = excluded.support, + updated_at = CURRENT_TIMESTAMP + """, + ( + match_id_row["id"], + player_id, + _stringify(player_payload.get("id")), + _stringify(_get_nested(player_payload, "team", "side")), + _coerce_int(player_payload.get("level")), + _coerce_int(player_payload.get("kills")), + _coerce_int(player_payload.get("deaths")), + _coerce_int(player_payload.get("teamkills")), + _coerce_int(player_payload.get("time_seconds")), + _coerce_float(player_payload.get("kills_per_minute")), + _coerce_float(player_payload.get("deaths_per_minute")), + _coerce_float(player_payload.get("kill_death_ratio")), + _coerce_int(player_payload.get("combat")), + _coerce_int(player_payload.get("offense")), + _coerce_int(player_payload.get("defense")), + _coerce_int(player_payload.get("support")), + ), + ) + if stat_exists is None: + player_rows_inserted += 1 + else: + player_rows_updated += 1 + + return { + "matches_inserted": 0 if match_exists else 1, + "matches_updated": 1 if match_exists else 0, + "player_rows_inserted": player_rows_inserted, + "player_rows_updated": player_rows_updated, + } + + +def get_refresh_cutoff_for_server( + server_slug: str, + *, + overlap_hours: int = DEFAULT_REFRESH_OVERLAP_HOURS, + db_path: Path | None = None, +) -> str | None: + """Return the timestamp used to stop incremental scans once older pages appear.""" + resolved_path = initialize_historical_storage(db_path=db_path) + with _connect(resolved_path) as connection: + server_row = _resolve_historical_server(connection, server_slug) + row = connection.execute( + """ + SELECT COALESCE(MAX(ended_at), MAX(started_at), MAX(created_at_source)) AS latest_seen_at + FROM historical_matches + WHERE historical_server_id = ? + """, + (server_row["id"],), + ).fetchone() + latest_seen_at = _stringify(row["latest_seen_at"] if row else None) + if not latest_seen_at: + return None + + cutoff = _parse_timestamp(latest_seen_at) - timedelta(hours=overlap_hours) + return cutoff.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") + + +def list_recent_historical_matches( + *, + server_slug: str | None = None, + limit: int = 20, + db_path: Path | None = None, +) -> list[dict[str, object]]: + """Return recent persisted matches for validation and later API work.""" + resolved_path = initialize_historical_storage(db_path=db_path) + where_clause = "" + params: list[object] = [] + if server_slug: + where_clause = "WHERE historical_servers.slug = ?" + params.append(server_slug) + params.append(limit) + + with _connect(resolved_path) as connection: + rows = connection.execute( + f""" + SELECT + historical_servers.slug AS server_slug, + historical_servers.display_name AS server_name, + historical_matches.external_match_id, + historical_matches.started_at, + historical_matches.ended_at, + historical_matches.map_pretty_name, + historical_matches.map_name, + historical_matches.allied_score, + historical_matches.axis_score, + COUNT(historical_player_match_stats.id) AS player_count + FROM historical_matches + INNER JOIN historical_servers + ON historical_servers.id = historical_matches.historical_server_id + LEFT JOIN historical_player_match_stats + ON historical_player_match_stats.historical_match_id = historical_matches.id + {where_clause} + GROUP BY historical_matches.id + ORDER BY COALESCE(historical_matches.ended_at, historical_matches.started_at) DESC + LIMIT ? + """, + params, + ).fetchall() + return [dict(row) for row in rows] + + +def list_weekly_top_kills( + *, + limit: int = 10, + server_id: str | None = None, + db_path: Path | None = None, +) -> dict[str, object]: + """Return ranked weekly kill totals from persisted historical match stats.""" + resolved_path = initialize_historical_storage(db_path=db_path) + window_end = datetime.now(timezone.utc) + window_start = window_end - timedelta(days=DEFAULT_WEEKLY_WINDOW_DAYS) + + where_clauses = [ + "COALESCE(historical_matches.ended_at, historical_matches.started_at, historical_matches.created_at_source) >= ?", + "COALESCE(historical_matches.ended_at, historical_matches.started_at, historical_matches.created_at_source) <= ?", + ] + params: list[object] = [ + window_start.isoformat().replace("+00:00", "Z"), + window_end.isoformat().replace("+00:00", "Z"), + ] + if server_id: + normalized_server_id = server_id.strip() + where_clauses.append( + "(historical_servers.slug = ? OR CAST(historical_servers.server_number AS TEXT) = ?)" + ) + params.extend([normalized_server_id, normalized_server_id]) + + with _connect(resolved_path) as connection: + rows = connection.execute( + f""" + WITH ranked_players AS ( + SELECT + historical_servers.slug AS server_slug, + historical_servers.display_name AS server_name, + historical_players.stable_player_key, + historical_players.display_name AS player_name, + historical_players.steam_id, + COUNT(DISTINCT historical_matches.id) AS matches_count, + COALESCE(SUM(historical_player_match_stats.kills), 0) AS kills, + ROW_NUMBER() OVER ( + PARTITION BY historical_servers.slug + ORDER BY + COALESCE(SUM(historical_player_match_stats.kills), 0) DESC, + COUNT(DISTINCT historical_matches.id) ASC, + historical_players.display_name ASC + ) AS ranking_position + FROM historical_player_match_stats + INNER JOIN historical_matches + ON historical_matches.id = historical_player_match_stats.historical_match_id + INNER JOIN historical_servers + ON historical_servers.id = historical_matches.historical_server_id + INNER JOIN historical_players + ON historical_players.id = historical_player_match_stats.historical_player_id + WHERE {" AND ".join(where_clauses)} + GROUP BY historical_servers.slug, historical_players.id + ) + SELECT * + FROM ranked_players + WHERE ranking_position <= ? + ORDER BY server_slug ASC, ranking_position ASC + """, + [*params, limit], + ).fetchall() + + items: list[dict[str, object]] = [] + for row in rows: + items.append( + { + "server": { + "slug": row["server_slug"], + "name": row["server_name"], + }, + "time_range": { + "start": window_start.isoformat().replace("+00:00", "Z"), + "end": window_end.isoformat().replace("+00:00", "Z"), + "window_days": DEFAULT_WEEKLY_WINDOW_DAYS, + }, + "player": { + "stable_player_key": row["stable_player_key"], + "name": row["player_name"], + "steam_id": row["steam_id"], + }, + "ranking_position": int(row["ranking_position"]), + "weekly_kills": int(row["kills"] or 0), + "matches_considered": int(row["matches_count"] or 0), + } + ) + + return { + "window_start": window_start.isoformat().replace("+00:00", "Z"), + "window_end": window_end.isoformat().replace("+00:00", "Z"), + "items": items, + } + + +def _connect(db_path: Path) -> sqlite3.Connection: + connection = sqlite3.connect(db_path) + connection.row_factory = sqlite3.Row + return connection + + +def _has_legacy_historical_schema(connection: sqlite3.Connection) -> bool: + columns = { + str(row["name"]) + for row in connection.execute("PRAGMA table_info(historical_matches)").fetchall() + } + return bool(columns) and "historical_server_id" not in columns + + +def _rename_legacy_historical_tables(connection: sqlite3.Connection) -> None: + rename_plan = ( + ("historical_player_match_stats", "historical_player_match_stats_legacy"), + ("historical_players", "historical_players_legacy"), + ("historical_matches", "historical_matches_legacy"), + ) + for current_name, legacy_name in rename_plan: + table_exists = connection.execute( + """ + SELECT 1 + FROM sqlite_master + WHERE type = 'table' AND name = ? + """, + (current_name,), + ).fetchone() + if not table_exists: + continue + + legacy_exists = connection.execute( + """ + SELECT 1 + FROM sqlite_master + WHERE type = 'table' AND name = ? + """, + (legacy_name,), + ).fetchone() + if legacy_exists: + continue + + connection.execute(f"ALTER TABLE {current_name} RENAME TO {legacy_name}") + + +def _migrate_legacy_historical_data(connection: sqlite3.Connection) -> None: + matches_table = connection.execute( + """ + SELECT 1 + FROM sqlite_master + WHERE type = 'table' AND name = 'historical_matches_legacy' + """ + ).fetchone() + if not matches_table: + return + + player_map: dict[int, int] = {} + for row in connection.execute( + """ + SELECT id, source_player_ref, canonical_name, last_seen_name + FROM historical_players_legacy + ORDER BY id ASC + """ + ).fetchall(): + stable_player_key = _stringify(row["source_player_ref"]) or f"legacy-player:{row['id']}" + display_name = _stringify(row["last_seen_name"]) or _stringify(row["canonical_name"]) or "Unknown player" + now = _utc_now_iso() + connection.execute( + """ + INSERT INTO historical_players ( + stable_player_key, + display_name, + steam_id, + source_player_id, + first_seen_at, + last_seen_at + ) VALUES (?, ?, NULL, NULL, ?, ?) + ON CONFLICT(stable_player_key) DO UPDATE SET + display_name = excluded.display_name, + last_seen_at = excluded.last_seen_at, + updated_at = CURRENT_TIMESTAMP + """, + (stable_player_key, display_name, now, now), + ) + new_row = connection.execute( + "SELECT id FROM historical_players WHERE stable_player_key = ?", + (stable_player_key,), + ).fetchone() + if new_row is not None: + player_map[int(row["id"])] = int(new_row["id"]) + + match_map: dict[int, int] = {} + for row in connection.execute( + """ + SELECT * + FROM historical_matches_legacy + ORDER BY id ASC + """ + ).fetchall(): + server_slug = _stringify(row["external_server_id"]) or "comunidad-hispana-01" + server_row = _resolve_historical_server(connection, server_slug) + connection.execute( + """ + INSERT INTO historical_matches ( + historical_server_id, + external_match_id, + historical_map_id, + created_at_source, + started_at, + ended_at, + map_name, + map_pretty_name, + game_mode, + image_name, + allied_score, + axis_score, + last_seen_at, + raw_payload_ref + ) VALUES (?, ?, NULL, ?, ?, ?, ?, ?, ?, NULL, NULL, NULL, ?, ?) + ON CONFLICT(historical_server_id, external_match_id) DO UPDATE SET + started_at = excluded.started_at, + ended_at = excluded.ended_at, + map_name = excluded.map_name, + map_pretty_name = excluded.map_pretty_name, + game_mode = excluded.game_mode, + last_seen_at = excluded.last_seen_at, + raw_payload_ref = excluded.raw_payload_ref, + updated_at = CURRENT_TIMESTAMP + """, + ( + server_row["id"], + _stringify(row["source_match_ref"]) or f"legacy-match:{row['id']}", + _stringify(row["created_at"]), + _stringify(row["started_at"]), + _stringify(row["ended_at"]), + _stringify(row["map_name"]), + _stringify(row["map_name"]), + _stringify(row["mode_name"]), + _utc_now_iso(), + _stringify(row["source_url"]), + ), + ) + new_row = connection.execute( + """ + SELECT id + FROM historical_matches + WHERE historical_server_id = ? AND external_match_id = ? + """, + ( + server_row["id"], + _stringify(row["source_match_ref"]) or f"legacy-match:{row['id']}", + ), + ).fetchone() + if new_row is not None: + match_map[int(row["id"])] = int(new_row["id"]) + + for row in connection.execute( + """ + SELECT * + FROM historical_player_match_stats_legacy + ORDER BY id ASC + """ + ).fetchall(): + new_match_id = match_map.get(int(row["match_id"])) + new_player_id = player_map.get(int(row["player_id"])) + if new_match_id is None or new_player_id is None: + continue + + connection.execute( + """ + INSERT INTO historical_player_match_stats ( + historical_match_id, + historical_player_id, + match_player_ref, + team_side, + level, + kills, + deaths, + teamkills, + time_seconds, + kills_per_minute, + deaths_per_minute, + kill_death_ratio, + combat, + offense, + defense, + support + ) VALUES (?, ?, NULL, NULL, NULL, ?, ?, NULL, ?, NULL, NULL, NULL, NULL, NULL, NULL, NULL) + ON CONFLICT(historical_match_id, historical_player_id) DO UPDATE SET + kills = excluded.kills, + deaths = excluded.deaths, + time_seconds = excluded.time_seconds, + updated_at = CURRENT_TIMESTAMP + """, + ( + new_match_id, + new_player_id, + _coerce_int(row["kills"]), + _coerce_int(row["deaths"]), + _coerce_int(row["time_seconds"]), + ), + ) + + +def _normalize_historical_player_identities(connection: sqlite3.Connection) -> None: + rows = connection.execute( + """ + SELECT id, stable_player_key, steam_id + FROM historical_players + ORDER BY id ASC + """ + ).fetchall() + for row in rows: + stable_player_key = _stringify(row["stable_player_key"]) + if not stable_player_key or ":" in stable_player_key: + continue + if stable_player_key.isdigit() and len(stable_player_key) >= 16: + normalized_key = f"steam:{stable_player_key}" + connection.execute( + """ + UPDATE historical_players + SET stable_player_key = ?, + steam_id = COALESCE(steam_id, ?), + updated_at = CURRENT_TIMESTAMP + WHERE id = ? + """, + (normalized_key, stable_player_key, row["id"]), + ) + + +def _seed_default_historical_servers(connection: sqlite3.Connection) -> None: + for server in DEFAULT_HISTORICAL_SERVERS: + connection.execute( + """ + INSERT INTO historical_servers ( + slug, + display_name, + scoreboard_base_url, + server_number, + source_kind + ) VALUES (?, ?, ?, ?, ?) + ON CONFLICT(slug) DO UPDATE SET + display_name = excluded.display_name, + scoreboard_base_url = excluded.scoreboard_base_url, + server_number = excluded.server_number, + source_kind = excluded.source_kind, + updated_at = CURRENT_TIMESTAMP + """, + ( + server.slug, + server.display_name, + server.scoreboard_base_url, + server.server_number, + server.source_kind, + ), + ) + + +def _resolve_historical_server( + connection: sqlite3.Connection, + server_slug: str, +) -> sqlite3.Row: + row = connection.execute( + """ + SELECT id, slug, scoreboard_base_url + FROM historical_servers + WHERE slug = ? + """, + (server_slug,), + ).fetchone() + if row is None: + raise ValueError(f"Unknown historical server slug: {server_slug}") + return row + + +def _upsert_historical_map( + connection: sqlite3.Connection, + match_payload: Mapping[str, object], +) -> int | None: + external_map_id = _stringify(_get_nested(match_payload, "map", "id")) + if not external_map_id: + return None + + connection.execute( + """ + INSERT INTO historical_maps ( + external_map_id, + map_name, + pretty_name, + game_mode, + image_name + ) VALUES (?, ?, ?, ?, ?) + ON CONFLICT(external_map_id) DO UPDATE SET + map_name = excluded.map_name, + pretty_name = excluded.pretty_name, + game_mode = excluded.game_mode, + image_name = excluded.image_name, + updated_at = CURRENT_TIMESTAMP + """, + ( + external_map_id, + _extract_map_name(match_payload), + _extract_map_pretty_name(match_payload), + _extract_map_game_mode(match_payload), + _extract_map_image_name(match_payload), + ), + ) + row = connection.execute( + "SELECT id FROM historical_maps WHERE external_map_id = ?", + (external_map_id,), + ).fetchone() + return int(row["id"]) if row is not None else None + + +def _upsert_historical_player( + connection: sqlite3.Connection, + player_payload: Mapping[str, object], +) -> int: + stable_player_key = _build_stable_player_key(player_payload) + display_name = _stringify(player_payload.get("player")) or "Unknown player" + steam_id = _stringify(_get_nested(player_payload, "steaminfo", "profile", "steamid")) + if not steam_id: + steam_id = _stringify(_get_nested(player_payload, "steaminfo", "id")) + source_player_id = _stringify(player_payload.get("player_id")) + seen_at = _utc_now_iso() + + connection.execute( + """ + INSERT INTO historical_players ( + stable_player_key, + display_name, + steam_id, + source_player_id, + first_seen_at, + last_seen_at + ) VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(stable_player_key) DO UPDATE SET + display_name = excluded.display_name, + steam_id = COALESCE(excluded.steam_id, historical_players.steam_id), + source_player_id = COALESCE(excluded.source_player_id, historical_players.source_player_id), + last_seen_at = excluded.last_seen_at, + updated_at = CURRENT_TIMESTAMP + """, + ( + stable_player_key, + display_name, + steam_id, + source_player_id, + seen_at, + seen_at, + ), + ) + row = connection.execute( + "SELECT id FROM historical_players WHERE stable_player_key = ?", + (stable_player_key,), + ).fetchone() + if row is None: + raise RuntimeError("Failed to persist historical player identity.") + return int(row["id"]) + + +def _build_stable_player_key(player_payload: Mapping[str, object]) -> str: + steam_id = _stringify(_get_nested(player_payload, "steaminfo", "profile", "steamid")) + if steam_id: + return f"steam:{steam_id}" + + steaminfo_id = _stringify(_get_nested(player_payload, "steaminfo", "id")) + if steaminfo_id: + return f"steaminfo:{steaminfo_id}" + + source_player_id = _stringify(player_payload.get("player_id")) + if source_player_id: + return f"crcon-player:{source_player_id}" + + player_name = _stringify(player_payload.get("player")) or "unknown-player" + normalized_name = "".join( + character.lower() if character.isalnum() else "-" + for character in player_name + ) + compact_name = "-".join(part for part in normalized_name.split("-") if part) + return f"name:{compact_name or 'unknown-player'}" + + +def _extract_map_name(match_payload: Mapping[str, object]) -> str | None: + return _stringify(match_payload.get("map_name")) or _stringify(_get_nested(match_payload, "map", "name")) + + +def _extract_map_pretty_name(match_payload: Mapping[str, object]) -> str | None: + return _stringify(_get_nested(match_payload, "map", "pretty_name")) or _extract_map_name(match_payload) + + +def _extract_map_game_mode(match_payload: Mapping[str, object]) -> str | None: + return _stringify(_get_nested(match_payload, "map", "game_mode")) + + +def _extract_map_image_name(match_payload: Mapping[str, object]) -> str | None: + return _stringify(_get_nested(match_payload, "map", "image_name")) + + +def _coerce_list(value: object) -> list[Mapping[str, object]]: + if not isinstance(value, list): + return [] + return [item for item in value if isinstance(item, Mapping)] + + +def _coerce_int(value: object) -> int | None: + if value in (None, ""): + return None + try: + return int(value) + except (TypeError, ValueError): + return None + + +def _coerce_float(value: object) -> float | None: + if value in (None, ""): + return None + try: + return float(value) + except (TypeError, ValueError): + return None + + +def _stringify(value: object) -> str | None: + if value is None: + return None + text = str(value).strip() + return text or None + + +def _normalize_timestamp(value: object) -> str | None: + text = _stringify(value) + if not text: + return None + try: + return _parse_timestamp(text).astimezone(timezone.utc).isoformat().replace("+00:00", "Z") + except ValueError: + return text + + +def _parse_timestamp(value: str) -> datetime: + normalized = value.strip().replace("Z", "+00:00") + parsed = datetime.fromisoformat(normalized) + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return parsed + + +def _get_nested(payload: Mapping[str, object], *path: str) -> object: + current: object = payload + for key in path: + if not isinstance(current, Mapping): + return None + current = current.get(key) + return current + + +def _utc_now_iso() -> str: + return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") diff --git a/backend/app/routes.py b/backend/app/routes.py index dda288d..2870c64 100644 --- a/backend/app/routes.py +++ b/backend/app/routes.py @@ -15,6 +15,7 @@ from .payloads import ( build_server_latest_payload, build_servers_payload, build_trailer_payload, + build_weekly_top_kills_payload, ) @@ -39,6 +40,13 @@ def resolve_get_payload(path: str) -> tuple[HTTPStatus | None, dict[str, object] return HTTPStatus.BAD_REQUEST, build_error_payload("Invalid limit parameter") return HTTPStatus.OK, build_server_history_payload(limit=limit) + if parsed.path == "/api/historical/weekly-top-kills": + limit = _parse_limit(parsed.query) + if limit is None: + return HTTPStatus.BAD_REQUEST, build_error_payload("Invalid limit parameter") + server_id = parse_qs(parsed.query).get("server", [None])[0] + return HTTPStatus.OK, build_weekly_top_kills_payload(limit=limit, server_id=server_id) + builder = GET_ROUTES.get(parsed.path) if builder is None: if parsed.path.startswith("/api/servers/") and parsed.path.endswith("/history"): diff --git a/docs/decisions.md b/docs/decisions.md index f5f799f..d92325c 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -99,3 +99,30 @@ futura agregacion semanal por servidor. A2S se mantiene como fuente de estado actual de servidores. El historico de partidas y rankings debe construirse en una linea separada basada en CRCON. La discovery detallada queda en `docs/historical-crcon-source-discovery.md`. + +## Decision 013: persistencia historica local separada del flujo live + +El backend mantiene el estado live de servidores y el historico CRCON en el +mismo SQLite local de desarrollo para no introducir infraestructura prematura, +pero ambas lineas quedan separadas por tablas y contratos distintos. + +El flujo live sigue usando `server_snapshots` via A2S. El flujo historico usa +tablas `historical_*` para: + +- servidores historicos configurados +- partidas +- mapas +- jugadores +- estadisticas por jugador y partida +- ejecuciones de ingesta + +Las claves estables son: + +- servidor: `historical_servers.slug` +- partida: `(historical_server_id, external_match_id)` +- jugador: `stable_player_key` +- estadistica por partida: `(historical_match_id, historical_player_id)` + +Esto permite bootstrap, refresco incremental e idempotencia sin mezclar +semanticas de estado actual con historico persistido. El modelo detallado queda +en `docs/historical-domain-model.md`. diff --git a/docs/historical-domain-model.md b/docs/historical-domain-model.md new file mode 100644 index 0000000..b751835 --- /dev/null +++ b/docs/historical-domain-model.md @@ -0,0 +1,149 @@ +# Historical Domain Model + +## Objective + +Definir la base minima de dominio y persistencia para historico de partidas y +metricas por jugador obtenidas desde la capa JSON publica de los scoreboards +CRCON de Comunidad Hispana. + +## Scope + +Esta capa cubre solo historico persistido en backend: + +- identidad estable de los 2 servidores historicos +- partidas cerradas o actualizadas desde CRCON +- mapas asociados a esas partidas +- identidad reutilizable de jugadores +- estadisticas de jugador por partida +- trazabilidad de ejecuciones de ingesta + +No sustituye ni modifica el flujo actual de snapshots live via A2S. + +## Stable Identities + +### Server + +- tabla: `historical_servers` +- clave estable: `slug` +- ejemplos: + - `comunidad-hispana-01` + - `comunidad-hispana-02` +- atributos de soporte: + - `scoreboard_base_url` + - `server_number` + - `source_kind` + +### Match + +- tabla: `historical_matches` +- clave estable: `(historical_server_id, external_match_id)` +- `external_match_id` corresponde al `id` devuelto por CRCON para cada partida +- razon: + - el `id` de partida es estable dentro de cada scoreboard + - se conserva separado por servidor para evitar asumir unicidad global sin + contrato formal + +### Player + +- tabla: `historical_players` +- clave estable: `stable_player_key` +- estrategia de identidad: + 1. `steam:{steamid}` cuando existe `steaminfo.profile.steamid` + 2. `steaminfo:{id}` cuando existe `steaminfo.id` + 3. `crcon-player:{player_id}` cuando existe `player_id` + 4. `name:{normalized-name}` como ultimo fallback + +La prioridad evita perder continuidad cuando CRCON expone SteamID. Los +fallbacks quedan documentados porque la calidad del origen no es totalmente +uniforme. + +### Player Stats Per Match + +- tabla: `historical_player_match_stats` +- clave estable: `(historical_match_id, historical_player_id)` +- efecto: + - la misma partida puede reingestarse sin duplicar filas + - si una partida cambia despues, la fila se actualiza por `UPSERT` + +### Ingestion Run + +- tabla: `historical_ingestion_runs` +- registra: + - tipo de ejecucion (`bootstrap` o `incremental`) + - inicio y fin + - estado + - paginas procesadas + - matches vistos + - inserts y updates + +## Data Model + +### `historical_servers` + +Fuente historica por scoreboard CRCON. + +### `historical_maps` + +Catalogo reutilizable de mapas usando `map.id` cuando existe. + +### `historical_matches` + +Partida historica persistida con: + +- servidor +- identidad externa +- tiempos (`creation_time`, `start`, `end`) +- mapa y metadatos visibles +- resultado axis/allied +- referencia de procedencia + +### `historical_players` + +Identidad reutilizable del jugador entre partidas y servidores. + +### `historical_player_match_stats` + +Metricas por jugador y partida con al menos: + +- kills +- deaths +- teamkills +- time_seconds +- kills_per_minute +- deaths_per_minute +- kill_death_ratio +- combat +- offense +- defense +- support + +### `historical_ingestion_runs` + +Trazabilidad operativa para bootstrap y refresh incremental. + +## Idempotency Strategy + +- servidores sembrados de forma declarativa y actualizables por `slug` +- partidas persistidas con `UPSERT` por `(historical_server_id, external_match_id)` +- jugadores persistidos con `UPSERT` por `stable_player_key` +- estadisticas por jugador actualizadas con `UPSERT` por + `(historical_match_id, historical_player_id)` +- el refresco incremental usa una ventana de solape temporal para volver a leer + partidas recientes y absorber cambios tardios sin rehacer todo el historico + +## Query Readiness + +La estructura soporta ya consultas futuras como: + +- top kills de la ultima semana por servidor +- partidas recientes por servidor +- mapas jugados y frecuencia +- agregados por jugador sobre ventanas temporales + +## Separation From Live State + +- live state actual: `server_snapshots` via A2S +- historico persistido: `historical_*` via CRCON scoreboard JSON + +Ambas lineas comparten el mismo SQLite local de desarrollo para reducir +complejidad operativa, pero mantienen tablas y contratos separados.