Split live AdminLog ingestion from historical materialization
This commit is contained in:
devRaGonSa
@@ -0,0 +1,128 @@
|
||||
---
|
||||
id: TASK-271
|
||||
title: Split RCON live ingestion from historical materialization
|
||||
status: in-progress
|
||||
type: backend
|
||||
team: Backend Senior
|
||||
supporting_teams: ["Arquitecto Python"]
|
||||
roadmap_item: current-match
|
||||
priority: high
|
||||
---
|
||||
|
||||
# TASK-271 - Split RCON live ingestion from historical materialization
|
||||
|
||||
## Goal
|
||||
|
||||
Separate near-real-time AdminLog ingestion for current-match freshness from the slow historical materialization path so `/api/current-match/kills` and `/api/current-match/players` can stay fresh without waiting on heavy historical work.
|
||||
|
||||
## Context
|
||||
|
||||
Production evidence shows the current combined worker design is starving the current-match read model:
|
||||
|
||||
- production service `hll-vietnam-rcon-historical-worker-1` currently runs `python -m app.rcon_historical_worker loop`
|
||||
- configured interval is `HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS=2`
|
||||
- configured AdminLog lookback is `HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES=10`
|
||||
- real `rcon_historical_capture_runs` durations are around `15-30` minutes
|
||||
- recent runs overlapped and one failed with PostgreSQL deadlock
|
||||
- each cycle materializes roughly `838k` player-stat rows and updates roughly `392-394` materialized matches
|
||||
- `/api/current-match/kills` and `/api/current-match/players` read `rcon_admin_log_events`, not live CRCON/RCON per public request
|
||||
|
||||
This means the current-match feed problem is not primarily frontend rendering. The source table is not being refreshed frequently enough because live AdminLog ingestion is coupled to heavy historical materialization.
|
||||
|
||||
This task must preserve the current product identity and must not change frontend layout, CSS, assets, RCON hosts, ports, passwords, server list, `27001`, ranking semantics, Elo/MMR activation, or reintroduce `comunidad-hispana-03`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Inspect the current historical worker, current-match worker, AdminLog ingestion/storage, materialization path, API read path and Portainer deployment.
|
||||
2. Implement a dedicated lightweight live AdminLog worker that only refreshes `rcon_admin_log_events`.
|
||||
3. Reduce repeated PostgreSQL schema initialization in the live path by moving obvious DDL/init work out of hot loops and request reads.
|
||||
4. Add no-overlap protection for heavy historical materialization so concurrent heavy runs skip cleanly instead of overlapping.
|
||||
5. Update Portainer deployment so live ingestion and heavy historical materialization run as separate services with safe cadences.
|
||||
6. Add or update focused tests and refresh the current-match freshness documentation with post-deploy validation commands.
|
||||
|
||||
## Files to Read First
|
||||
|
||||
- `AGENTS.md`
|
||||
- `ai/architecture-index.md`
|
||||
- `ai/repo-context.md`
|
||||
- `ai/orchestrator/backend-senior.md`
|
||||
- `backend/app/rcon_historical_worker.py`
|
||||
- `backend/app/rcon_current_match_worker.py`
|
||||
- `backend/app/rcon_admin_log_ingestion.py`
|
||||
- `backend/app/rcon_admin_log_storage.py`
|
||||
- `backend/app/postgres_rcon_storage.py`
|
||||
- `backend/app/payloads.py`
|
||||
- `deploy/portainer/docker-compose.nas.yml`
|
||||
- `docs/current-match-adminlog-freshness.md`
|
||||
|
||||
## Expected Files to Modify
|
||||
|
||||
- `ai/tasks/in-progress/TASK-271-split-rcon-live-ingestion-from-historical-materialization.md`
|
||||
- `backend/app/rcon_current_match_worker.py`
|
||||
- `backend/app/rcon_admin_log_storage.py`
|
||||
- `backend/app/postgres_rcon_storage.py`
|
||||
- `backend/app/rcon_historical_storage.py`
|
||||
- `backend/app/rcon_historical_worker.py`
|
||||
- `backend/tests/test_rcon_current_match_worker.py`
|
||||
- `backend/tests/test_rcon_historical_worker.py`
|
||||
- `backend/tests/test_current_match_payload.py`
|
||||
- `deploy/portainer/docker-compose.nas.yml`
|
||||
- `docs/current-match-adminlog-freshness.md`
|
||||
|
||||
## Constraints
|
||||
|
||||
- Do not run `ai-platform run`.
|
||||
- Do not commit or push.
|
||||
- Do not use `git add .`.
|
||||
- Do not touch `ai/system-metrics.md`.
|
||||
- Do not include `tmp/`.
|
||||
- Do not include TASK-204, TASK-242 or unrelated in-progress/done tasks.
|
||||
- Do not touch weapon assets, map assets or clan assets.
|
||||
- Do not touch frontend layout or CSS.
|
||||
- Do not change RCON hosts, ports, passwords, `27001` or the configured server list.
|
||||
- Do not reintroduce `comunidad-hispana-03`.
|
||||
- Do not reactivate Elo/MMR.
|
||||
- Do not change ranking semantics or fabricate support rankings.
|
||||
- Do not change annual tops/player search behavior except where explicitly needed to stop them from blocking live ingestion.
|
||||
- Keep the change backend/deployment/documentation scoped and reviewable.
|
||||
|
||||
## Validation
|
||||
|
||||
Before completing the task ensure:
|
||||
|
||||
- `python -m compileall backend/app`
|
||||
- `cd backend; python -m unittest tests.test_current_match_payload`
|
||||
- `cd backend; python -m unittest tests.test_rcon_historical_worker`
|
||||
- `cd backend; python -m unittest tests.test_rcon_current_match_worker`
|
||||
- `git diff --name-only` matches the intended scope
|
||||
- no unrelated dirty files were modified
|
||||
- documentation reflects the new live worker split and post-deploy checks
|
||||
|
||||
## Outcome
|
||||
|
||||
Implemented scope:
|
||||
|
||||
- dedicated live AdminLog worker path kept in `backend/app/rcon_current_match_worker.py`
|
||||
- live worker storage initialization moved out of the hot loop body and out of per-target persistence calls
|
||||
- live worker no longer waits on the shared long-running backend writer lock
|
||||
- current-match persistence stays idempotent through the existing `rcon_admin_log_events` dedupe path
|
||||
- historical capture/materialization path gained a single-running historical guard and returns `skipped` with `already-running` when a heavy run is still active
|
||||
- Portainer deployment now splits `rcon-live-adminlog-worker` from `rcon-historical-worker`
|
||||
- heavy historical cadence was set explicitly to `900` seconds in the Portainer compose file
|
||||
|
||||
Validation completed:
|
||||
|
||||
- `python -m compileall backend/app`
|
||||
- `cd backend; python -m unittest tests.test_current_match_payload`
|
||||
- `cd backend; python -m unittest tests.test_rcon_historical_worker`
|
||||
- `cd backend; python -m unittest tests.test_rcon_current_match_worker`
|
||||
|
||||
Follow-up intentionally left out of central scope:
|
||||
|
||||
- `TEAM KILL` parser correctness remains a separate follow-up task unless handled later with a tiny isolated patch
|
||||
|
||||
## Change Budget
|
||||
|
||||
- Prefer fewer than 5 modified files when feasible, but accept a slightly larger backend/deploy/docs scope if required to complete the split safely.
|
||||
- Prefer focused edits over broad refactors.
|
||||
- Split out follow-up correctness issues instead of expanding scope unnecessarily.
|
||||
Reference in New Issue
Block a user