comunidadhll/ai/tasks/done/TASK-271-split-rcon-live-ingestion-from-historical-materialization.md

id, title, status, type, team, supporting_teams, roadmap_item, priority

id	title	status	type	team	supporting_teams	roadmap_item	priority
TASK-271	Split RCON live ingestion from historical materialization	in-progress	backend	Backend Senior	Arquitecto Python	current-match	high

TASK-271 - Split RCON live ingestion from historical materialization

Goal

Separate near-real-time AdminLog ingestion for current-match freshness from the slow historical materialization path so /api/current-match/kills and /api/current-match/players can stay fresh without waiting on heavy historical work.

Context

Production evidence shows the current combined worker design is starving the current-match read model:

• production service hll-vietnam-rcon-historical-worker-1 currently runs python -m app.rcon_historical_worker loop
• configured interval is HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS=2
• configured AdminLog lookback is HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES=10
• real rcon_historical_capture_runs durations are around 15-30 minutes
• recent runs overlapped and one failed with PostgreSQL deadlock
• each cycle materializes roughly 838k player-stat rows and updates roughly 392-394 materialized matches
• /api/current-match/kills and /api/current-match/players read rcon_admin_log_events, not live CRCON/RCON per public request

This means the current-match feed problem is not primarily frontend rendering. The source table is not being refreshed frequently enough because live AdminLog ingestion is coupled to heavy historical materialization.

This task must preserve the current product identity and must not change frontend layout, CSS, assets, RCON hosts, ports, passwords, server list, 27001, ranking semantics, Elo/MMR activation, or reintroduce comunidad-hispana-03.

Steps

1. Inspect the current historical worker, current-match worker, AdminLog ingestion/storage, materialization path, API read path and Portainer deployment.
2. Implement a dedicated lightweight live AdminLog worker that only refreshes rcon_admin_log_events.
3. Reduce repeated PostgreSQL schema initialization in the live path by moving obvious DDL/init work out of hot loops and request reads.
4. Add no-overlap protection for heavy historical materialization so concurrent heavy runs skip cleanly instead of overlapping.
5. Update Portainer deployment so live ingestion and heavy historical materialization run as separate services with safe cadences.
6. Add or update focused tests and refresh the current-match freshness documentation with post-deploy validation commands.

Files to Read First

• AGENTS.md
• ai/architecture-index.md
• ai/repo-context.md
• ai/orchestrator/backend-senior.md
• backend/app/rcon_historical_worker.py
• backend/app/rcon_current_match_worker.py
• backend/app/rcon_admin_log_ingestion.py
• backend/app/rcon_admin_log_storage.py
• backend/app/postgres_rcon_storage.py
• backend/app/payloads.py
• deploy/portainer/docker-compose.nas.yml
• docs/current-match-adminlog-freshness.md

Expected Files to Modify

• ai/tasks/in-progress/TASK-271-split-rcon-live-ingestion-from-historical-materialization.md
• backend/app/rcon_current_match_worker.py
• backend/app/rcon_admin_log_storage.py
• backend/app/postgres_rcon_storage.py
• backend/app/rcon_historical_storage.py
• backend/app/rcon_historical_worker.py
• backend/tests/test_rcon_current_match_worker.py
• backend/tests/test_rcon_historical_worker.py
• backend/tests/test_current_match_payload.py
• deploy/portainer/docker-compose.nas.yml
• docs/current-match-adminlog-freshness.md

Constraints

• Do not run ai-platform run.
• Do not commit or push.
• Do not use git add ..
• Do not touch ai/system-metrics.md.
• Do not include tmp/.
• Do not include TASK-204, TASK-242 or unrelated in-progress/done tasks.
• Do not touch weapon assets, map assets or clan assets.
• Do not touch frontend layout or CSS.
• Do not change RCON hosts, ports, passwords, 27001 or the configured server list.
• Do not reintroduce comunidad-hispana-03.
• Do not reactivate Elo/MMR.
• Do not change ranking semantics or fabricate support rankings.
• Do not change annual tops/player search behavior except where explicitly needed to stop them from blocking live ingestion.
• Keep the change backend/deployment/documentation scoped and reviewable.

Validation

Before completing the task ensure:

• python -m compileall backend/app
• cd backend; python -m unittest tests.test_current_match_payload
• cd backend; python -m unittest tests.test_rcon_historical_worker
• cd backend; python -m unittest tests.test_rcon_current_match_worker
• git diff --name-only matches the intended scope
• no unrelated dirty files were modified
• documentation reflects the new live worker split and post-deploy checks

Outcome

Implemented scope:

• dedicated live AdminLog worker path kept in backend/app/rcon_current_match_worker.py
• live worker storage initialization moved out of the hot loop body and out of per-target persistence calls
• live worker no longer waits on the shared long-running backend writer lock
• current-match persistence stays idempotent through the existing rcon_admin_log_events dedupe path
• historical capture/materialization path gained a single-running historical guard and returns skipped with already-running when a heavy run is still active
• Portainer deployment now splits rcon-live-adminlog-worker from rcon-historical-worker
• heavy historical cadence was set explicitly to 900 seconds in the Portainer compose file

Validation completed:

• python -m compileall backend/app
• cd backend; python -m unittest tests.test_current_match_payload
• cd backend; python -m unittest tests.test_rcon_historical_worker
• cd backend; python -m unittest tests.test_rcon_current_match_worker

Follow-up intentionally left out of central scope:

• TEAM KILL parser correctness remains a separate follow-up task unless handled later with a tiny isolated patch

Change Budget

• Prefer fewer than 5 modified files when feasible, but accept a slightly larger backend/deploy/docs scope if required to complete the split safely.
• Prefer focused edits over broad refactors.
• Split out follow-up correctness issues instead of expanding scope unnecessarily.