From 61b9a56c629d468776b5a7742d1b8acaefe4aea7 Mon Sep 17 00:00:00 2001 From: devRaGonSa Date: Fri, 19 Jun 2026 16:32:53 +0200 Subject: [PATCH] Revert "Restore current-live RCON worker deployment" This reverts commit 7c021f7bde70aa64e36c8df0badeef2603155388. --- ...ore-current-live-rcon-worker-deployment.md | 253 ------------------ deploy/portainer/docker-compose.nas.yml | 24 +- docs/current-match-adminlog-freshness.md | 107 ++++---- 3 files changed, 52 insertions(+), 332 deletions(-) delete mode 100644 ai/tasks/done/TASK-270-restore-current-live-rcon-worker-deployment.md diff --git a/ai/tasks/done/TASK-270-restore-current-live-rcon-worker-deployment.md b/ai/tasks/done/TASK-270-restore-current-live-rcon-worker-deployment.md deleted file mode 100644 index 35918d0..0000000 --- a/ai/tasks/done/TASK-270-restore-current-live-rcon-worker-deployment.md +++ /dev/null @@ -1,253 +0,0 @@ ---- -id: TASK-270 -title: Restore current-live RCON worker deployment -status: done -type: documentation -team: Backend Senior -supporting_teams: ["Arquitecto Python"] -roadmap_item: current-match -priority: high ---- - -# TASK-270 - Restore current-live RCON worker deployment - -## Goal - -Restore the existing production-good `rcon-historical-worker` deployment behavior so current-match AdminLog freshness uses the worker's supported `current-live` mode instead of the heavy historical loop. - -## Context - -The current GitHub/test deployment starts `rcon-historical-worker` as `python -m app.rcon_historical_worker loop`, which resolves to historical capture with heavy materialization enabled. Logs described in the task context show `capture_mode=historical`, `materialization_skipped=false`, and `interval_seconds=2`, which is the wrong runtime shape for current-match freshness. - -The backend code already supports the lower-risk production-good path: - -- `backend/app/rcon_historical_worker.py` supports `--capture-mode current-live`, `--interval`, `--retries`, and `--retry-delay` -- `backend/app/config.py` resolves `HLL_RCON_CURRENT_MATCH_MODE=true` or `HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION=true` to `current-live` -- `current-live` always skips heavy historical materialization and uses the short current-match writer-lock timeout - -Mandatory comparison result: - -1. Uploaded ZIP artifact: no ZIP file matching the described production export was present in the local workspace, so an exact filesystem path inside the ZIP could not be cited directly. -2. Known-good repository reference matching the described ZIP behavior: `deploy/jta/docker-compose.yml` -3. Current repo/deployment file missing or overriding that behavior: `deploy/portainer/docker-compose.nas.yml` -4. Root `docker-compose.yml` is not the likely current Portainer stack file because it sets `container_name: hll-vietnam-rcon-historical-worker`, while the running container name provided in the request is `hll-vietnam-rcon-historical-worker-1` -5. The running container name strongly matches the Compose service `rcon-historical-worker` from a stack deployment without `container_name`, which fits `deploy/portainer/docker-compose.nas.yml` -6. `deploy/jta/docker-compose.yml` already contains the production-good `current-live` command and env pattern and serves as the checked-in known-good comparison point - -## Steps - -1. Compare the worker CLI/config support with the deployment compose variants in the repository. -2. Update only the deployment compose file that likely backs the current Portainer stack. -3. Keep `backend/app/rcon_current_match_worker.py` inactive in deployment. -4. Update documentation to reflect that current-match freshness should come from `rcon-historical-worker` in `current-live` mode. -5. Validate compose syntax and confirm no unrelated files changed. - -## Files to Read First - -- `AGENTS.md` -- `ai/architecture-index.md` -- `ai/repo-context.md` -- `ai/orchestrator/backend-senior.md` -- `backend/app/rcon_historical_worker.py` -- `backend/app/config.py` -- `deploy/portainer/docker-compose.nas.yml` -- `deploy/jta/docker-compose.yml` -- `docs/current-match-adminlog-freshness.md` -- `ai/tasks/done/TASK-269-implement-current-match-adminlog-freshness-worker.md` - -## Expected Files to Modify - -- `deploy/portainer/docker-compose.nas.yml` -- `docs/current-match-adminlog-freshness.md` -- `ai/tasks/done/TASK-270-restore-current-live-rcon-worker-deployment.md` - -## Constraints - -- Do not run `ai-platform run`. -- Do not commit or push. -- Do not touch frontend except for documentation references. -- Do not touch physical assets or `frontend/assets/img/`. -- Do not change RCON hosts, ports, passwords, `external_server_id`, `target_key`, server list, or `27001`. -- Do not change `historical-runner`. -- Do not activate `backend/app/rcon_current_match_worker.py` in this task. -- Do not reactivate Elo/MMR. -- Do not reintroduce `comunidad-hispana-03`. -- Do not touch `ai/system-metrics.md`. -- Do not include `tmp/` or unrelated pending changes. - -## Validation - -Before completing the task ensure: - -- `python -m app.rcon_historical_worker --help` -- `docker compose -f deploy/portainer/docker-compose.nas.yml --env-file deploy/portainer/stack.env.example config` -- optional backend sanity because Python code was not changed: - - `python -m compileall backend/app` - - `cd backend; python -m unittest tests.test_rcon_historical_worker tests.test_current_match_payload` -- `git diff --name-only` matches the expected scope - -## Outcome - -What was wrong in the current deployment: - -- `rcon-historical-worker` was deployed as the heavy historical loop instead of the existing lightweight `current-live` path -- this lets frequent materialization hold the shared writer lock -- it blocks `historical_runner` and also collides with the new dedicated TASK-269 worker if both are active -- deadlock risk increases under PostgreSQL because multiple writers contend around the same persisted read-model refresh path - -What the known-good production behavior did correctly: - -- reused `python -m app.rcon_historical_worker` instead of introducing a second always-on worker -- ran it in `current-live` mode -- skipped heavy historical materialization -- used a short interval and short writer-lock timeout to keep current-match AdminLog/read-model freshness moving - -Exact service changed: - -- service: `rcon-historical-worker` -- likely running container: `hll-vietnam-rcon-historical-worker-1` -- file changed: `deploy/portainer/docker-compose.nas.yml` - -Exact command/env before: - -- command: `python -m app.rcon_historical_worker loop` -- env: - - `HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS=${HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS:-600}` - - `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES=${HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES:-2}` - - `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS=${HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS:-15}` - - no `HLL_RCON_CURRENT_MATCH_MODE` - - no `HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION` - - no `HLL_RCON_CURRENT_MATCH_CAPTURE_INTERVAL_SECONDS` - - no `HLL_RCON_CURRENT_MATCH_WRITER_LOCK_TIMEOUT_SECONDS` - -Exact command/env after: - -- command: `python -m app.rcon_historical_worker loop --capture-mode current-live --interval 5 --retries 0 --retry-delay 0` -- env: - - `HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS=5` - - `HLL_RCON_CURRENT_MATCH_CAPTURE_INTERVAL_SECONDS=5` - - `HLL_RCON_CURRENT_MATCH_MODE=true` - - `HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION=true` - - `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES=0` - - `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS=0` - - `HLL_RCON_CURRENT_MATCH_WRITER_LOCK_TIMEOUT_SECONDS=4` - - `HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES` left unchanged - -Why this is the correct approach: - -- TASK-269 added a dedicated worker, but it still competes for the same shared writer lock if the historical worker is left in heavy mode -- the known-good current production behavior already solved current-match freshness with the existing `rcon_historical_worker` `current-live` mode -- restoring that existing mode is lower-risk than adding a second permanent live worker immediately -- the bad deployment shape is `capture_mode=historical` with `materialization_skipped=false` at a near-live cadence, which is the wrong design for current-match freshness -- the correct split is: - - current-live worker: fast interval, no heavy materialization, current-match AdminLog freshness - - historical-runner: scheduled heavy historical refresh/snapshot work - -Confirmation of protected settings: - -- RCON hosts unchanged -- RCON ports unchanged -- RCON credentials unchanged -- server list unchanged -- `external_server_id` values unchanged -- `target_key` values unchanged -- `27001` unchanged -- `historical-runner` command and interval unchanged -- TASK-269 dedicated worker not activated - -Post-deploy validation commands for the operator: - -```bash -docker inspect -f '{{.Path}} {{json .Args}}' hll-vietnam-rcon-historical-worker-1 -docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' hll-vietnam-rcon-historical-worker-1 | sort | grep -Ei 'CAPTURE|CURRENT|RCON|ADMIN|MATERIAL|INTERVAL|RETRY|LOOKBACK|LOCK' -docker logs --tail=200 hll-vietnam-rcon-historical-worker-1 -docker exec -it hll-vietnam-backend-1 sh -lc 'ls -l /app/data/hll_vietnam_dev.writer.lock || true; cat /app/data/hll_vietnam_dev.writer.lock || true' -``` - -Expected runtime signals: - -- command includes `--capture-mode current-live` -- `materialization_skipped=true` -- `interval_seconds=5` -- `retries=0` -- `retry_delay=0` -- targets remain `comunidad-hispana-01` and `comunidad-hispana-02` -- no repeated heavy materialization every 5 seconds - -Current-match API freshness validation script: - -```powershell -cd "D:\Proyectos\HLL Vietnam" - -$killsBase = "https://comunidadhll.devzamode.es/api/current-match/kills?server=comunidad-hispana-02&limit=18" -$playersBase = "https://comunidadhll.devzamode.es/api/current-match/players?server=comunidad-hispana-02" - -1..18 | ForEach-Object { - $now = Get-Date - $ts = [DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds() - - $kills = Invoke-RestMethod "$killsBase&_ts=$ts" -Headers @{ "Cache-Control" = "no-cache"; "Pragma" = "no-cache" } - $killItems = @($kills.data.items) - $topKill = $killItems | Select-Object -First 1 - - $players = Invoke-RestMethod "$playersBase&_ts=$ts" -Headers @{ "Cache-Control" = "no-cache"; "Pragma" = "no-cache" } - $playerItems = @($players.data.items) - $topPlayers = $playerItems | Sort-Object -Property kills -Descending | Select-Object -First 5 | ForEach-Object { - "$($_.player_name):K=$($_.kills):D=$($_.deaths):Team=$($_.team)" - } - - $killLocalTime = "" - $lagSeconds = "" - if ($topKill -and $topKill.server_time) { - $killDate = [DateTimeOffset]::FromUnixTimeSeconds([int64]$topKill.server_time).ToLocalTime() - $killLocalTime = $killDate.ToString("HH:mm:ss") - $lagSeconds = [int](([DateTimeOffset]::Now - $killDate).TotalSeconds) - } - - [PSCustomObject]@{ - sample = $_ - sampled_at = $now.ToString("HH:mm:ss") - kills_scope = $kills.data.scope - kills_count = $killItems.Count - first_id = if ($killItems.Count -gt 0) { $killItems[0].event_id } else { "" } - first_kill_time = $killLocalTime - estimated_lag_seconds = $lagSeconds - first_kill = if ($topKill) { "$($topKill.killer_name)->$($topKill.victim_name):$($topKill.weapon)" } else { "" } - players_count = $playerItems.Count - top_players = ($topPlayers -join " | ") - } - - Start-Sleep -Seconds 10 -} -``` - -Acceptance target: - -- when CRCON shows a new kill, `/api/current-match/kills` should advance in roughly `5-30` seconds -- `/api/current-match/players` should follow shortly after -- not `5-10` minutes later - -Risks: - -- if Portainer stack-level environment variables explicitly override the new current-live env values in a different hidden stack file, runtime may still need operator verification after redeploy -- if another untracked deployment file is actually used instead of `deploy/portainer/docker-compose.nas.yml`, the compose change here will not affect the running stack -- `hll-vietnam-backend-1` lock-file command assumes the standard container name pattern and may need adaptation if the stack/project name differs - -Validation completed: - -- `python -m app.rcon_historical_worker --help` -- `docker compose -f deploy/portainer/docker-compose.nas.yml --env-file deploy/portainer/stack.env.example config` -- `python -m compileall backend/app` -- `cd backend; python -m unittest tests.test_rcon_historical_worker tests.test_current_match_payload` -- No RCON host changed -- No RCON port changed -- No RCON credentials changed -- No `27001` changed -- No server list changed -- TASK-269 worker was not activated - -## Change Budget - -- Modified files kept to the requested minimal deployment/docs/task scope. -- No backend Python changes. -- No frontend or asset changes. diff --git a/deploy/portainer/docker-compose.nas.yml b/deploy/portainer/docker-compose.nas.yml index 4f41331..c01edde 100644 --- a/deploy/portainer/docker-compose.nas.yml +++ b/deploy/portainer/docker-compose.nas.yml @@ -94,32 +94,16 @@ services: - advanced build: context: ../../backend - command: - - python - - -m - - app.rcon_historical_worker - - loop - - --capture-mode - - current-live - - --interval - - "5" - - --retries - - "0" - - --retry-delay - - "0" + command: ["python", "-m", "app.rcon_historical_worker", "loop"] environment: HLL_BACKEND_DATABASE_URL: ${HLL_BACKEND_DATABASE_URL:?HLL_BACKEND_DATABASE_URL is required} HLL_BACKEND_LIVE_DATA_SOURCE: ${HLL_BACKEND_LIVE_DATA_SOURCE:-rcon} HLL_BACKEND_HISTORICAL_DATA_SOURCE: ${HLL_BACKEND_HISTORICAL_DATA_SOURCE:-rcon} HLL_BACKEND_RCON_TIMEOUT_SECONDS: ${HLL_BACKEND_RCON_TIMEOUT_SECONDS:-20} HLL_BACKEND_RCON_TARGETS: ${HLL_BACKEND_RCON_TARGETS:?HLL_BACKEND_RCON_TARGETS is required} - HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS: "5" - HLL_RCON_CURRENT_MATCH_CAPTURE_INTERVAL_SECONDS: "5" - HLL_RCON_CURRENT_MATCH_MODE: "true" - HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION: "true" - HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES: "0" - HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS: "0" - HLL_RCON_CURRENT_MATCH_WRITER_LOCK_TIMEOUT_SECONDS: "4" + HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS: ${HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS:-600} + HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES: ${HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES:-2} + HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS: ${HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS:-15} HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES: ${HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES:-10} depends_on: postgres: diff --git a/docs/current-match-adminlog-freshness.md b/docs/current-match-adminlog-freshness.md index 1439283..597a226 100644 --- a/docs/current-match-adminlog-freshness.md +++ b/docs/current-match-adminlog-freshness.md @@ -18,105 +18,95 @@ Observed behavior from TASK-268 matched this design: ## Solution -Restore the existing `rcon-historical-worker` deployment to the code path that already supports lightweight current-match ingestion: - -- module: `python -m app.rcon_historical_worker` -- command: - -```text -python -m app.rcon_historical_worker loop --capture-mode current-live --interval 5 --retries 0 --retry-delay 0 -``` +Add a dedicated lightweight worker: +- module: `python -m app.rcon_current_match_worker` - scope: only trusted current-match servers - `comunidad-hispana-01` - `comunidad-hispana-02` -- uses the same AdminLog ingestion and persistence path that already feeds `rcon_admin_log_events` -- skips heavy historical materialization on each near-live loop -- keeps the shared writer lock held only for short current-match ingestion windows +- fetches recent AdminLog directly from existing configured RCON targets +- reuses existing parsing and `persist_rcon_admin_log_entries(...)` +- writes into the same `rcon_admin_log_events` table +- relies on existing dedupe/idempotency for overlap-safe windows -Deployment settings for the current-live worker: +Default worker settings: -- `HLL_RCON_CURRENT_MATCH_MODE=true` -- `HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION=true` -- `HLL_RCON_CURRENT_MATCH_CAPTURE_INTERVAL_SECONDS=5` -- `HLL_RCON_CURRENT_MATCH_WRITER_LOCK_TIMEOUT_SECONDS=4` -- `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES=0` -- `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS=0` +- `CURRENT_MATCH_ADMINLOG_INTERVAL_SECONDS=10` +- `CURRENT_MATCH_ADMINLOG_LOOKBACK_SECONDS=180` +- `CURRENT_MATCH_ADMINLOG_ENABLED=false` -The dedicated worker added in TASK-269 stays in the repository, but it is not the primary deployment fix and is not activated by this change. +The worker is opt-in. This task does not change Compose to start it automatically. ## Design Notes -This deployment change intentionally avoids: +The worker intentionally avoids: - changing RCON hosts, ports or passwords - introducing a second AdminLog schema -- changing `historical-runner` +- reducing the heavy historical worker interval globally - pushing ingestion into `/api/current-match/*` request handlers -- activating `python -m app.rcon_current_match_worker` in production Each iteration: 1. filters configured RCON targets down to trusted current-match servers -2. captures fresh live server state and recent AdminLog entries +2. fetches recent AdminLog entries with a small overlap window 3. persists through the existing idempotent storage path -4. skips heavy historical match materialization for the current-live loop +4. logs per-target counts for: + - `entries_seen` + - `events_inserted` + - `duplicate_events` + - `duration_ms` 5. continues if one server fails -The heavy historical/snapshot path remains the responsibility of `historical-runner` on its scheduled cadence. - ## Run Commands -One-shot validation using the same worker path: +Local one-shot validation: ```powershell cd backend -python -m app.rcon_historical_worker capture --capture-mode current-live +python -m app.rcon_current_match_worker once ``` -Local near-live loop: +Local loop: ```powershell cd backend -python -m app.rcon_historical_worker loop --capture-mode current-live --interval 5 --retries 0 --retry-delay 0 +python -m app.rcon_current_match_worker loop --interval 10 --lookback-seconds 180 ``` -Docker/Compose-style ad hoc command: +Docker/Compose-style ad hoc command without changing deployment defaults: ```powershell -docker compose run --rm rcon-historical-worker python -m app.rcon_historical_worker loop --capture-mode current-live --interval 5 --retries 0 --retry-delay 0 +docker compose run --rm backend python -m app.rcon_current_match_worker loop --interval 10 --lookback-seconds 180 ``` -Expected runtime signals after deployment: +Example service snippet for a future approved deployment: -- `capture_mode: current-live` -- `materialization_skipped: true` -- `interval_seconds: 5` -- no repeated heavy materialization every cycle +```yaml +current-match-adminlog-worker: + profiles: ["advanced"] + build: + context: ./backend + command: ["python", "-m", "app.rcon_current_match_worker", "loop"] + environment: + HLL_BACKEND_DATABASE_URL: ${HLL_BACKEND_DATABASE_URL} + HLL_BACKEND_RCON_TARGETS: ${HLL_BACKEND_RCON_TARGETS} + CURRENT_MATCH_ADMINLOG_ENABLED: ${CURRENT_MATCH_ADMINLOG_ENABLED:-false} + CURRENT_MATCH_ADMINLOG_INTERVAL_SECONDS: ${CURRENT_MATCH_ADMINLOG_INTERVAL_SECONDS:-10} + CURRENT_MATCH_ADMINLOG_LOOKBACK_SECONDS: ${CURRENT_MATCH_ADMINLOG_LOOKBACK_SECONDS:-180} +``` + +The snippet is documented only. It is not applied automatically by this task. ## Post-Deployment Validation -Verify the current-live worker command and env: +Watch worker logs for per-target counts and errors. The useful signals are: -```bash -docker inspect -f '{{.Path}} {{json .Args}}' hll-vietnam-rcon-historical-worker-1 -docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' hll-vietnam-rcon-historical-worker-1 | sort | grep -Ei 'CAPTURE|CURRENT|RCON|ADMIN|MATERIAL|INTERVAL|RETRY|LOOKBACK|LOCK' -``` - -Watch worker logs for: - -- `capture_mode: current-live` -- `materialization_skipped: true` -- `interval_seconds: 5` -- `target_count: 2` -- `comunidad-hispana-01` -- `comunidad-hispana-02` - -Confirm the worker is not doing heavy historical materialization every 5 seconds and that the writer lock is not held for long periods: - -```bash -docker exec -it hll-vietnam-backend-1 sh -lc 'ls -l /app/data/hll_vietnam_dev.writer.lock || true; cat /app/data/hll_vietnam_dev.writer.lock || true' -``` +- `entries_seen` +- `events_inserted` +- `duplicate_events` +- `target_key` +- `duration_ms` Sample the public API every 10 seconds: @@ -165,7 +155,6 @@ $playersBase = "https://comunidadhll.devzamode.es/api/current-match/players?serv Expected outcome after deployment: -- new CRCON kills should appear in `/api/current-match/kills` within roughly `5-30` seconds +- new CRCON kills should appear in `/api/current-match/kills` within roughly `10-30` seconds - `/api/current-match/players` should advance accordingly - the frontend should reflect it without further polling changes -- `historical-runner` should remain responsible for hourly or scheduled heavy refresh work