rgonsal/comunidadhll
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restore current-live RCON worker deployment

Browse Source
This commit is contained in:
devRaGonSa
2026-06-18 10:11:44 +02:00
parent 7f37d198e4
commit 7c021f7bde
3 changed files with 332 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

253
ai/tasks/done/TASK-270-restore-current-live-rcon-worker-deployment.md Normal file
View File

@@ -0,0 +1,253 @@
---
id: TASK-270
title: Restore current-live RCON worker deployment
status: done
type: documentation
team: Backend Senior
supporting_teams: ["Arquitecto Python"]
roadmap_item: current-match
priority: high
---
# TASK-270 - Restore current-live RCON worker deployment
## Goal
Restore the existing production-good `rcon-historical-worker` deployment behavior so current-match AdminLog freshness uses the worker's supported `current-live` mode instead of the heavy historical loop.
## Context
The current GitHub/test deployment starts `rcon-historical-worker` as `python -m app.rcon_historical_worker loop`, which resolves to historical capture with heavy materialization enabled. Logs described in the task context show `capture_mode=historical`, `materialization_skipped=false`, and `interval_seconds=2`, which is the wrong runtime shape for current-match freshness.
The backend code already supports the lower-risk production-good path:
- `backend/app/rcon_historical_worker.py` supports `--capture-mode current-live`, `--interval`, `--retries`, and `--retry-delay`
- `backend/app/config.py` resolves `HLL_RCON_CURRENT_MATCH_MODE=true` or `HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION=true` to `current-live`
- `current-live` always skips heavy historical materialization and uses the short current-match writer-lock timeout
Mandatory comparison result:
1. Uploaded ZIP artifact: no ZIP file matching the described production export was present in the local workspace, so an exact filesystem path inside the ZIP could not be cited directly.
2. Known-good repository reference matching the described ZIP behavior: `deploy/jta/docker-compose.yml`
3. Current repo/deployment file missing or overriding that behavior: `deploy/portainer/docker-compose.nas.yml`
4. Root `docker-compose.yml` is not the likely current Portainer stack file because it sets `container_name: hll-vietnam-rcon-historical-worker`, while the running container name provided in the request is `hll-vietnam-rcon-historical-worker-1`
5. The running container name strongly matches the Compose service `rcon-historical-worker` from a stack deployment without `container_name`, which fits `deploy/portainer/docker-compose.nas.yml`
6. `deploy/jta/docker-compose.yml` already contains the production-good `current-live` command and env pattern and serves as the checked-in known-good comparison point
## Steps
1. Compare the worker CLI/config support with the deployment compose variants in the repository.
2. Update only the deployment compose file that likely backs the current Portainer stack.
3. Keep `backend/app/rcon_current_match_worker.py` inactive in deployment.
4. Update documentation to reflect that current-match freshness should come from `rcon-historical-worker` in `current-live` mode.
5. Validate compose syntax and confirm no unrelated files changed.
## Files to Read First
- `AGENTS.md`
- `ai/architecture-index.md`
- `ai/repo-context.md`
- `ai/orchestrator/backend-senior.md`
- `backend/app/rcon_historical_worker.py`
- `backend/app/config.py`
- `deploy/portainer/docker-compose.nas.yml`
- `deploy/jta/docker-compose.yml`
- `docs/current-match-adminlog-freshness.md`
- `ai/tasks/done/TASK-269-implement-current-match-adminlog-freshness-worker.md`
## Expected Files to Modify
- `deploy/portainer/docker-compose.nas.yml`
- `docs/current-match-adminlog-freshness.md`
- `ai/tasks/done/TASK-270-restore-current-live-rcon-worker-deployment.md`
## Constraints
- Do not run `ai-platform run`.
- Do not commit or push.
- Do not touch frontend except for documentation references.
- Do not touch physical assets or `frontend/assets/img/`.
- Do not change RCON hosts, ports, passwords, `external_server_id`, `target_key`, server list, or `27001`.
- Do not change `historical-runner`.
- Do not activate `backend/app/rcon_current_match_worker.py` in this task.
- Do not reactivate Elo/MMR.
- Do not reintroduce `comunidad-hispana-03`.
- Do not touch `ai/system-metrics.md`.
- Do not include `tmp/` or unrelated pending changes.
## Validation
Before completing the task ensure:
- `python -m app.rcon_historical_worker --help`
- `docker compose -f deploy/portainer/docker-compose.nas.yml --env-file deploy/portainer/stack.env.example config`
- optional backend sanity because Python code was not changed:
- `python -m compileall backend/app`
- `cd backend; python -m unittest tests.test_rcon_historical_worker tests.test_current_match_payload`
- `git diff --name-only` matches the expected scope
## Outcome
What was wrong in the current deployment:
- `rcon-historical-worker` was deployed as the heavy historical loop instead of the existing lightweight `current-live` path
- this lets frequent materialization hold the shared writer lock
- it blocks `historical_runner` and also collides with the new dedicated TASK-269 worker if both are active
- deadlock risk increases under PostgreSQL because multiple writers contend around the same persisted read-model refresh path
What the known-good production behavior did correctly:
- reused `python -m app.rcon_historical_worker` instead of introducing a second always-on worker
- ran it in `current-live` mode
- skipped heavy historical materialization
- used a short interval and short writer-lock timeout to keep current-match AdminLog/read-model freshness moving
Exact service changed:
- service: `rcon-historical-worker`
- likely running container: `hll-vietnam-rcon-historical-worker-1`
- file changed: `deploy/portainer/docker-compose.nas.yml`
Exact command/env before:
- command: `python -m app.rcon_historical_worker loop`
- env:
- `HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS=${HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS:-600}`
- `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES=${HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES:-2}`
- `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS=${HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS:-15}`
- no `HLL_RCON_CURRENT_MATCH_MODE`
- no `HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION`
- no `HLL_RCON_CURRENT_MATCH_CAPTURE_INTERVAL_SECONDS`
- no `HLL_RCON_CURRENT_MATCH_WRITER_LOCK_TIMEOUT_SECONDS`
Exact command/env after:
- command: `python -m app.rcon_historical_worker loop --capture-mode current-live --interval 5 --retries 0 --retry-delay 0`
- env:
- `HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS=5`
- `HLL_RCON_CURRENT_MATCH_CAPTURE_INTERVAL_SECONDS=5`
- `HLL_RCON_CURRENT_MATCH_MODE=true`
- `HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION=true`
- `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES=0`
- `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS=0`
- `HLL_RCON_CURRENT_MATCH_WRITER_LOCK_TIMEOUT_SECONDS=4`
- `HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES` left unchanged
Why this is the correct approach:
- TASK-269 added a dedicated worker, but it still competes for the same shared writer lock if the historical worker is left in heavy mode
- the known-good current production behavior already solved current-match freshness with the existing `rcon_historical_worker` `current-live` mode
- restoring that existing mode is lower-risk than adding a second permanent live worker immediately
- the bad deployment shape is `capture_mode=historical` with `materialization_skipped=false` at a near-live cadence, which is the wrong design for current-match freshness
- the correct split is:
- current-live worker: fast interval, no heavy materialization, current-match AdminLog freshness
- historical-runner: scheduled heavy historical refresh/snapshot work
Confirmation of protected settings:
- RCON hosts unchanged
- RCON ports unchanged
- RCON credentials unchanged
- server list unchanged
- `external_server_id` values unchanged
- `target_key` values unchanged
- `27001` unchanged
- `historical-runner` command and interval unchanged
- TASK-269 dedicated worker not activated
Post-deploy validation commands for the operator:
```bash
docker inspect -f '{{.Path}} {{json .Args}}' hll-vietnam-rcon-historical-worker-1
docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' hll-vietnam-rcon-historical-worker-1 | sort | grep -Ei 'CAPTURE|CURRENT|RCON|ADMIN|MATERIAL|INTERVAL|RETRY|LOOKBACK|LOCK'
docker logs --tail=200 hll-vietnam-rcon-historical-worker-1
docker exec -it hll-vietnam-backend-1 sh -lc 'ls -l /app/data/hll_vietnam_dev.writer.lock || true; cat /app/data/hll_vietnam_dev.writer.lock || true'
```
Expected runtime signals:
- command includes `--capture-mode current-live`
- `materialization_skipped=true`
- `interval_seconds=5`
- `retries=0`
- `retry_delay=0`
- targets remain `comunidad-hispana-01` and `comunidad-hispana-02`
- no repeated heavy materialization every 5 seconds
Current-match API freshness validation script:
```powershell
cd "D:\Proyectos\HLL Vietnam"
$killsBase = "https://comunidadhll.devzamode.es/api/current-match/kills?server=comunidad-hispana-02&limit=18"
$playersBase = "https://comunidadhll.devzamode.es/api/current-match/players?server=comunidad-hispana-02"
1..18 | ForEach-Object {
$now = Get-Date
$ts = [DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds()
$kills = Invoke-RestMethod "$killsBase&_ts=$ts" -Headers @{ "Cache-Control" = "no-cache"; "Pragma" = "no-cache" }
$killItems = @($kills.data.items)
$topKill = $killItems | Select-Object -First 1
$players = Invoke-RestMethod "$playersBase&_ts=$ts" -Headers @{ "Cache-Control" = "no-cache"; "Pragma" = "no-cache" }
$playerItems = @($players.data.items)
$topPlayers = $playerItems | Sort-Object -Property kills -Descending | Select-Object -First 5 | ForEach-Object {
"$($_.player_name):K=$($_.kills):D=$($_.deaths):Team=$($_.team)"
}
$killLocalTime = ""
$lagSeconds = ""
if ($topKill -and $topKill.server_time) {
$killDate = [DateTimeOffset]::FromUnixTimeSeconds([int64]$topKill.server_time).ToLocalTime()
$killLocalTime = $killDate.ToString("HH:mm:ss")
$lagSeconds = [int](([DateTimeOffset]::Now - $killDate).TotalSeconds)
}
[PSCustomObject]@{
sample = $_
sampled_at = $now.ToString("HH:mm:ss")
kills_scope = $kills.data.scope
kills_count = $killItems.Count
first_id = if ($killItems.Count -gt 0) { $killItems[0].event_id } else { "" }
first_kill_time = $killLocalTime
estimated_lag_seconds = $lagSeconds
first_kill = if ($topKill) { "$($topKill.killer_name)->$($topKill.victim_name):$($topKill.weapon)" } else { "" }
players_count = $playerItems.Count
top_players = ($topPlayers -join " | ")
}
Start-Sleep -Seconds 10
}
```
Acceptance target:
- when CRCON shows a new kill, `/api/current-match/kills` should advance in roughly `5-30` seconds
- `/api/current-match/players` should follow shortly after
- not `5-10` minutes later
Risks:
- if Portainer stack-level environment variables explicitly override the new current-live env values in a different hidden stack file, runtime may still need operator verification after redeploy
- if another untracked deployment file is actually used instead of `deploy/portainer/docker-compose.nas.yml`, the compose change here will not affect the running stack
- `hll-vietnam-backend-1` lock-file command assumes the standard container name pattern and may need adaptation if the stack/project name differs
Validation completed:
- `python -m app.rcon_historical_worker --help`
- `docker compose -f deploy/portainer/docker-compose.nas.yml --env-file deploy/portainer/stack.env.example config`
- `python -m compileall backend/app`
- `cd backend; python -m unittest tests.test_rcon_historical_worker tests.test_current_match_payload`
- No RCON host changed
- No RCON port changed
- No RCON credentials changed
- No `27001` changed
- No server list changed
- TASK-269 worker was not activated
## Change Budget
- Modified files kept to the requested minimal deployment/docs/task scope.
- No backend Python changes.
- No frontend or asset changes.

24
deploy/portainer/docker-compose.nas.yml
View File

@@ -94,16 +94,32 @@ services:
- advanced - advanced
build: build:
context: ../../backend context: ../../backend
command: ["python", "-m", "app.rcon_historical_worker", "loop"] command:
- python
- -m
- app.rcon_historical_worker
- loop
- --capture-mode
- current-live
- --interval
- "5"
- --retries
- "0"
- --retry-delay
- "0"
environment: environment:
HLL_BACKEND_DATABASE_URL: ${HLL_BACKEND_DATABASE_URL:?HLL_BACKEND_DATABASE_URL is required} HLL_BACKEND_DATABASE_URL: ${HLL_BACKEND_DATABASE_URL:?HLL_BACKEND_DATABASE_URL is required}
HLL_BACKEND_LIVE_DATA_SOURCE: ${HLL_BACKEND_LIVE_DATA_SOURCE:-rcon} HLL_BACKEND_LIVE_DATA_SOURCE: ${HLL_BACKEND_LIVE_DATA_SOURCE:-rcon}
HLL_BACKEND_HISTORICAL_DATA_SOURCE: ${HLL_BACKEND_HISTORICAL_DATA_SOURCE:-rcon} HLL_BACKEND_HISTORICAL_DATA_SOURCE: ${HLL_BACKEND_HISTORICAL_DATA_SOURCE:-rcon}
HLL_BACKEND_RCON_TIMEOUT_SECONDS: ${HLL_BACKEND_RCON_TIMEOUT_SECONDS:-20} HLL_BACKEND_RCON_TIMEOUT_SECONDS: ${HLL_BACKEND_RCON_TIMEOUT_SECONDS:-20}
HLL_BACKEND_RCON_TARGETS: ${HLL_BACKEND_RCON_TARGETS:?HLL_BACKEND_RCON_TARGETS is required} HLL_BACKEND_RCON_TARGETS: ${HLL_BACKEND_RCON_TARGETS:?HLL_BACKEND_RCON_TARGETS is required}
HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS: ${HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS:-600} HLL_RCON_HISTORICAL_CAPTURE_INTERVAL_SECONDS: "5"
HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES: ${HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES:-2} HLL_RCON_CURRENT_MATCH_CAPTURE_INTERVAL_SECONDS: "5"
HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS: ${HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS:-15} HLL_RCON_CURRENT_MATCH_MODE: "true"
HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION: "true"
HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES: "0"
HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS: "0"
HLL_RCON_CURRENT_MATCH_WRITER_LOCK_TIMEOUT_SECONDS: "4"
HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES: ${HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES:-10} HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES: ${HLL_BACKEND_RCON_ADMIN_LOG_LOOKBACK_MINUTES:-10}
depends_on: depends_on:
postgres: postgres:

107
docs/current-match-adminlog-freshness.md
View File

@@ -18,95 +18,105 @@ Observed behavior from TASK-268 matched this design:
## Solution ## Solution
Add a dedicated lightweight worker: Restore the existing `rcon-historical-worker` deployment to the code path that already supports lightweight current-match ingestion:
- module: `python -m app.rcon_historical_worker`
- command:
```text
python -m app.rcon_historical_worker loop --capture-mode current-live --interval 5 --retries 0 --retry-delay 0
```
- module: `python -m app.rcon_current_match_worker`
- scope: only trusted current-match servers - scope: only trusted current-match servers
- `comunidad-hispana-01` - `comunidad-hispana-01`
- `comunidad-hispana-02` - `comunidad-hispana-02`
- fetches recent AdminLog directly from existing configured RCON targets - uses the same AdminLog ingestion and persistence path that already feeds `rcon_admin_log_events`
- reuses existing parsing and `persist_rcon_admin_log_entries(...)` - skips heavy historical materialization on each near-live loop
- writes into the same `rcon_admin_log_events` table - keeps the shared writer lock held only for short current-match ingestion windows
- relies on existing dedupe/idempotency for overlap-safe windows
Default worker settings: Deployment settings for the current-live worker:
- `CURRENT_MATCH_ADMINLOG_INTERVAL_SECONDS=10` - `HLL_RCON_CURRENT_MATCH_MODE=true`
- `CURRENT_MATCH_ADMINLOG_LOOKBACK_SECONDS=180` - `HLL_RCON_SKIP_HISTORICAL_MATERIALIZATION=true`
- `CURRENT_MATCH_ADMINLOG_ENABLED=false` - `HLL_RCON_CURRENT_MATCH_CAPTURE_INTERVAL_SECONDS=5`
- `HLL_RCON_CURRENT_MATCH_WRITER_LOCK_TIMEOUT_SECONDS=4`
- `HLL_RCON_HISTORICAL_CAPTURE_MAX_RETRIES=0`
- `HLL_RCON_HISTORICAL_CAPTURE_RETRY_DELAY_SECONDS=0`
The worker is opt-in. This task does not change Compose to start it automatically. The dedicated worker added in TASK-269 stays in the repository, but it is not the primary deployment fix and is not activated by this change.
## Design Notes ## Design Notes
The worker intentionally avoids: This deployment change intentionally avoids:
- changing RCON hosts, ports or passwords - changing RCON hosts, ports or passwords
- introducing a second AdminLog schema - introducing a second AdminLog schema
- reducing the heavy historical worker interval globally - changing `historical-runner`
- pushing ingestion into `/api/current-match/*` request handlers - pushing ingestion into `/api/current-match/*` request handlers
- activating `python -m app.rcon_current_match_worker` in production
Each iteration: Each iteration:
1. filters configured RCON targets down to trusted current-match servers 1. filters configured RCON targets down to trusted current-match servers
2. fetches recent AdminLog entries with a small overlap window 2. captures fresh live server state and recent AdminLog entries
3. persists through the existing idempotent storage path 3. persists through the existing idempotent storage path
4. logs per-target counts for: 4. skips heavy historical match materialization for the current-live loop
- `entries_seen`
- `events_inserted`
- `duplicate_events`
- `duration_ms`
5. continues if one server fails 5. continues if one server fails
The heavy historical/snapshot path remains the responsibility of `historical-runner` on its scheduled cadence.
## Run Commands ## Run Commands
Local one-shot validation: One-shot validation using the same worker path:
```powershell ```powershell
cd backend cd backend
python -m app.rcon_current_match_worker once python -m app.rcon_historical_worker capture --capture-mode current-live
``` ```
Local loop: Local near-live loop:
```powershell ```powershell
cd backend cd backend
python -m app.rcon_current_match_worker loop --interval 10 --lookback-seconds 180 python -m app.rcon_historical_worker loop --capture-mode current-live --interval 5 --retries 0 --retry-delay 0
``` ```
Docker/Compose-style ad hoc command without changing deployment defaults: Docker/Compose-style ad hoc command:
```powershell ```powershell
docker compose run --rm backend python -m app.rcon_current_match_worker loop --interval 10 --lookback-seconds 180 docker compose run --rm rcon-historical-worker python -m app.rcon_historical_worker loop --capture-mode current-live --interval 5 --retries 0 --retry-delay 0
``` ```
Example service snippet for a future approved deployment: Expected runtime signals after deployment:
```yaml - `capture_mode: current-live`
current-match-adminlog-worker: - `materialization_skipped: true`
profiles: ["advanced"] - `interval_seconds: 5`
build: - no repeated heavy materialization every cycle
context: ./backend
command: ["python", "-m", "app.rcon_current_match_worker", "loop"]
environment:
HLL_BACKEND_DATABASE_URL: ${HLL_BACKEND_DATABASE_URL}
HLL_BACKEND_RCON_TARGETS: ${HLL_BACKEND_RCON_TARGETS}
CURRENT_MATCH_ADMINLOG_ENABLED: ${CURRENT_MATCH_ADMINLOG_ENABLED:-false}
CURRENT_MATCH_ADMINLOG_INTERVAL_SECONDS: ${CURRENT_MATCH_ADMINLOG_INTERVAL_SECONDS:-10}
CURRENT_MATCH_ADMINLOG_LOOKBACK_SECONDS: ${CURRENT_MATCH_ADMINLOG_LOOKBACK_SECONDS:-180}
```
The snippet is documented only. It is not applied automatically by this task.
## Post-Deployment Validation ## Post-Deployment Validation
Watch worker logs for per-target counts and errors. The useful signals are: Verify the current-live worker command and env:
- `entries_seen` ```bash
- `events_inserted` docker inspect -f '{{.Path}} {{json .Args}}' hll-vietnam-rcon-historical-worker-1
- `duplicate_events` docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' hll-vietnam-rcon-historical-worker-1 | sort | grep -Ei 'CAPTURE|CURRENT|RCON|ADMIN|MATERIAL|INTERVAL|RETRY|LOOKBACK|LOCK'
- `target_key` ```
- `duration_ms`
Watch worker logs for:
- `capture_mode: current-live`
- `materialization_skipped: true`
- `interval_seconds: 5`
- `target_count: 2`
- `comunidad-hispana-01`
- `comunidad-hispana-02`
Confirm the worker is not doing heavy historical materialization every 5 seconds and that the writer lock is not held for long periods:
```bash
docker exec -it hll-vietnam-backend-1 sh -lc 'ls -l /app/data/hll_vietnam_dev.writer.lock || true; cat /app/data/hll_vietnam_dev.writer.lock || true'
```
Sample the public API every 10 seconds: Sample the public API every 10 seconds:
@@ -155,6 +165,7 @@ $playersBase = "https://comunidadhll.devzamode.es/api/current-match/players?serv
Expected outcome after deployment: Expected outcome after deployment:
- new CRCON kills should appear in `/api/current-match/kills` within roughly `10-30` seconds - new CRCON kills should appear in `/api/current-match/kills` within roughly `5-30` seconds
- `/api/current-match/players` should advance accordingly - `/api/current-match/players` should advance accordingly
- the frontend should reflect it without further polling changes - the frontend should reflect it without further polling changes
- `historical-runner` should remain responsible for hourly or scheduled heavy refresh work