chore: add database maintenance and home copy tasks
This commit is contained in:
236
ai/tasks/pending/TASK-database-maintenance-cleanup-command.md
Normal file
236
ai/tasks/pending/TASK-database-maintenance-cleanup-command.md
Normal file
@@ -0,0 +1,236 @@
|
|||||||
|
---
|
||||||
|
id: TASK-database-maintenance-cleanup-command
|
||||||
|
title: Add database maintenance cleanup command
|
||||||
|
status: pending
|
||||||
|
type: backend
|
||||||
|
team: Backend Senior
|
||||||
|
supporting_teams: [Arquitecto de Base de Datos, Arquitecto Python]
|
||||||
|
roadmap_item: historical-data-maintenance
|
||||||
|
priority: high
|
||||||
|
---
|
||||||
|
|
||||||
|
# TASK-database-maintenance-cleanup-command - Add database maintenance cleanup command
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Implement a safe backend cleanup command with dry-run/apply modes to remove database data that is no longer needed while preserving all data required for visible history and leaderboards.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The project should keep the database bounded without manually deleting data.
|
||||||
|
|
||||||
|
The command will later be scheduled from historical-runner, but this task only implements the cleanup module and tests.
|
||||||
|
|
||||||
|
The cleanup must not rely on native PostgreSQL scheduling or a new Docker service.
|
||||||
|
|
||||||
|
The cleanup must protect:
|
||||||
|
|
||||||
|
- latest 100 closed/materialized matches;
|
||||||
|
- current month;
|
||||||
|
- previous month during the first 7 days of the new month;
|
||||||
|
- current week;
|
||||||
|
- previous week if fallback weekly logic may need it;
|
||||||
|
- child/player stats for protected matches;
|
||||||
|
- current live/RCON data.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Inspect the listed files first.
|
||||||
|
2. Add a backend module for database maintenance.
|
||||||
|
3. Provide a CLI entry point:
|
||||||
|
- `python -m app.database_maintenance cleanup --dry-run`
|
||||||
|
- `python -m app.database_maintenance cleanup --apply`
|
||||||
|
4. Make dry-run the default behavior.
|
||||||
|
5. Add JSON logs for planning, batches and completion.
|
||||||
|
6. Add tests.
|
||||||
|
7. Validate locally.
|
||||||
|
8. Commit and push the task branch.
|
||||||
|
|
||||||
|
## Files to Read First
|
||||||
|
|
||||||
|
- `AGENTS.md`
|
||||||
|
- `backend/app/config.py`
|
||||||
|
- `backend/app/postgres_display_storage.py`
|
||||||
|
- `backend/app/historical_runner.py`
|
||||||
|
- `backend/app/rcon_historical_read_model.py`
|
||||||
|
- `backend/app/rcon_historical_backfill.py`
|
||||||
|
- `backend/tests/`
|
||||||
|
|
||||||
|
## Expected Files to Modify
|
||||||
|
|
||||||
|
- `backend/app/database_maintenance.py`
|
||||||
|
- `backend/app/config.py`
|
||||||
|
- `backend/tests/test_database_maintenance.py`
|
||||||
|
|
||||||
|
Optional only if required by existing test helpers:
|
||||||
|
|
||||||
|
- `backend/tests/conftest.py`
|
||||||
|
- `backend/tests/test_*.py`
|
||||||
|
|
||||||
|
Do not modify frontend files in this task.
|
||||||
|
|
||||||
|
## Functional Requirements
|
||||||
|
|
||||||
|
Create a cleanup command:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
python -m app.database_maintenance cleanup --dry-run
|
||||||
|
python -m app.database_maintenance cleanup --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
Supported options:
|
||||||
|
|
||||||
|
- `--recent-matches-keep`
|
||||||
|
- `--admin-log-noncritical-retention-days`
|
||||||
|
- `--admin-log-critical-retention-days`
|
||||||
|
- `--server-snapshot-retention-days`
|
||||||
|
- `--batch-size`
|
||||||
|
- `--vacuum-analyze`
|
||||||
|
- `--now`
|
||||||
|
|
||||||
|
Environment defaults:
|
||||||
|
|
||||||
|
- `HLL_RECENT_MATCHES_KEEP=100`
|
||||||
|
- `HLL_ADMIN_LOG_NONCRITICAL_RETENTION_DAYS=30`
|
||||||
|
- `HLL_ADMIN_LOG_CRITICAL_RETENTION_DAYS=90`
|
||||||
|
- `HLL_SERVER_SNAPSHOT_RETENTION_DAYS=14`
|
||||||
|
- `HLL_DB_MAINTENANCE_BATCH_SIZE=5000`
|
||||||
|
|
||||||
|
Dry-run behavior:
|
||||||
|
|
||||||
|
- calculate all candidate rows;
|
||||||
|
- do not delete anything;
|
||||||
|
- log exactly what would be deleted;
|
||||||
|
- return status ok.
|
||||||
|
|
||||||
|
Apply behavior:
|
||||||
|
|
||||||
|
- delete only candidate rows;
|
||||||
|
- delete in batches;
|
||||||
|
- use transactions safely;
|
||||||
|
- do not block indefinitely;
|
||||||
|
- log deleted row counts;
|
||||||
|
- return status ok even if deleted counts are zero.
|
||||||
|
|
||||||
|
Retention Logic
|
||||||
|
|
||||||
|
Protected materialized matches:
|
||||||
|
|
||||||
|
- latest N closed matches, default N = 100;
|
||||||
|
- current month matches;
|
||||||
|
- previous month matches if now.day <= 7;
|
||||||
|
- current week matches;
|
||||||
|
- previous week matches if weekly fallback may need them;
|
||||||
|
- any match whose end/start timestamp cannot be safely interpreted should be skipped/protected, not deleted.
|
||||||
|
|
||||||
|
Delete materialized matches only if:
|
||||||
|
|
||||||
|
- not in protected latest N;
|
||||||
|
- not in protected current/previous month window;
|
||||||
|
- not in protected current/previous week window;
|
||||||
|
- older than the relevant protected windows;
|
||||||
|
- child stats/events that depend on the match are handled first.
|
||||||
|
|
||||||
|
AdminLog critical events:
|
||||||
|
|
||||||
|
- `kill`
|
||||||
|
- `match_start`
|
||||||
|
- `match_end`
|
||||||
|
|
||||||
|
Critical events may only be deleted after `HLL_ADMIN_LOG_CRITICAL_RETENTION_DAYS` and only if they are not needed for protected materialized matches/windows.
|
||||||
|
|
||||||
|
AdminLog non-critical events may be deleted after `HLL_ADMIN_LOG_NONCRITICAL_RETENTION_DAYS`.
|
||||||
|
|
||||||
|
Server snapshots may be deleted after `HLL_SERVER_SNAPSHOT_RETENTION_DAYS`.
|
||||||
|
|
||||||
|
`displayed_historical_snapshots` should not be routinely deleted because it is keyed/upserted. Only skip or report suspicious legacy/corrupt records; do not aggressively delete snapshots in this task.
|
||||||
|
|
||||||
|
## Safety Requirements
|
||||||
|
|
||||||
|
- If an optional table does not exist, log a controlled skip and continue.
|
||||||
|
- If PostgreSQL URL is not configured and the app is in SQLite mode, either support SQLite safely or return a clear unsupported message, depending on existing architecture.
|
||||||
|
- Do not leave orphan rows.
|
||||||
|
- Do not delete from unrelated Elo/MMR tables unless the audit proves exact dependency and task scope is explicitly updated.
|
||||||
|
- Use existing project lock/writer-lock patterns if present.
|
||||||
|
- Cleanup failure must produce JSON error output.
|
||||||
|
|
||||||
|
## Required JSON Log Events
|
||||||
|
|
||||||
|
At minimum:
|
||||||
|
|
||||||
|
- `database-maintenance-started`
|
||||||
|
- `database-maintenance-plan`
|
||||||
|
- `database-maintenance-table-skipped`
|
||||||
|
- `database-maintenance-delete-batch`
|
||||||
|
- `database-maintenance-completed`
|
||||||
|
- `database-maintenance-error`
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
Create `backend/tests/test_database_maintenance.py`.
|
||||||
|
|
||||||
|
Minimum cases:
|
||||||
|
|
||||||
|
- dry-run does not delete.
|
||||||
|
- apply deletes old server snapshots.
|
||||||
|
- apply deletes old non-critical admin log events.
|
||||||
|
- apply preserves critical events within retention.
|
||||||
|
- apply preserves latest 100 materialized matches.
|
||||||
|
- apply preserves current month matches.
|
||||||
|
- apply preserves previous month when now.day <= 7.
|
||||||
|
- apply preserves current week.
|
||||||
|
- apply preserves previous week when fallback protection applies.
|
||||||
|
- apply deletes an old non-protected materialized match.
|
||||||
|
- child/player stats for deleted matches are deleted first or handled without orphans.
|
||||||
|
- missing optional tables are logged and do not crash cleanup.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
PYTHONPATH=backend python -m unittest backend.tests.test_database_maintenance
|
||||||
|
python -m compileall backend/app
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Also manually run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
docker compose exec backend python -m app.database_maintenance cleanup --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
or, if backend container is unavailable:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd backend
|
||||||
|
python -m app.database_maintenance cleanup --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
Document which one was used.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Document:
|
||||||
|
|
||||||
|
- validation commands run;
|
||||||
|
- dry-run output summary;
|
||||||
|
- any existing unrelated test failures;
|
||||||
|
- exact files changed;
|
||||||
|
- any follow-up task needed instead of expanding scope.
|
||||||
|
|
||||||
|
Codex CLI must commit and push the completed task branch.
|
||||||
|
|
||||||
|
Suggested implementation branch:
|
||||||
|
|
||||||
|
`task/database-maintenance-cleanup-command`
|
||||||
|
|
||||||
|
Suggested commit message:
|
||||||
|
|
||||||
|
`feat: add database maintenance cleanup command`
|
||||||
|
|
||||||
|
## Change Budget
|
||||||
|
|
||||||
|
Prefer fewer than 5 modified files.
|
||||||
|
Prefer focused backend-only changes.
|
||||||
|
Split follow-up tasks if implementation exceeds safe scope.
|
||||||
152
ai/tasks/pending/TASK-database-maintenance-docs.md
Normal file
152
ai/tasks/pending/TASK-database-maintenance-docs.md
Normal file
@@ -0,0 +1,152 @@
|
|||||||
|
---
|
||||||
|
id: TASK-database-maintenance-docs
|
||||||
|
title: Document database maintenance operations
|
||||||
|
status: pending
|
||||||
|
type: documentation
|
||||||
|
team: PM
|
||||||
|
supporting_teams: [Backend Senior, Arquitecto de Base de Datos]
|
||||||
|
roadmap_item: historical-data-maintenance
|
||||||
|
priority: medium
|
||||||
|
---
|
||||||
|
|
||||||
|
# TASK-database-maintenance-docs - Document database maintenance operations
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Document how database maintenance works, how to audit database growth, and how to run dry-run/apply safely.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Database cleanup will be implemented as application-level maintenance and scheduled inside historical-runner.
|
||||||
|
|
||||||
|
Operators must understand:
|
||||||
|
|
||||||
|
- what data is deleted;
|
||||||
|
- what data is protected;
|
||||||
|
- how to preview deletes;
|
||||||
|
- how to enable/disable scheduled cleanup;
|
||||||
|
- how to verify table sizes;
|
||||||
|
- what not to do in Docker/Portainer.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Inspect the listed files first.
|
||||||
|
2. Add or update operational documentation.
|
||||||
|
3. Include exact commands for PowerShell/Docker.
|
||||||
|
4. Include safety warnings.
|
||||||
|
5. Validate docs diff.
|
||||||
|
6. Commit and push the task branch.
|
||||||
|
|
||||||
|
## Files to Read First
|
||||||
|
|
||||||
|
- `README.md`
|
||||||
|
- `backend/README.md`
|
||||||
|
- `ai/README.md`
|
||||||
|
- `backend/app/database_maintenance.py`
|
||||||
|
- `backend/app/historical_runner.py`
|
||||||
|
- `docker-compose.yml`
|
||||||
|
|
||||||
|
## Expected Files to Modify
|
||||||
|
|
||||||
|
Preferred:
|
||||||
|
|
||||||
|
- `docs/database-maintenance.md`
|
||||||
|
|
||||||
|
Also update if relevant:
|
||||||
|
|
||||||
|
- `README.md`
|
||||||
|
- `backend/README.md`
|
||||||
|
|
||||||
|
Do not modify runtime code in this task.
|
||||||
|
|
||||||
|
## Required Documentation Content
|
||||||
|
|
||||||
|
Include sections:
|
||||||
|
|
||||||
|
- Overview
|
||||||
|
- Why cleanup is application-level and not `pg_cron`
|
||||||
|
- Scheduled cleanup inside `historical-runner`
|
||||||
|
- Environment variables
|
||||||
|
- Protected data
|
||||||
|
- Deleted data
|
||||||
|
- Dry-run command
|
||||||
|
- Apply command
|
||||||
|
- Table-size audit SQL
|
||||||
|
- Row-count/age audit SQL
|
||||||
|
- Logs to inspect
|
||||||
|
- Docker/Portainer warnings
|
||||||
|
- Rollback/restore considerations
|
||||||
|
|
||||||
|
Required environment variables:
|
||||||
|
|
||||||
|
- `HLL_DB_MAINTENANCE_ENABLED=false`
|
||||||
|
- `HLL_DB_MAINTENANCE_INTERVAL_SECONDS=43200`
|
||||||
|
- `HLL_RECENT_MATCHES_KEEP=100`
|
||||||
|
- `HLL_ADMIN_LOG_NONCRITICAL_RETENTION_DAYS=30`
|
||||||
|
- `HLL_ADMIN_LOG_CRITICAL_RETENTION_DAYS=90`
|
||||||
|
- `HLL_SERVER_SNAPSHOT_RETENTION_DAYS=14`
|
||||||
|
- `HLL_DB_MAINTENANCE_BATCH_SIZE=5000`
|
||||||
|
|
||||||
|
Required warnings:
|
||||||
|
|
||||||
|
- Never use `docker compose down -v` unless intentionally deleting volumes.
|
||||||
|
- Always review dry-run output before enabling apply in production.
|
||||||
|
- Do not manually delete protected match/stat rows from PostgreSQL.
|
||||||
|
- Keep backups before changing retention settings.
|
||||||
|
- Do not add Comunidad Hispana #03 back into RCON targets in this task.
|
||||||
|
|
||||||
|
Required SQL:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
select
|
||||||
|
schemaname,
|
||||||
|
relname as table_name,
|
||||||
|
pg_size_pretty(pg_total_relation_size(relid)) as total_size,
|
||||||
|
pg_size_pretty(pg_relation_size(relid)) as table_size,
|
||||||
|
pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as indexes_size,
|
||||||
|
n_live_tup as estimated_rows,
|
||||||
|
n_dead_tup as estimated_dead_rows
|
||||||
|
from pg_stat_user_tables
|
||||||
|
order by pg_total_relation_size(relid) desc;
|
||||||
|
```
|
||||||
|
|
||||||
|
Also include table-specific examples for:
|
||||||
|
|
||||||
|
- admin log events by type/date;
|
||||||
|
- materialized matches by server/date;
|
||||||
|
- server snapshots by date;
|
||||||
|
- displayed snapshots count.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
If README files are touched, ensure examples are consistent with the current Compose profiles.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Document:
|
||||||
|
|
||||||
|
- files changed;
|
||||||
|
- docs sections added;
|
||||||
|
- validation performed;
|
||||||
|
- any follow-up needed.
|
||||||
|
|
||||||
|
Codex CLI must commit and push the completed task branch.
|
||||||
|
|
||||||
|
Suggested implementation branch:
|
||||||
|
|
||||||
|
`task/database-maintenance-docs`
|
||||||
|
|
||||||
|
Suggested commit message:
|
||||||
|
|
||||||
|
`docs: document database maintenance operations`
|
||||||
|
|
||||||
|
## Change Budget
|
||||||
|
|
||||||
|
- Documentation only.
|
||||||
|
- Prefer fewer than 3 modified files.
|
||||||
184
ai/tasks/pending/TASK-db-retention-audit.md
Normal file
184
ai/tasks/pending/TASK-db-retention-audit.md
Normal file
@@ -0,0 +1,184 @@
|
|||||||
|
---
|
||||||
|
id: TASK-db-retention-audit
|
||||||
|
title: Audit database retention strategy
|
||||||
|
status: pending
|
||||||
|
type: research
|
||||||
|
team: Arquitecto de Base de Datos
|
||||||
|
supporting_teams: [Backend Senior, Arquitecto Python, PM]
|
||||||
|
roadmap_item: historical-data-maintenance
|
||||||
|
priority: high
|
||||||
|
---
|
||||||
|
|
||||||
|
# TASK-db-retention-audit - Audit database retention strategy
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Audit the current HLL Vietnam database/storage model and produce a precise retention strategy before any destructive cleanup is implemented.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The project stores live and historical RCON data, materialized matches, precomputed snapshots, player/event data and server snapshots.
|
||||||
|
|
||||||
|
We need to prevent the database from growing indefinitely, but we must not delete data required for:
|
||||||
|
|
||||||
|
- the latest 100 visible matches;
|
||||||
|
- current weekly leaderboards;
|
||||||
|
- previous-week fallback leaderboards;
|
||||||
|
- current monthly leaderboards;
|
||||||
|
- previous-month fallback leaderboards during the first 7 days of a new month;
|
||||||
|
- current or recent RCON materialization;
|
||||||
|
- current-match/live pages.
|
||||||
|
|
||||||
|
Important architectural decision:
|
||||||
|
Do not use native database scheduling, `pg_cron` or an external cron service for now. The future cleanup should be application-level, versioned, logged and executed from the existing `historical-runner`.
|
||||||
|
|
||||||
|
Known snapshot context:
|
||||||
|
Precomputed display snapshots should already be bounded because PostgreSQL stores them in `displayed_historical_snapshots` using a logical primary key `(server_key, snapshot_type, metric, snapshot_window)` and persistence uses upsert semantics. File-based snapshots also use deterministic paths by server/type/metric. This task must verify that assumption against the current code.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Inspect the listed files first.
|
||||||
|
2. Identify all tables/files involved in:
|
||||||
|
- RCON admin log events;
|
||||||
|
- RCON materialized matches;
|
||||||
|
- materialized player stats;
|
||||||
|
- server snapshots;
|
||||||
|
- displayed historical snapshots;
|
||||||
|
- player event raw ledger;
|
||||||
|
- Elo/MMR tables if present;
|
||||||
|
- monthly rankings/checkpoints if present.
|
||||||
|
3. Determine which data is safe to delete and when.
|
||||||
|
4. Determine which data must be protected.
|
||||||
|
5. Determine the safe delete order to avoid orphan data.
|
||||||
|
6. Determine whether any foreign keys/cascade rules exist or are missing.
|
||||||
|
7. Create a concise retention audit document.
|
||||||
|
|
||||||
|
## Files to Read First
|
||||||
|
|
||||||
|
- `AGENTS.md`
|
||||||
|
- `ai/repo-context.md`
|
||||||
|
- `ai/architecture-index.md`
|
||||||
|
- `backend/app/postgres_display_storage.py`
|
||||||
|
- `backend/app/historical_snapshot_storage.py`
|
||||||
|
- `backend/app/historical_runner.py`
|
||||||
|
- `backend/app/rcon_historical_worker.py`
|
||||||
|
- `backend/app/rcon_historical_read_model.py`
|
||||||
|
- `backend/app/rcon_historical_backfill.py`
|
||||||
|
- `backend/app/player_event_aggregates.py`
|
||||||
|
|
||||||
|
## Expected Files to Modify
|
||||||
|
|
||||||
|
- `ai/tasks/pending/TASK-db-retention-audit.md`
|
||||||
|
- `ai/reports/database-retention-audit.md`
|
||||||
|
|
||||||
|
If `ai/reports/*.md` is ignored by Git, create the report in:
|
||||||
|
|
||||||
|
- `ai/tasks/review/TASK-db-retention-audit-report.md`
|
||||||
|
|
||||||
|
and explain why in the outcome.
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- Do not implement cleanup logic in this task.
|
||||||
|
- Do not delete data.
|
||||||
|
- Do not modify backend runtime behavior.
|
||||||
|
- Do not add Docker services.
|
||||||
|
- Do not touch `main` directly.
|
||||||
|
- Keep the report factual and based on the actual schema/code.
|
||||||
|
- Clearly separate confirmed facts from assumptions.
|
||||||
|
- Do not expand Elo/MMR functionality.
|
||||||
|
|
||||||
|
## Retention Rules To Validate
|
||||||
|
|
||||||
|
The future cleanup must protect:
|
||||||
|
|
||||||
|
- latest 100 closed/materialized matches;
|
||||||
|
- all matches from current month;
|
||||||
|
- all matches from previous month while `now.day <= 7`;
|
||||||
|
- all matches from current week;
|
||||||
|
- all matches from previous week if weekly fallback logic may still need them;
|
||||||
|
- child/player-stat rows belonging to protected matches;
|
||||||
|
- snapshot records currently addressed by frontend endpoints;
|
||||||
|
- current RCON/live data.
|
||||||
|
|
||||||
|
Candidate cleanup areas:
|
||||||
|
|
||||||
|
- old `server_snapshots`;
|
||||||
|
- non-critical old `rcon_admin_log_events`;
|
||||||
|
- old critical admin log events only after retention and protection checks;
|
||||||
|
- old materialized matches not protected by latest-100/month/week rules;
|
||||||
|
- child stats for deleted materialized matches;
|
||||||
|
- legacy/corrupt duplicate snapshots only if actually found.
|
||||||
|
|
||||||
|
AdminLog critical events:
|
||||||
|
|
||||||
|
- `kill`
|
||||||
|
- `match_start`
|
||||||
|
- `match_end`
|
||||||
|
|
||||||
|
AdminLog non-critical events:
|
||||||
|
|
||||||
|
- `chat`
|
||||||
|
- `message`
|
||||||
|
- `connected`
|
||||||
|
- `disconnected`
|
||||||
|
- `team_switch`
|
||||||
|
- `unknown`
|
||||||
|
- `kick`
|
||||||
|
- `ban`
|
||||||
|
- any other event not required for materialized stats/rankings.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
python -m compileall backend/app
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Also include recommended SQL audit commands in the report:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
select
|
||||||
|
schemaname,
|
||||||
|
relname as table_name,
|
||||||
|
pg_size_pretty(pg_total_relation_size(relid)) as total_size,
|
||||||
|
pg_size_pretty(pg_relation_size(relid)) as table_size,
|
||||||
|
pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as indexes_size,
|
||||||
|
n_live_tup as estimated_rows,
|
||||||
|
n_dead_tup as estimated_dead_rows
|
||||||
|
from pg_stat_user_tables
|
||||||
|
order by pg_total_relation_size(relid) desc;
|
||||||
|
```
|
||||||
|
|
||||||
|
and table-specific row/age queries for the actual tables found.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Document:
|
||||||
|
|
||||||
|
- files inspected;
|
||||||
|
- tables/storage areas audited;
|
||||||
|
- safe-to-delete data;
|
||||||
|
- protected data;
|
||||||
|
- required delete order;
|
||||||
|
- risks;
|
||||||
|
- missing indexes/constraints if any;
|
||||||
|
- recommended implementation plan for the cleanup command.
|
||||||
|
|
||||||
|
Codex CLI must commit and push the completed task branch.
|
||||||
|
|
||||||
|
Suggested implementation branch:
|
||||||
|
|
||||||
|
`task/db-retention-audit`
|
||||||
|
|
||||||
|
Suggested commit message:
|
||||||
|
|
||||||
|
`docs: audit database retention strategy`
|
||||||
|
|
||||||
|
## Change Budget
|
||||||
|
|
||||||
|
Prefer fewer than 3 modified files.
|
||||||
|
No runtime changes.
|
||||||
|
No product code changes.
|
||||||
145
ai/tasks/pending/TASK-final-maintenance-and-copy-validation.md
Normal file
145
ai/tasks/pending/TASK-final-maintenance-and-copy-validation.md
Normal file
@@ -0,0 +1,145 @@
|
|||||||
|
---
|
||||||
|
id: TASK-final-maintenance-and-copy-validation
|
||||||
|
title: Final maintenance and copy validation
|
||||||
|
status: pending
|
||||||
|
type: platform
|
||||||
|
team: PM
|
||||||
|
supporting_teams: [Backend Senior, Frontend Senior, Arquitecto de Base de Datos]
|
||||||
|
roadmap_item: validation
|
||||||
|
priority: high
|
||||||
|
---
|
||||||
|
|
||||||
|
# TASK-final-maintenance-and-copy-validation - Final maintenance and copy validation
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Validate the database maintenance implementation, historical-runner integration, documentation and home copy update together before merge to main.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
This task is a final validation task. It should not expand scope. It should verify that previous tasks work together and that the branch is safe to push/merge.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Inspect the listed files first.
|
||||||
|
2. Run backend validations.
|
||||||
|
3. Run frontend validations.
|
||||||
|
4. Run Docker/local smoke checks where available.
|
||||||
|
5. Verify no unexpected files changed.
|
||||||
|
6. Commit/push only if this task produces documentation or small validation notes. Otherwise report validation results.
|
||||||
|
|
||||||
|
## Files to Read First
|
||||||
|
|
||||||
|
- `AGENTS.md`
|
||||||
|
- `ai/tasks/done/`
|
||||||
|
- `backend/app/database_maintenance.py`
|
||||||
|
- `backend/app/historical_runner.py`
|
||||||
|
- `docs/database-maintenance.md`
|
||||||
|
- `frontend/index.html`
|
||||||
|
|
||||||
|
## Expected Files to Modify
|
||||||
|
|
||||||
|
Normally none.
|
||||||
|
|
||||||
|
Optional only if the project convention requires a validation note:
|
||||||
|
|
||||||
|
- `ai/tasks/review/TASK-final-maintenance-and-copy-validation-report.md`
|
||||||
|
|
||||||
|
Do not modify runtime code unless a blocking validation issue is discovered. If a bug is discovered, create a follow-up task instead of expanding this validation task.
|
||||||
|
|
||||||
|
## Required Validation Commands
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
python -m compileall backend/app
|
||||||
|
PYTHONPATH=backend python -m unittest backend.tests.test_database_maintenance
|
||||||
|
PYTHONPATH=backend python -m unittest backend.tests.test_historical_runner_maintenance
|
||||||
|
PYTHONPATH=backend python -m unittest discover -s backend/tests -p "*historical*"
|
||||||
|
node --check frontend/assets/js/main.js
|
||||||
|
node --check frontend/assets/js/historico.js
|
||||||
|
node --check frontend/assets/js/partida-actual.js
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
If a known unrelated test fails, document:
|
||||||
|
|
||||||
|
- exact failing test;
|
||||||
|
- why it is unrelated;
|
||||||
|
- whether this task introduced it.
|
||||||
|
|
||||||
|
Docker/local smoke checks:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
docker compose up -d --build backend frontend postgres
|
||||||
|
docker compose --profile advanced up -d --build historical-runner rcon-historical-worker
|
||||||
|
docker compose --profile advanced ps
|
||||||
|
docker compose logs --tail=200 historical-runner
|
||||||
|
```
|
||||||
|
|
||||||
|
Endpoint checks:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
Invoke-RestMethod "http://127.0.0.1:8000/api/historical/snapshots/recent-matches?server=all-servers&limit=10" | ConvertTo-Json -Depth 10
|
||||||
|
Invoke-RestMethod "http://127.0.0.1:8000/api/historical/snapshots/leaderboard?server=all-servers&timeframe=weekly&metric=kills&limit=10" | ConvertTo-Json -Depth 10
|
||||||
|
Invoke-RestMethod "http://127.0.0.1:8000/api/historical/snapshots/leaderboard?server=all-servers&timeframe=monthly&metric=kills&limit=10" | ConvertTo-Json -Depth 10
|
||||||
|
```
|
||||||
|
|
||||||
|
Maintenance dry-run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
docker compose exec backend python -m app.database_maintenance cleanup --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not run destructive production apply during this validation unless the user explicitly asks.
|
||||||
|
|
||||||
|
Frontend visual checks:
|
||||||
|
|
||||||
|
`http://localhost:8080/?v=maintenance-copy-validation`
|
||||||
|
|
||||||
|
`http://localhost:8080/historico.html?v=maintenance-copy-validation`
|
||||||
|
|
||||||
|
`http://localhost:8080/partida-actual.html?server=comunidad-hispana-01&v=maintenance-copy-validation`
|
||||||
|
|
||||||
|
Expected frontend result:
|
||||||
|
|
||||||
|
- home does not show “HLL Vietnam” in the main hero;
|
||||||
|
- trailer still says “HLL Vietnam”;
|
||||||
|
- historical page loads;
|
||||||
|
- current match page loads.
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- Do not add new services.
|
||||||
|
- Do not touch main directly.
|
||||||
|
- Do not run production destructive cleanup.
|
||||||
|
- Do not expand scope.
|
||||||
|
- Do not reintroduce RCON server #03.
|
||||||
|
- Do not modify unrelated UI.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Document:
|
||||||
|
|
||||||
|
- validation commands run;
|
||||||
|
- pass/fail result;
|
||||||
|
- Docker status;
|
||||||
|
- maintenance dry-run summary;
|
||||||
|
- endpoint summaries;
|
||||||
|
- any unrelated failures;
|
||||||
|
- whether branch was pushed.
|
||||||
|
|
||||||
|
Codex CLI must push the completed validation branch if any files changed.
|
||||||
|
|
||||||
|
Suggested implementation branch:
|
||||||
|
|
||||||
|
`task/final-maintenance-and-copy-validation`
|
||||||
|
|
||||||
|
Suggested commit message if a validation note is created:
|
||||||
|
|
||||||
|
`test: validate database maintenance and home copy`
|
||||||
|
|
||||||
|
## Change Budget
|
||||||
|
|
||||||
|
- Prefer no code changes.
|
||||||
|
- Validation/report only.
|
||||||
160
ai/tasks/pending/TASK-historical-runner-maintenance-scheduler.md
Normal file
160
ai/tasks/pending/TASK-historical-runner-maintenance-scheduler.md
Normal file
@@ -0,0 +1,160 @@
|
|||||||
|
---
|
||||||
|
id: TASK-historical-runner-maintenance-scheduler
|
||||||
|
title: Schedule database maintenance inside historical runner
|
||||||
|
status: pending
|
||||||
|
type: backend
|
||||||
|
team: Arquitecto Python
|
||||||
|
supporting_teams: [Backend Senior, Arquitecto de Base de Datos]
|
||||||
|
roadmap_item: historical-data-maintenance
|
||||||
|
priority: high
|
||||||
|
---
|
||||||
|
|
||||||
|
# TASK-historical-runner-maintenance-scheduler - Schedule database maintenance inside historical runner
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Integrate the database maintenance cleanup command into the existing historical-runner loop without adding a new Docker service or external scheduler.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The project already has historical-runner as an advanced service. We want database maintenance to run as an internal assistant task from that runner.
|
||||||
|
|
||||||
|
No new container/service should be added.
|
||||||
|
|
||||||
|
The runner must continue working even if maintenance fails.
|
||||||
|
|
||||||
|
The maintenance must be disabled by default and enabled only by environment configuration.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Inspect the listed files first.
|
||||||
|
2. Add configuration for maintenance scheduler.
|
||||||
|
3. Integrate a due-check into historical-runner.
|
||||||
|
4. Call the cleanup command in apply mode only when enabled and due.
|
||||||
|
5. Ensure cleanup failures do not crash the runner.
|
||||||
|
6. Add tests.
|
||||||
|
7. Validate.
|
||||||
|
8. Commit and push the task branch.
|
||||||
|
|
||||||
|
## Files to Read First
|
||||||
|
|
||||||
|
- `AGENTS.md`
|
||||||
|
- `backend/app/historical_runner.py`
|
||||||
|
- `backend/app/config.py`
|
||||||
|
- `backend/app/database_maintenance.py`
|
||||||
|
- `backend/tests/`
|
||||||
|
- `docker-compose.yml`
|
||||||
|
- `deploy/portainer/docker-compose.nas.yml` if present
|
||||||
|
|
||||||
|
## Expected Files to Modify
|
||||||
|
|
||||||
|
- `backend/app/historical_runner.py`
|
||||||
|
- `backend/app/config.py`
|
||||||
|
- `backend/tests/test_historical_runner_maintenance.py`
|
||||||
|
|
||||||
|
Optional only if documentation is needed for env examples:
|
||||||
|
|
||||||
|
- `backend/.env.example`
|
||||||
|
|
||||||
|
Do not add a new Docker service in this task.
|
||||||
|
|
||||||
|
## Functional Requirements
|
||||||
|
|
||||||
|
Add env/config:
|
||||||
|
|
||||||
|
- `HLL_DB_MAINTENANCE_ENABLED=false`
|
||||||
|
- `HLL_DB_MAINTENANCE_INTERVAL_SECONDS=43200`
|
||||||
|
- `HLL_RECENT_MATCHES_KEEP=100`
|
||||||
|
- `HLL_ADMIN_LOG_NONCRITICAL_RETENTION_DAYS=30`
|
||||||
|
- `HLL_ADMIN_LOG_CRITICAL_RETENTION_DAYS=90`
|
||||||
|
- `HLL_SERVER_SNAPSHOT_RETENTION_DAYS=14`
|
||||||
|
- `HLL_DB_MAINTENANCE_BATCH_SIZE=5000`
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- if `HLL_DB_MAINTENANCE_ENABLED` is false or missing, runner logs skip and does nothing;
|
||||||
|
- if enabled, runner checks whether maintenance is due;
|
||||||
|
- default interval is 12 hours;
|
||||||
|
- if due, runner invokes database maintenance apply mode;
|
||||||
|
- if not due, runner logs not-due at low verbosity or structured event;
|
||||||
|
- if maintenance fails, runner logs error and continues normal historical processing;
|
||||||
|
- maintenance must not run concurrently with snapshot refresh/backfill if existing writer locks apply;
|
||||||
|
- if lock is busy, skip or fail gracefully without leaving runner stuck.
|
||||||
|
|
||||||
|
Required logs:
|
||||||
|
|
||||||
|
- `database-maintenance-scheduler-skipped-disabled`
|
||||||
|
- `database-maintenance-scheduler-skipped-not-due`
|
||||||
|
- `database-maintenance-scheduler-started`
|
||||||
|
- `database-maintenance-scheduler-completed`
|
||||||
|
- `database-maintenance-scheduler-failed`
|
||||||
|
- `database-maintenance-scheduler-lock-busy` if applicable
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- Do not add `database-maintenance` as a new compose service.
|
||||||
|
- Do not use `pg_cron`.
|
||||||
|
- Do not use host cron.
|
||||||
|
- Do not change frontend.
|
||||||
|
- Do not change public API contracts.
|
||||||
|
- Do not run cleanup by default unless env explicitly enables it.
|
||||||
|
- Do not block the runner indefinitely.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
Create tests for:
|
||||||
|
|
||||||
|
- scheduler disabled does not call cleanup;
|
||||||
|
- scheduler enabled but not due does not call cleanup;
|
||||||
|
- scheduler enabled and due calls cleanup;
|
||||||
|
- cleanup exception is logged and runner continues;
|
||||||
|
- interval parsing handles invalid values safely;
|
||||||
|
- maintenance state is tracked in-process or persisted safely according to existing runner design.
|
||||||
|
|
||||||
|
Use mocks/stubs to avoid real destructive cleanup in scheduler tests.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
PYTHONPATH=backend python -m unittest backend.tests.test_historical_runner_maintenance
|
||||||
|
PYTHONPATH=backend python -m unittest backend.tests.test_database_maintenance
|
||||||
|
python -m compileall backend/app
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Optional runtime validation:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
docker compose --profile advanced up -d --build historical-runner
|
||||||
|
docker compose logs --tail=200 historical-runner
|
||||||
|
```
|
||||||
|
|
||||||
|
Only enable maintenance in local test with a short interval and safe dataset.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Document:
|
||||||
|
|
||||||
|
- validation commands run;
|
||||||
|
- whether a local runner smoke test was performed;
|
||||||
|
- logs observed;
|
||||||
|
- exact files changed;
|
||||||
|
- any follow-up task needed.
|
||||||
|
|
||||||
|
Codex CLI must commit and push the completed task branch.
|
||||||
|
|
||||||
|
Suggested implementation branch:
|
||||||
|
|
||||||
|
`task/historical-runner-maintenance-scheduler`
|
||||||
|
|
||||||
|
Suggested commit message:
|
||||||
|
|
||||||
|
`feat: schedule database maintenance in historical runner`
|
||||||
|
|
||||||
|
## Change Budget
|
||||||
|
|
||||||
|
- Prefer fewer than 4 modified files.
|
||||||
|
- No Docker service addition.
|
||||||
|
- Keep scheduling logic small and testable.
|
||||||
139
ai/tasks/pending/TASK-home-remove-vietnam-copy-except-trailer.md
Normal file
139
ai/tasks/pending/TASK-home-remove-vietnam-copy-except-trailer.md
Normal file
@@ -0,0 +1,139 @@
|
|||||||
|
---
|
||||||
|
id: TASK-home-remove-vietnam-copy-except-trailer
|
||||||
|
title: Remove Vietnam copy from home except trailer
|
||||||
|
status: pending
|
||||||
|
type: frontend
|
||||||
|
team: Frontend Senior
|
||||||
|
supporting_teams: [Experto en interfaz]
|
||||||
|
roadmap_item: frontend-copy-adjustments
|
||||||
|
priority: medium
|
||||||
|
---
|
||||||
|
|
||||||
|
# TASK-home-remove-vietnam-copy-except-trailer - Remove Vietnam copy from home except trailer
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Temporarily remove “Vietnam” from the home page branding copy, except in the trailer block.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The home page should no longer show “HLL Vietnam” in the main community branding. The trailer section must continue to refer to HLL Vietnam because the trailer itself is still about HLL Vietnam.
|
||||||
|
|
||||||
|
This is a small frontend copy-only change.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. Inspect the listed files first.
|
||||||
|
2. Modify only the required home page copy.
|
||||||
|
3. Preserve trailer copy.
|
||||||
|
4. Validate diff and encoding.
|
||||||
|
5. Commit and push the task branch.
|
||||||
|
|
||||||
|
## Files to Read First
|
||||||
|
|
||||||
|
- `AGENTS.md`
|
||||||
|
- `frontend/index.html`
|
||||||
|
- `frontend/assets/css/styles.css`
|
||||||
|
- `frontend/assets/css/hero-header-compact.css`
|
||||||
|
|
||||||
|
## Expected Files to Modify
|
||||||
|
|
||||||
|
- `frontend/index.html`
|
||||||
|
|
||||||
|
Do not modify CSS unless the copy change causes an actual layout issue. If CSS is needed, explain why in the outcome.
|
||||||
|
|
||||||
|
## Exact Required Changes
|
||||||
|
|
||||||
|
In `frontend/index.html`, change:
|
||||||
|
|
||||||
|
`content="Comunidad hispana de HLL Vietnam. Accede al Discord oficial y descubre el trailer del proyecto."`
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
`content="Comunidad hispana de HLL. Accede al Discord oficial y descubre el trailer del proyecto."`
|
||||||
|
|
||||||
|
Change:
|
||||||
|
|
||||||
|
`<title>Comunidad Hispana - HLL Vietnam</title>`
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
`<title>Comunidad Hispana - HLL</title>`
|
||||||
|
|
||||||
|
Change:
|
||||||
|
|
||||||
|
`alt="Logo oficial de la comunidad HLL Vietnam"`
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
`alt="Logo oficial de la comunidad HLL"`
|
||||||
|
|
||||||
|
Change:
|
||||||
|
|
||||||
|
`<span class="hero__title-accent">HLL Vietnam</span>`
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
`<span class="hero__title-accent">HLL</span>`
|
||||||
|
|
||||||
|
Do not change:
|
||||||
|
|
||||||
|
`Primer vistazo a HLL Vietnam`
|
||||||
|
|
||||||
|
Do not change:
|
||||||
|
|
||||||
|
`Trailer HLL Vietnam`
|
||||||
|
|
||||||
|
Do not change the trailer iframe URL.
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- Do not change backend.
|
||||||
|
- Do not change JS.
|
||||||
|
- Do not change layout unless required by the shorter title.
|
||||||
|
- Do not corrupt UTF-8.
|
||||||
|
- Do not introduce BOM if the existing file does not require one.
|
||||||
|
- Do not touch unrelated pages.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
git diff -- frontend/index.html
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Manual check:
|
||||||
|
|
||||||
|
`http://localhost:8080/?v=home-copy`
|
||||||
|
|
||||||
|
Expected:
|
||||||
|
|
||||||
|
- hero title says Comunidad Hispana HLL;
|
||||||
|
- page title says Comunidad Hispana - HLL;
|
||||||
|
- trailer still says Primer vistazo a HLL Vietnam;
|
||||||
|
- iframe title still says Trailer HLL Vietnam.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Document:
|
||||||
|
|
||||||
|
- exact copy changed;
|
||||||
|
- validation commands run;
|
||||||
|
- whether local visual check was performed.
|
||||||
|
|
||||||
|
Codex CLI must commit and push the completed task branch.
|
||||||
|
|
||||||
|
Suggested implementation branch:
|
||||||
|
|
||||||
|
`task/home-remove-vietnam-copy-except-trailer`
|
||||||
|
|
||||||
|
Suggested commit message:
|
||||||
|
|
||||||
|
`fix: remove Vietnam copy from home except trailer`
|
||||||
|
|
||||||
|
## Change Budget
|
||||||
|
|
||||||
|
- One modified file expected.
|
||||||
|
- No runtime logic changes.
|
||||||
Reference in New Issue
Block a user