feat: add database maintenance and update home copy
This commit is contained in:
devRaGonSa
236
ai/tasks/done/TASK-database-maintenance-cleanup-command.md
Normal file
236
ai/tasks/done/TASK-database-maintenance-cleanup-command.md
Normal file
@@ -0,0 +1,236 @@
|
||||
---
|
||||
id: TASK-database-maintenance-cleanup-command
|
||||
title: Add database maintenance cleanup command
|
||||
status: pending
|
||||
type: backend
|
||||
team: Backend Senior
|
||||
supporting_teams: [Arquitecto de Base de Datos, Arquitecto Python]
|
||||
roadmap_item: historical-data-maintenance
|
||||
priority: high
|
||||
---
|
||||
|
||||
# TASK-database-maintenance-cleanup-command - Add database maintenance cleanup command
|
||||
|
||||
## Goal
|
||||
|
||||
Implement a safe backend cleanup command with dry-run/apply modes to remove database data that is no longer needed while preserving all data required for visible history and leaderboards.
|
||||
|
||||
## Context
|
||||
|
||||
The project should keep the database bounded without manually deleting data.
|
||||
|
||||
The command will later be scheduled from historical-runner, but this task only implements the cleanup module and tests.
|
||||
|
||||
The cleanup must not rely on native PostgreSQL scheduling or a new Docker service.
|
||||
|
||||
The cleanup must protect:
|
||||
|
||||
- latest 100 closed/materialized matches;
|
||||
- current month;
|
||||
- previous month during the first 7 days of the new month;
|
||||
- current week;
|
||||
- previous week if fallback weekly logic may need it;
|
||||
- child/player stats for protected matches;
|
||||
- current live/RCON data.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Inspect the listed files first.
|
||||
2. Add a backend module for database maintenance.
|
||||
3. Provide a CLI entry point:
|
||||
- `python -m app.database_maintenance cleanup --dry-run`
|
||||
- `python -m app.database_maintenance cleanup --apply`
|
||||
4. Make dry-run the default behavior.
|
||||
5. Add JSON logs for planning, batches and completion.
|
||||
6. Add tests.
|
||||
7. Validate locally.
|
||||
8. Commit and push the task branch.
|
||||
|
||||
## Files to Read First
|
||||
|
||||
- `AGENTS.md`
|
||||
- `backend/app/config.py`
|
||||
- `backend/app/postgres_display_storage.py`
|
||||
- `backend/app/historical_runner.py`
|
||||
- `backend/app/rcon_historical_read_model.py`
|
||||
- `backend/app/rcon_historical_backfill.py`
|
||||
- `backend/tests/`
|
||||
|
||||
## Expected Files to Modify
|
||||
|
||||
- `backend/app/database_maintenance.py`
|
||||
- `backend/app/config.py`
|
||||
- `backend/tests/test_database_maintenance.py`
|
||||
|
||||
Optional only if required by existing test helpers:
|
||||
|
||||
- `backend/tests/conftest.py`
|
||||
- `backend/tests/test_*.py`
|
||||
|
||||
Do not modify frontend files in this task.
|
||||
|
||||
## Functional Requirements
|
||||
|
||||
Create a cleanup command:
|
||||
|
||||
```powershell
|
||||
python -m app.database_maintenance cleanup --dry-run
|
||||
python -m app.database_maintenance cleanup --apply
|
||||
```
|
||||
|
||||
Supported options:
|
||||
|
||||
- `--recent-matches-keep`
|
||||
- `--admin-log-noncritical-retention-days`
|
||||
- `--admin-log-critical-retention-days`
|
||||
- `--server-snapshot-retention-days`
|
||||
- `--batch-size`
|
||||
- `--vacuum-analyze`
|
||||
- `--now`
|
||||
|
||||
Environment defaults:
|
||||
|
||||
- `HLL_RECENT_MATCHES_KEEP=100`
|
||||
- `HLL_ADMIN_LOG_NONCRITICAL_RETENTION_DAYS=30`
|
||||
- `HLL_ADMIN_LOG_CRITICAL_RETENTION_DAYS=90`
|
||||
- `HLL_SERVER_SNAPSHOT_RETENTION_DAYS=14`
|
||||
- `HLL_DB_MAINTENANCE_BATCH_SIZE=5000`
|
||||
|
||||
Dry-run behavior:
|
||||
|
||||
- calculate all candidate rows;
|
||||
- do not delete anything;
|
||||
- log exactly what would be deleted;
|
||||
- return status ok.
|
||||
|
||||
Apply behavior:
|
||||
|
||||
- delete only candidate rows;
|
||||
- delete in batches;
|
||||
- use transactions safely;
|
||||
- do not block indefinitely;
|
||||
- log deleted row counts;
|
||||
- return status ok even if deleted counts are zero.
|
||||
|
||||
Retention Logic
|
||||
|
||||
Protected materialized matches:
|
||||
|
||||
- latest N closed matches, default N = 100;
|
||||
- current month matches;
|
||||
- previous month matches if now.day <= 7;
|
||||
- current week matches;
|
||||
- previous week matches if weekly fallback may need them;
|
||||
- any match whose end/start timestamp cannot be safely interpreted should be skipped/protected, not deleted.
|
||||
|
||||
Delete materialized matches only if:
|
||||
|
||||
- not in protected latest N;
|
||||
- not in protected current/previous month window;
|
||||
- not in protected current/previous week window;
|
||||
- older than the relevant protected windows;
|
||||
- child stats/events that depend on the match are handled first.
|
||||
|
||||
AdminLog critical events:
|
||||
|
||||
- `kill`
|
||||
- `match_start`
|
||||
- `match_end`
|
||||
|
||||
Critical events may only be deleted after `HLL_ADMIN_LOG_CRITICAL_RETENTION_DAYS` and only if they are not needed for protected materialized matches/windows.
|
||||
|
||||
AdminLog non-critical events may be deleted after `HLL_ADMIN_LOG_NONCRITICAL_RETENTION_DAYS`.
|
||||
|
||||
Server snapshots may be deleted after `HLL_SERVER_SNAPSHOT_RETENTION_DAYS`.
|
||||
|
||||
`displayed_historical_snapshots` should not be routinely deleted because it is keyed/upserted. Only skip or report suspicious legacy/corrupt records; do not aggressively delete snapshots in this task.
|
||||
|
||||
## Safety Requirements
|
||||
|
||||
- If an optional table does not exist, log a controlled skip and continue.
|
||||
- If PostgreSQL URL is not configured and the app is in SQLite mode, either support SQLite safely or return a clear unsupported message, depending on existing architecture.
|
||||
- Do not leave orphan rows.
|
||||
- Do not delete from unrelated Elo/MMR tables unless the audit proves exact dependency and task scope is explicitly updated.
|
||||
- Use existing project lock/writer-lock patterns if present.
|
||||
- Cleanup failure must produce JSON error output.
|
||||
|
||||
## Required JSON Log Events
|
||||
|
||||
At minimum:
|
||||
|
||||
- `database-maintenance-started`
|
||||
- `database-maintenance-plan`
|
||||
- `database-maintenance-table-skipped`
|
||||
- `database-maintenance-delete-batch`
|
||||
- `database-maintenance-completed`
|
||||
- `database-maintenance-error`
|
||||
|
||||
## Tests
|
||||
|
||||
Create `backend/tests/test_database_maintenance.py`.
|
||||
|
||||
Minimum cases:
|
||||
|
||||
- dry-run does not delete.
|
||||
- apply deletes old server snapshots.
|
||||
- apply deletes old non-critical admin log events.
|
||||
- apply preserves critical events within retention.
|
||||
- apply preserves latest 100 materialized matches.
|
||||
- apply preserves current month matches.
|
||||
- apply preserves previous month when now.day <= 7.
|
||||
- apply preserves current week.
|
||||
- apply preserves previous week when fallback protection applies.
|
||||
- apply deletes an old non-protected materialized match.
|
||||
- child/player stats for deleted matches are deleted first or handled without orphans.
|
||||
- missing optional tables are logged and do not crash cleanup.
|
||||
|
||||
## Validation
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
PYTHONPATH=backend python -m unittest backend.tests.test_database_maintenance
|
||||
python -m compileall backend/app
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Also manually run:
|
||||
|
||||
```powershell
|
||||
docker compose exec backend python -m app.database_maintenance cleanup --dry-run
|
||||
```
|
||||
|
||||
or, if backend container is unavailable:
|
||||
|
||||
```powershell
|
||||
cd backend
|
||||
python -m app.database_maintenance cleanup --dry-run
|
||||
```
|
||||
|
||||
Document which one was used.
|
||||
|
||||
## Outcome
|
||||
|
||||
Document:
|
||||
|
||||
- validation commands run;
|
||||
- dry-run output summary;
|
||||
- any existing unrelated test failures;
|
||||
- exact files changed;
|
||||
- any follow-up task needed instead of expanding scope.
|
||||
|
||||
Codex CLI must commit and push the completed task branch.
|
||||
|
||||
Suggested implementation branch:
|
||||
|
||||
`task/database-maintenance-cleanup-command`
|
||||
|
||||
Suggested commit message:
|
||||
|
||||
`feat: add database maintenance cleanup command`
|
||||
|
||||
## Change Budget
|
||||
|
||||
Prefer fewer than 5 modified files.
|
||||
Prefer focused backend-only changes.
|
||||
Split follow-up tasks if implementation exceeds safe scope.
|
||||
Reference in New Issue
Block a user