237 lines
6.6 KiB
Markdown
237 lines
6.6 KiB
Markdown
---
|
|
id: TASK-database-maintenance-cleanup-command
|
|
title: Add database maintenance cleanup command
|
|
status: pending
|
|
type: backend
|
|
team: Backend Senior
|
|
supporting_teams: [Arquitecto de Base de Datos, Arquitecto Python]
|
|
roadmap_item: historical-data-maintenance
|
|
priority: high
|
|
---
|
|
|
|
# TASK-database-maintenance-cleanup-command - Add database maintenance cleanup command
|
|
|
|
## Goal
|
|
|
|
Implement a safe backend cleanup command with dry-run/apply modes to remove database data that is no longer needed while preserving all data required for visible history and leaderboards.
|
|
|
|
## Context
|
|
|
|
The project should keep the database bounded without manually deleting data.
|
|
|
|
The command will later be scheduled from historical-runner, but this task only implements the cleanup module and tests.
|
|
|
|
The cleanup must not rely on native PostgreSQL scheduling or a new Docker service.
|
|
|
|
The cleanup must protect:
|
|
|
|
- latest 100 closed/materialized matches;
|
|
- current month;
|
|
- previous month during the first 7 days of the new month;
|
|
- current week;
|
|
- previous week if fallback weekly logic may need it;
|
|
- child/player stats for protected matches;
|
|
- current live/RCON data.
|
|
|
|
## Steps
|
|
|
|
1. Inspect the listed files first.
|
|
2. Add a backend module for database maintenance.
|
|
3. Provide a CLI entry point:
|
|
- `python -m app.database_maintenance cleanup --dry-run`
|
|
- `python -m app.database_maintenance cleanup --apply`
|
|
4. Make dry-run the default behavior.
|
|
5. Add JSON logs for planning, batches and completion.
|
|
6. Add tests.
|
|
7. Validate locally.
|
|
8. Commit and push the task branch.
|
|
|
|
## Files to Read First
|
|
|
|
- `AGENTS.md`
|
|
- `backend/app/config.py`
|
|
- `backend/app/postgres_display_storage.py`
|
|
- `backend/app/historical_runner.py`
|
|
- `backend/app/rcon_historical_read_model.py`
|
|
- `backend/app/rcon_historical_backfill.py`
|
|
- `backend/tests/`
|
|
|
|
## Expected Files to Modify
|
|
|
|
- `backend/app/database_maintenance.py`
|
|
- `backend/app/config.py`
|
|
- `backend/tests/test_database_maintenance.py`
|
|
|
|
Optional only if required by existing test helpers:
|
|
|
|
- `backend/tests/conftest.py`
|
|
- `backend/tests/test_*.py`
|
|
|
|
Do not modify frontend files in this task.
|
|
|
|
## Functional Requirements
|
|
|
|
Create a cleanup command:
|
|
|
|
```powershell
|
|
python -m app.database_maintenance cleanup --dry-run
|
|
python -m app.database_maintenance cleanup --apply
|
|
```
|
|
|
|
Supported options:
|
|
|
|
- `--recent-matches-keep`
|
|
- `--admin-log-noncritical-retention-days`
|
|
- `--admin-log-critical-retention-days`
|
|
- `--server-snapshot-retention-days`
|
|
- `--batch-size`
|
|
- `--vacuum-analyze`
|
|
- `--now`
|
|
|
|
Environment defaults:
|
|
|
|
- `HLL_RECENT_MATCHES_KEEP=100`
|
|
- `HLL_ADMIN_LOG_NONCRITICAL_RETENTION_DAYS=30`
|
|
- `HLL_ADMIN_LOG_CRITICAL_RETENTION_DAYS=90`
|
|
- `HLL_SERVER_SNAPSHOT_RETENTION_DAYS=14`
|
|
- `HLL_DB_MAINTENANCE_BATCH_SIZE=5000`
|
|
|
|
Dry-run behavior:
|
|
|
|
- calculate all candidate rows;
|
|
- do not delete anything;
|
|
- log exactly what would be deleted;
|
|
- return status ok.
|
|
|
|
Apply behavior:
|
|
|
|
- delete only candidate rows;
|
|
- delete in batches;
|
|
- use transactions safely;
|
|
- do not block indefinitely;
|
|
- log deleted row counts;
|
|
- return status ok even if deleted counts are zero.
|
|
|
|
Retention Logic
|
|
|
|
Protected materialized matches:
|
|
|
|
- latest N closed matches, default N = 100;
|
|
- current month matches;
|
|
- previous month matches if now.day <= 7;
|
|
- current week matches;
|
|
- previous week matches if weekly fallback may need them;
|
|
- any match whose end/start timestamp cannot be safely interpreted should be skipped/protected, not deleted.
|
|
|
|
Delete materialized matches only if:
|
|
|
|
- not in protected latest N;
|
|
- not in protected current/previous month window;
|
|
- not in protected current/previous week window;
|
|
- older than the relevant protected windows;
|
|
- child stats/events that depend on the match are handled first.
|
|
|
|
AdminLog critical events:
|
|
|
|
- `kill`
|
|
- `match_start`
|
|
- `match_end`
|
|
|
|
Critical events may only be deleted after `HLL_ADMIN_LOG_CRITICAL_RETENTION_DAYS` and only if they are not needed for protected materialized matches/windows.
|
|
|
|
AdminLog non-critical events may be deleted after `HLL_ADMIN_LOG_NONCRITICAL_RETENTION_DAYS`.
|
|
|
|
Server snapshots may be deleted after `HLL_SERVER_SNAPSHOT_RETENTION_DAYS`.
|
|
|
|
`displayed_historical_snapshots` should not be routinely deleted because it is keyed/upserted. Only skip or report suspicious legacy/corrupt records; do not aggressively delete snapshots in this task.
|
|
|
|
## Safety Requirements
|
|
|
|
- If an optional table does not exist, log a controlled skip and continue.
|
|
- If PostgreSQL URL is not configured and the app is in SQLite mode, either support SQLite safely or return a clear unsupported message, depending on existing architecture.
|
|
- Do not leave orphan rows.
|
|
- Do not delete from unrelated Elo/MMR tables unless the audit proves exact dependency and task scope is explicitly updated.
|
|
- Use existing project lock/writer-lock patterns if present.
|
|
- Cleanup failure must produce JSON error output.
|
|
|
|
## Required JSON Log Events
|
|
|
|
At minimum:
|
|
|
|
- `database-maintenance-started`
|
|
- `database-maintenance-plan`
|
|
- `database-maintenance-table-skipped`
|
|
- `database-maintenance-delete-batch`
|
|
- `database-maintenance-completed`
|
|
- `database-maintenance-error`
|
|
|
|
## Tests
|
|
|
|
Create `backend/tests/test_database_maintenance.py`.
|
|
|
|
Minimum cases:
|
|
|
|
- dry-run does not delete.
|
|
- apply deletes old server snapshots.
|
|
- apply deletes old non-critical admin log events.
|
|
- apply preserves critical events within retention.
|
|
- apply preserves latest 100 materialized matches.
|
|
- apply preserves current month matches.
|
|
- apply preserves previous month when now.day <= 7.
|
|
- apply preserves current week.
|
|
- apply preserves previous week when fallback protection applies.
|
|
- apply deletes an old non-protected materialized match.
|
|
- child/player stats for deleted matches are deleted first or handled without orphans.
|
|
- missing optional tables are logged and do not crash cleanup.
|
|
|
|
## Validation
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
PYTHONPATH=backend python -m unittest backend.tests.test_database_maintenance
|
|
python -m compileall backend/app
|
|
git diff --check
|
|
```
|
|
|
|
Also manually run:
|
|
|
|
```powershell
|
|
docker compose exec backend python -m app.database_maintenance cleanup --dry-run
|
|
```
|
|
|
|
or, if backend container is unavailable:
|
|
|
|
```powershell
|
|
cd backend
|
|
python -m app.database_maintenance cleanup --dry-run
|
|
```
|
|
|
|
Document which one was used.
|
|
|
|
## Outcome
|
|
|
|
Document:
|
|
|
|
- validation commands run;
|
|
- dry-run output summary;
|
|
- any existing unrelated test failures;
|
|
- exact files changed;
|
|
- any follow-up task needed instead of expanding scope.
|
|
|
|
Codex CLI must commit and push the completed task branch.
|
|
|
|
Suggested implementation branch:
|
|
|
|
`task/database-maintenance-cleanup-command`
|
|
|
|
Suggested commit message:
|
|
|
|
`feat: add database maintenance cleanup command`
|
|
|
|
## Change Budget
|
|
|
|
Prefer fewer than 5 modified files.
|
|
Prefer focused backend-only changes.
|
|
Split follow-up tasks if implementation exceeds safe scope.
|