comunidadhll/ai/tasks/review/TASK-db-retention-audit-report.md

---
id: TASK-db-retention-audit-report
source_task: TASK-db-retention-audit
status: review
type: report
---

# Database Retention Audit

## Scope

This report audits the current storage model used by HLL Vietnam for:

- live server snapshots;
- public-scoreboard historical matches and player stats;
- RCON AdminLog events and materialized matches;
- displayed historical snapshots;
- player event raw ledger;
- paused Elo/MMR storage surfaces.

Confirmed facts below are based on the current repository code, not on assumptions.

## Files Inspected

- `AGENTS.md`
- `ai/repo-context.md`
- `ai/architecture-index.md`
- `backend/app/config.py`
- `backend/app/historical_runner.py`
- `backend/app/historical_storage.py`
- `backend/app/historical_snapshot_storage.py`
- `backend/app/player_event_aggregates.py`
- `backend/app/player_event_storage.py`
- `backend/app/postgres_display_storage.py`
- `backend/app/postgres_rcon_storage.py`
- `backend/app/rcon_admin_log_materialization.py`
- `backend/app/rcon_admin_log_storage.py`
- `backend/app/rcon_historical_backfill.py`
- `backend/app/rcon_historical_leaderboards.py`
- `backend/app/rcon_historical_read_model.py`
- `backend/app/rcon_historical_worker.py`
- `backend/app/writer_lock.py`

## Confirmed Storage Areas

### Live and displayed data

- `server_snapshots`
  - Present in SQLite live storage and PostgreSQL displayed storage.
  - Used for live/current server status history.
- `displayed_historical_snapshots`
  - PostgreSQL primary key is `(server_key, snapshot_type, metric, snapshot_window)`.
  - Persistence uses upsert semantics, so normal operation is already bounded by logical identity.

### Public-scoreboard historical fallback

- `historical_servers`
- `historical_maps`
- `historical_matches`
- `historical_players`
- `historical_player_match_stats`
- `historical_ingestion_runs`
- `historical_backfill_progress`

These remain part of the displayed historical fallback/read path.

### RCON historical primary path

- `rcon_historical_targets`
- `rcon_historical_capture_runs`
- `rcon_historical_samples`
- `rcon_historical_checkpoints`
- `rcon_historical_competitive_windows`
- `rcon_admin_log_events`
- `rcon_player_profile_snapshots`
- `rcon_materialized_matches`
- `rcon_match_player_stats`
- `rcon_scoreboard_match_candidates`

### Player-event ledger

- `player_event_raw_ledger`
- `player_event_ingestion_runs`
- `player_event_backfill_progress`

### Elo/MMR paused surfaces

Confirmed in backend documentation and runtime payloads as preserved but paused.
This audit did not find cleanup code or explicit retention logic for Elo/MMR tables in the task’s read-first files, so they should stay out of cleanup scope.

## Snapshot Boundedness Findings

### `displayed_historical_snapshots`

Confirmed bounded in PostgreSQL by primary key plus upsert:

- same logical snapshot identity overwrites previous payload;
- normal scheduled refresh does not append unlimited rows;
- routine cleanup is not required for healthy records.

Safe policy:

- do not bulk-delete this table in the first maintenance command;
- only report suspicious legacy/corrupt rows if found later.

### File-based snapshots

Confirmed bounded by deterministic file paths under `backend/data/snapshots/<server>/...`.
Normal writes replace files in place.

Safe policy:

- do not add file cleanup in the first database-maintenance command.

## Retention Windows Confirmed In Runtime Code

Two different historical paths currently exist, and they do not use identical monthly rules.

### Public-scoreboard fallback (`historical_*`)

Confirmed in `backend/app/historical_storage.py`:

- weekly fallback can use previous week when current-week sample is insufficient;
- monthly fallback uses previous month only when current month has zero closed matches;
- `is_early_month` is computed with `current_time.day <= 3`, but selection still depends on zero current-month matches.

### RCON materialized leaderboards (`rcon_materialized_matches`)

Confirmed in `backend/app/rcon_historical_leaderboards.py`:

- monthly leaderboard selection uses previous month through day 7 inclusive;
- weekly selection falls back to previous week when current week lacks the configured minimum closed matches.

## Cleanup Policy Recommendation

Because the new maintenance task explicitly requires:

- latest 100 materialized matches;
- current month;
- previous month during first 7 days of a new month;
- current week;
- previous week when weekly fallback may need it;

the cleanup command should implement the stricter policy directly instead of copying the older `historical_storage.py` month fallback behavior.

## Safe-To-Delete Areas

### 1. `server_snapshots`

Safe to delete old rows by `captured_at` age once outside retention.

Why:

- live UI only needs recent history;
- no dependent child tables were found;
- latest snapshot queries use recent rows only.

Recommended initial retention:

- keep last 14 days by default.

### 2. Non-critical `rcon_admin_log_events`

Safe to delete by event age after retention.

Confirmed critical event types:

- `kill`
- `match_start`
- `match_end`

Everything else should be treated as non-critical for first-pass cleanup:

- `chat`
- `message`
- `connected`
- `disconnected`
- `team_switch`
- `unknown`
- `kick`
- `ban`
- any other non-materialization event type.

Recommended initial retention:

- non-critical: 30 days.

### 3. Critical `rcon_admin_log_events`

Conditionally safe to delete only when both are true:

- older than the critical retention threshold;
- not needed by protected materialized matches/windows.

Recommended initial retention:

- critical: 90 days.

### 4. Old `rcon_materialized_matches`

Conditionally safe to delete when all of the following are true:

- not in latest N closed matches;
- not in current month;
- not in previous month during the first 7 days of a new month;
- not in current week;
- not in previous week when weekly fallback may still be needed.

### 5. Child rows in `rcon_match_player_stats`

Safe to delete only as a dependent delete for non-protected materialized matches.

Delete these before deleting parent matches because no cascade constraint was found.

## Areas That Must Be Protected

### RCON materialized read path

Protect:

- latest 100 closed `rcon_materialized_matches`;
- all `rcon_match_player_stats` for protected matches;
- `kill`, `match_start`, `match_end` events that still support protected matches or currently relevant windows.

### Live/current match surfaces

Protect recent/current RCON data:

- recent `rcon_admin_log_events` still used by current-match kill feed and current-match player stats;
- recent `server_snapshots`.

Reason:

- current-match pages read recent AdminLog windows directly;
- premature deletion would break kill feed or partial player stats.

### Displayed snapshots

Protect:

- `displayed_historical_snapshots` as a bounded read cache;
- file-based snapshots under `backend/data/snapshots/`.

### Public-scoreboard `historical_*`

Do not delete in the first cleanup command.

Reason:

- these tables still back visible historical endpoints and fallback behavior;
- the task’s cleanup requirements are expressed around materialized matches and AdminLog, not around legacy scoreboard imports;
- deleting this domain now would widen scope materially.

### Player-event ledger

Do not delete in the first cleanup command.

Reason:

- it feeds visible player-event snapshots and monthly MVP V2 support;
- no task requirement currently defines safe retention windows for this ledger;
- deleting it without product retention rules is risky.

### Elo/MMR paused data

Do not delete.

Reason:

- task explicitly warns against unrelated Elo/MMR deletions;
- runtime is paused, but persisted surfaces are intentionally preserved.

## Foreign Keys, Cascades, and Orphan Risk

### Confirmed foreign keys

Public-scoreboard historical tables have foreign keys:

- `historical_matches.historical_server_id -> historical_servers.id`
- `historical_matches.historical_map_id -> historical_maps.id`
- `historical_player_match_stats.historical_match_id -> historical_matches.id`
- `historical_player_match_stats.historical_player_id -> historical_players.id`

RCON/postgres tables also define references for some capture-oriented tables.

### Missing cascade behavior relevant to cleanup

No `ON DELETE CASCADE` behavior was found for:

- `rcon_materialized_matches`
- `rcon_match_player_stats`

The materialization code explicitly deletes/rebuilds child player rows itself, which is another signal that cleanup must manage delete order manually.

### Required delete order

For materialized RCON cleanup:

1. identify protected match keys;
2. identify candidate old matches outside protected windows;
3. delete `rcon_match_player_stats` for candidate match keys;
4. delete candidate rows from `rcon_materialized_matches`;
5. delete eligible `rcon_admin_log_events`;
6. delete eligible `server_snapshots`.

This order avoids orphaned player-stat rows.

## Risks and Edge Cases

### Timestamp parsing risk

Some match rows may have:

- missing `started_at`;
- missing `ended_at`;
- only server-time fields;
- malformed or non-ISO timestamps.

Safe rule:

- if a materialized match timestamp cannot be interpreted safely, protect it and skip deletion.

### Cross-path monthly rule mismatch

There is a confirmed mismatch today:

- public-scoreboard fallback code does not strictly use previous month through day 7;
- RCON leaderboard code does.

Cleanup should follow the task requirement, not the older fallback implementation.

### Current-match dependency on recent AdminLog

Current match pages query recent AdminLog rows directly, not only materialized matches.
Deleting recent critical events too aggressively could break live/current-match UI even if historical leaderboards still work.

### Optional table presence

Some environments may not have every migrated or optional table yet.
Maintenance code should treat absent tables as controlled skips, not hard failures.

### SQLite versus PostgreSQL

The repo still supports SQLite fallback when `HLL_BACKEND_DATABASE_URL` is unset.
Maintenance code should either:

- support SQLite safely for the targeted tables; or
- emit a clear unsupported message.

Given the codebase still keeps local SQLite testability, SQLite support is preferable for the cleanup command and its tests.

## Recommended Cleanup Scope For First Implementation

Implement only:

- `server_snapshots`;
- `rcon_admin_log_events`;
- `rcon_materialized_matches`;
- `rcon_match_player_stats`.

Explicitly skip in first implementation:

- `displayed_historical_snapshots`;
- `historical_*`;
- `player_event_raw_ledger` and related worker tables;
- Elo/MMR tables;
- `rcon_historical_samples`, `rcon_historical_competitive_windows`, `rcon_player_profile_snapshots`, `rcon_scoreboard_match_candidates`.

Reason:

- keeps scope aligned to the task;
- avoids deleting sources still needed for visible history fallback or paused domains;
- stays within the repository’s change-budget expectations.

## Implementation Plan For `app.database_maintenance`

1. Add maintenance config accessors for all retention values and batch size.
2. Add a CLI with `cleanup --dry-run` and `cleanup --apply`.
3. Build one plan phase that calculates:
   - protected materialized match keys;
   - candidate match keys;
   - candidate admin-log rows by criticality and age;
   - candidate server snapshots by age.
4. Log the plan in structured JSON before any deletes.
5. On apply:
   - acquire the shared backend writer lock;
   - delete `rcon_match_player_stats` in batches for candidate match keys;
   - delete `rcon_materialized_matches` in batches;
   - delete `rcon_admin_log_events` in batches;
   - delete `server_snapshots` in batches;
   - optionally run `VACUUM ANALYZE` only when explicitly requested.
6. Treat missing optional tables as logged skips.
7. Keep dry-run as default behavior.

## Recommended SQL Audit Commands

### Table-size audit

```sql
select
  schemaname,
  relname as table_name,
  pg_size_pretty(pg_total_relation_size(relid)) as total_size,
  pg_size_pretty(pg_relation_size(relid)) as table_size,
  pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as indexes_size,
  n_live_tup as estimated_rows,
  n_dead_tup as estimated_dead_rows
from pg_stat_user_tables
order by pg_total_relation_size(relid) desc;
```

### AdminLog by type and age

```sql
select
  event_type,
  count(*) as row_count,
  min(event_timestamp) as first_event_timestamp,
  max(event_timestamp) as last_event_timestamp,
  min(server_time) as first_server_time,
  max(server_time) as last_server_time
from rcon_admin_log_events
group by event_type
order by row_count desc, event_type asc;
```

### Materialized matches by server and closure time

```sql
select
  target_key,
  source_basis,
  count(*) as matches,
  min(coalesce(ended_at, started_at)) as first_closed_at,
  max(coalesce(ended_at, started_at)) as last_closed_at
from rcon_materialized_matches
group by target_key, source_basis
order by target_key asc, source_basis asc;
```

### Materialized player stats coverage

```sql
select
  target_key,
  count(*) as player_stat_rows,
  count(distinct match_key) as match_count
from rcon_match_player_stats
group by target_key
order by target_key asc;
```

### Live server snapshots by age

```sql
select
  server_id,
  min(captured_at) as first_captured_at,
  max(captured_at) as last_captured_at,
  count(*) as snapshot_rows
from server_snapshots
group by server_id
order by last_captured_at desc;
```

### Displayed snapshot coverage

```sql
select
  snapshot_type,
  metric,
  snapshot_window,
  count(*) as snapshot_rows,
  min(generated_at) as first_generated_at,
  max(generated_at) as last_generated_at
from displayed_historical_snapshots
group by snapshot_type, metric, snapshot_window
order by snapshot_type asc, metric asc, snapshot_window asc;
```

### Public-scoreboard fallback historical matches

```sql
select
  hs.slug as server_slug,
  count(*) as matches,
  min(coalesce(hm.ended_at, hm.started_at, hm.created_at_source)) as first_match_at,
  max(coalesce(hm.ended_at, hm.started_at, hm.created_at_source)) as last_match_at
from historical_matches hm
join historical_servers hs on hs.id = hm.historical_server_id
group by hs.slug
order by hs.slug asc;
```

### Player-event ledger age

```sql
select
  event_type,
  count(*) as row_count,
  min(occurred_at) as first_occurred_at,
  max(occurred_at) as last_occurred_at
from player_event_raw_ledger
group by event_type
order by row_count desc, event_type asc;
```

## Summary

The first cleanup command can be implemented safely if it stays narrow:

- clean `server_snapshots`;
- clean aged `rcon_admin_log_events`;
- clean non-protected `rcon_materialized_matches`;
- delete `rcon_match_player_stats` first for those matches;
- skip bounded snapshot caches and broader fallback domains for now.

The main correctness risk is not delete syntax. It is retention-window selection and preserving current/live dependencies. The cleanup command should therefore be plan-first, dry-run-first, and lock-protected.