feat: migrate displayed historical data to postgres

ai/tasks/done/TASK-POSTGRES-HISTORICAL-PHASE-2.md

@@ -0,0 +1,135 @@
+---
+id: TASK-POSTGRES-HISTORICAL-PHASE-2
+title: Migrate displayed historical data to PostgreSQL
+status: done
+type: backend
+team: Backend Senior
+supporting_teams:
+  - Arquitecto de Base de Datos
+  - Arquitecto Python
+roadmap_item: foundation
+priority: high
+---
+
+# TASK-POSTGRES-HISTORICAL-PHASE-2 - Migrate displayed historical data to PostgreSQL
+
+## Goal
+
+Make PostgreSQL authoritative for historical and server data currently displayed
+by the frontend, add an idempotent SQLite/file migration command, and close the
+phase-1 datetime serialization and materialized detail lookup regressions.
+
+## Context
+
+Phase 1 moved the live RCON historical write path to PostgreSQL, but historical
+frontend pages still read older SQLite and file-backed data for ranking,
+scoreboard fallback, snapshots, server cache, and some match-detail continuity.
+PostgreSQL reads also expose native datetime values that can terminate JSON API
+responses. The phase-2 migration must preserve existing historical URLs and safe
+scoreboard links without making SQLite the active source of truth again.
+
+## Steps
+
+1. Inspect the PostgreSQL RCON schema, SQLite historical schema, snapshot/server
+   reads, and affected API routes.
+2. Add PostgreSQL-backed displayed read models plus deterministic migration from
+   existing SQLite/files.
+3. Fix JSON-safe datetime handling and recent/detail materialized match
+   consistency with focused regression tests.
+4. Update diagnostics and docs, then run the requested local and Docker checks
+   where available.
+
+## Files to Read First
+
+- `ai/architecture-index.md`
+- `ai/repo-context.md`
+- `backend/app/postgres_rcon_storage.py`
+- `backend/app/historical_api.py`
+- `backend/app/historical_snapshots.py`
+- `backend/app/storage_diagnostics.py`
+
+## Expected Files to Modify
+
+- `backend/app/postgres_rcon_storage.py`
+- `backend/app/sqlite_to_postgres_migration.py`
+- `backend/app/storage_diagnostics.py`
+- backend historical/server storage and API modules needed to move displayed
+  SQLite/file reads to PostgreSQL
+- focused backend tests for datetime and detail lookup regressions
+- `backend/README.md`
+- `docs/decisions.md`
+
+## Constraints
+
+- Do not return displayed historical data to SQLite as the main fix.
+- Preserve old match ids, match keys, public scoreboard safe URLs, and existing
+  historical data where present.
+- Do not re-enable Comunidad Hispana #03 as an active configured target.
+- Keep Elo/MMR out of phase 2 unless a visible API requires its PostgreSQL data.
+- Do not modify unrelated frontend behavior or commit runtime database files.
+
+## Validation
+
+- Run the requested compile, diagnostics, migration, Node syntax, RCON pipeline,
+  integration, Docker Compose, container diagnostics, materialization, and HTTP
+  smoke commands where the local environment permits them.
+- Review `git diff --name-only` and confirm changed files match this task.
+
+## Outcome
+
+Implemented PostgreSQL phase 2 for the displayed historical surface. The new
+displayed PostgreSQL schema stores migrated public-scoreboard match/player
+tables, historical snapshot payloads, live server cache rows and player-event
+ledger rows alongside the phase-1 RCON tables. API storage facades now read
+those PostgreSQL stores when `HLL_BACKEND_DATABASE_URL` is configured; SQLite
+and snapshot JSON remain migration inputs or explicit local fixtures.
+
+`python -m app.sqlite_to_postgres_migration` now discovers legacy SQLite files
+and historical snapshot JSON under the configured backend data directory, copies
+stable IDs/keys with idempotent conflict handling, advances PostgreSQL
+sequences, skips visible `comunidad-hispana-03` legacy scope, preserves safe
+scoreboard URLs and prints JSON row counts/errors. A repeat container run read
+the existing source rows and reported zero inserted rows with no errors.
+
+Required fixes included global HTTP JSON encoding for PostgreSQL `date` and
+`datetime` values, a materialized detail lookup guard for match keys that embed
+the requested server key, reduced repeated RCON PostgreSQL read-time DDL to
+avoid schema-lock chains during materialization/diagnostics, and focused tests
+for JSON encoding plus recent/detail match-id consistency.
+
+Post-migration container diagnostics showed PostgreSQL counts including:
+
+- `admin_log_events=21242`
+- `materialized_matches=58`
+- `player_stats=3824`
+- `public_scoreboard_historical_matches=10030`
+- ranking source stats `1090704`
+- displayed snapshots `51`
+- player-event ledger `14715`
+- safe scoreboard candidates `400`
+
+Remaining SQLite/file-backed boundaries are non-visible phase-3 scope only:
+public-scoreboard import run/backfill bookkeeping and paused Elo/MMR tables.
+
+Validation completed:
+
+- `python -m compileall backend/app`
+- `python -m unittest discover -s tests -p test_json_serialization.py`
+  from `backend/`
+- `python -m app.storage_diagnostics` locally and in the backend container
+- `node --check` for the four requested frontend JavaScript files
+- `powershell -ExecutionPolicy Bypass -File scripts/run-rcon-data-pipeline-tests.ps1`
+- `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1`
+- requested advanced Docker Compose down/up/build/ps flow
+- container migration, diagnostics, AdminLog ingestion and materialization
+- requested HTTP probes for `/health`, recent matches and both known detail keys
+- direct PostgreSQL-backed snapshot and direct monthly leaderboard HTTP probes
+
+The RCON pipeline script still prints existing SQLite `ResourceWarning` noise
+from its unittest fallback suites, but the script and suites passed.
+
+## Change Budget
+
+This migration intentionally crosses backend schema, read models, diagnostics,
+tests, and docs. Keep each change tied to displayed data ownership or the
+required migration/serialization/detail consistency fixes.