comunidadhll/ai/tasks/done/TASK-POSTGRES-HISTORICAL-PHASE-2.md

id, title, status, type, team, supporting_teams, roadmap_item, priority

id	title	status	type	team	supporting_teams	roadmap_item	priority
TASK-POSTGRES-HISTORICAL-PHASE-2	Migrate displayed historical data to PostgreSQL	done	backend	Backend Senior	Arquitecto de Base de Datos Arquitecto Python	foundation	high

TASK-POSTGRES-HISTORICAL-PHASE-2 - Migrate displayed historical data to PostgreSQL

Goal

Make PostgreSQL authoritative for historical and server data currently displayed by the frontend, add an idempotent SQLite/file migration command, and close the phase-1 datetime serialization and materialized detail lookup regressions.

Context

Phase 1 moved the live RCON historical write path to PostgreSQL, but historical frontend pages still read older SQLite and file-backed data for ranking, scoreboard fallback, snapshots, server cache, and some match-detail continuity. PostgreSQL reads also expose native datetime values that can terminate JSON API responses. The phase-2 migration must preserve existing historical URLs and safe scoreboard links without making SQLite the active source of truth again.

Steps

1. Inspect the PostgreSQL RCON schema, SQLite historical schema, snapshot/server reads, and affected API routes.
2. Add PostgreSQL-backed displayed read models plus deterministic migration from existing SQLite/files.
3. Fix JSON-safe datetime handling and recent/detail materialized match consistency with focused regression tests.
4. Update diagnostics and docs, then run the requested local and Docker checks where available.

Files to Read First

• ai/architecture-index.md
• ai/repo-context.md
• backend/app/postgres_rcon_storage.py
• backend/app/historical_api.py
• backend/app/historical_snapshots.py
• backend/app/storage_diagnostics.py

Expected Files to Modify

• backend/app/postgres_rcon_storage.py
• backend/app/sqlite_to_postgres_migration.py
• backend/app/storage_diagnostics.py
• backend historical/server storage and API modules needed to move displayed SQLite/file reads to PostgreSQL
• focused backend tests for datetime and detail lookup regressions
• backend/README.md
• docs/decisions.md

Constraints

• Do not return displayed historical data to SQLite as the main fix.
• Preserve old match ids, match keys, public scoreboard safe URLs, and existing historical data where present.
• Do not re-enable Comunidad Hispana #03 as an active configured target.
• Keep Elo/MMR out of phase 2 unless a visible API requires its PostgreSQL data.
• Do not modify unrelated frontend behavior or commit runtime database files.

Validation

• Run the requested compile, diagnostics, migration, Node syntax, RCON pipeline, integration, Docker Compose, container diagnostics, materialization, and HTTP smoke commands where the local environment permits them.
• Review git diff --name-only and confirm changed files match this task.

Outcome

Implemented PostgreSQL phase 2 for the displayed historical surface. The new displayed PostgreSQL schema stores migrated public-scoreboard match/player tables, historical snapshot payloads, live server cache rows and player-event ledger rows alongside the phase-1 RCON tables. API storage facades now read those PostgreSQL stores when HLL_BACKEND_DATABASE_URL is configured; SQLite and snapshot JSON remain migration inputs or explicit local fixtures.

python -m app.sqlite_to_postgres_migration now discovers legacy SQLite files and historical snapshot JSON under the configured backend data directory, copies stable IDs/keys with idempotent conflict handling, advances PostgreSQL sequences, skips visible comunidad-hispana-03 legacy scope, preserves safe scoreboard URLs and prints JSON row counts/errors. A repeat container run read the existing source rows and reported zero inserted rows with no errors.

Required fixes included global HTTP JSON encoding for PostgreSQL date and datetime values, a materialized detail lookup guard for match keys that embed the requested server key, reduced repeated RCON PostgreSQL read-time DDL to avoid schema-lock chains during materialization/diagnostics, and focused tests for JSON encoding plus recent/detail match-id consistency.

Post-migration container diagnostics showed PostgreSQL counts including:

• admin_log_events=21242
• materialized_matches=58
• player_stats=3824
• public_scoreboard_historical_matches=10030
• ranking source stats 1090704
• displayed snapshots 51
• player-event ledger 14715
• safe scoreboard candidates 400

Remaining SQLite/file-backed boundaries are non-visible phase-3 scope only: public-scoreboard import run/backfill bookkeeping and paused Elo/MMR tables.

Validation completed:

• python -m compileall backend/app
• python -m unittest discover -s tests -p test_json_serialization.py from backend/
• python -m app.storage_diagnostics locally and in the backend container
• node --check for the four requested frontend JavaScript files
• powershell -ExecutionPolicy Bypass -File scripts/run-rcon-data-pipeline-tests.ps1
• powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1
• requested advanced Docker Compose down/up/build/ps flow
• container migration, diagnostics, AdminLog ingestion and materialization
• requested HTTP probes for /health, recent matches and both known detail keys
• direct PostgreSQL-backed snapshot and direct monthly leaderboard HTTP probes

The RCON pipeline script still prints existing SQLite ResourceWarning noise from its unittest fallback suites, but the script and suites passed.

Change Budget

This migration intentionally crosses backend schema, read models, diagnostics, tests, and docs. Keep each change tied to displayed data ownership or the required migration/serialization/detail consistency fixes.