136 lines
5.6 KiB
Markdown
136 lines
5.6 KiB
Markdown
---
|
|
id: TASK-POSTGRES-HISTORICAL-PHASE-2
|
|
title: Migrate displayed historical data to PostgreSQL
|
|
status: done
|
|
type: backend
|
|
team: Backend Senior
|
|
supporting_teams:
|
|
- Arquitecto de Base de Datos
|
|
- Arquitecto Python
|
|
roadmap_item: foundation
|
|
priority: high
|
|
---
|
|
|
|
# TASK-POSTGRES-HISTORICAL-PHASE-2 - Migrate displayed historical data to PostgreSQL
|
|
|
|
## Goal
|
|
|
|
Make PostgreSQL authoritative for historical and server data currently displayed
|
|
by the frontend, add an idempotent SQLite/file migration command, and close the
|
|
phase-1 datetime serialization and materialized detail lookup regressions.
|
|
|
|
## Context
|
|
|
|
Phase 1 moved the live RCON historical write path to PostgreSQL, but historical
|
|
frontend pages still read older SQLite and file-backed data for ranking,
|
|
scoreboard fallback, snapshots, server cache, and some match-detail continuity.
|
|
PostgreSQL reads also expose native datetime values that can terminate JSON API
|
|
responses. The phase-2 migration must preserve existing historical URLs and safe
|
|
scoreboard links without making SQLite the active source of truth again.
|
|
|
|
## Steps
|
|
|
|
1. Inspect the PostgreSQL RCON schema, SQLite historical schema, snapshot/server
|
|
reads, and affected API routes.
|
|
2. Add PostgreSQL-backed displayed read models plus deterministic migration from
|
|
existing SQLite/files.
|
|
3. Fix JSON-safe datetime handling and recent/detail materialized match
|
|
consistency with focused regression tests.
|
|
4. Update diagnostics and docs, then run the requested local and Docker checks
|
|
where available.
|
|
|
|
## Files to Read First
|
|
|
|
- `ai/architecture-index.md`
|
|
- `ai/repo-context.md`
|
|
- `backend/app/postgres_rcon_storage.py`
|
|
- `backend/app/historical_api.py`
|
|
- `backend/app/historical_snapshots.py`
|
|
- `backend/app/storage_diagnostics.py`
|
|
|
|
## Expected Files to Modify
|
|
|
|
- `backend/app/postgres_rcon_storage.py`
|
|
- `backend/app/sqlite_to_postgres_migration.py`
|
|
- `backend/app/storage_diagnostics.py`
|
|
- backend historical/server storage and API modules needed to move displayed
|
|
SQLite/file reads to PostgreSQL
|
|
- focused backend tests for datetime and detail lookup regressions
|
|
- `backend/README.md`
|
|
- `docs/decisions.md`
|
|
|
|
## Constraints
|
|
|
|
- Do not return displayed historical data to SQLite as the main fix.
|
|
- Preserve old match ids, match keys, public scoreboard safe URLs, and existing
|
|
historical data where present.
|
|
- Do not re-enable Comunidad Hispana #03 as an active configured target.
|
|
- Keep Elo/MMR out of phase 2 unless a visible API requires its PostgreSQL data.
|
|
- Do not modify unrelated frontend behavior or commit runtime database files.
|
|
|
|
## Validation
|
|
|
|
- Run the requested compile, diagnostics, migration, Node syntax, RCON pipeline,
|
|
integration, Docker Compose, container diagnostics, materialization, and HTTP
|
|
smoke commands where the local environment permits them.
|
|
- Review `git diff --name-only` and confirm changed files match this task.
|
|
|
|
## Outcome
|
|
|
|
Implemented PostgreSQL phase 2 for the displayed historical surface. The new
|
|
displayed PostgreSQL schema stores migrated public-scoreboard match/player
|
|
tables, historical snapshot payloads, live server cache rows and player-event
|
|
ledger rows alongside the phase-1 RCON tables. API storage facades now read
|
|
those PostgreSQL stores when `HLL_BACKEND_DATABASE_URL` is configured; SQLite
|
|
and snapshot JSON remain migration inputs or explicit local fixtures.
|
|
|
|
`python -m app.sqlite_to_postgres_migration` now discovers legacy SQLite files
|
|
and historical snapshot JSON under the configured backend data directory, copies
|
|
stable IDs/keys with idempotent conflict handling, advances PostgreSQL
|
|
sequences, skips visible `comunidad-hispana-03` legacy scope, preserves safe
|
|
scoreboard URLs and prints JSON row counts/errors. A repeat container run read
|
|
the existing source rows and reported zero inserted rows with no errors.
|
|
|
|
Required fixes included global HTTP JSON encoding for PostgreSQL `date` and
|
|
`datetime` values, a materialized detail lookup guard for match keys that embed
|
|
the requested server key, reduced repeated RCON PostgreSQL read-time DDL to
|
|
avoid schema-lock chains during materialization/diagnostics, and focused tests
|
|
for JSON encoding plus recent/detail match-id consistency.
|
|
|
|
Post-migration container diagnostics showed PostgreSQL counts including:
|
|
|
|
- `admin_log_events=21242`
|
|
- `materialized_matches=58`
|
|
- `player_stats=3824`
|
|
- `public_scoreboard_historical_matches=10030`
|
|
- ranking source stats `1090704`
|
|
- displayed snapshots `51`
|
|
- player-event ledger `14715`
|
|
- safe scoreboard candidates `400`
|
|
|
|
Remaining SQLite/file-backed boundaries are non-visible phase-3 scope only:
|
|
public-scoreboard import run/backfill bookkeeping and paused Elo/MMR tables.
|
|
|
|
Validation completed:
|
|
|
|
- `python -m compileall backend/app`
|
|
- `python -m unittest discover -s tests -p test_json_serialization.py`
|
|
from `backend/`
|
|
- `python -m app.storage_diagnostics` locally and in the backend container
|
|
- `node --check` for the four requested frontend JavaScript files
|
|
- `powershell -ExecutionPolicy Bypass -File scripts/run-rcon-data-pipeline-tests.ps1`
|
|
- `powershell -ExecutionPolicy Bypass -File scripts/run-integration-tests.ps1`
|
|
- requested advanced Docker Compose down/up/build/ps flow
|
|
- container migration, diagnostics, AdminLog ingestion and materialization
|
|
- requested HTTP probes for `/health`, recent matches and both known detail keys
|
|
- direct PostgreSQL-backed snapshot and direct monthly leaderboard HTTP probes
|
|
|
|
The RCON pipeline script still prints existing SQLite `ResourceWarning` noise
|
|
from its unittest fallback suites, but the script and suites passed.
|
|
|
|
## Change Budget
|
|
|
|
This migration intentionally crosses backend schema, read models, diagnostics,
|
|
tests, and docs. Keep each change tied to displayed data ownership or the
|
|
required migration/serialization/detail consistency fixes.
|