diff --git a/memory-bank/activeContext.md b/memory-bank/activeContext.md index d607ca4..dfec323 100644 --- a/memory-bank/activeContext.md +++ b/memory-bank/activeContext.md @@ -8,7 +8,7 @@ Pounce Terminal fully functional with complete monitoring & notification system. - [x] Database models (User, Domain, DomainCheck, Subscription, TLDPrice, DomainHealthCache) - [x] Domain checker service (WHOIS + RDAP + DNS) - [x] Domain health checker (DNS, HTTP, SSL layers) -- [x] Authentication system (JWT + OAuth) +- [x] Authentication system (HttpOnly cookies + OAuth) - [x] API endpoints for domain management - [x] Tiered scheduler for domain checks (Scout=daily, Trader=hourly, Tycoon=10min) - [x] Next.js frontend with dark terminal theme @@ -18,8 +18,26 @@ Pounce Terminal fully functional with complete monitoring & notification system. - [x] **Watchlist with automatic monitoring & alerts** - [x] **Health check overlays with complete DNS/HTTP/SSL details** - [x] **Instant alert toggle (no refresh needed)** +- [x] **Performance Phase 0–2 applied (scheduler split, DB/index fixes, cached health, dashboard summary, metrics, job queue scaffolding)** -## Recent Changes (Dec 2024) +## Recent Changes (Dec 2025) + +### Security hardening +- **HttpOnly cookie auth** (no JWT in URLs / no token in `localStorage`) +- **OAuth redirect hardening** (state + redirect validation) +- **Blog HTML sanitization** on backend +- **Secrets removed from repo history** + `.gitignore` hardened + +### Performance & architecture phases (0 → 2) +- **Scheduler split**: API runs with `ENABLE_SCHEDULER=false`, scheduler runs as separate process/container +- **Market feed**: bounded DB queries + pagination (no full table loads) +- **Health**: bulk cached endpoint (`/domains/health-cache`) + cache-first per-domain health +- **Radar**: single-call dashboard payload (`/dashboard/summary`) → fewer frontend round-trips +- **DB migrations**: idempotent indexes + optional columns for existing DBs +- **Auction scoring**: persisted `pounce_score` populated by scraper +- **Admin**: removed N+1 patterns in user listing/export +- **Observability**: Prometheus metrics (`/metrics`) + optional DB query timing +- **Job queue**: Redis + ARQ worker scaffolding + admin scraping enqueue ### Watchlist & Monitoring 1. **Automatic domain checks**: Runs based on subscription tier @@ -46,9 +64,9 @@ Pounce Terminal fully functional with complete monitoring & notification system. ## Next Steps 1. **Configure SMTP on server** - Required for email alerts to work -2. **Test email delivery** - Verify alerts are sent correctly -3. **Consider SMS alerts** - Would require Twilio integration -4. **Monitor scheduler health** - Check logs for job execution +2. **Run production stack with scheduler + worker** (Docker Compose includes `scheduler`, `worker`, `redis`) +3. **Monitor `/metrics`** and set alerts (p95 latency, DB query time, job failures) +4. **Run load test** (`loadtest/k6/api-smoke.js`) after each deployment ## Server Deployment Checklist - [ ] Set `SMTP_*` environment variables (see `env.example`) @@ -69,4 +87,4 @@ Pounce Terminal fully functional with complete monitoring & notification system. - Email alerts require SMTP configuration - Some TLDs (.ch, .de) don't publish expiration dates publicly - SSL checks may fail on local dev (certificate chain issues) -- Scheduler starts automatically with uvicorn +- Scheduler should not run in the API process in production (avoid duplicate jobs with multiple API workers) diff --git a/memory-bank/progress.md b/memory-bank/progress.md index 7e1739b..ce276a7 100644 --- a/memory-bank/progress.md +++ b/memory-bank/progress.md @@ -1,30 +1,25 @@ -# DomainWatch - Progress +# Pounce - Progress ## What Works -- ✅ Full backend API -- ✅ User registration and login -- ✅ JWT authentication -- ✅ Public domain availability check -- ✅ Domain watchlist (add/remove/refresh) -- ✅ Subscription tiers and limits -- ✅ Daily scheduled domain checks -- ✅ Frontend with responsive design -- ✅ Dashboard with domain list -- ✅ Pricing page +- ✅ FastAPI backend + Next.js frontend (Terminal + public pages) +- ✅ Cookie-based auth (HttpOnly) + OAuth +- ✅ Market feed + auctions + listings + TLD intel +- ✅ Watchlist + cached health checks (bulk + per-domain cache-first) +- ✅ Separate scheduler process (avoid duplicate jobs with multiple API workers) +- ✅ Redis-backed job queue scaffolding (ARQ worker) + admin scraping enqueue +- ✅ Prometheus metrics endpoint (`/metrics`) +- ✅ Load test scaffolding (`loadtest/k6`) ## What's Left -- ⏳ Email notifications -- ⏳ Payment integration -- ⏳ Domain check history view -- ⏳ Password reset functionality -- ⏳ User settings page -- ⏳ Admin dashboard +- ⏳ SMTP configuration + deliverability verification +- ⏳ Production observability rollout (dashboards/alerts around `/metrics`) +- ⏳ Optional: migrate periodic background jobs from APScheduler → queue/worker for scaling ## Current Issues - None known - awaiting first test run ## Performance Notes -- WHOIS queries: ~1-3 seconds per domain -- DNS queries: ~0.1-0.5 seconds per domain -- Scheduler configured for 6:00 AM daily checks +- WHOIS queries are inherently slow (external I/O): expect ~1–3s/domain +- DB hotspots were reduced (market feed bounded queries, N+1 removed in price tracking/admin) +- Use `/metrics` + `loadtest/k6` to track p95 after deployments diff --git a/memory-bank/systemPatterns.md b/memory-bank/systemPatterns.md index b1d2494..dd35a89 100644 --- a/memory-bank/systemPatterns.md +++ b/memory-bank/systemPatterns.md @@ -1,18 +1,24 @@ -# DomainWatch - System Patterns +# Pounce - System Patterns ## Architecture ``` -┌─────────────────┐ ┌─────────────────┐ -│ Next.js App │────▶│ FastAPI Backend │ -│ (Port 3000) │◀────│ (Port 8000) │ -└─────────────────┘ └────────┬────────┘ - │ - ┌────────────┼────────────┐ - │ │ │ - ┌─────▼─────┐ ┌────▼────┐ ┌────▼────┐ - │ SQLite/ │ │ WHOIS │ │ DNS │ - │ Postgres │ │ Lookups │ │ Lookups │ - └───────────┘ └─────────┘ └─────────┘ +┌─────────────────┐ ┌────────────────────┐ +│ Next.js App │◀────▶│ FastAPI API │ +│ (Port 3000) │ │ (Port 8000) │ +└─────────────────┘ └──────────┬─────────┘ + │ + ┌───────────────┼────────────────┐ + │ │ │ + ┌─────▼─────┐ ┌────▼────┐ ┌─────▼─────┐ + │ Postgres │ │ Redis │ │ External │ + │ (primary) │ │ queue + │ │ I/O (DNS, │ + └───────────┘ │ limiter │ │ WHOIS, HTTP│ + └─────────┘ └───────────┘ + +Separate processes (recommended): +- API: `ENABLE_SCHEDULER=false` +- Scheduler: `python backend/run_scheduler.py` +- Worker: `arq app.jobs.worker.WorkerSettings` ``` ## Design Patterns @@ -32,10 +38,10 @@ ## Authentication Flow ``` 1. User registers → Creates user + free subscription -2. User logs in → Receives JWT token -3. Token stored in localStorage -4. API requests include Bearer token -5. Backend validates token → Returns user data +2. User logs in → Backend sets HttpOnly auth cookie (JWT inside cookie) +3. Frontend calls API with `credentials: 'include'` +4. Backend validates cookie → Returns user data +5. OAuth uses `state` + validated redirects, then sets cookie (no JWT in URL) ``` ## Domain Checking Strategy @@ -49,16 +55,12 @@ ## Scheduler Pattern ``` -APScheduler (AsyncIO mode) +APScheduler (AsyncIO mode) in separate scheduler process │ - └── CronTrigger (daily at 06:00) - │ - └── check_all_domains() - │ - ├── Fetch all domains - ├── Check each with 0.5s delay - ├── Update statuses - └── Log newly available domains + ├── Domain checks (tier-based frequency) + ├── TLD price scrape + change detection + ├── Auction scrape + cleanup + └── Health cache refresh (writes DomainHealthCache used by UI) ``` ## Database Models diff --git a/memory-bank/techContext.md b/memory-bank/techContext.md index a28b811..03af259 100644 --- a/memory-bank/techContext.md +++ b/memory-bank/techContext.md @@ -1,4 +1,4 @@ -# DomainWatch - Technical Context +# Pounce - Technical Context ## Tech Stack @@ -6,8 +6,10 @@ - **Framework:** FastAPI (Python 3.11+) - **Database:** SQLite (development) / PostgreSQL (production) - **ORM:** SQLAlchemy 2.0 with async support -- **Authentication:** JWT with python-jose, bcrypt for password hashing -- **Scheduling:** APScheduler for background jobs +- **Authentication:** HttpOnly cookie auth (JWT in cookie) + OAuth (Google/GitHub) +- **Scheduling:** APScheduler (recommended as separate scheduler process) +- **Job Queue (optional):** Redis + ARQ worker for non-blocking admin/long jobs +- **Observability:** Prometheus metrics endpoint (`/metrics`) ### Frontend - **Framework:** Next.js 14 (App Router) @@ -30,6 +32,8 @@ hushen_test/ │ │ ├── models/ # Database models │ │ ├── schemas/ # Pydantic schemas │ │ ├── services/ # Business logic +│ │ ├── jobs/ # ARQ worker + tasks (optional) +│ │ ├── observability/ # Metrics (Prometheus) │ │ ├── config.py # Settings │ │ ├── database.py # DB configuration │ │ ├── main.py # FastAPI app @@ -45,7 +49,7 @@ hushen_test/ └── memory-bank/ # Project documentation ``` -## API Endpoints +## API Endpoints (high-level) ### Public - `POST /api/v1/check/` - Check domain availability @@ -53,14 +57,16 @@ hushen_test/ ### Authenticated - `POST /api/v1/auth/register` - Register user -- `POST /api/v1/auth/login` - Get JWT token +- `POST /api/v1/auth/login` - Sets HttpOnly cookie session - `GET /api/v1/auth/me` - Current user info - `GET /api/v1/domains/` - List monitored domains +- `GET /api/v1/domains/health-cache` - Bulk cached health reports for watchlist UI - `POST /api/v1/domains/` - Add domain to watchlist - `DELETE /api/v1/domains/{id}` - Remove domain - `POST /api/v1/domains/{id}/refresh` - Manual refresh - `GET /api/v1/subscription/` - User subscription info - `GET /api/v1/subscription/tiers` - Available plans +- `GET /api/v1/dashboard/summary` - Single-call payload for `/terminal/radar` ## Development @@ -81,8 +87,9 @@ npm run dev ``` ## Production Deployment -- Backend: uvicorn with gunicorn -- Frontend: next build && next start +- Backend: run API + scheduler (separate) + optional worker +- Frontend: Next.js (`output: 'standalone'` for Docker) - Database: PostgreSQL recommended +- Redis: recommended (rate limiting storage + job queue) - Reverse proxy: nginx recommended