pounce/README.md
yves.gugger f0cc69ac95 feat: TLD price scraper, .ch domain fix, DB integration
Major changes:
- Add TLD price scraper with Porkbun API (886+ TLDs, no API key needed)
- Fix .ch domain checker using rdap.nic.ch custom RDAP
- Integrate database for TLD price history tracking
- Add admin endpoints for manual scrape and stats
- Extend scheduler with daily TLD price scrape job (03:00 UTC)
- Update API to use DB data with static fallback
- Update README with complete documentation

New files:
- backend/app/services/tld_scraper/ (scraper package)
- TLD_TRACKING_PLAN.md (implementation plan)

API changes:
- POST /admin/scrape-tld-prices - trigger manual scrape
- GET /admin/tld-prices/stats - database statistics
- GET /tld-prices/overview now uses DB data
2025-12-08 09:12:44 +01:00

15 KiB

pounce — Domain Availability Monitoring

A professional full-stack application for monitoring domain name availability with TLD price tracking and intelligence.

Features

Core Functionality

  • Domain Availability Monitoring — Track any domain and get notified when it becomes available
  • TLD Price Intelligence — Compare prices across 886+ TLDs from Porkbun API
  • Automated Price Scraping — Daily cronjob scrapes real TLD prices from public APIs
  • Expiration Tracking — Monitor domain expiration dates and plan ahead
  • Real-time Checks — RDAP, WHOIS, and DNS-based availability verification
  • Swiss Domain Support — Special RDAP integration for .ch/.li domains via nic.ch
  • Historical Data — Track price trends and availability history over time (12 months)

User Experience

  • Modern UI — Clean, minimalist dark-mode design with smooth animations
  • Responsive — Fully optimized for desktop, tablet, and mobile
  • Authentication — Secure JWT-based auth with subscription tiers
  • Dashboard — Personal watchlist with status indicators and actions

Tech Stack

Backend

Technology Purpose
Python 3.12+ Runtime
FastAPI Async web framework
SQLAlchemy 2.0 Async ORM
SQLite/PostgreSQL Database
APScheduler Background job scheduling
python-whois WHOIS domain lookups
whodap RDAP domain lookups
httpx Async HTTP client (for custom RDAP)
BeautifulSoup4 Web scraping (backup)
Pydantic Data validation
JWT (python-jose) Authentication

Frontend

Technology Purpose
Next.js 14 React framework (App Router)
TypeScript Type safety
Tailwind CSS Styling
Zustand State management
Lucide React Icons (outlined style)
clsx Conditional classes

Project Structure

pounce/
├── backend/
│   ├── app/
│   │   ├── api/                 # API endpoints
│   │   │   ├── auth.py          # Authentication (register, login, me)
│   │   │   ├── check.py         # Domain availability check
│   │   │   ├── domains.py       # Domain watchlist CRUD
│   │   │   ├── subscription.py  # User subscription management
│   │   │   ├── tld_prices.py    # TLD pricing data
│   │   │   └── admin.py         # Admin endpoints
│   │   ├── models/              # SQLAlchemy models
│   │   │   ├── user.py          # User model
│   │   │   ├── domain.py        # Domain watchlist model
│   │   │   ├── subscription.py  # Subscription tiers
│   │   │   └── tld_price.py     # TLD pricing model
│   │   ├── schemas/             # Pydantic schemas
│   │   ├── services/            # Business logic
│   │   │   ├── auth.py          # Auth service (JWT, hashing)
│   │   │   ├── domain_checker.py # RDAP/WHOIS/DNS checks
│   │   │   └── tld_scraper/     # TLD price scraping
│   │   │       ├── __init__.py
│   │   │       ├── base.py      # Base scraper class
│   │   │       ├── porkbun.py   # Porkbun API scraper
│   │   │       ├── tld_list.py  # TLD-List scraper (placeholder)
│   │   │       └── aggregator.py # Combines sources, saves to DB
│   │   ├── config.py            # Environment settings
│   │   ├── database.py          # Database connection
│   │   ├── main.py              # FastAPI app entry
│   │   └── scheduler.py         # Background jobs
│   ├── requirements.txt
│   ├── Dockerfile
│   └── env.example
├── frontend/
│   ├── src/
│   │   ├── app/                 # Next.js App Router pages
│   │   │   ├── page.tsx         # Homepage with domain checker
│   │   │   ├── layout.tsx       # Root layout
│   │   │   ├── dashboard/       # User dashboard
│   │   │   ├── login/           # Login page
│   │   │   ├── register/        # Registration page
│   │   │   ├── pricing/         # Pricing plans
│   │   │   └── tld-pricing/     # TLD price intelligence
│   │   │       ├── page.tsx     # TLD overview
│   │   │       └── [tld]/       # TLD detail page
│   │   ├── components/          # React components
│   │   │   ├── Header.tsx       # Navigation header
│   │   │   ├── Footer.tsx       # Site footer
│   │   │   └── DomainChecker.tsx # Domain search component
│   │   └── lib/                 # Utilities
│   │       ├── api.ts           # API client
│   │       └── store.ts         # Zustand store
│   ├── tailwind.config.ts       # Tailwind configuration
│   ├── package.json
│   ├── Dockerfile
│   └── env.example
├── docker-compose.yml           # Docker deployment
├── DEPLOYMENT.md                # Deployment guide
└── README.md

Pages Overview

Route Description Auth Required
/ Homepage with domain checker, features, pricing preview No
/login User login No
/register User registration No
/dashboard Personal domain watchlist Yes
/pricing Subscription plans with FAQ No
/tld-pricing TLD price overview with trends No*
/tld-pricing/[tld] TLD detail with registrar comparison Yes

*Unauthenticated users see limited data with shimmer effects


Quick Start

Prerequisites

  • Python 3.12+
  • Node.js 18+
  • npm or yarn

1. Clone Repository

git clone https://git.6bit.ch/yvg/pounce.git
cd pounce

2. Backend Setup

cd backend

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # Linux/macOS
# .\venv\Scripts\activate  # Windows

# Install dependencies
pip install -r requirements.txt

# Create environment file
cp env.example .env

# Generate secret key and add to .env
python -c "import secrets; print(f'SECRET_KEY={secrets.token_hex(32)}')"

Edit .env:

DATABASE_URL=sqlite+aiosqlite:///./domainwatch.db
SECRET_KEY=your-generated-secret-key
CORS_ORIGINS=http://localhost:3000,http://127.0.0.1:3000

Start backend:

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

3. Frontend Setup

cd frontend

# Install dependencies
npm install

# Create environment file
echo "NEXT_PUBLIC_API_URL=http://localhost:8000" > .env.local

# Start development server
npm run dev

4. Access Application


API Endpoints

Authentication

Method Endpoint Description
POST /api/v1/auth/register Register new user
POST /api/v1/auth/login Login (returns JWT)
GET /api/v1/auth/me Get current user

Domain Check

Method Endpoint Description
POST /api/v1/check Check domain availability

Domain Watchlist

Method Endpoint Description
GET /api/v1/domains List watched domains
POST /api/v1/domains Add domain to watchlist
DELETE /api/v1/domains/{id} Remove domain
POST /api/v1/domains/{id}/refresh Refresh domain status
GET /api/v1/domains/{id}/history Get check history

TLD Prices

Method Endpoint Description
GET /api/v1/tld-prices/overview Get all TLDs overview
GET /api/v1/tld-prices/trending Get trending TLDs
GET /api/v1/tld-prices/{tld} Get TLD details
GET /api/v1/tld-prices/{tld}/history Get price history
GET /api/v1/tld-prices/{tld}/compare Compare registrar prices

Subscription

Method Endpoint Description
GET /api/v1/subscription Get current subscription
POST /api/v1/subscription/upgrade Upgrade plan

Admin

Method Endpoint Description
GET /api/v1/admin/users List all users
POST /api/v1/admin/upgrade-user Upgrade user subscription
POST /api/v1/admin/scrape-tld-prices Manually trigger TLD price scrape
GET /api/v1/admin/tld-prices/stats Get TLD price database stats

Subscription Tiers

Feature Starter (Free) Professional ($4.99/mo) Enterprise ($9.99/mo)
Domains 3 25 100
Check Frequency Daily Daily Hourly
Notifications Email Priority Email Priority Email
WHOIS Data Basic Full Full
Check History 30 days Unlimited
Expiration Tracking
API Access

Environment Variables

Backend (.env)

Variable Description Default
DATABASE_URL Database connection string sqlite+aiosqlite:///./domainwatch.db
SECRET_KEY JWT signing key (min 32 chars) Required
CORS_ORIGINS Allowed origins (comma-separated) http://localhost:3000
ACCESS_TOKEN_EXPIRE_MINUTES JWT token lifetime 10080 (7 days)
SCHEDULER_CHECK_INTERVAL_HOURS Domain check interval 24
SMTP_HOST Email server host Optional
SMTP_PORT Email server port 587
SMTP_USER Email username Optional
SMTP_PASSWORD Email password Optional

Frontend (.env.local)

Variable Description Default
NEXT_PUBLIC_API_URL Backend API URL http://localhost:8000

Production Deployment

# Clone repository
git clone https://git.6bit.ch/yvg/pounce.git
cd pounce

# Set environment variables
export SECRET_KEY=$(python3 -c "import secrets; print(secrets.token_hex(32))")

# Build and start
docker-compose up -d --build

Manual Deployment

See DEPLOYMENT.md for detailed instructions including:

  • Systemd service setup
  • PM2 for Node.js
  • Nginx reverse proxy configuration
  • SSL/HTTPS with Let's Encrypt
  • PostgreSQL database setup

Domain Checking Methods

pounce uses multiple methods to verify domain availability:

  1. Custom RDAP — For TLDs with own RDAP servers (e.g., .ch, .li via rdap.nic.ch)
  2. RDAP via whodap — Modern protocol with detailed data, supported by major registries
  3. WHOIS (Fallback) — Traditional protocol for domains without RDAP support
  4. DNS (Quick check) — Fast initial availability check via DNS queries

The system automatically falls back to the next method if one fails.

Supported TLDs with Custom RDAP

TLD RDAP Server Notes
.ch rdap.nic.ch Swiss domains - WHOIS blocked
.li rdap.nic.ch Liechtenstein - same registry

Standard RDAP TLDs

.com, .net, .org, .info, .biz, .io, .co, .ai, .app, .dev, .xyz, .me, and many more.


TLD Price Scraping

pounce automatically scrapes TLD prices from public sources for price intelligence and trend tracking.

Data Source

Primary: Porkbun Public API (no API key required!)

POST https://api.porkbun.com/api/json/v3/pricing/get
  • 886+ TLDs with pricing
  • Registration, Renewal, Transfer prices
  • No authentication needed
  • Updated daily via scheduler

Scheduler Jobs

Job Schedule Description
Domain Check Configurable (default: daily) Checks all watched domains
TLD Price Scrape Daily at 03:00 UTC Scrapes current TLD prices

Manual Scrape

Trigger a manual TLD price scrape:

curl -X POST http://localhost:8000/api/v1/admin/scrape-tld-prices

Response:

{
  "message": "TLD price scrape completed",
  "status": "success",
  "tlds_scraped": 886,
  "prices_saved": 886,
  "sources_succeeded": 1
}

Database Stats

Get current TLD price data statistics:

curl http://localhost:8000/api/v1/admin/tld-prices/stats

Response:

{
  "total_records": 886,
  "unique_tlds": 886,
  "unique_registrars": 1,
  "latest_record": "2025-12-08T08:04:54",
  "data_range_days": 0
}

Price Data Model

TLDPrice:
  - tld: str              # e.g., "com", "io", "ai"
  - registrar: str        # e.g., "porkbun"
  - registration_price: float
  - renewal_price: float
  - transfer_price: float
  - currency: str         # "USD"
  - recorded_at: datetime # For historical tracking

Future Data Sources

The scraper architecture supports multiple sources:

  1. Porkbun API (active)
  2. TLD-List.com (blocked - placeholder)
  3. Namecheap Affiliate (requires signup)
  4. Cloudflare (requires account)

Design System

Colors

  • Background: Dark (#0a0a0a)
  • Foreground: Light (#fafafa)
  • Accent: Teal (#00d4aa)
  • Warning: Orange (#f97316)
  • Success: Green (same as accent)

Typography

  • Display: Playfair Display (serif)
  • Body: Inter (sans-serif)
  • Mono: JetBrains Mono (code)

Components

  • All icons use outlined style (Lucide React)
  • Minimalist, award-winning aesthetic
  • Smooth animations and transitions
  • Responsive design (mobile-first)

Development

Backend

cd backend
source venv/bin/activate

# Run with auto-reload
uvicorn app.main:app --reload

# API docs available at:
# http://localhost:8000/docs (Swagger)
# http://localhost:8000/redoc (ReDoc)

Frontend

cd frontend

# Development server
npm run dev

# Lint
npm run lint

# Build for production
npm run build

# Start production server
npm start

Troubleshooting

Backend won't start

  1. Check Python version: python3 --version (needs 3.12+)
  2. Ensure virtual environment is activated
  3. Check .env file exists and has valid SECRET_KEY
  4. Check for syntax errors in config files

Frontend can't connect to backend

  1. Ensure backend is running on port 8000
  2. Check NEXT_PUBLIC_API_URL in .env.local
  3. Verify CORS origins in backend .env
  4. Check browser console for CORS errors

Domain checks failing

  1. Some TLDs may have rate limits
  2. Certain ccTLDs require special handling (see below)
  3. Check backend logs for specific errors

Swiss/Liechtenstein domains (.ch, .li):

  • WHOIS is blocked for automated requests
  • pounce uses the official rdap.nic.ch API instead
  • This is automatic - no configuration needed

Other ccTLDs:

  • Some country code TLDs don't support RDAP
  • pounce falls back to DNS checking
  • Results may be less detailed (no registrar info)

Database errors

  1. Delete domainwatch.db to reset (dev only)
  2. Check database URL format in .env
  3. Ensure write permissions on database directory

License

MIT License


Support

For issues and feature requests, please open a GitHub issue or contact support@pounce.dev