Project

Development Setup

Get a working development environment in under a minute.

Prerequisites

  • Python 3.9+ (3.11+ recommended for best performance)

  • uv – fast Python package manager from Astral

  • git with pre-commit support

Clone & Install

git clone https://github.com/wyattowalsh/proxywhirl.git
cd proxywhirl
uv sync
pre-commit install

Verify Installation

# Run the test suite
uv run pytest tests/ -q

# Check linting
uv run ruff check proxywhirl/ tests/

# Check types
uv run ty check proxywhirl/

Make Commands

All common operations are available via make:

Command

Equivalent

Purpose

make test

uv run pytest tests/ -q

Run all tests

make lint

uv run ruff check proxywhirl/ tests/

Lint with ruff

make format

uv run ruff format proxywhirl/ tests/

Auto-format code

make type-check

uv run ty check proxywhirl/

Type check with ty (Astral)

make quality-gates

all of the above

Full pre-merge validation

make commit

conventional commit

Commitizen-guided commit

Always use uv run

Never run bare pytest, python, or pip. Always prefix with uv run to use the project’s virtual environment.

Contributing

Workflow

  1. Fork & clone the repository

  2. Create a branch from main: git checkout -b feature/my-feature or fix/my-fix

  3. Write tests first – every new feature needs test coverage

  4. Implement your changes

  5. Run quality gates: make quality-gates

  6. Commit using conventional commits: <type>(<scope>): <description>

  7. Open a PR against main

Branch Naming

Pattern

Use

feature/*

New features

fix/*

Bug fixes

claude/*

AI-assisted changes

develop

Integration branch

Commit Convention

Commits follow Conventional Commits:

feat(strategies): add failover chain support
fix(cache): handle L2 encryption key rotation
docs(guides): expand async client examples
test(rotator): add property tests for weighted selection
refactor(storage): simplify SQLite connection pooling
perf(strategies): reduce EMA calculation overhead
chore(deps): update httpx to 0.28

Types: feat, fix, docs, chore, refactor, test, perf

Code Style

  • Line length: 100 characters

  • Quotes: double quotes

  • Indent: 4 spaces

  • Naming: PascalCase (classes), snake_case (functions), UPPER_SNAKE_CASE (constants)

  • Imports: sorted by ruff (isort-compatible)

  • Types: always type-hint public functions; use from __future__ import annotations

  • Models: Pydantic with ConfigDict(frozen=True, extra="forbid")

  • Logging: use loguru (never print)

Pre-commit Hooks

The following hooks run automatically on every commit:

  • ruff – lint and format check

  • commitizen – conventional commit message validation

  • private key detection – blocks accidental secret commits

  • large file check – prevents binary bloat

Testing Guidelines

Marker

Purpose

Example

(none)

Unit tests

tests/unit/test_strategies.py

@pytest.mark.slow

Long-running tests

Large pool simulations

@pytest.mark.integration

Integration tests

tests/integration/

@pytest.mark.network

Requires network

Live proxy validation

HTTP mocking uses respx (not responses). Async tests run with asyncio_mode = "auto".

# Run specific test file
uv run pytest tests/unit/test_strategies.py -v

# Run tests matching a pattern
uv run pytest -k "test_weighted" -v

# Run with coverage
uv run pytest tests/ --cov=proxywhirl --cov-report=term-missing

Extending with Custom Strategies

ProxyWhirl supports a plugin architecture via StrategyRegistry. Implement the RotationStrategy protocol and register your strategy:

from proxywhirl import StrategyRegistry, RotationStrategy

class MyStrategy:
    def select(self, pool, context=None):
        return pool.get_healthy_proxies()[0]

    def record_result(self, proxy, success, response_time_ms):
        pass

registry = StrategyRegistry()
registry.register_strategy("my-strategy", MyStrategy)

See Python API for the full RotationStrategy protocol and Advanced Rotation Strategies for composite strategy patterns.

CI/CD Pipelines

All workflows live in .github/workflows/:

Workflow

Trigger

Purpose

ci.yml

push / PR

Lint, type-check, test matrix (Python 3.9-3.13)

security.yml

push / schedule

Dependency audit, secret scanning

generate-proxies.yml

cron (every 6h)

Fetch proxies from all built-in sources, update proxywhirl.db

validate-sources.yml

push / PR

Verify proxy source URLs are reachable

update-readme-stats.yml

after proxy generation

Update README badge counts

release.yml

tag push

Build and publish to PyPI

cd.yml

release

Deploy docs and dashboard

Quality Gate Pipeline

The CI pipeline enforces this dependency order:

Level 0: lint + format-check + type-check  (parallel, no deps)
    |
Level 1: unit tests  (depends on Level 0)
    |
Level 2: integration + property tests  (depends on Level 1)
    |
Level 3: coverage + build  (depends on all tests)

Architecture

Module Map

Module

Purpose

models/

Pydantic models: Proxy, ProxyPool, Session, SelectionContext, *Config

strategies/

9 rotation strategies + RotationStrategy protocol + StrategyRegistry

rotator/

ProxyWhirl (sync) and AsyncProxyWhirl (async) – core HTTP clients

storage.py

SQLiteStorage, FileStorage, ProxyTable (SQLModel ORM)

fetchers.py

ProxyFetcher, ProxyValidator, parsers (JSON, CSV, PlainText, HTMLTable)

sources.py

Hundreds of pre-configured ProxySourceConfig instances (ALL_SOURCES, RECOMMENDED_SOURCES)

cache/

CacheManager with L1 in-memory, L2 encrypted disk, L3 SQLite tiers

circuit_breaker/

CircuitBreaker, AsyncCircuitBreaker with state machine

retry/

RetryExecutor, RetryPolicy, backoff strategies (exponential, linear)

rate_limiting/

RateLimiter (token bucket), sync and async variants

api/

FastAPI REST API (/api/v1/* routes) with OpenAPI

mcp/

MCP server for AI assistant integration (Claude, GPT)

cli.py

Typer CLI (proxywhirl command, 9 subcommands)

config.py

TOML config loader, CLIConfig, DataStorageConfig

exceptions.py

ProxyWhirlError hierarchy, ProxyErrorCode enum, URL redaction

safe_regex.py

ReDoS-safe regex utilities

geo.py

IP geolocation enrichment (GeoIP2)

For the full public API surface, see Python API.

Key Exports

The top-level proxywhirl package exports:

  • Rotators: ProxyWhirl, AsyncProxyWhirl

  • Models: Proxy, ProxyPool, Session, ProxySource, HealthStatus, SelectionContext, ProxyCredentials, ProxyConfiguration

  • Strategies: RoundRobinStrategy, RandomStrategy, WeightedStrategy, LeastUsedStrategy, PerformanceBasedStrategy, SessionPersistenceStrategy, GeoTargetedStrategy, CostAwareStrategy, CompositeStrategy

  • Config: StrategyConfig, CircuitBreakerConfig, RetryPolicy, CacheConfig, DataStorageConfig

  • Components: CacheManager, CircuitBreaker, AsyncCircuitBreaker, RetryExecutor, ProxyFetcher, ProxyValidator, BrowserRenderer

  • Sources: ALL_SOURCES, RECOMMENDED_SOURCES, ALL_HTTP_SOURCES, ALL_SOCKS4_SOURCES, ALL_SOCKS5_SOURCES, API_SOURCES

  • Utilities: configure_logging, encrypt_credentials, decrypt_credentials, deduplicate_proxies

Environment Variables

Variable

Component

Purpose

PROXYWHIRL_KEY

CLI/config

Master encryption key

PROXYWHIRL_CACHE_ENCRYPTION_KEY

cache

Fernet key for L2 cache

PROXYWHIRL_STORAGE_PATH

api

SQLite database path

PROXYWHIRL_API_KEY

api

API authentication key

PROXYWHIRL_MCP_API_KEY

mcp

MCP server auth key

PROXYWHIRL_MCP_DB

mcp

MCP database path

See Configuration Reference for the full TOML configuration reference and environment variable override behavior.

Current Health

  • Tests: 2700+ passing across unit, integration, property, contract, and benchmark suites

  • Performance: All selection strategies operate within 2.8-26 us (<5 ms target)

  • Proxy sources: Hundreds of sources, refreshed every 6 hours via CI

  • Python: Tested on 3.9, 3.10, 3.11, 3.12, and 3.13

Changelog Highlights

Recent

  • Composite strategies – chain filters and selectors with <5 us overhead via CompositeStrategy

  • Cost-aware strategy – budget-optimize proxy selection with CostAwareStrategy

  • MCP server – Model Context Protocol integration for AI assistants

  • Free proxy lists – auto-updated every 6 hours from hundreds of sources

  • Interactive dashboard – browse and export at proxywhirl.com

  • Rate limiting API – token bucket algorithm, per-proxy and global limits

  • Cache encryption – Fernet-encrypted L2 cache tier

Phase Completion Status

Phase

Status

Notable wins

Phase 6 – Performance-based strategy

Complete

EMA-driven scoring delivers 15-25% faster selection with <26 us overhead.

Phase 7 – Session persistence

Complete

Sticky sessions with TTL cleanup and zero request leakage at 99.9% stickiness.

Phase 8 – Geo-targeted routing

Complete

Region-aware filtering validated against 100% accuracy in SC-006.

Phase 9 – Strategy composition

Complete

Composite pipelines, hot-swapping under load, and plugin registry (SC-009/010).

Phase 10 – Polish & validation

In progress

Optional tasks: coverage uplift, large-scale property tests, release packaging.

License

ProxyWhirl is open source software released under the MIT License.

Related: Automation & CI/CD Runbook for CI integration | Configuration Reference for config reference | Getting Started for quickstart

Historical reports

Detailed phase completion reports live alongside the source tree under docs/ (e.g., PHASE_9_COMPLETION.md) and capture acceptance criteria per milestone.