Troubleshooting Guide

This guide helps you diagnose and resolve common issues with ProxyWhirl. Use the error codes and solutions below to quickly fix problems.


Table of Contents


Error Codes Reference

ProxyWhirl uses structured error codes for programmatic error handling. All exceptions inherit from ProxyWhirlError and include metadata for debugging.

PROXY_POOL_EMPTY

Error Class: ProxyPoolEmptyError Retry Recommended: No

Cause: No proxies are available in the pool for selection.

Solutions:

  1. Add proxies to the pool:

    rotator.add_proxy("http://proxy1.example.com:8080")
    
  2. Enable auto-fetch from sources:

    proxywhirl fetch
    
  3. Load from our free proxy lists:

    import httpx
    response = httpx.get("https://your-docs-site.com/proxy-lists/http.txt")
    for proxy in response.text.strip().split("\n"):
        rotator.add_proxy(f"http://{proxy}")
    
  4. Check if proxies were filtered out by health checks:

    proxywhirl pool list
    proxywhirl health
    

Related Code:


PROXY_VALIDATION_FAILED

Error Class: ProxyValidationError Retry Recommended: No

Cause: Proxy URL or configuration is invalid.

Common Issues:

  • Invalid URL format (missing scheme, port, or hostname)

  • Unsupported protocol (only http://, https://, socks4://, socks5:// are supported)

  • Improperly encoded credentials

Solutions:

  1. Verify URL format:

    # ✅ Correct formats
    "http://proxy.example.com:8080"
    "socks5://user:pass@proxy.example.com:1080"
    
    # ❌ Invalid formats
    "proxy.example.com:8080"  # Missing scheme
    "ftp://proxy.example.com:8080"  # Unsupported protocol
    
  2. Check credential encoding:

    from urllib.parse import quote
    username = quote("user@domain.com")
    password = quote("p@ssw0rd!")
    proxy_url = f"http://{username}:{password}@proxy.example.com:8080"
    
  3. Validate using CLI:

    proxywhirl pool test http://proxy.example.com:8080
    

Related Code:


PROXY_CONNECTION_FAILED

Error Class: ProxyConnectionError Retry Recommended: Yes

Cause: Unable to establish connection through the proxy.

Common Issues:

  • Proxy server is down or unreachable

  • Network connectivity issues

  • Firewall blocking proxy ports

  • Proxy doesn’t support the target protocol

Solutions:

  1. Test proxy connectivity:

    proxywhirl pool test http://proxy.example.com:8080
    
  2. Check network accessibility:

    curl -x http://proxy.example.com:8080 https://httpbin.org/ip
    
  3. Increase timeout:

    rotator = ProxyWhirl(
        proxies=["http://proxy1:8080"],
        timeout=60  # Default is 30 seconds
    )
    
  4. Verify proxy supports target protocol:

    • HTTP proxies cannot tunnel HTTPS CONNECT requests if not configured

    • SOCKS proxies may have protocol restrictions

  5. Enable verbose logging:

    from loguru import logger
    logger.add("debug.log", level="DEBUG")
    

Related Code:


PROXY_AUTH_FAILED

Error Class: ProxyAuthenticationError Retry Recommended: No

Cause: Proxy authentication failed.

Common Issues:

  • Incorrect username or password

  • Credentials expired

  • Proxy requires different auth method (e.g., IP whitelist instead of user/pass)

Solutions:

  1. Verify credentials:

    from pydantic import SecretStr
    proxy = Proxy(
        url="http://proxy.example.com:8080",
        username=SecretStr("correct_username"),
        password=SecretStr("correct_password")
    )
    
  2. Test with curl:

    curl -x http://user:pass@proxy.example.com:8080 https://httpbin.org/ip
    
  3. Check if proxy uses IP authentication:

    • Some proxies whitelist IPs instead of using credentials

    • Contact your proxy provider to verify authentication method

  4. Ensure credentials aren’t expired:

    • Rotating residential proxies may have time-limited credentials

Related Code:


PROXY_FETCH_FAILED

Error Class: ProxyFetchError Retry Recommended: Yes

Cause: Fetching proxies from external sources failed.

Common Issues:

  • Source URL is unreachable

  • API credentials invalid or expired

  • Response format doesn’t match expectations

  • Rate limit hit

Solutions:

  1. Test sources manually:

    proxywhirl sources --validate
    
  2. Check specific source:

    curl https://api.proxyscrape.com/v2/?request=get&protocol=http
    
  3. Review source health:

    proxywhirl sources -v -f  # Fail on unhealthy sources
    
  4. Use offline mode with cached proxies:

    rotator = ProxyWhirl(
        proxies=[],  # Empty, will load from storage
        storage_path="proxywhirl.db"
    )
    

Related Code:


PROXY_STORAGE_FAILED

Error Class: ProxyStorageError Retry Recommended: No

Cause: Proxy storage operations failed.

Common Issues:

  • Insufficient file system permissions

  • Disk space full

  • Database corruption

  • Path doesn’t exist or isn’t writable

Solutions:

  1. Check file permissions:

    ls -la proxywhirl.db
    chmod 644 proxywhirl.db
    
  2. Verify disk space:

    df -h .
    
  3. Check directory is writable:

    mkdir -p .cache/proxies
    chmod 755 .cache/proxies
    
  4. Reinitialize database:

    rm proxywhirl.db
    proxywhirl fetch --save-db
    

Related Code:


CACHE_CORRUPTED

Error Class: CacheCorruptionError Retry Recommended: No

Cause: Cache data is corrupted and cannot be recovered.

Common Issues:

  • Disk corruption

  • Interrupted write operation

  • Version incompatibility after upgrade

  • Encryption key mismatch

Solutions:

  1. Clear cache:

    rm -rf .cache/proxies/*
    rm -rf .cache/db/proxywhirl.db
    
  2. Reinitialize cache:

    from proxywhirl.cache import CacheManager
    cache = CacheManager(
        l1_max_entries=1000,
        l2_dir=".cache/proxies",
        l3_db_path=".cache/db/proxywhirl.db"
    )
    await cache.clear()  # Clear all tiers
    
  3. Check for disk errors:

    # macOS
    diskutil verifyVolume /
    
    # Linux
    sudo fsck /dev/sda1
    

Related Code:


CACHE_STORAGE_FAILED

Error Class: CacheStorageError Retry Recommended: Yes

Cause: Cache storage backend is unavailable.

Common Issues:

  • Redis/Memcached server down

  • Network connectivity issues

  • Invalid credentials

  • Backend overloaded

Solutions:

  1. Verify cache backend is running:

    # Redis
    redis-cli ping
    
    # Check connection
    telnet localhost 6379
    
  2. Disable external cache, use local:

    # Use file-based L2/L3 cache instead
    config = CLIConfig(
        cache_enabled=True,
        cache_l2_dir=".cache/proxies",
        cache_l3_db_path=".cache/db/proxywhirl.db"
    )
    
  3. Check credentials:

    export REDIS_PASSWORD="your-password"
    

Related Code:


CACHE_VALIDATION_FAILED

Error Class: CacheValidationError Retry Recommended: No

Cause: Cache entry fails validation.

Common Issues:

  • Schema version mismatch after upgrade

  • Invalid data types

  • Missing required fields

Solutions:

  1. Clear cache and rebuild:

    rm -rf .cache/
    proxywhirl fetch
    
  2. Check ProxyWhirl version:

    pip show proxywhirl
    pip install --upgrade proxywhirl
    

Related Code:


TIMEOUT

Error Code: TIMEOUT Retry Recommended: Yes

Cause: Request timed out.

Common Issues:

  • Slow proxy server

  • Network congestion

  • Target server slow to respond

  • Timeout value too low

Solutions:

  1. Increase timeout:

    rotator = ProxyWhirl(
        proxies=["http://proxy1:8080"],
        timeout=60  # Increase from default 30s
    )
    
  2. Configure via CLI:

    proxywhirl config set timeout 60
    
  3. Per-request timeout:

    proxywhirl request --timeout 90 https://example.com
    
  4. Check proxy latency:

    proxywhirl health --target-url https://httpbin.org/ip
    

Related Code:


NETWORK_ERROR

Error Code: NETWORK_ERROR Retry Recommended: Yes

Cause: Generic network error.

Common Issues:

  • DNS resolution failed

  • Network interface down

  • ISP blocking connections

  • Firewall rules

Solutions:

  1. Test network connectivity:

    ping 8.8.8.8
    curl https://httpbin.org/ip
    
  2. Check DNS:

    nslookup proxy.example.com
    dig proxy.example.com
    
  3. Verify firewall rules:

    # macOS
    sudo pfctl -sr
    
    # Linux (iptables)
    sudo iptables -L
    

Related Code:


INVALID_CONFIGURATION

Error Code: INVALID_CONFIGURATION Retry Recommended: No

Cause: Configuration file or settings are invalid.

Common Issues:

  • Malformed TOML syntax

  • Invalid field values

  • Missing required fields

  • Type mismatch

Solutions:

  1. Validate configuration:

    proxywhirl config show
    
  2. Reinitialize config:

    proxywhirl config init
    
  3. Check TOML syntax:

    # Use a TOML validator
    python -c "import tomllib; tomllib.load(open('.proxywhirl.toml', 'rb'))"
    
  4. Review allowed values:

    • rotation_strategy: round-robin, random, weighted, least-used

    • default_format: text, json, csv (human and table are deprecated aliases for text)

    • storage_backend: file, sqlite, memory

Related Code:


Common Issues

Issue: All proxies marked as unhealthy

Symptoms:

  • ProxyPoolEmptyError after some usage

  • Health check shows all proxies failed

Diagnosis:

proxywhirl health --verbose
proxywhirl pool list

Solutions:

  1. Check if proxies are actually dead:

    for proxy in $(cat proxies.txt); do
        curl -x http://$proxy https://httpbin.org/ip -m 5 || echo "$proxy FAILED"
    done
    
  2. Lower health check sensitivity:

    from proxywhirl import ProxyWhirl
    from proxywhirl.circuit_breaker import CircuitBreaker
    
    # Increase failure threshold
    rotator = ProxyWhirl(
        proxies=[...],
        circuit_breaker=CircuitBreaker(
            failure_threshold=10,  # Default is 5
            recovery_timeout=60    # Retry after 1 min
        )
    )
    
  3. Use more lenient target URL:

    # Instead of a strict endpoint
    proxywhirl health --target-url https://example.com
    
  4. Disable health checks temporarily:

    rotator = ProxyWhirl(
        proxies=[...],
        health_check_interval=0  # Disable automatic checks
    )
    

Issue: Rate limiting / 429 errors

Symptoms:

  • HTTP 429 Too Many Requests

  • Target blocking your requests

Diagnosis:

from loguru import logger
logger.add("requests.log", level="DEBUG")
# Check request frequency in logs

Solutions:

  1. Enable rate limiting:

    from proxywhirl.rate_limiting import RateLimiter
    
    rate_limiter = RateLimiter(
        max_requests_per_minute=30,
        max_requests_per_hour=500,
        burst_size=10
    )
    
    rotator = ProxyWhirl(
        proxies=[...],
        rate_limiter=rate_limiter
    )
    
  2. Add delays between requests:

    import time
    for url in urls:
        response = rotator.get(url)
        time.sleep(2)  # 2 second delay
    
  3. Rotate proxies more aggressively:

    # Use random strategy instead of round-robin
    rotator = ProxyWhirl(
        proxies=[...],
        strategy="random"
    )
    
  4. Use session persistence:

    from proxywhirl.strategies import SessionPersistenceStrategy
    
    rotator = ProxyWhirl(
        proxies=[...],
        strategy=SessionPersistenceStrategy(
            session_duration=300  # 5 minutes per session
        )
    )
    

Issue: SSL/TLS verification errors

Symptoms:

  • SSLError: [SSL: CERTIFICATE_VERIFY_FAILED]

  • HTTPS requests fail but HTTP works

Diagnosis:

proxywhirl pool test http://proxy.example.com:8080 --target-url https://httpbin.org/get

Solutions:

  1. Disable SSL verification (NOT RECOMMENDED for production):

    rotator = ProxyWhirl(
        proxies=[...],
        verify_ssl=False
    )
    
  2. Use HTTP proxy for HTTPS targets:

    # HTTP proxy can still tunnel HTTPS via CONNECT
    rotator.add_proxy("http://proxy.example.com:8080")
    response = rotator.get("https://secure-site.com")
    
  3. Install CA certificates:

    # macOS
    brew install ca-certificates
    
    # Ubuntu/Debian
    sudo apt-get install ca-certificates
    
    # Update certifi
    pip install --upgrade certifi
    
  4. Use SOCKS proxy instead:

    rotator.add_proxy("socks5://proxy.example.com:1080")
    

Issue: Memory leaks / growing memory usage

Symptoms:

  • Memory usage grows over time

  • Eventually crashes with MemoryError

Diagnosis:

import tracemalloc
tracemalloc.start()

# Run your code
# ...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
    print(stat)

Solutions:

  1. Use LRU client pool (built-in):

    # ProxyWhirl automatically uses LRUClientPool (max 100 clients)
    rotator = ProxyWhirl(proxies=[...])
    
  2. Close rotator when done:

    try:
        rotator = ProxyWhirl(proxies=[...])
        # ... use rotator
    finally:
        rotator.close()  # Close all httpx clients
    
  3. Use context manager:

    with ProxyWhirl(proxies=[...]) as rotator:
        response = rotator.get("https://example.com")
    
  4. Clear cache periodically:

    await rotator.cache_manager.cleanup()
    

Issue: Database locked (SQLite)

Symptoms:

  • sqlite3.OperationalError: database is locked

  • Concurrent access failures

Diagnosis:

lsof proxywhirl.db  # Check what's accessing it

Solutions:

  1. Use WAL mode (Write-Ahead Logging):

    import sqlite3
    conn = sqlite3.connect("proxywhirl.db")
    conn.execute("PRAGMA journal_mode=WAL")
    conn.close()
    
  2. Increase timeout:

    from proxywhirl.storage import SQLiteStorage
    
    storage = SQLiteStorage(
        "proxywhirl.db",
        timeout=30.0  # Default is 5.0
    )
    
  3. Use file locking:

    # CLI already uses file locks by default
    proxywhirl fetch
    
    # Disable with --no-lock if needed
    proxywhirl fetch --no-lock
    
  4. Close connections properly:

    async with SQLiteStorage("proxywhirl.db") as storage:
        # Use storage
        pass
    # Automatically closed
    

Issue: Proxy works in browser but not in ProxyWhirl

Symptoms:

  • Browser proxy settings work fine

  • ProxyWhirl gets connection errors

Diagnosis:

# Test with curl (same as ProxyWhirl)
curl -x http://proxy.example.com:8080 https://httpbin.org/ip -v

Possible Causes:

  1. Browser uses system proxy settings:

    • Browsers may use PAC (Proxy Auto-Config) files

    • ProxyWhirl requires explicit proxy URL

  2. Browser handles auth differently:

    • Browser may cache credentials

    • ProxyWhirl requires credentials in URL or config

  3. Browser uses SOCKS proxy:

    # Try SOCKS instead of HTTP
    rotator.add_proxy("socks5://proxy.example.com:1080")
    
  4. Check proxy type:

    # Test with different protocols
    curl -x http://proxy:8080 https://httpbin.org/ip
    curl -x socks5://proxy:1080 https://httpbin.org/ip
    

Issue: Config file not found

Symptoms:

  • Config file not found: /path/to/.proxywhirl.toml

  • CLI falls back to defaults

Solutions:

  1. Check discovery order:

    • --config /path/to/file (explicit)

    • ./pyproject.toml (project-local, needs [tool.proxywhirl] section)

    • ~/.config/proxywhirl/config.toml (user-global)

  2. Initialize config:

    proxywhirl config init
    
  3. Use explicit path:

    proxywhirl --config ./custom.toml pool list
    
  4. Verify file exists:

    ls -la .proxywhirl.toml
    cat .proxywhirl.toml
    

Debugging Tips

Enable Debug Logging

Method 1: Environment Variable

export LOGURU_LEVEL=DEBUG
proxywhirl fetch

Method 2: Programmatic

from loguru import logger

# Remove default handler
logger.remove()

# Add with DEBUG level
logger.add(
    "debug.log",
    level="DEBUG",
    format="{time} {level} {message}",
    rotation="10 MB"
)

# Now all ProxyWhirl operations will log verbosely
from proxywhirl import ProxyWhirl
rotator = ProxyWhirl(proxies=["http://proxy1:8080"])

Method 3: CLI Flag

proxywhirl --verbose pool list
proxywhirl -v health

Inspect Exception Metadata

All ProxyWhirl exceptions include rich metadata:

from proxywhirl import ProxyWhirl
from proxywhirl.exceptions import ProxyWhirlError

try:
    rotator = ProxyWhirl(proxies=[])
    rotator.get("https://example.com")
except ProxyWhirlError as e:
    # Access error details
    print(f"Error Code: {e.error_code}")
    print(f"Proxy URL: {e.proxy_url}")  # Redacted
    print(f"Error Type: {e.error_type}")
    print(f"Retry Recommended: {e.retry_recommended}")
    print(f"Attempt Count: {e.attempt_count}")
    print(f"Metadata: {e.metadata}")

    # Convert to dict for logging
    import json
    print(json.dumps(e.to_dict(), indent=2))

Use Test Mode

Test proxies without making actual requests:

# Test individual proxy
proxywhirl pool test http://proxy.example.com:8080

# Test with custom target
proxywhirl pool test http://proxy:8080 --target-url https://api.example.com

# Allow testing against localhost (SSRF protection disabled)
proxywhirl pool test http://proxy:8080 --allow-private

Monitor Health in Real-Time

# Continuous health monitoring
proxywhirl health --continuous --interval 30

# Output:
# Check #1
# Status: 5 healthy | 2 degraded | 1 failed
#
# Check #2
# Status: 4 healthy | 3 degraded | 1 failed

Export Metrics for Analysis

from proxywhirl import RetryMetrics

# Access retry statistics
metrics = rotator.retry_executor.metrics
print(f"Total attempts: {metrics.total_attempts}")
print(f"Success rate: {metrics.success_rate:.2%}")
print(f"Avg attempts: {metrics.average_attempts_per_operation:.2f}")

# Export to JSON
import json
with open("metrics.json", "w") as f:
    json.dump(metrics.to_dict(), f, indent=2)

Validate Sources

Check which proxy sources are working:

# List all sources
proxywhirl sources

# Validate sources (slow, tests all 60+)
proxywhirl sources --validate

# CI mode (exit with error if any unhealthy)
proxywhirl sources --validate --fail-on-unhealthy

Check Database Health

# Open SQLite database
sqlite3 proxywhirl.db

# Check tables
.tables

# View proxy stats
SELECT url, health_status, total_requests, success_rate
FROM proxies
ORDER BY success_rate DESC
LIMIT 10;

# Check database size
.dbinfo

# Exit
.exit

FAQ

How do I add custom proxy sources?

Answer:

Edit your sources configuration in proxywhirl/sources.py or use the API:

from proxywhirl.sources import ProxySource
from proxywhirl.fetchers import fetch_from_source

# Define custom source
custom_source = ProxySource(
    name="MyCustomSource",
    url="https://my-proxy-api.com/list",
    format="plain_text",  # or "json"
    protocol="http"
)

# Fetch proxies
proxies = await fetch_from_source(custom_source)

# Add to rotator
for proxy in proxies:
    rotator.add_proxy(proxy)

How do I configure rotation strategies?

Answer:

ProxyWhirl supports multiple strategies:

1. Round-Robin (default):

rotator = ProxyWhirl(proxies=[...], strategy="round-robin")

2. Random:

rotator = ProxyWhirl(proxies=[...], strategy="random")

3. Weighted (performance-based):

rotator = ProxyWhirl(proxies=[...], strategy="weighted")
# Faster proxies get more traffic

4. Least-Used:

rotator = ProxyWhirl(proxies=[...], strategy="least-used")
# Balances load across proxies

5. Session Persistence:

from proxywhirl.strategies import SessionPersistenceStrategy

strategy = SessionPersistenceStrategy(
    session_duration=300,  # 5 minutes
    identifier_fn=lambda req: req.url.host  # Sticky sessions per domain
)
rotator = ProxyWhirl(proxies=[...], strategy=strategy)

6. Geo-Targeted:

from proxywhirl.strategies import GeoTargetedStrategy

strategy = GeoTargetedStrategy(
    target_countries=["US", "GB", "CA"]
)
rotator = ProxyWhirl(proxies=[...], strategy=strategy)

7. Performance-Based:

from proxywhirl.strategies import PerformanceBasedStrategy

strategy = PerformanceBasedStrategy(
    latency_weight=0.7,
    success_rate_weight=0.3
)
rotator = ProxyWhirl(proxies=[...], strategy=strategy)

See proxywhirl/strategies.py for implementation details.


How do I persist proxies across runs?

Answer:

Use SQLite storage:

from proxywhirl import ProxyWhirl
from proxywhirl.storage import SQLiteStorage

# Initialize storage
storage = SQLiteStorage("proxywhirl.db")
await storage.initialize()

# Create rotator with storage
rotator = ProxyWhirl(
    proxies=[],  # Will load from storage
    storage=storage
)

# Add proxies (automatically persisted)
rotator.add_proxy("http://proxy1:8080")

# Proxies are now saved in proxywhirl.db

Or use the CLI:

# Fetch and save to database
proxywhirl fetch --save-db --db proxywhirl.db

# Later, load from database
python -c "
from proxywhirl.storage import SQLiteStorage
import asyncio

async def load():
    storage = SQLiteStorage('proxywhirl.db')
    await storage.initialize()
    proxies = await storage.load()
    print(f'Loaded {len(proxies)} proxies')

asyncio.run(load())
"

How do I handle proxy authentication?

Answer:

Method 1: Inline credentials

rotator.add_proxy("http://user:pass@proxy.example.com:8080")

Method 2: Separate fields (recommended for config files)

from pydantic import SecretStr
from proxywhirl.models import Proxy

proxy = Proxy(
    url="http://proxy.example.com:8080",
    username=SecretStr("myuser"),
    password=SecretStr("mypass")
)
rotator.add_proxy(proxy)

Method 3: Configuration file

# .proxywhirl.toml
encrypt_credentials = true

[[proxies]]
url = "http://proxy.example.com:8080"
username = "myuser"
password = "mypass"

Credentials are automatically encrypted when encrypt_credentials = true.


How do I handle different proxy protocols?

Answer:

ProxyWhirl supports HTTP, HTTPS, SOCKS4, and SOCKS5:

# HTTP proxy
rotator.add_proxy("http://proxy.example.com:8080")

# HTTPS proxy
rotator.add_proxy("https://proxy.example.com:8443")

# SOCKS4 proxy
rotator.add_proxy("socks4://proxy.example.com:1080")

# SOCKS5 proxy
rotator.add_proxy("socks5://proxy.example.com:1080")

# SOCKS5 with auth
rotator.add_proxy("socks5://user:pass@proxy.example.com:1080")

Protocol Selection:

  • Use HTTP/HTTPS for web scraping

  • Use SOCKS5 for maximum compatibility

  • Use SOCKS4 for legacy systems

Important: Not all proxies support all protocols. Test before deploying:

proxywhirl pool test socks5://proxy.example.com:1080

Why are my proxies slow?

Answer:

Diagnosis:

proxywhirl health --verbose

Common Causes:

  1. Geographic distance:

    • Proxy in different continent

    • Use geo-targeted strategy

  2. Overloaded proxy:

    • Free proxies are often slow

    • Consider premium proxies

  3. Poor connectivity:

    • Check network latency: ping proxy.example.com

  4. Target site slow:

    • Test different targets: proxywhirl pool test http://proxy:8080 --target-url https://example.com

Solutions:

  1. Use performance-based strategy:

    rotator = ProxyWhirl(
        proxies=[...],
        strategy="weighted"  # Faster proxies get priority
    )
    
  2. Filter slow proxies:

    from proxywhirl.models import HealthStatus
    
    # Only use proxies faster than 2 seconds
    fast_proxies = [
        p for p in rotator.pool.proxies
        if p.average_response_time_ms < 2000
        and p.health_status == HealthStatus.HEALTHY
    ]
    
  3. Use parallel validation:

    proxywhirl fetch --concurrency 1000 --timeout 3
    

How do I use ProxyWhirl with async code?

Answer:

Use AsyncProxyWhirl:

import asyncio
from proxywhirl import AsyncProxyWhirl

async def main():
    async with AsyncProxyWhirl(
        proxies=["http://proxy1:8080", "http://proxy2:8080"]
    ) as rotator:
        # Make async requests
        response = await rotator.get("https://httpbin.org/ip")
        print(response.json())

        # Multiple concurrent requests
        urls = ["https://httpbin.org/ip"] * 10
        tasks = [rotator.get(url) for url in urls]
        responses = await asyncio.gather(*tasks)

asyncio.run(main())

See examples.ipynb for more examples.


How do I integrate with existing httpx code?

Answer:

Option 1: Use ProxyWhirl’s built-in client

rotator = ProxyWhirl(proxies=[...])
response = rotator.get("https://api.example.com")
# Same interface as httpx

Option 2: Get proxy URL and use with httpx

import httpx
from proxywhirl import ProxyWhirl

rotator = ProxyWhirl(proxies=[...])
proxy = rotator.get_next()

# Use with httpx directly
with httpx.Client(proxy=proxy.url) as client:
    response = client.get("https://api.example.com")

Option 3: Use as httpx transport

# Coming soon: ProxyWhirlTransport

Can I use ProxyWhirl in production?

Answer:

Yes, with precautions:

  1. Use premium proxies for reliability:

    • Free proxies have high failure rates

    • Consider rotating residential proxies

  2. Enable health monitoring:

    rotator = ProxyWhirl(
        proxies=[...],
        health_check_interval=60  # Check every minute
    )
    
  3. Configure retries:

    from proxywhirl import RetryPolicy
    
    rotator = ProxyWhirl(
        proxies=[...],
        retry_policy=RetryPolicy(
            max_attempts=5,
            multiplier=2.0,
            max_backoff_delay=30.0
        )
    )
    
  4. Monitor metrics:

    metrics = rotator.retry_executor.metrics
    if metrics.success_rate < 0.8:
        logger.warning("Low success rate, check proxies")
    
  5. Use circuit breakers:

    from proxywhirl.circuit_breaker import CircuitBreaker
    
    rotator = ProxyWhirl(
        proxies=[...],
        circuit_breaker=CircuitBreaker(
            failure_threshold=5,
            recovery_timeout=30
        )
    )
    
  6. Enable persistence:

    storage = SQLiteStorage("proxywhirl.db")
    rotator = ProxyWhirl(proxies=[...], storage=storage)
    

Getting Help

Check Existing Documentation

Search Issues

Check if your issue is already reported:

Enable Debugging

Before asking for help, collect debug information:

# Enable verbose logging
export LOGURU_LEVEL=DEBUG

# Run your command with debug output
proxywhirl --verbose pool list > debug.log 2>&1

# Collect system info
uv --version
python --version
pip show proxywhirl

Create an Issue

If you can’t find a solution:

  1. Search existing issues first

  2. Include debug logs

  3. Provide minimal reproduction

  4. Specify your environment:

    • ProxyWhirl version

    • Python version

    • Operating system

    • Proxy type (free/premium/residential)

Template:

## Description
Brief description of the issue

## Reproduction
```python
# Minimal code to reproduce
from proxywhirl import ProxyWhirl
rotator = ProxyWhirl(proxies=["http://proxy:8080"])
rotator.get("https://example.com")  # Error here

Environment

  • ProxyWhirl version: 0.1.0

  • Python version: 3.11.5

  • OS: macOS 14.0

Logs

[Paste debug logs here]

Expected Behavior

What should happen

Actual Behavior

What actually happens


---

## Related Documentation

- **Getting Started:** {doc}`/getting-started/index`
- **Configuration:** {doc}`/reference/configuration`
- **Error Reference:** {doc}`/reference/exceptions`
- **CLI Reference:** {doc}`/guides/cli-reference`
- **Examples:** [examples.ipynb](https://github.com/wyattowalsh/proxywhirl/blob/main/examples.ipynb)