proxywhirl.rotator.async

Async proxy rotation implementation using httpx.AsyncClient.

Classes

AsyncProxyWhirl

Async proxy rotator with automatic failover and intelligent rotation.

LRUAsyncClientPool

LRU cache for httpx.AsyncClient instances with automatic eviction.

Module Contents

class proxywhirl.rotator.async_.AsyncProxyWhirl(proxies=None, strategy=None, config=None, retry_policy=None, bootstrap=None)[source]

Bases: proxywhirl.rotator.base.ProxyRotatorBase

Async proxy rotator with automatic failover and intelligent rotation.

Provides async HTTP methods (GET, POST, PUT, DELETE, PATCH) that automatically rotate through a pool of proxies, with intelligent failover on connection errors.

Example:

from proxywhirl import AsyncProxyWhirl, Proxy

async with AsyncProxyWhirl() as rotator:
    await rotator.add_proxy("http://proxy1.example.com:8080")
    await rotator.add_proxy("http://proxy2.example.com:8080")

    response = await rotator.get("https://httpbin.org/ip")
    print(response.json())

Initialize AsyncProxyWhirl.

Parameters:

  • proxies (list[proxywhirl.models.Proxy] | None) – Initial list of proxies (optional)

  • strategy (proxywhirl.strategies.RotationStrategy | str | None) – Rotation strategy instance or name string. Supported names: round-robin, random, weighted, least-used, performance-based, session, geo-targeted. Defaults to RoundRobinStrategy.

  • config (proxywhirl.models.ProxyConfiguration | None) – Configuration settings (default: ProxyConfiguration())

  • retry_policy (proxywhirl.retry.RetryPolicy | None) – Retry policy configuration (default: RetryPolicy())

  • bootstrap (proxywhirl.models.BootstrapConfig | bool | None) – Bootstrap configuration for lazy proxy fetching. False disables auto-bootstrap (for manual proxy management). True or None uses default BootstrapConfig.

async add_proxy(proxy)[source]

Add a proxy to the pool.

Parameters:

proxy (proxywhirl.models.Proxy | str) – Proxy instance or URL string

Return type:

None

async clear_unhealthy_proxies()[source]

Remove all unhealthy and dead proxies from the pool.

Returns:

Number of proxies removed

Return type:

int

async delete(url, **kwargs)[source]

Make async DELETE request.

Parameters:

  • url (str)

  • kwargs (Any)

Return type:

httpx.Response

async get(url, **kwargs)[source]

Make async GET request.

Parameters:

  • url (str)

  • kwargs (Any)

Return type:

httpx.Response

get_circuit_breaker_states()[source]

Get circuit breaker states for all proxies.

Returns:

Mapping of proxy IDs to their circuit breaker instances.

Return type:

dict[str, CircuitBreaker]

get_pool_stats()[source]

Get statistics about the proxy pool.

Returns:

Pool statistics including total_proxies, healthy_proxies, unhealthy_proxies, dead_proxies, total_requests, total_successes, total_failures, and average_success_rate.

Return type:

dict[str, Any]

async get_proxy()[source]

Get the next proxy from the pool using the rotation strategy.

Returns:

Next proxy to use

Raises:

ProxyPoolEmptyError – If no healthy proxies available

Return type:

proxywhirl.models.Proxy

get_retry_metrics()[source]

Get retry metrics.

Returns:

RetryMetrics instance with current metrics

Return type:

proxywhirl.retry.RetryMetrics

get_statistics()[source]

Get comprehensive statistics including source breakdown (FR-050).

Returns:

All stats from get_pool_stats() plus source_breakdown mapping source names to proxy counts.

Return type:

dict[str, Any]

async head(url, **kwargs)[source]

Make async HEAD request.

Parameters:

  • url (str)

  • kwargs (Any)

Return type:

httpx.Response

async options(url, **kwargs)[source]

Make async OPTIONS request.

Parameters:

  • url (str)

  • kwargs (Any)

Return type:

httpx.Response

async patch(url, **kwargs)[source]

Make async PATCH request.

Parameters:

  • url (str)

  • kwargs (Any)

Return type:

httpx.Response

async post(url, **kwargs)[source]

Make async POST request.

Parameters:

  • url (str)

  • kwargs (Any)

Return type:

httpx.Response

async put(url, **kwargs)[source]

Make async PUT request.

Parameters:

  • url (str)

  • kwargs (Any)

Return type:

httpx.Response

async remove_proxy(proxy_id)[source]

Remove a proxy from the pool.

Parameters:

proxy_id (str) – UUID of proxy to remove

Return type:

None

reset_circuit_breaker(proxy_id)[source]

Manually reset a circuit breaker to CLOSED state.

Parameters:

proxy_id (str) – ID of the proxy whose circuit breaker to reset

Raises:

KeyError – If proxy_id not found

Return type:

None

set_strategy(strategy, *, atomic=True)[source]

Hot-swap the rotation strategy without restarting.

This method implements atomic strategy swapping to ensure: - New requests immediately use the new strategy - In-flight requests complete with their original strategy - No requests are dropped during the swap - Swap completes in <100ms (SC-009)

Parameters:

  • strategy (proxywhirl.strategies.RotationStrategy | str) – New strategy instance or name string. Supported names: round-robin, random, weighted, least-used, performance-based, session, geo-targeted.

  • atomic (bool) – If True (default), ensures atomic swap. If False, allows immediate replacement (faster but may affect in-flight requests)

Return type:

None

Example

>>> async with AsyncProxyWhirl(strategy="round-robin") as rotator:
...     # ... after some requests ...
...     rotator.set_strategy("performance-based")  # Hot-swap
...     # New requests now use performance-based strategy
Thread Safety:

Thread-safe via atomic reference swap. Multiple threads can call this method safely without race conditions.

Performance:

Target: <100ms for hot-swap completion (SC-009) Typical: <10ms for strategy instance creation and assignment

class proxywhirl.rotator.async_.LRUAsyncClientPool(maxsize=100)[source]

LRU cache for httpx.AsyncClient instances with automatic eviction.

When the pool reaches maxsize, the least recently used client is closed and removed to prevent unbounded memory growth.

Supports dictionary-like access for backward compatibility with tests.

Initialize LRU async client pool.

Parameters:

maxsize (int) – Maximum number of clients to cache (default: 100)

async clear()[source]

Close all clients and clear the pool.

Return type:

None

async get(proxy_id)[source]

Get a client from the pool, marking it as recently used.

Parameters:

proxy_id (str) – Proxy ID to look up

Returns:

Client if found, None otherwise

Return type:

httpx.AsyncClient | None

async put(proxy_id, client)[source]

Add a client to the pool, evicting LRU client if at capacity.

Parameters:

  • proxy_id (str) – Proxy ID to store under

  • client (httpx.AsyncClient) – Client instance to store

Return type:

None

async remove(proxy_id)[source]

Remove and close a client from the pool.

Parameters:

proxy_id (str) – Proxy ID to remove

Return type:

None