proxywhirl.sources¶

Built-in proxy source configurations.

This module provides pre-configured ProxySourceConfig instances for popular free proxy list APIs and websites. These can be used directly with ProxyFetcher.

Example

>>> from proxywhirl import ProxyFetcher
>>> from proxywhirl.sources import PROXY_SCRAPE_HTTP, GEONODE_HTTP
>>> fetcher = ProxyFetcher(sources=[PROXY_SCRAPE_HTTP, GEONODE_HTTP])
>>> proxies = await fetcher.fetch_all()

Classes¶

`SourceValidationReport`	Aggregate report of source validation.
`SourceValidationResult`	Result of validating a single proxy source.

Functions¶

`fetch_all_sources`([validate, timeout, max_concurrent, ...])	Fetch proxies from all built-in sources.
`validate_source`(source[, timeout, client])	Validate a single proxy source.
`validate_sources`([sources, timeout, concurrency])	Validate multiple proxy sources concurrently.
`validate_sources_sync`([sources, timeout, concurrency])	Synchronous wrapper for validate_sources.

Module Contents¶

class proxywhirl.sources.SourceValidationReport[source]¶

Aggregate report of source validation.

property all_healthy: bool[source]¶

Check if all sources are healthy.

Return type:: bool

property healthy: list[SourceValidationResult][source]¶

Get all healthy sources.

Return type:: list[SourceValidationResult]

property unhealthy: list[SourceValidationResult][source]¶

Get all unhealthy sources.

Return type:: list[SourceValidationResult]

class proxywhirl.sources.SourceValidationResult[source]¶

Result of validating a single proxy source.

property is_healthy: bool[source]¶

Check if source is healthy (returns valid proxy data).

Return type:: bool

async proxywhirl.sources.fetch_all_sources(validate=True, timeout=10, max_concurrent=100, sources=None, fetch_progress_callback=None, validate_progress_callback=None, test_url=None)[source]¶

Fetch proxies from all built-in sources.

Parameters:

validate (bool) – Whether to validate proxies
timeout (int) – Validation timeout in seconds
max_concurrent (int) – Maximum concurrent validation requests
sources (list[proxywhirl.models.ProxySourceConfig] | None) – List of sources to fetch from (defaults to ALL_SOURCES)
fetch_progress_callback (Any | None) – Optional callback(completed, total, proxies_found) for fetch progress
validate_progress_callback (Any | None) – Optional callback(completed, total, valid_count) for validation progress
test_url (str | None) – Optional URL to validate against (defaults to http://www.gstatic.com/generate_204)

Returns:

List of proxy dictionaries

Return type:

list[dict[str, Any]]

async proxywhirl.sources.validate_source(source, timeout=15.0, client=None)[source]¶

Validate a single proxy source.

Parameters:

source (proxywhirl.models.ProxySourceConfig) – The proxy source configuration to validate
timeout (float) – Request timeout in seconds
client (httpx.AsyncClient | None) – Optional shared HTTP client

Returns:

SourceValidationResult with validation details

Return type:

SourceValidationResult

async proxywhirl.sources.validate_sources(sources=None, timeout=15.0, concurrency=20)[source]¶

Validate multiple proxy sources concurrently.

Parameters:

sources (list[proxywhirl.models.ProxySourceConfig] | None) – List of sources to validate (defaults to ALL_SOURCES)
timeout (float) – Request timeout per source in seconds
concurrency (int) – Maximum concurrent requests

Returns:

SourceValidationReport with all validation results

Return type:

SourceValidationReport

proxywhirl.sources.validate_sources_sync(sources=None, timeout=15.0, concurrency=20)[source]¶

Synchronous wrapper for validate_sources.

Parameters:

sources (list[proxywhirl.models.ProxySourceConfig] | None) – List of sources to validate (defaults to ALL_SOURCES)
timeout (float) – Request timeout per source in seconds
concurrency (int) – Maximum concurrent requests

Returns:

SourceValidationReport with all validation results

Return type:

SourceValidationReport