Fetch¶

Fetch service orchestration for pyfetcher.

Purpose:

Compose header providers, retry policies, rate limiters, and transports into a single reusable service object that handles the full lifecycle of HTTP requests.

Design:

Retry logic sits above transports via tenacity.
Rate limiting is applied before each request attempt.
A function-first API is layered on top of this service.
Requests are normalized through central contracts.
Backends can be swapped per request.

Examples

>>> service = FetchService()
>>> hasattr(service, "afetch")
True

class pyfetcher.fetch.service.FetchService(*, header_provider=None, httpx_transport=None, aiohttp_transport=None, rate_limiter=None)[source]¶

Reusable fetch orchestration service.

Composes header generation, rate limiting, retry logic, and transport execution into a single service. Supports both synchronous and asynchronous operation, batch fetching with bounded concurrency, and streaming.

Parameters:

header_provider (HeaderProvider | None) – Optional header provider for generating request headers. Defaults to BrowserHeaderProvider.
httpx_transport (HttpxTransport | None) – Optional HTTPX transport instance. Created lazily if not provided.
aiohttp_transport (AiohttpTransport | None) – Optional aiohttp transport instance. Created lazily if not provided.
rate_limiter (DomainRateLimiter | None) – Optional per-domain rate limiter. Pass None to disable rate limiting (the default).

Examples

>>> service = FetchService()
>>> hasattr(service, "fetch")
True

fetch(request)[source]¶

Fetch synchronously with retries and optional rate limiting.

Prepares the request with provider headers, applies rate limiting if configured, then executes the request through the appropriate transport with Tenacity retry logic.

Parameters:: request (FetchRequest) – The fetch request to execute.
Returns:: A normalized FetchResponse.
Raises:: Exception – The final transport or retry failure after all attempts are exhausted.
Return type:: FetchResponse

Examples

>>> service = FetchService()
>>> hasattr(service, "fetch")
True

async afetch(request)[source]¶

Fetch asynchronously with retries and optional rate limiting.

Prepares the request with provider headers, applies async rate limiting if configured, then executes the request through the appropriate transport with async Tenacity retry logic.

Parameters:: request (FetchRequest) – The fetch request to execute.
Returns:: A normalized FetchResponse.
Raises:: Exception – The final transport or retry failure after all attempts are exhausted.
Return type:: FetchResponse

Examples

>>> service = FetchService()
>>> hasattr(service, "afetch")
True

async afetch_many(batch)[source]¶

Fetch many requests asynchronously with bounded concurrency.

Executes all requests in the batch concurrently, limited by the batch concurrency setting or the first request’s pool concurrency. Individual failures are captured per-item without aborting the batch.

Parameters:: batch (BatchFetchRequest) – The batch fetch request containing all requests.
Returns:: A BatchFetchResponse preserving input order.
Return type:: BatchFetchResponse

Examples

>>> service = FetchService()
>>> hasattr(service, "afetch_many")
True

async astream(request)[source]¶

Stream a request asynchronously.

Prepares the request and delegates to the transport’s streaming interface. Rate limiting is applied once before streaming begins.

Parameters:: request (FetchRequest) – The fetch request to stream.
Yields:: StreamChunk objects.
Raises:: Exception – Streaming or backend failures.
Return type:: AsyncIterator[StreamChunk]

Examples

>>> service = FetchService()
>>> hasattr(service, "astream")
True

close()[source]¶

Close owned synchronous resources.

Closes any internally-managed transport clients. Externally provided transports are not closed.

Return type:: None

async aclose()[source]¶

Close owned asynchronous resources.

Closes any internally-managed async transport clients. Externally provided transports are not closed.

Return type:: None

Function-first fetch API for pyfetcher.

Purpose:

Provide a small ergonomic surface for common fetch, batch, and stream operations without requiring callers to manage a service object directly.

Design:

Functions delegate to a module-level default service.
Request dictionaries are validated through Pydantic models.
Advanced callers can still instantiate FetchService directly.

Examples

>>> request = FetchRequest(url="https://example.com")
>>> request.method
'GET'

pyfetcher.fetch.functions.get_default_fetch_service()[source]¶

Return the module-level default fetch service (cached singleton).

Returns:: A cached FetchService instance.
Return type:: FetchService

Examples

>>> isinstance(get_default_fetch_service(), FetchService)
True

pyfetcher.fetch.functions.fetch(url, /, **kwargs)[source]¶

Fetch a URL synchronously using the default service.

Convenience function that creates a FetchRequest from the given URL string (or accepts a pre-built request) and delegates to the default FetchService.

Parameters:

url (str | FetchRequest) – URL string or pre-built FetchRequest.
**kwargs (object) – Additional request fields when url is a string.

Returns:

A normalized FetchResponse.

Raises:

Exception – Any underlying fetch failure.

Return type:

FetchResponse

Examples

>>> callable(fetch)
True

async pyfetcher.fetch.functions.afetch(url, /, **kwargs)[source]¶

Fetch a URL asynchronously using the default service.

Convenience function that creates a FetchRequest from the given URL string (or accepts a pre-built request) and delegates to the default FetchService.

Parameters:

url (str | FetchRequest) – URL string or pre-built FetchRequest.
**kwargs (object) – Additional request fields when url is a string.

Returns:

A normalized FetchResponse.

Raises:

Exception – Any underlying fetch failure.

Return type:

FetchResponse

Examples

>>> callable(afetch)
True

async pyfetcher.fetch.functions.afetch_many(requests, /, **kwargs)[source]¶

Fetch many URLs asynchronously using the default service.

Convenience function for batch fetching with bounded concurrency.

Parameters:

requests (list[FetchRequest] | BatchFetchRequest) – A list of FetchRequest objects or a pre-built BatchFetchRequest.
**kwargs (object) – Additional batch fields when passing a plain request list.

Returns:

A BatchFetchResponse preserving input order.

Raises:

Exception – Any service-level failure.

Return type:

BatchFetchResponse

Examples

>>> callable(afetch_many)
True

async pyfetcher.fetch.functions.astream(url, /, **kwargs)[source]¶

Stream a URL asynchronously using the default service.

Convenience function for streaming responses as chunks.

Parameters:

url (str | FetchRequest) – URL string or pre-built FetchRequest.
**kwargs (object) – Additional request fields when url is a string.

Yields:

StreamChunk objects.

Raises:

Exception – Any streaming failure.

Return type:

AsyncIterator[StreamChunk]

Examples

>>> callable(astream)
True

Batch convenience helpers for pyfetcher.

Purpose:: Provide lightweight wrappers for constructing batch fetch requests.

Examples

>>> callable(build_batch_request)
True

pyfetcher.fetch.batch.build_batch_request(requests, *, concurrency=None)[source]¶

Build a batch fetch request from a list of individual requests.

Parameters:

requests (list[FetchRequest]) – Individual requests to execute concurrently.
concurrency (int | None) – Optional concurrency override limiting the number of simultaneous in-flight requests.

Returns:

A validated BatchFetchRequest.

Return type:

BatchFetchRequest

Examples

>>> req = FetchRequest(url="https://example.com")
>>> build_batch_request([req]).requests[0].url.host
'example.com'

Streaming convenience helpers for pyfetcher.

Purpose:: Provide small helpers for consuming async stream chunks into a single bytes object.

Examples

>>> collect_bytes([b"a", b"b"])
b'ab'

pyfetcher.fetch.stream.collect_bytes(chunks)[source]¶

Concatenate a list of byte chunks into a single bytes object.

Parameters:: chunks (list[bytes]) – List of byte chunks.
Returns:: All chunks concatenated.
Return type:: bytes

Examples

>>> collect_bytes([b"a", b"b"])
b'ab'

async pyfetcher.fetch.stream.collect_stream_bytes(chunks)[source]¶

Collect an async byte stream into a single bytes object.

Iterates the async iterable and accumulates all bytes into a single contiguous buffer.

Parameters:: chunks (AsyncIterable[bytes]) – Async iterable yielding byte chunks.
Returns:: All chunks concatenated.
Return type:: bytes

Examples

>>> async def _demo():
...     async def _gen():
...         yield b"a"
...         yield b"b"
...     return await collect_stream_bytes(_gen())