Headers¶

Browser profile definitions for pyfetcher.

Purpose:

Define complete browser identity profiles that bundle User-Agent strings with matching Client Hints (Sec-CH-UA-), Sec-Fetch- metadata, and Accept headers into coherent, realistic browser identities. Profiles ensure that all identity-related headers are internally consistent, which is critical for avoiding detection by anti-bot systems.

Design:

Each profile represents a specific browser + version + platform combination.
Headers within a profile are guaranteed to be mutually consistent (e.g. a Chrome UA will have matching Sec-CH-UA and platform values).
Profiles are immutable dataclass instances for safety and hashability.
The get_profile function selects profiles by name; list_profiles enumerates all available profile names.

Examples

>>> profile = get_profile("chrome_win")
>>> "Chrome" in profile.user_agent
True
>>> profile.platform
'Windows'

class pyfetcher.headers.profiles.BrowserProfile(name, browser, platform, mobile=False, user_agent='', sec_ch_ua='', sec_ch_ua_full_version_list='', sec_ch_ua_mobile='?0', sec_ch_ua_platform='""', sec_ch_ua_platform_version='""', sec_ch_ua_model='""', accept='text/html, application/xhtml+xml, application/xml;q=0.9, image/avif, image/webp, image/apng, */*;q=0.8', accept_language_options=('en-US, en;q=0.9', 'en-US, en;q=0.8', 'en-GB, en;q=0.9', 'en-CA, en;q=0.9', 'en-AU, en;q=0.9'), accept_encoding='gzip, deflate, br, zstd', sec_fetch_dest='document', sec_fetch_mode='navigate', sec_fetch_site='none', sec_fetch_user='?1', upgrade_insecure_requests='1', extra_headers=<factory>)[source]¶

A complete browser identity profile.

Bundles all headers that form a browser’s identity fingerprint into a single coherent object. Anti-bot systems check for consistency across these headers, so they must all agree on browser type, version, and platform.

Parameters:

name (str) – Unique profile identifier (e.g. 'chrome_win').
browser (str) – Browser family name (e.g. 'chrome', 'firefox', 'safari').
platform (str) – Operating system name (e.g. 'Windows', 'macOS', 'Linux').
mobile (bool) – Whether this is a mobile browser profile.
user_agent (str) – The User-Agent header string.
sec_ch_ua (str) – The Sec-CH-UA client hint header value.
sec_ch_ua_full_version_list (str) – The Sec-CH-UA-Full-Version-List header.
sec_ch_ua_mobile (str) – The Sec-CH-UA-Mobile header ('?0' or '?1').
sec_ch_ua_platform (str) – The Sec-CH-UA-Platform header (quoted string).
sec_ch_ua_platform_version (str) – The Sec-CH-UA-Platform-Version header.
sec_ch_ua_model (str) – The Sec-CH-UA-Model header (empty for desktop).
accept (str) – The Accept header for document requests.
accept_language_options (tuple[str, ...]) – Pool of Accept-Language values for randomization.
accept_encoding (str) – The Accept-Encoding header value.
sec_fetch_dest (str) – Default Sec-Fetch-Dest value.
sec_fetch_mode (str) – Default Sec-Fetch-Mode value.
sec_fetch_site (str) – Default Sec-Fetch-Site value.
sec_fetch_user (str) – Default Sec-Fetch-User value.
upgrade_insecure_requests (str) – The Upgrade-Insecure-Requests header.
extra_headers (dict[str, str]) – Additional browser-specific headers.

Examples

>>> profile = BrowserProfile(
...     name="test", browser="chrome", platform="Windows",
...     user_agent="Mozilla/5.0", sec_ch_ua='"Chrome";v="133"',
...     sec_ch_ua_full_version_list='"Chrome";v="133.0"',
...     sec_ch_ua_mobile="?0", sec_ch_ua_platform='"Windows"',
...     sec_ch_ua_platform_version='"10.0.0"',
... )
>>> profile.browser
'chrome'

to_headers()[source]¶

Convert this profile to a complete headers dictionary.

Returns a dictionary containing all identity-related headers appropriate for a top-level navigation request. The Accept-Encoding header is intentionally omitted because transport libraries (httpx, aiohttp) handle content negotiation and decompression automatically.

Returns:: A dictionary mapping header names to values.
Return type:: dict[str, str]

Examples

>>> profile = get_profile("chrome_win")
>>> headers = profile.to_headers()
>>> "user-agent" in headers
True

pyfetcher.headers.profiles.get_profile(name)[source]¶

Get a browser profile by name.

Parameters:: name (str) – Profile name (e.g. 'chrome_win', 'firefox_mac').
Returns:: The matching BrowserProfile instance.
Raises:: KeyError – If no profile matches the given name.
Return type:: BrowserProfile

Examples

>>> get_profile("chrome_win").browser
'chrome'

pyfetcher.headers.profiles.list_profiles()[source]¶

List all available profile names.

Returns:: A sorted list of registered profile name strings.
Return type:: list[str]

Examples

>>> "chrome_win" in list_profiles()
True

Browser-like header providers for pyfetcher.

Purpose:

Provide header providers that generate realistic browser-like headers using the profile system. Supports both fixed-profile and randomized header generation with small per-request variations.

Design:

Browser profiles are sourced from pyfetcher.headers.profiles.
Small randomized fields (Accept-Language, color scheme preference) are applied at build time to add realistic variation.
Providers remain separate from transport and retry concerns.

Examples

>>> provider = BrowserHeaderProvider("chrome_win")
>>> headers = provider.build(request=FetchRequest(url="https://example.com"))
>>> "user-agent" in headers
True

pyfetcher.headers.browser.get_best_browser_headers(impersonate='chrome_win')[source]¶

Return browser-like headers with small randomized fields.

Generates a complete set of browser headers from the named profile, then adds randomized Accept-Language and Sec-CH-Prefers-Color-Scheme values for realistic per-request variation.

Parameters:: impersonate (str) – Profile name or legacy alias.
Returns:: A browser-like headers dictionary suitable for HTTP requests.
Return type:: dict[str, str]

Examples

>>> headers = get_best_browser_headers("chrome_win")
>>> "accept-language" in headers
True

class pyfetcher.headers.browser.BrowserHeaderProvider(impersonate='chrome_win')[source]¶

Provide browser-like headers for fetch requests.

Wraps a browser profile and generates consistent, realistic headers for every request. Per-request headers from the FetchRequest take precedence over profile headers when both are present.

Parameters:: impersonate (str) – Profile name or legacy alias (default 'chrome_win').

Examples

>>> provider = BrowserHeaderProvider("chrome_win")
>>> headers = provider.build(request=FetchRequest(url="https://example.com"))
>>> "user-agent" in headers
True

property profile: BrowserProfile¶

Return the resolved browser profile.

Returns:: The BrowserProfile used by this provider.

build(*, request)[source]¶

Build browser-like headers for the given request.

Generates headers from the profile, adds randomized variation, then merges with any per-request headers (request headers win on conflict).

Parameters:: request (FetchRequest) – The fetch request for which to build headers.
Returns:: A complete headers dictionary.
Return type:: dict[str, str]

Examples

>>> provider = BrowserHeaderProvider()
>>> headers = provider.build(request=FetchRequest(url="https://example.com"))
>>> "accept" in headers
True

Rotating header provider for pyfetcher.

Purpose:

Provide a header provider that rotates through browser profiles to distribute requests across multiple browser identities, reducing the likelihood of fingerprint-based blocking.

Design:

Rotation happens at the session level: a profile is selected per call and all headers for that request are internally consistent.
Profile selection uses market-share weights by default but can be configured with an explicit profile list.
The provider maintains no mutable state beyond the configuration, so it is safe to share across threads/tasks.

Examples

>>> provider = RotatingHeaderProvider()
>>> headers = provider.build(request=FetchRequest(url="https://example.com"))
>>> "user-agent" in headers
True

class pyfetcher.headers.rotating.RotatingHeaderProvider(profiles=None, weights=None)[source]¶

Header provider that rotates through browser profiles.

Each call to build() selects a random profile from the configured pool (weighted by market share) and returns a complete, internally consistent set of headers for that browser identity.

Parameters:

profiles (list[BrowserProfile] | None) – Optional list of profiles to rotate through. Defaults to all desktop profiles.
weights (list[float] | None) – Optional list of weights for profile selection. Must have the same length as profiles. Defaults to market-share weights.

Examples

>>> provider = RotatingHeaderProvider()
>>> headers = provider.build(request=FetchRequest(url="https://example.com"))
>>> "user-agent" in headers
True

property profiles: list[BrowserProfile]¶

Return the configured profile pool.

Returns:: The list of profiles available for rotation.

build(*, request)[source]¶

Build headers using a randomly selected browser profile.

Selects a profile, generates its full header set with randomized variation, then merges per-request headers on top.

Parameters:: request (FetchRequest) – The fetch request for which to build headers.
Returns:: A complete headers dictionary from the selected profile.
Return type:: dict[str, str]

Examples

>>> provider = RotatingHeaderProvider()
>>> headers = provider.build(request=FetchRequest(url="https://example.com"))
>>> "accept" in headers
True

Static header providers for pyfetcher.

Purpose:: Provide the simplest possible provider that always returns the same header set regardless of the request.

Examples

>>> provider = StaticHeaderProvider({"user-agent": "ua"})
>>> provider.build(request=FetchRequest(url="https://example.com"))["user-agent"]
'ua'

class pyfetcher.headers.static.StaticHeaderProvider(headers)[source]¶

Always return the same header set.

Useful for testing or when a fixed set of headers is required for all requests (e.g. an API token header).

Parameters:: headers (dict[str, str]) – Static headers to return for every request.

Examples

>>> provider = StaticHeaderProvider({"x-demo": "1"})
>>> provider.build(request=FetchRequest(url="https://example.com"))["x-demo"]
'1'

build(*, request)[source]¶

Return static headers.

Parameters:: request (FetchRequest) – The fetch request (ignored).
Returns:: A shallow copy of the configured headers.
Return type:: dict[str, str]

User-Agent generation utilities for pyfetcher.

Purpose:

Provide functions for generating realistic User-Agent strings and selecting random profiles weighted by real-world browser market share.

Design:

User-Agent generation leverages the profile system for consistency.
Random selection uses market-share weights to produce realistic distributions of browser identities.
All randomization is done at the function call level, so each call may produce a different result.

Examples

>>> ua = random_user_agent()
>>> "Mozilla" in ua
True

pyfetcher.headers.ua.random_user_agent(*, browser=None, platform=None, mobile=None)[source]¶

Generate a random realistic User-Agent string.

Optionally filter by browser family, platform, or mobile/desktop. When no filters are specified, profiles are selected using real-world market share weights.

Parameters:

browser (str | None) – Filter to a specific browser family (e.g. 'chrome', 'firefox', 'safari', 'edge'). Case-insensitive.
platform (str | None) – Filter to a specific platform (e.g. 'Windows', 'macOS', 'Linux', 'Android', 'iOS'). Case-insensitive.
mobile (bool | None) – If True, only mobile profiles. If False, only desktop profiles. If None, any profile.

Returns:

A realistic User-Agent string.

Raises:

ValueError – If no profiles match the given filters.

Return type:

str

Examples

>>> ua = random_user_agent(browser="chrome")
>>> "Chrome" in ua
True
>>> ua = random_user_agent(mobile=True)
>>> "Mobile" in ua or "iPhone" in ua
True

pyfetcher.headers.ua.random_profile(*, browser=None, platform=None, mobile=None)[source]¶

Select a random browser profile, optionally filtered.

Profiles are selected using market-share weights when no filters are applied. When filters narrow the candidate set, weights are renormalized across matching profiles.

Parameters:

browser (str | None) – Filter by browser family (case-insensitive).
platform (str | None) – Filter by platform name (case-insensitive).
mobile (bool | None) – Filter by mobile/desktop.

Returns:

A randomly selected BrowserProfile.

Raises:

ValueError – If no profiles match the given filters.

Return type:

BrowserProfile

Examples

>>> profile = random_profile(browser="firefox")
>>> profile.browser
'firefox'

pyfetcher.headers.ua.user_agents_for_browser(browser)[source]¶

Return all User-Agent strings for a given browser family.

Parameters:: browser (str) – Browser family name (case-insensitive).
Returns:: A list of User-Agent strings.
Return type:: list[str]

Examples

>>> uas = user_agents_for_browser("chrome")
>>> all("Chrome" in ua for ua in uas)
True

Base header-provider protocols for pyfetcher.

Purpose:: Define the small protocol surface used by fetch services to obtain headers for a request. Any object with a build method accepting a FetchRequest and returning a dict[str, str] satisfies the protocol.

Examples

>>> from pyfetcher.headers.static import StaticHeaderProvider
>>> provider = StaticHeaderProvider({"x-test": "1"})
>>> hasattr(provider, "build")
True

class pyfetcher.headers.base.HeaderProvider(*args, **kwargs)[source]¶

Protocol for header-provider strategies.

Any object that implements build(*, request: FetchRequest) -> dict[str, str] satisfies this protocol and can be used with FetchService.

Examples

>>> from pyfetcher.headers.static import StaticHeaderProvider
>>> provider = StaticHeaderProvider({"user-agent": "test"})
>>> provider.build(request=FetchRequest(url="https://example.com"))["user-agent"]
'test'

build(*, request)[source]¶

Build request headers for the given fetch request.

Parameters:: request (FetchRequest) – The fetch request for which headers are needed.
Returns:: A dictionary mapping header names to values.
Return type:: dict[str, str]