Crawler
=======

.. automodule:: pyfetcher.crawler.frontier
   :members:

.. automodule:: pyfetcher.crawler.spider
   :members:

.. automodule:: pyfetcher.crawler.dedup
   :members:

.. automodule:: pyfetcher.crawler.politeness
   :members:

.. automodule:: pyfetcher.crawler.feeds
   :members:

.. automodule:: pyfetcher.crawler.discovery
   :members: