Skip to main content
Version: 4.0

AsyncThread

Run an asyncio event loop in a dedicated background thread.

This lets synchronous Scrapy callbacks drive asynchronous Apify and Crawlee coroutines. The scheduler and the HTTP cache storage each own their own AsyncThread, so the request queue and the key-value store never share an event loop; they only share the read-only global Configuration. A single shared loop would also work but would couple their lifecycles.

Index

Methods

__init__

  • __init__(default_timeout): None
  • Parameters

    • optionaldefault_timeout: timedelta = timedelta(seconds=60)

    Returns None

close

  • close(timeout): None
  • Close the event loop and its thread gracefully.

    This method cancels all pending tasks, stops the event loop, and waits for the thread to exit. If the thread does not exit within the given timeout, a forced shutdown is attempted.


    Parameters

    • optionaltimeout: timedelta | None = None

      The maximum time to wait for the event loop thread to exit. Pass None to use the default_timeout passed to the constructor.

    Returns None

run_coro

  • run_coro(coro, timeout): Any
  • Run a coroutine on an event loop running in a separate thread.

    This method schedules the coroutine to run on the event loop and blocks until the coroutine completes or the specified timeout is reached.


    Parameters

    • coro: Coroutine

      The coroutine to run.

    • optionaltimeout: timedelta | Literal[default] = 'default'

      The maximum time to wait for the coroutine to finish. Pass 'default' to use the default_timeout passed to the constructor.

    Returns Any

    The result returned by the coroutine.

submit_coro

  • submit_coro(coro): futures.Future
  • Schedule a coroutine on the event loop without waiting for its result.

    Use this for work nothing depends on, so the calling thread is not blocked by the round trip. Failures are logged, as there is no caller to propagate them to, and close cancels whatever is still pending - call wait_for_submitted first if that matters.


    Parameters

    • coro: Coroutine

      The coroutine to run.

    Returns futures.Future

    The future of the scheduled coroutine, for callers that want to inspect its outcome later.

wait_for_submitted

  • wait_for_submitted(timeout): None
  • Block until the coroutines submitted via submit_coro have finished.

    Use this before anything that would observe their effects, or before close, which cancels whatever is still running. Coroutines that do not finish within the timeout stay tracked for the next call.


    Parameters

    • optionaltimeout: timedelta | None = None

      The maximum time to wait for the submitted coroutines. Pass None to use the default_timeout passed to the constructor.

    Returns None