Skip to main content
Version: 2.3

SessionPool

Handles the rotation, creation and persistence of user-like sessions. Creates a pool of Session instances, that are randomly rotated. When some session is marked as blocked, it is removed and new one is created instead (the pool never returns an unusable session). Learn more in the Session management guide.

You can create one by calling the Apify.openSessionPool function.

Session pool is already integrated into crawlers, and it can significantly improve your scraper performance with just 2 lines of code.

Example usage:

const crawler = new Apify.CheerioCrawler({
    useSessionPool: true,
    persistCookiesPerSession: true,
    // ...
});

You can configure the pool with many options. See the SessionPoolOptions. Session pool is by default persisted in default KeyValueStore. If you want to have one pool for all runs you have to specify SessionPoolOptions.persistStateKeyValueStoreId.

Advanced usage:

const sessionPool = await Apify.openSessionPool({
    maxPoolSize: 25,
    sessionOptions: {
        maxAgeSecs: 10,
        maxUsageCount: 150, // for example when you know that the site blocks after 150 requests.
    },
    persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',
    persistStateKey: 'my-session-pool',
});

// Get random session from the pool
const session1 = await sessionPool.getSession();
const session2 = await sessionPool.getSession();
const session3 = await sessionPool.getSession();

// Now you can mark the session either failed or successful

// Marks session as bad after unsuccessful usage -> it increases error count (soft retire)
session1.markBad();

// Marks as successful.
session2.markGood();

// Retires session -> session is removed from the pool
session3.retire();

sessionPool.sessions​

sessionPool.forceCloud​

sessionPool.usableSessionsCount​

Gets count of usable sessions in the pool.

Returns:

number

sessionPool.retiredSessionsCount​

Gets count of retired sessions in the pool.

Returns:

number

sessionPool.initialize()​

Starts periodic state persistence and potentially loads SessionPool state from KeyValueStore. It is called automatically by the Apify.openSessionPool function.

Returns:

Promise<void>

sessionPool.addSession([options])​

Adds a new session to the session pool. The pool automatically creates sessions up to the maximum size of the pool, but this allows you to add more sessions once the max pool size is reached. This also allows you to add session with overridden session options (e.g. with specific session id).

Parameters:

  • [options]: Session | SessionOptions - The configuration options for the session being added to the session pool.

sessionPool.getSession([sessionId])​

Gets session. If there is space for new session, it creates and returns new session. If the session pool is full, it picks a session from the pool, If the picked session is usable it is returned, otherwise it creates and returns a new one.

Parameters:

  • [sessionId]: String - If provided, it returns the usable session with this id, undefined otherwise.

Returns:

Promise<Session>

sessionPool.getState()​

Returns an object representing the internal state of the SessionPool instance. Note that the object's fields can change in future releases.

sessionPool.persistState()​

Persists the current state of the SessionPool into the default KeyValueStore. The state is persisted automatically in regular intervals.

Returns:

Promise<void>

sessionPool.teardown()​

Removes listener from persistState event. This function should be called after you are done with using the SessionPool instance.