Skip to main content

🦜🔗 LangChain integration

Copy for LLM

For more information on LangChain visit its documentation. The Apify integration lives in the langchain-apify repository.

Help keep this page up to date

This integration uses a third-party service. If you find outdated content, please submit an issue on GitHub.

In this example, we'll use the Website Content Crawler Actor, which can deeply crawl websites such as documentation, knowledge bases, help centers, or blogs and extract text content from the web pages. Then we feed the documents into a vector index and answer questions from it.

This example demonstrates how to integrate Apify with LangChain in Python.

Python only

The langchain-apify package is currently available for Python only.

Before we start with the integration, we need to install all dependencies:

pip install langchain-openai langchain-apify

After successful installation of all dependencies, we can start writing code.

First, import all required packages:

import os

from langchain_apify import ApifyWrapper
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings

Find your Apify API token and OpenAI API key and initialize these into environment variable:

os.environ["OPENAI_API_KEY"] = "Your OpenAI API key"
os.environ["APIFY_TOKEN"] = "Your Apify API token"

Run the Actor, wait for it to finish, and fetch its results from the Apify dataset into a LangChain document loader.

Note that if you already have some results in an Apify dataset, you can load them directly using ApifyDatasetLoader, as shown in this notebook. In that notebook, you'll also find the explanation of the dataset_mapping_function, which is used to map fields from the Apify dataset records to LangChain Document fields.

apify = ApifyWrapper()
llm = ChatOpenAI(model="gpt-5.4-mini")

loader = apify.call_actor(
    actor_id="apify/website-content-crawler",
    run_input={"startUrls": [{"url": "https://docs.langchain.com/oss/python/langchain/quickstart"}], "maxCrawlPages": 10, "crawlerType": "cheerio"},
    dataset_mapping_function=lambda item: Document(
        page_content=item["text"] or "", metadata={"source": item["url"]}
    ),
)
Crawling may take some time

The Actor call may take some time as it crawls the LangChain documentation website.

Initialize the vector index from the crawled documents:

vector_store = InMemoryVectorStore.from_documents(loader.load(), OpenAIEmbeddings())
retriever = vector_store.as_retriever()

prompt = ChatPromptTemplate.from_template(
    "Answer the question using only the context below.\n\n"
    "Context:\n{context}\n\nQuestion: {question}"
)

And finally, query the vector index:

query = "What is LangChain?"
docs = retriever.invoke(query)
context = "\n\n".join(doc.page_content for doc in docs)
answer = (prompt | llm).invoke({"context": context, "question": query}).content
sources = ", ".join({doc.metadata["source"] for doc in docs})

print("answer:", answer)
print("source:", sources)

If you want to test the whole example, you can simply create a new file, langchain_integration.py, and copy the whole code into it.

import os

from langchain_apify import ApifyWrapper
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings

os.environ["OPENAI_API_KEY"] = "Your OpenAI API key"
os.environ["APIFY_TOKEN"] = "Your Apify API token"

apify = ApifyWrapper()
llm = ChatOpenAI(model="gpt-5.4-mini")

print("Call website content crawler ...")
loader = apify.call_actor(
    actor_id="apify/website-content-crawler",
    run_input={"startUrls": [{"url": "https://docs.langchain.com/oss/python/langchain/quickstart"}], "maxCrawlPages": 10, "crawlerType": "cheerio"},
    dataset_mapping_function=lambda item: Document(page_content=item["text"] or "", metadata={"source": item["url"]}),
)
print("Compute embeddings...")
vector_store = InMemoryVectorStore.from_documents(loader.load(), OpenAIEmbeddings())
retriever = vector_store.as_retriever()

prompt = ChatPromptTemplate.from_template(
    "Answer the question using only the context below.\n\n"
    "Context:\n{context}\n\nQuestion: {question}"
)
query = "What is LangChain?"
docs = retriever.invoke(query)
context = "\n\n".join(doc.page_content for doc in docs)
answer = (prompt | llm).invoke({"context": context, "question": query}).content
sources = ", ".join({doc.metadata["source"] for doc in docs})

print("answer:", answer)
print("source:", sources)

To run it, you can use the following command: python langchain_integration.py

After running the code, you should see the following output:

answer: LangChain is a framework designed for developing applications powered by large language models (LLMs). It simplifies the
 entire application lifecycle, from development to productionization and deployment. LangChain provides open-source components and integrates with various third-party tools, making it easier to build and optimize applications using language models.

source: https://docs.langchain.com/oss/python/langchain/quickstart

LangChain is a standard interface through which you can interact with a variety of large language models (LLMs). It provides modules you can use to build language model applications as well as chains and agents with memory capabilities.

You can use all of Apify’s Actors as document loaders in LangChain. For example, to incorporate web browsing functionality, you can use the RAG-Web-Browser Actor. This allows you to either crawl and scrape top pages from Google Search results or directly scrape text content from a URL and return it as markdown. To set this up, change the actor_id to apify/rag-web-browser and specify the run_input.

loader = apify.call_actor(
    actor_id="apify/rag-web-browser",
    run_input={"query": "apify langchain web browser", "maxResults": 3},
    dataset_mapping_function=lambda item: Document(page_content=item["markdown"] or "", metadata={"source": item["metadata"]["url"]}),
)
print("Documents:", loader.load())

Similarly, you can use other Apify Actors to load data into LangChain and query the vector index.

Use Actors as LangChain tools

The ApifyWrapper shown above loads Actor output into a vector index. For agent workflows, the langchain-apify package also ships dedicated tools that wrap specific Actors behind a simplified, LLM-friendly interface, so an agent can call them without knowing Actor IDs or input schemas. Each tool is a standard LangChain BaseTool, so it works anywhere LangChain tools do.

Choose the right tool set

The package provides 19 tools split across three lists, so you can bind a focused subset to an agent instead of importing everything:

Tool setToolsUse case
APIFY_CORE_TOOLS6Run any Actor or saved task, fetch dataset items, scrape a single URL to markdown
APIFY_SEARCH_TOOLS6Google Search, Google Maps, YouTube, multi-page website crawling, RAG web browsing, e-commerce
APIFY_SOCIAL_TOOLS7Instagram, LinkedIn, Twitter/X, TikTok, Facebook

A model selects tools based on their names and descriptions. The more tools you register, the larger the decision space, which can lead to wrong tool selection, slower responses, and higher token usage. Register only the tools your agent needs.

Each list holds tool classes, so instantiate them before passing them to an agent. There are three ways to compose your tool list:

  1. Bind a whole tool set when your agent needs the full category:

    from langchain_apify import APIFY_SEARCH_TOOLS

    tools = [tool_cls() for tool_cls in APIFY_SEARCH_TOOLS]

  2. Import individual tools for tighter control:

    from langchain_apify import ApifyScrapeUrlTool, ApifyGoogleSearchTool

    tools = [ApifyScrapeUrlTool(), ApifyGoogleSearchTool()]

  3. Mix a tool set with individual tools:

    from langchain_apify import APIFY_CORE_TOOLS, ApifyTwitterScraperTool

    tools = [tool_cls() for tool_cls in APIFY_CORE_TOOLS] + [ApifyTwitterScraperTool()]

Call a tool directly

Each tool can run on its own, which is the quickest way to see what it returns. Every call runs a real Actor on the Apify platform, so it may take from seconds to minutes. Set your Apify API token and invoke the tool with its input:

import json
import os

from langchain_apify import ApifyGoogleSearchTool

os.environ["APIFY_TOKEN"] = "Your Apify API token"

tool = ApifyGoogleSearchTool()
result = tool.invoke({"query": "what is langchain", "max_results": 3})

payload = json.loads(result)
print("Run status:", payload["run"]["status"])
print("Items returned:", len(payload["items"]))

Most tools return a JSON string with two keys: run (run metadata such as status and dataset_id) and items (the Actor's dataset items).

Some components return a different shape

ApifyScrapeUrlTool follows the same JSON envelope, with the scraped markdown in the single item's content field. ApifySearchRetriever and ApifyCrawlLoader are the exceptions: they return LangChain Document objects instead.

Give the tools to an agent

To let a model decide when to call the tools, bind a tool list to an agent. The example below uses LangGraph's prebuilt ReAct agent, so install it alongside the previous dependencies:

pip install langgraph
import os

from langchain_apify import APIFY_SEARCH_TOOLS
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent

os.environ["OPENAI_API_KEY"] = "Your OpenAI API key"
os.environ["APIFY_TOKEN"] = "Your Apify API token"

model = ChatOpenAI(model="gpt-5.4-mini")
tools = [tool_cls() for tool_cls in APIFY_SEARCH_TOOLS]
agent = create_react_agent(model, tools)

response = agent.invoke(
    {"messages": [("human", "Search the web and tell me what Apify is.")]}
)
print(response["messages"][-1].content)

Retrieve documents for RAG

For retrieval-augmented generation, ApifySearchRetriever returns LangChain Document objects directly, so it plugs into a RAG chain in place of the vector index built earlier:

import os

from langchain_apify import ApifySearchRetriever

os.environ["APIFY_TOKEN"] = "Your Apify API token"

retriever = ApifySearchRetriever(max_results=3)
docs = retriever.invoke("What is LangChain?")
print(docs)

To crawl a whole site into documents on demand, use ApifyCrawlLoader instead.

Tool reference

Every dedicated tool below is importable from langchain_apify.

Core tools (APIFY_CORE_TOOLS)

Generic platform primitives that run any Actor or task and read datasets.

  • ApifyRunActorTool - start any Actor by ID and return run metadata only (run ID, status, dataset ID). Pair with ApifyGetDatasetItemsTool.
  • ApifyRunActorAndGetDatasetTool - run any Actor and return both run metadata and dataset items in one call.
  • ApifyGetDatasetItemsTool - read items from an existing dataset by ID, with limit / offset pagination.
  • ApifyScrapeUrlTool - scrape a single URL and return its markdown content in a JSON envelope (in the item's content field).
  • ApifyRunTaskTool - run a saved Actor task by ID and return run metadata.
  • ApifyRunTaskAndGetDatasetTool - run a saved task and return both run metadata and dataset items.

Search and crawling tools (APIFY_SEARCH_TOOLS)

Web search, crawling, and platform-specific search.

Social media tools (APIFY_SOCIAL_TOOLS)

Scrape posts and profiles from major social platforms.

Run any other Actor

For Actors without a dedicated tool, use ApifyActorsTool. Construct it with the Actor ID; it builds an input schema from the Actor and accepts a run_input dict matching that Actor's input:

from langchain_apify import ApifyActorsTool

tool = ApifyActorsTool("apify/google-trends-scraper")
result = tool.invoke({"run_input": {"searchTerms": ["web scraping", "data extraction"]}})

Resources