GuideLangChainPythonRAG

LangChain web scraping: load web pages into LangChain Documents

A small LangChain document loader on top of the MarkUDown API: scrape one URL or crawl a site, get clean Markdown with source metadata, and split it by headings so every chunk keeps its context.

Updated November 16, 2026 · By Scrape Technology

Why Markdown instead of raw HTML

A web page's HTML is mostly navigation, scripts and styling. Feed it to a model and you pay for those tokens and dilute the retrieval index. MarkUDown returns only the main content as Markdown, so headings survive as # lines and LangChain's MarkdownHeaderTextSplitter can cut the page where the author did. When a page needs JavaScript, MarkUDown moves from a plain fetch to a headless browser and then to an Abrasio managed browser session on its own; your loader code does not change.

PageRaw HTML tokensMarkUDown tokensFewer tokens
books.toscrape.com (home)9,9391,96480%
books.toscrape.com (book page)2,12337083%
quotes.toscrape.com2,90722292%
quotes.toscrape.com (page 2)3,61360483%
Wikipedia: Web scraping74,03110,98785%
Wikipedia: Markdown102,18810,03890%
Python docs: json32,9758,73274%
Python tutorial: introduction21,4253,95882%
Total (8 pages)249,20136,87585%

Measured on 2026-11-16: raw HTML fetched directly, Markdown from /scrape with the SDK defaults, both counted with tiktoken (o200k_base). Median time per /scrape call: 2.6 s, measured from the client.

1. Install and set your key

Python 3.10 or newer. Get a key at MarkUDown (500 free credits, no credit card); the auth docs explain the X-API-KEY header the SDK sends for you.

bash
pip install markudown langchain-core langchain-text-splitters
export MARKUDOWN_API_KEY="your-api-key"   # PowerShell: $env:MARKUDOWN_API_KEY="your-api-key"

2. The loader

Subclassing BaseLoader and implementing lazy_load() gives you .load() and .lazy_load() like any LangChain loader. In scrape mode each URL becomes one Document; in crawl mode MarkUDown follows internal links from the start URL and every page becomes a Document. Pages that came back empty are skipped.

markudown_loader.py
import os
from typing import Iterator, Literal

from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document
from markudown import MarkUDown


class MarkUDownLoader(BaseLoader):
    """Load web pages as LangChain Documents through the MarkUDown API."""

    def __init__(self, urls, *, mode: Literal["scrape", "crawl"] = "scrape",
                 limit: int = 10, max_depth: int = 2, api_key: str | None = None):
        self.urls = [urls] if isinstance(urls, str) else list(urls)
        self.mode, self.limit, self.max_depth = mode, limit, max_depth
        self.client = MarkUDown(api_key=api_key or os.environ["MARKUDOWN_API_KEY"])

    def lazy_load(self) -> Iterator[Document]:
        for url in self.urls:
            if self.mode == "scrape":
                pages = [self.client.scrape(url)]
            else:  # crawl: the SDK polls the job until it completes
                job = self.client.crawl(url, limit=self.limit, max_depth=self.max_depth)
                pages = job["data"]["data"]
            for page in pages:
                if page.get("markdown"):
                    yield self._to_document(page)

    @staticmethod
    def _to_document(page: dict) -> Document:
        meta = page.get("metadata") or {}
        return Document(
            page_content=page["markdown"],
            metadata={
                "source": page.get("url"),
                "title": meta.get("title"),
                "description": meta.get("description"),
                "status_code": meta.get("statusCode"),
            },
        )

For reference, these are the response shapes the loader reads (see the scrape and crawl docs):

response shapes
# md.scrape(url) returns
{"url": "...", "markdown": "# Title\n...", "links": [...],
 "metadata": {"title": "...", "description": "...", "source": "...", "statusCode": 200}}

# md.crawl(url) returns, once the job is completed
{"success": True, "id": "...", "status": "completed",
 "data": {"total": 5, "data": [{"url": "...", "markdown": "...", "metadata": {...}}, ...]}}

3. Scrape, crawl and split

The example loads one documentation page and a five-page crawl of a public practice site, then splits everything in two passes: by Markdown heading (the heading text lands in the chunk metadata as h1/h2/h3) and by size, so no chunk is longer than 1,000 characters.

load_and_split.py
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

from markudown_loader import MarkUDownLoader

# 1. One page -> one Document (1 credit)
docs = MarkUDownLoader("https://docs.python.org/3/tutorial/introduction.html").load()
print(docs[0].metadata)

# 2. A small crawl -> one Document per page (up to 5 credits)
site = MarkUDownLoader("https://quotes.toscrape.com/", mode="crawl", limit=5, max_depth=1).load()
for d in site:
    print(d.metadata["source"], len(d.page_content), "chars")

# 3. Split on Markdown headings first, then cap the chunk size
by_heading = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")], strip_headers=False
)
by_size = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)

chunks = []
for doc in docs + site:
    for section in by_heading.split_text(doc.page_content):
        section.metadata.update(doc.metadata)  # keep source/title on every chunk
        chunks.extend(by_size.split_documents([section]))

print(len(chunks), "chunks")
print(chunks[0].metadata)

The chunks list is ready for any vector store or retriever. Each chunk carries source and title, so answers can cite the page they came from.

4. Optional: search the chunks locally

To check retrieval without an embeddings API, BM25 ranks chunks by keyword overlap and runs entirely on your machine. Swap it for a vector store when you move to semantic search; the chunks do not change.

search_local.py
# pip install langchain-community rank_bm25
from langchain_community.retrievers import BM25Retriever

from load_and_split import chunks

retriever = BM25Retriever.from_documents(chunks, k=3)
for hit in retriever.invoke("How do I slice a string in Python?"):
    print(hit.metadata["source"], "|", hit.metadata.get("h2"), "|", hit.page_content[:80].replace("\n", " "))

Before you run it on more pages

  • Credits: a scrape costs 1 credit. A crawl deducts its limit in credits when it starts, so set limit to the number of pages you actually want. Full table on the pricing page.
  • Long crawls: pass wait=False to crawl() to get a job id back at once, then poll with get_job_status("crawl", job_id), or give a callback_url and receive a webhook (async jobs).
  • Many single URLs: batch_scrape(urls) scrapes a list in one job, or use AsyncMarkUDown with asyncio.gather.
  • Errors: the SDK raises RateLimitError on 429 and InsufficientCreditsError on 402/403, both importable from markudown.
  • Filtering a crawl: include_only and blocked_words keep it to the sections you care about (crawl docs).
  • Agents: if your LangChain or LangGraph agent should decide when to scrape, the MarkUDown MCP server (npx markudown-mcp) exposes the same endpoints as tools.

Comparing options

Firecrawl ships an official LangChain package; we compare the two services, including where Firecrawl is the better pick, in MarkUDown vs Firecrawl. If you need to drive the browser yourself (log in, click, fill forms) rather than load content, see Abrasio vs Browserbase.

Load your first site into LangChain

Create a MarkUDown API key with 500 free credits. No credit card.