LangChain web scraping: load web pages into LangChain Documents
A small LangChain document loader on top of the MarkUDown API: scrape one URL or crawl a site, get clean Markdown with source metadata, and split it by headings so every chunk keeps its context.
Updated November 16, 2026 · By Scrape Technology
Why Markdown instead of raw HTML
A web page's HTML is mostly navigation, scripts and styling. Feed it to a model and you pay for those tokens and dilute the retrieval index. MarkUDown returns only the main content as Markdown, so headings survive as # lines and LangChain's MarkdownHeaderTextSplitter can cut the page where the author did. When a page needs JavaScript, MarkUDown moves from a plain fetch to a headless browser and then to an Abrasio managed browser session on its own; your loader code does not change.
| Page | Raw HTML tokens | MarkUDown tokens | Fewer tokens |
|---|---|---|---|
| books.toscrape.com (home) | 9,939 | 1,964 | 80% |
| books.toscrape.com (book page) | 2,123 | 370 | 83% |
| quotes.toscrape.com | 2,907 | 222 | 92% |
| quotes.toscrape.com (page 2) | 3,613 | 604 | 83% |
| Wikipedia: Web scraping | 74,031 | 10,987 | 85% |
| Wikipedia: Markdown | 102,188 | 10,038 | 90% |
| Python docs: json | 32,975 | 8,732 | 74% |
| Python tutorial: introduction | 21,425 | 3,958 | 82% |
| Total (8 pages) | 249,201 | 36,875 | 85% |
Measured on 2026-11-16: raw HTML fetched directly, Markdown from /scrape with the SDK defaults, both counted with tiktoken (o200k_base). Median time per /scrape call: 2.6 s, measured from the client.
1. Install and set your key
Python 3.10 or newer. Get a key at MarkUDown (500 free credits, no credit card); the auth docs explain the X-API-KEY header the SDK sends for you.
pip install markudown langchain-core langchain-text-splitters
export MARKUDOWN_API_KEY="your-api-key" # PowerShell: $env:MARKUDOWN_API_KEY="your-api-key"2. The loader
Subclassing BaseLoader and implementing lazy_load() gives you .load() and .lazy_load() like any LangChain loader. In scrape mode each URL becomes one Document; in crawl mode MarkUDown follows internal links from the start URL and every page becomes a Document. Pages that came back empty are skipped.
import os
from typing import Iterator, Literal
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document
from markudown import MarkUDown
class MarkUDownLoader(BaseLoader):
"""Load web pages as LangChain Documents through the MarkUDown API."""
def __init__(self, urls, *, mode: Literal["scrape", "crawl"] = "scrape",
limit: int = 10, max_depth: int = 2, api_key: str | None = None):
self.urls = [urls] if isinstance(urls, str) else list(urls)
self.mode, self.limit, self.max_depth = mode, limit, max_depth
self.client = MarkUDown(api_key=api_key or os.environ["MARKUDOWN_API_KEY"])
def lazy_load(self) -> Iterator[Document]:
for url in self.urls:
if self.mode == "scrape":
pages = [self.client.scrape(url)]
else: # crawl: the SDK polls the job until it completes
job = self.client.crawl(url, limit=self.limit, max_depth=self.max_depth)
pages = job["data"]["data"]
for page in pages:
if page.get("markdown"):
yield self._to_document(page)
@staticmethod
def _to_document(page: dict) -> Document:
meta = page.get("metadata") or {}
return Document(
page_content=page["markdown"],
metadata={
"source": page.get("url"),
"title": meta.get("title"),
"description": meta.get("description"),
"status_code": meta.get("statusCode"),
},
)For reference, these are the response shapes the loader reads (see the scrape and crawl docs):
# md.scrape(url) returns
{"url": "...", "markdown": "# Title\n...", "links": [...],
"metadata": {"title": "...", "description": "...", "source": "...", "statusCode": 200}}
# md.crawl(url) returns, once the job is completed
{"success": True, "id": "...", "status": "completed",
"data": {"total": 5, "data": [{"url": "...", "markdown": "...", "metadata": {...}}, ...]}}3. Scrape, crawl and split
The example loads one documentation page and a five-page crawl of a public practice site, then splits everything in two passes: by Markdown heading (the heading text lands in the chunk metadata as h1/h2/h3) and by size, so no chunk is longer than 1,000 characters.
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
from markudown_loader import MarkUDownLoader
# 1. One page -> one Document (1 credit)
docs = MarkUDownLoader("https://docs.python.org/3/tutorial/introduction.html").load()
print(docs[0].metadata)
# 2. A small crawl -> one Document per page (up to 5 credits)
site = MarkUDownLoader("https://quotes.toscrape.com/", mode="crawl", limit=5, max_depth=1).load()
for d in site:
print(d.metadata["source"], len(d.page_content), "chars")
# 3. Split on Markdown headings first, then cap the chunk size
by_heading = MarkdownHeaderTextSplitter(
headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")], strip_headers=False
)
by_size = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = []
for doc in docs + site:
for section in by_heading.split_text(doc.page_content):
section.metadata.update(doc.metadata) # keep source/title on every chunk
chunks.extend(by_size.split_documents([section]))
print(len(chunks), "chunks")
print(chunks[0].metadata)The chunks list is ready for any vector store or retriever. Each chunk carries source and title, so answers can cite the page they came from.
4. Optional: search the chunks locally
To check retrieval without an embeddings API, BM25 ranks chunks by keyword overlap and runs entirely on your machine. Swap it for a vector store when you move to semantic search; the chunks do not change.
# pip install langchain-community rank_bm25
from langchain_community.retrievers import BM25Retriever
from load_and_split import chunks
retriever = BM25Retriever.from_documents(chunks, k=3)
for hit in retriever.invoke("How do I slice a string in Python?"):
print(hit.metadata["source"], "|", hit.metadata.get("h2"), "|", hit.page_content[:80].replace("\n", " "))Before you run it on more pages
- Credits: a scrape costs 1 credit. A crawl deducts its
limitin credits when it starts, so setlimitto the number of pages you actually want. Full table on the pricing page. - Long crawls: pass
wait=Falsetocrawl()to get a job id back at once, then poll withget_job_status("crawl", job_id), or give acallback_urland receive a webhook (async jobs). - Many single URLs:
batch_scrape(urls)scrapes a list in one job, or useAsyncMarkUDownwithasyncio.gather. - Errors: the SDK raises
RateLimitErroron 429 andInsufficientCreditsErroron 402/403, both importable frommarkudown. - Filtering a crawl:
include_onlyandblocked_wordskeep it to the sections you care about (crawl docs). - Agents: if your LangChain or LangGraph agent should decide when to scrape, the MarkUDown MCP server (
npx markudown-mcp) exposes the same endpoints as tools.
Comparing options
Firecrawl ships an official LangChain package; we compare the two services, including where Firecrawl is the better pick, in MarkUDown vs Firecrawl. If you need to drive the browser yourself (log in, click, fill forms) rather than load content, see Abrasio vs Browserbase.
Load your first site into LangChain
Create a MarkUDown API key with 500 free credits. No credit card.