MarkUDown
Crawl API — MarkUDown website crawler API docs
POST
/api/crawlCrawl
Recursively follow internal links from a starting URL, returning Markdown for every page discovered. Ideal for converting entire documentation sites or blogs.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
| url | array<string> | required | — | Starting URL, as a one-item array (only the first item is used). |
| max_depth | integer | optional | 2 | Maximum link depth to follow from the starting URL. |
| limit | integer | optional | 5 | Maximum pages to crawl. The job is charged this many credits when it starts. |
| blocked_words | array<string> | optional | [] | Skip URLs that contain any of these words, e.g. ["archive","legacy"]. |
| exclude_tags | array<string> | optional | ["header","nav","footer"] | CSS selectors or tag names to strip, e.g. ["nav",".ads","footer"]. |
| include_html | boolean | optional | false | Also return each page's HTML. |
| main_content | boolean | optional | true | Extract only the main content (drops navigation, header and footer). |
| timeout | integer | optional | 60 | Per-page timeout in seconds. |
| callback_url | string | optional | null | Webhook URL to POST the result to when the job completes. |
Request
{
"url": ["https://docs.example.com/"],
"max_depth": 2,
"limit": 20,
"blocked_words": ["archive"]
}Response
{
"success": true,
"id": "abc123",
"url": "/api/crawl/abc123"
}
// Poll GET /api/crawl/{id}
{
"success": true,
"id": "abc123",
"status": "completed",
"data": {
"success": true,
"status": "completed",
"total": 2,
"data": [
{
"url": "https://docs.example.com/intro",
"markdown": "# Introduction\n\n...",
"metadata": { "title": "Introduction", "statusCode": 200 }
}
]
}
}