MarkUDown

Crawl API — MarkUDown website crawler API docs

POST/api/crawl

Crawl

Recursively follow internal links from a starting URL, returning Markdown for every page discovered. Ideal for converting entire documentation sites or blogs.

ParameterTypeRequiredDefaultDescription
urlarray<string>required—Starting URL, as a one-item array (only the first item is used).
max_depthintegeroptional2Maximum link depth to follow from the starting URL.
limitintegeroptional5Maximum pages to crawl. The job is charged this many credits when it starts.
blocked_wordsarray<string>optional[]Skip URLs that contain any of these words, e.g. ["archive","legacy"].
exclude_tagsarray<string>optional["header","nav","footer"]CSS selectors or tag names to strip, e.g. ["nav",".ads","footer"].
include_htmlbooleanoptionalfalseAlso return each page's HTML.
main_contentbooleanoptionaltrueExtract only the main content (drops navigation, header and footer).
timeoutintegeroptional60Per-page timeout in seconds.
callback_urlstringoptionalnullWebhook URL to POST the result to when the job completes.

Request

{
  "url": ["https://docs.example.com/"],
  "max_depth": 2,
  "limit": 20,
  "blocked_words": ["archive"]
}

Response

{
  "success": true,
  "id": "abc123",
  "url": "/api/crawl/abc123"
}

// Poll GET /api/crawl/{id}
{
  "success": true,
  "id": "abc123",
  "status": "completed",
  "data": {
    "success": true,
    "status": "completed",
    "total": 2,
    "data": [
      {
        "url": "https://docs.example.com/intro",
        "markdown": "# Introduction\n\n...",
        "metadata": { "title": "Introduction", "statusCode": 200 }
      }
    ]
  }
}