Crawl4AI is the most popular open-source web crawler on GitHub — 61,900+ stars, more than Scrapy, more than Playwright. It was built from the ground up for LLM consumption: every page becomes clean markdown, not HTML soup. And since v0.8, it has a built-in MCP server that exposes its full capabilities directly to AI agents.

The catch: there’s no hosted option. You run Crawl4AI yourself via Docker, and the MCP integration is still maturing. But the underlying crawler is battle-tested, completely free, and handles things that commercial alternatives charge per-page for.

I’ve been comparing it against Firecrawl, Tavily, and Playwright across web scraping tasks. Here’s what I found.

What It Does

The Crawl4AI MCP server exposes seven tools through its Docker deployment:

  • md — Generate clean markdown from any URL. The core capability — Crawl4AI’s markdown generation includes “Fit Markdown” (heuristic noise filtering), numbered citation references, and configurable content filters. This is what 61,000 people starred the project for.
  • html — Extract preprocessed HTML from a page. Useful when you need the DOM structure rather than markdown — form analysis, layout inspection, or feeding into CSS/XPath extraction strategies.
  • screenshot — Capture full-page screenshots of any URL. Returns the visual state of the page, useful for debugging JavaScript-heavy sites or verifying that dynamic content rendered correctly.
  • pdf — Generate PDF documents from web pages. Captures the page as a printable document, preserving layout and styling that markdown conversion strips out.
  • execute_js — Run JavaScript on a web page. Click buttons, fill forms, scroll to trigger lazy loading, dismiss cookie banners — anything that requires interaction before the content you need becomes visible.
  • crawl — Multi-URL crawling with configurable concurrency. Process multiple URLs in parallel with Crawl4AI’s resource-aware dispatching. Supports depth control, adaptive crawling (automatically stops when enough content is gathered), and crash recovery for long-running jobs.
  • ask — Query the Crawl4AI library documentation. A meta-tool — ask how to use Crawl4AI itself. Useful for agents that need to construct complex crawl configurations without prior knowledge of the API.

The tool count is modest (seven) compared to Firecrawl (12+), but the scope is different. Crawl4AI gives you raw web interaction primitives — render a page, grab the markdown, run some JavaScript, take a screenshot — rather than higher-level abstractions like “extract structured data matching this schema.” The extraction intelligence lives in the crawler engine, not the MCP tool definitions.

Setup

Crawl4AI’s MCP server runs exclusively through Docker. There’s no npx one-liner, no hosted URL, no pip install.

Step 1: Start the Docker container.

docker run -d -p 11235:11235 --name crawl4ai \
  --env-file .llm.env --shm-size=1g \
  unclecode/crawl4ai:latest

The .llm.env file holds API keys for any LLM providers you want to use with extraction strategies (OpenAI, Anthropic, Ollama, etc.). If you’re only using the markdown and screenshot tools, you can skip the env file entirely.

Step 2: Connect via MCP. Two transport options:

# SSE (Server-Sent Events)
claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse

# WebSocket
claude mcp add --transport sse c4ai-ws ws://localhost:11235/mcp/ws

Setup difficulty: Moderate. You need Docker installed and running, and you need to understand which transport your MCP client supports. There’s no remote hosted option — the server runs on your machine or your infrastructure. The container requires --shm-size=1g because Playwright (which powers the browser) needs shared memory for rendering.

Configuration options worth knowing:

  • Tool schemas are available at http://localhost:11235/mcp/schema — check this endpoint for the full parameter documentation.
  • The container supports .llm.env for configuring LLM-based extraction with any LiteLLM-compatible provider (OpenAI, Anthropic, Ollama, hundreds of models).
  • config.yml in the container controls port and server behavior — the MCP server runs on port 11235 by default.

What Works Well

The markdown extraction is genuinely best-in-class. Crawl4AI’s “Fit Markdown” goes beyond simple HTML-to-markdown conversion. It uses heuristics to strip navigation, footers, sidebars, and boilerplate — the noise that wastes context window tokens. The result is clean, focused content that’s immediately useful in a RAG pipeline. This is the feature that earned 61,900 stars, and it’s as good through MCP as it is through the Python API.

Completely free with no credit limits. No API keys to manage, no monthly credit budgets, no per-page charges. Crawl a thousand pages or ten thousand — it costs you compute, not credits. Compare this to Firecrawl (non-renewable 500 free credits, then paid plans), Tavily (1,000 credits/month that expire), or Browserbase ($20+/month). For high-volume scraping, the cost difference is enormous.

JavaScript execution unlocks dynamic content. The execute_js tool is a genuine differentiator over simpler scrapers. Modern websites hide content behind cookie banners, “load more” buttons, infinite scroll, and JavaScript-rendered SPAs. Being able to run arbitrary JavaScript before extracting content means Crawl4AI can handle sites that return empty pages to basic HTTP scrapers. Playwright runs under the hood, so you get a full browser environment.

Adaptive crawling is intelligent. The crawl tool doesn’t just blindly follow links — it can automatically determine when sufficient content has been gathered and stop. For agents that need “enough information about topic X” rather than “every page on this site,” adaptive crawling saves time and compute.

LLM-based extraction strategies (via the Python API). While the MCP tools expose the basics, Crawl4AI’s underlying engine supports structured data extraction using any LLM via LiteLLM — define a Pydantic schema, point it at a page, and get structured JSON back. It also supports LLM-free extraction via CSS selectors, XPath, and regex. These capabilities are accessible through the crawl tool’s configuration parameters.

Crash recovery for long crawls. Deep crawls fail — networks drop, pages timeout, containers restart. Crawl4AI v0.8 added resume_state and on_state_change callbacks that let you pick up where you left off. No other MCP-accessible crawler offers this.

What Doesn’t Work Well

Docker is a hard requirement. No Docker, no Crawl4AI MCP server. This rules out environments where Docker isn’t available — many corporate laptops, some CI/CD environments, Codespaces with limited permissions. Every other scraping MCP server we’ve reviewed (Firecrawl, Playwright, Puppeteer, Browserbase) offers at least an npx or pip installation path.

The MCP integration is still maturing. GitHub issue #1316 documents SSE connection failures where the server shows “connected” but tool calls fail with “Unexpected message” errors. Issue #1311 reports missing type fields in MCP function schemas, breaking compatibility with Google Gemini CLI. The underlying crawler is stable, but the MCP layer has rough edges. Workarounds exist (port configuration fixes), but this isn’t plug-and-play yet.

No stdio transport. Crawl4AI’s MCP server supports SSE and WebSocket, but not stdio — the most widely supported MCP transport. Claude Desktop, most VS Code extensions, and many MCP clients default to stdio. You need a client that supports SSE or WebSocket connections, which limits compatibility.

No hosted or remote option. Firecrawl has a cloud API. Tavily has a remote MCP server you can connect to with a URL. Browserbase is cloud-native. Crawl4AI requires you to run and maintain your own Docker container. For teams that want to share a scraping service across multiple agents, you need to set up your own infrastructure.

The MCP tools are thin wrappers. Seven tools sounds reasonable, but they’re lower-level than competitors. Firecrawl’s MCP server includes an autonomous research agent, batch scraping with status polling, and LLM-powered extraction — all as distinct tools. Crawl4AI’s MCP layer exposes “crawl a URL, get markdown” and leaves the sophistication to the agent. The powerful extraction strategies and chunking capabilities exist in the Python API but aren’t fully surfaced as MCP tools.

Community fragmentation. Search “Crawl4AI MCP” on GitHub and you’ll find a dozen community implementations — sadiuysal, BjornMelin, coleam00, vivmagarwal, walksoda, and more. Each wraps Crawl4AI differently with different tool sets, different transports, and different levels of maturity. The built-in MCP server should be the canonical choice, but the documentation doesn’t clearly direct users to it over the community alternatives.

Compared to Alternatives

vs. Firecrawl: Firecrawl is the more polished MCP experience — 12+ tools, an autonomous research agent, LLM-powered extraction as a first-class tool, hosted cloud option, and both stdio and HTTP transports. But Firecrawl charges per page (500 non-renewable free credits, then $19+/month). For high-volume scraping where you control the infrastructure, Crawl4AI is dramatically cheaper. Firecrawl also lacks JavaScript execution and crash recovery.

vs. Playwright: Playwright’s MCP server offers 25+ tools with precise, deterministic browser control — CSS selectors, accessibility tree snapshots, network interception. It’s free and doesn’t need Docker. But Playwright gives you raw browser automation, not web scraping. You get HTML, not clean markdown. Crawl4AI handles the HTML-to-useful-content conversion that Playwright leaves to you.

vs. Tavily: Tavily combines search + extraction + crawling in a hosted package with a one-line setup. It’s easier to start with and doesn’t require Docker. But Tavily is credit-based (1,000/month free, then paid), has keyword-only search, and you can’t customize the extraction pipeline. Crawl4AI gives you full control over how content is extracted and processed, at zero ongoing cost.

vs. Puppeteer: Puppeteer’s MCP server is the simpler browser automation option — 14 tools, stdio transport, no Docker required. Like Playwright, it gives you browser control, not web scraping. Crawl4AI’s markdown extraction, adaptive crawling, and LLM extraction strategies go well beyond what Puppeteer offers.

vs. Fetch (built-in): The reference Fetch MCP server converts URLs to markdown with zero setup. But it can’t handle JavaScript-rendered content, doesn’t support multi-page crawling, has no extraction strategies, and produces basic markdown without noise filtering. Crawl4AI is what you upgrade to when Fetch isn’t enough.

Who Should Use This

Yes, use it if:

  • You need high-volume web scraping without per-page costs
  • You want the best HTML-to-markdown conversion available
  • You need JavaScript execution to handle dynamic content
  • You’re comfortable running Docker on your infrastructure
  • You want LLM-based structured extraction via LiteLLM
  • You need crash recovery for long-running crawl jobs

Skip it if:

  • You can’t run Docker in your environment
  • You need a hosted/cloud scraping service (use Firecrawl)
  • You want a plug-and-play MCP experience with stdio transport (use Playwright or Puppeteer)
  • You need search + scraping in one server (use Tavily)
  • You need the simplest possible setup (use Fetch)
  • MCP stability is critical — the MCP layer has known issues (#1316, #1311)
3.5 / 5 — The most powerful free web scraper, with an MCP layer that's still catching up
Crawl4AI is the most popular open-source web crawler for a reason — its markdown extraction is best-in-class, it handles JavaScript-heavy sites through Playwright, and it costs nothing no matter how many pages you crawl. The underlying engine is battle-tested across 61,900+ stargazers. But the MCP integration is the newest layer on a mature project, and it shows: SSE connection bugs, missing schema fields, no stdio transport, Docker-only deployment, and thin tool abstractions that don’t expose the full power of the Python API. If you’re comfortable with Docker and can work around the MCP rough edges, you get the best free web scraper in the ecosystem. If you need a polished, production-ready MCP experience, Firecrawl or Playwright are safer choices today.