Overview
Crawl4AI takes a URL and returns clean markdown you can feed straight to an LLM, handling the JavaScript rendering and DOM noise that make raw HTML useless in a research pipeline. The Docker server exposes MCP endpoints, so a research agent can call it directly. There's no GUI, so expect to use Python or Docker.
Key Features
- ✓Converts any page to clean, LLM-ready markdown
- ✓First-party MCP endpoints (SSE and WebSocket) in the Docker server
- ✓crwl command-line interface for scripting
- ✓Self-hosted FastAPI REST server, no per-page cost
- ✓Apache-2.0 licensed and free to run
Related tools
Other AI Research & Intelligence tools to consider:
Exa AI
Embeddings-first search API built for AI agents and applications
Free (1K searches/mo, Pro from $40/mo)
Firecrawl
Turn any website into clean markdown for AI pipelines
Free (500 lifetime credits); paid from $19/mo
Happenstance
AI search across your own Gmail, LinkedIn and X network to find warm intros
Free; $30/mo Pro ($24 billed yearly)
NotebookLM
Google's AI research assistant that synthesizes your documents into insights
Free / $19.99/mo via Google One AI Premium
Outline
Team wiki with a built-in MCP server for AI assistants
$10/mo Starter cloud; free self-hosted