Firecrawl

Open-source crawler that turns the web into clean, LLM-ready Markdown.

Crawl4AI is an open-source (Apache 2.0) web crawler and scraper built for AI pipelines, converting pages into clean Markdown or structured data for RAG, agents, and data pipelines. The core runs locally with no API key, handles JS rendering, and supports optional LLM-based extraction with any provider. It installs as a Python library/CLI or deploys as a Dockerized FastAPI server; a hosted Cloud API is in closed beta.

Full-stack web scraping and browser automation platform for AI data.

A cloud platform for web scraping, data extraction, and browser automation built around 'Actors' — serverless programs that crawl sites and return structured data. Its store offers tens of thousands of ready-made Actors, and outputs clean Markdown or JSON that feed LLMs, vector databases, and RAG pipelines via LangChain and LlamaIndex. The company also maintains the open-source Crawlee crawling library for local development.

Turn any webpage into structured data with one prompt-driven API call.

ScrapeGraphAI is an AI web-scraping tool that extracts structured data from pages and documents using natural-language prompts instead of CSS selectors or XPath, orchestrating LLMs in graph-style pipelines (single-page, multi-page, search, crawl). The core library is open-source under the MIT license with Python and Node SDKs; a hosted API adds a credit-based free tier and paid plans, plus integrations with LangChain, LlamaIndex, n8n, and an MCP server.

Search-foundation APIs — Reader, embeddings, and reranker — for grounding LLMs.

A suite of search-foundation APIs for retrieval and RAG: a Reader that turns any URL or web search into LLM-ready markdown, multilingual multimodal embeddings, and a reranker. One key spans every service, the Reader is open source, and the embedding models are also released as open weights for self-hosting.