A single-page scrape is a request and a response. A crawl is a job: it discovers pages, fetches them, and takes minutes, not milliseconds. So crawls are asynchronous.
You start a crawl and get a job back. From there you can follow it live with server-sent events, or let it run and receive a webhook when it finishes. Each page comes back as clean Markdown, the same output as the single-page scrape.
Limits that matter
A crawl returns up to 5,000 pages. That covers marketing sites, documentation, blogs and most knowledge bases. Pages count against your monthly quota one for one, so a 400-page crawl is 400 pages.
How discovery works
The crawler starts from the URL you give it, reads the sitemap when there is one, and follows internal links. You get a count of pages found before you commit, which is exactly what the home page tool shows you.
Common jobs
- Site migration. Pull every page out of the old site as Markdown before you rebuild. See site migration.
- LLM knowledge bases. Crawl a docs site once, chunk it, embed it. See LLM ingestion.
- Content audits. Get every page’s text into one place and look for gaps.
When you do not need an account
Run the tool on the home page. It tells you how many pages the site has. Pick an export tier, $9 for up to 50 pages, $29 for 250 or $79 for 1,000, and download a ZIP. If you will do this regularly, a subscription is cheaper per page.
Be a good citizen: respect the sites you crawl. Our acceptable use policy spells out what that means.
Planning a crawl so it finishes cleanly
Most crawl problems are scoping problems. Before you start a job, decide what a successful result looks like. If you want the documentation and nothing else, start from the docs URL rather than the site root, so the crawler does not wander into the marketing pages and the blog. If you want everything, start at the root and let the sitemap do the discovery. Either way, look at the page count before you commit. A site that reports 40 pages when you expected 400 usually means a section is rendered by script or hidden behind a login, and it is better to learn that before you spend quota.
Think about quota too. Because pages count one for one, a crawl is the easiest way to burn a free month. The Free plan covers 100 pages, which is enough to crawl a small site end to end or to sample a large one. For a real migration or a knowledge base build, a paid plan is cheaper than the same pages on the one-time export tiers once you are past a few hundred of them.
Following a job
A crawl is a job with a state, so treat it like one. Start it, store the job ID, and follow progress over server-sent events if a person is watching, or wait for the webhook if a system is. Webhooks are part of the Scale plan. On the other plans, the event stream gives you the same signal without a callback URL. Do not poll the job in a tight loop. The stream exists so you do not have to.
What you do with the output
Each page arrives as Markdown with its URL, so the output maps cleanly onto a folder of files, a table of rows or a set of chunks for embedding. Keep the URL with every chunk. When a model answers from your knowledge base, a link back to the source page is what makes the answer checkable.
Etiquette and limits
You are fetching someone else’s server. Crawl sites you own, sites you have permission to read, or public pages within what the acceptable use policy allows. A crawl is not a license to ignore a site’s terms. If a site asks not to be crawled, leave it alone.