Retrieval-augmented generation is only as good as the text you put in. Raw HTML is noisy: menus, scripts and footers repeated on every page poison the embeddings and cost tokens.
Crawl a site and every page comes back as clean Markdown. Headings are intact, so you can chunk on them. Linked PDFs go through the same account.
A typical build
- Crawl the docs site as an async job.
- Split each page on headings.
- Embed and store the chunks with the source URL.
- Re-crawl on a schedule to catch changes.
Keep it honest
Only ingest content you are allowed to use. Check the site’s terms, and see our acceptable use policy. For internal knowledge you own, there is no question.
If you need one site once, you do not need an account: use the home page tool and buy an export from $9.