Overview
Automatically crawl and extract content from your entire website - all pages, subpages, and docs.Quick Start
1
Enter Root URL
E.g., https://example.com
2
Configure Crawler
Set depth, exclude patterns
3
Start Crawl
Process can take minutes to hours
Configuration
Crawl Depth
- Level 1: Homepage only
- Level 2: Homepage + direct links
- Level 3: Up to 3 clicks deep
- Unlimited: Entire site (Pro plans)