Crawler

Key modules in omnivore-core: - crawler::frontier: URL frontier and prioritization - crawler::scheduler: task scheduling and worker pools - crawler::worker: fetchers and response handling - crawler::robots: robots.txt and politeness - crawler::browser: dynamic crawling via browser automation

Configuration: see CrawlConfig and PolitenessConfig.