Crawler documentation
Meet WebAtlasBot.
The official crawler of the MevaSearch search engine, building a safe and filter-free map of the internet.
Crawler documentation
The official crawler of the MevaSearch search engine, building a safe and filter-free map of the internet.
WebAtlasBot identifies itself with the following User-Agent.
WebAtlasBot/2.0 (+https://webatlasindex.pl/bot.php)
All WebAtlas crawlers connect only from the IP addresses listed in ip-ranges.json. A request with our User-Agent from any other address does not come from WebAtlas.
The crawler minimizes server load. It respects robots.txt directives, honors Crawl-delay, and checks noindex/canonical tags before saving a page.
Collected content feeds the WebAtlas Index — our own database used for search and semantic analysis by MevaSearch.
Block or throttle WebAtlasBot with standard robots.txt rules:
User-agent: WebAtlasBot
Disallow: /private/
Crawl-delay: 5The bot also honors the meta tag <meta name="robots" content="noindex">.
WebAtlasBot collects data for the MevaSearch engine, focusing on high-quality, educational and technical content, aiming to build a knowledge base resistant to algorithmic manipulation.
AdultBot and WebOrbitonBot are sibling crawlers of the same project, run by the same team and held to the same standards described on this page.