The RssSitemapBot.

Reads RSS, Atom, and sitemap files to feed our global indexing queue.

How it works

Identification

WebAtlasRssSitemapBot identifies itself with the following User-Agent.

WebAtlasRssSitemapBot/1.1 (+https://webatlasindex.pl/rsssitemap-bot.php)

✓

Verification

All WebAtlas crawlers connect only from the IP addresses listed in ip-ranges.json. A request with our User-Agent from any other address does not come from WebAtlas.

⟲

Purpose & logic

A lightweight crawler whose job is to read submitted RSS/Atom feeds and sitemap files, then extract the URLs they contain.

⫶

Filtering layers

Extracted URLs are automatically filtered through three layers:

Blocklisted ad/social domains.
Technical paths (e.g. /admin, /wp-login, /api).
Heuristic ad/tracking parameters (gclid, utm_source, clickid).

⌫

Webmaster control

To block WebAtlasRssSitemapBot entirely, add this to your robots.txt:

User-agent: WebAtlasRssSitemapBot
Disallow: /