# AAN Botpass

official crawler IP ranges, polled daily, compiled for the rate limiter

TODO: add an endpoint to pull AAN end-user IPs (the old aan-users-1year-whitelist-ip.py job — internal logged-in-user IPs are not yet compiled into these lists).

19services
5,850entries
1failing
2026-08-13 20:23last successful fetch
15 6 * * *daily poll (cron)
Combined lists: all.txt sorted_ipv4.txt sorted_ipv6.txt
Service Category Entries Last run Last good data Sources polledLists
manual
Manual Whitelist
 170 ok 2026-08-13 20:23 manual-whitelist-ips.txt (project root, edited by hand)
georaw
google
Google Crawlers
 AI Bots, Cloud Services, Google Bots, Search Engines 2,155 ok 2026-08-13 20:23 developers.google.com/static/crawling/ipranges/common-crawlers.jsondevelopers.google.com/static/crawling/ipranges/special-crawlers.jsondevelopers.google.com/static/crawling/ipranges/user-triggered-fetchers.jsondevelopers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.jsondevelopers.google.com/static/crawling/ipranges/user-triggered-agents.json
georaw
bing
Bingbot
 Search Engines 28 ok 2026-08-13 20:23 www.bing.com/toolbox/bingbot.json
georaw
duckduckgo
DuckDuckBot
 AI Bots, Search Engines 481 ok 2026-08-13 20:23 duckduckgo.com/duckduckbot.json
georaw
apple
Applebot
 Search Engines 12 ok 2026-08-13 20:23 search.developer.apple.com/applebot.json
georaw
openai
OpenAI Crawlers
 AI Bots 272 ok 2026-08-13 20:23 openai.com/gptbot.jsonopenai.com/chatgpt-user.jsonopenai.com/searchbot.jsonopenai.com/adsbot.json
georaw
anthropic
Anthropic ClaudeBot
 AI Bots 21 ok 2026-08-13 20:23 claude.com/crawling/bots.json
georaw
perplexity
Perplexity
 AI Bots 12 ok 2026-08-13 20:23 www.perplexity.ai/perplexitybot.jsonwww.perplexity.ai/perplexity-user.json
georaw
meta
Meta Crawlers (AS32934)
 Social Bots 569 ok 2026-08-13 20:23 stat.ripe.net/data/announced-prefixes/data.json?resource=AS32934
georaw
amazon
Amazon Bots
 Cloud Services, Search Engines 2,059 ok 2026-08-13 20:23 developer.amazon.com/amazonbot/ip-addresses/developer.amazon.com/amazonbot/searchbot-ip-addresses/developer.amazon.com/amazonbot/live-ip-addresses/
georaw
x-twitter
Twitterbot (X)
 Social Bots 3 ok 2026-08-13 20:23 web.archive.org/web/2024/https://developer.twitter.com/en/docs/twitter-for-websites/cards/guides/troubleshooting-cards
georaw
pinterest
Pinterestbot
 Social Bots 1 ok 2026-08-13 20:23 help.pinterest.com/en/business/article/pinterestbot
georaw
internetarchive
Internet Archive (archive.org_bot)
 Other Agents 2 ok 2026-08-13 20:23 rdap.arin.net/registry/ip/207.241.224.1rdap.arin.net/registry/ip/208.70.24.1
georaw
turnitin
TurnitinBot
 Other Agents 1 ok 2026-08-13 20:23 www.turnitin.com/robot/crawlerinfo.html
georaw
ahrefs
AhrefsBot
 SEO Tools 51 ok 2026-08-13 20:23 api.ahrefs.com/v3/public/crawler-ip-ranges
georaw
semrush
Semrush SiteAuditBot
 1 ok 2026-08-13 20:23 www.semrush.com/kb/681-site-audit-troubleshooting
georaw
mojeek
MojeekBot
 Search Engines 1 ok 2026-08-13 20:23 www.mojeek.com/mojeekbot.json
georaw
youbot
YouBot
 AI Bots 1 ok 2026-08-13 20:23 you.com/docs/youbot
georaw
liner
LinerBot
 AI Bots 10 stale — serving last goodFailed to fetch https://docs.getliner.com/linerbot.json after 3 attempts: HTTP 525 from https://docs.getliner.com/linerbot.json 2026-08-07 02:33 docs.getliner.com/linerbot.json
georaw

Sorted by importance by default — click a column header to re-sort, click again to flip, third click restores the default order. A failing service keeps serving its last good snapshot until a poll succeeds.

Limited to once per 30 minutes (enforced server-side). CLI is unrestricted: node cli.js poll