List of official public crawlers and generic AI bots - Fluid Topics - Latest

Fluid Topics Analytics Guide

Category
Reference Guides
Audience
public
Version
Latest

All API calls related to indexing activity from public crawlers and generic AI bots are excluded from API calls count and invoices.

Below is list of official public crawlers and generic AI bots which are consequently dismissed by the Fluid Topic analytics:

  • Adidxbot
  • AdsBot-Google
  • AhrefsBot
  • AhrefsSiteAudit
  • Amazonbot
  • Applebot
  • Baiduspider
  • Bingbot
  • BingPreview
  • BingSearchBot
  • Bot.html
  • ClaudeBot
  • Cohere-ai
  • Diffbot
  • Discordbot
  • DotBot
  • DuckAssistBot
  • DuckDuckBot
  • Exabot
  • FacebookBot
  • Facebookexternalhit
  • Facebot
  • Gigabot
  • Google-Extended
  • Googlebot
  • Googlebot-Image
  • Googlebot-News
  • Googlebot-Video
  • GoogleOther
  • GPTBot
  • Grunk
  • IbouBot
  • LinkedInBot
  • Meta-ExternalAgent
  • Microsoftbot
  • MJ12bot
  • Omgili
  • PerplexityBot
  • PetalBot
  • Qwantify
  • Screaming Frog SEO Spider
  • SemrushBot
  • SemrushBot-SA
  • SeznamBot
  • Sitebulb
  • Slackbot
  • Slurp
  • Sogou
  • Twitterbot
  • YandexAccessibilityBot
  • YandexBot
  • YandexImages
  • Yeti
  • YouBot

Bot detection relies on the User-Agent HTTP header. If the user agent associated to the user event belongs to the list of public crawlers and generic AI bots, then it is considered as coming from a bot.