List of supported crawlers - Fluid Topics - Latest

Fluid Topics Configuration and Administration Guide
Category
Reference Guides
Audience
public
Version
Latest

Since Fluid Topics is a single-page application, and crawlers cannot always crawl such applications correctly, Fluid Topics detects crawlers based on their user-agent string.

Fluid Topics recognizes user-agents matching any of the following values as crawlers, and serves them HTML-only content:

  • Any user-agent value containing the word bot
  • AdsBot-Google
  • AhrefsBot
  • anthropic
  • Applebot
  • BecomeBot
  • bingbot
  • BingPreview
  • Claude
  • cohere-ai
  • crawler
  • facebookexternalhit
  • Feedfetcher
  • Gemini
  • Go-http-client
  • Google Web Preview
  • Google-Agent
  • Googlebot-Mobile
  • Googlebot
  • GPTBot
  • Java/
  • Mediapartners-Google
  • meta-externalagent
  • meta-webindexer
  • MistralAI
  • Nutch
  • OAI-SearchBot
  • Perplexity
  • Phantom
  • python-requests
  • Seznam
  • Slurp
  • spider
  • Teoma
  • ThumbSniper
  • Twiceler
  • Web Preview
  • Webzio-Extended
  • WordPress.com mShots
  • YandexBot
  • YandexMarket

To check what a crawler gets, it is possible to update the browser's user agent to view pages as Googlebot or Bingbot.

  • User-agent matching is case-insensitive.
  • For in-house search engines (any search engine not meant for the web), Fluid Topics provides the List maps web service to index the content of a Fluid Topics tenant.