Since Fluid Topics is a single-page application, and crawlers cannot always crawl such applications correctly, Fluid Topics detects crawlers based on their user-agent string.
Fluid Topics recognizes user-agents matching any of the following values as crawlers, and serves them HTML-only content:
- Any user-agent value containing the word
bot AdsBot-GoogleAhrefsBotApplebotBecomeBotbingbotBingPreviewcrawlerfacebookexternalhitFeedfetcherGo-http-clientGoogle-InspectionToolGoogle Web PreviewGooglebotIframelyJava/Mediapartners-GoogleNutchPhantompython-requestsSeznamSlurpspiderTeomaThumbSniperTwicelerWeb PreviewWordPress.com mShotsYandexBotYandexMarketyandex.com/bots
- To optimize content crawling by AI user agents, Fluid Topics can return Markdown instead of HTML. See Activate Markdown for AI crawlers.
- To check what a crawler gets, it is possible to update the browser's user agent to view pages as Googlebot or Bingbot.
- User-agent matching is case-insensitive and uses substring matching (for example,
Googlebotmatches any user agent containing that string). - For in-house search engines (any search engine not meant for the web), Fluid Topics provides the List maps web service to index the content of a Fluid Topics tenant.