By defining default robots meta tags, ADMIN users can control how bots crawl content on tenants where crawling is allowed.
Fluid Topics supports the following meta tags:
noindexnofollownoarchive
Use case
It can be useful to define default robots meta tags when deciding to index all content but not cache it for security reasons.
- Since Fluid Topics is a single-page application, and crawlers cannot always crawl such applications correctly, Fluid Topics detects crawlers based on their user-agent string. See List of supported crawlers.
- There are never any robots meta tags for the most common user agents.
- Meta tags do not apply to documents that require authentication.
- To check what a crawler gets, it is possible to update the browser's user agent to view pages as Googlebot or Bingbot.
- For details about robots meta tags, see Google's Robots meta tags documentation.
- All major search engines support the robots meta tags specification.
- The
<meta name="robots" content="noindex">tag is automatically applied to external documents.