The robots.txt file gives crawlers instructions about what parts of the portal to index.
If an administrator allows the crawling of the content of the portal, the default content of the robots.txt file is:
User-agent: *
Disallow: /
Allow: /$
Allow: /home
Allow: /r/*
Allow: /v/u/*
Allow: /go/*
Allow: /access/sources/*/*
Allow: /reader/*/root
Allow: /reader/*/*
Allow: /viewer/document/*
Allow: /p/*
Allow: /api/khub/documents/*/content
Allow: /sitemap.xml
Allow: /sitemap/home.xml
Allow: /sitemap/pages.xml
Allow: /sitemap/structured/*.xml
Allow: /sitemap/unstructured/*.xml
Allow: /favicon.ico
Sitemap: https://REPLACE_BY_ANOTHER_URL.com/sitemap.xml
This configuration allows all bots access to:
- The Homepage
- The Reader page
- The Viewer page
When crawling is disallowed, the default content of the robots.txt file is:
User-agent: *
Disallow: /
This configuration forbids all bots from crawling any part of the portal.
The default content of the robots.txt file cannot be modified. However, it is possible to append custom directives to the robots.txt file. See Custom robots file.
The robots.txt file is only a suggestion to crawlers. Compliant bots honor it, but malicious or non-compliant crawlers can ignore it entirely. Configuring content access rights is the only reliable way to prevent content from being crawled and indexed.
The robots.txt file also provides the path to a sitemap which allows bots to retrieve URLs to the Reader page and the Viewer page.