In the Custom robots file section of the Web search engines menu, it is possible to add new directives to the robots.txt file to better control crawler behavior. Fluid Topics appends custom directives to the default content of the robots.txt file.
The following table presents directives to use in the robots.txt file:
| Directive | Description |
|---|---|
User-agent |
Specifies which crawler the following rules apply to. Starts a new group of rules that ends with another User-agent directive. Takes a bot name as value, or * to target all bots. When a bot matches a specific User-agent, it ignores rules from the * group. |
Disallow |
Forbids bots from crawling the specified URL path. |
Allow |
Allows bots to crawl the specified URL path, overriding a Disallow rule. |
Crawl-delay |
Requests bots to wait the specified number of seconds between requests (not supported by all bots). |
For example:
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /blog/
User-agent: Bingbot
Crawl-delay: 5
In this example, all bots are forbidden from crawling the /admin/ path, and the /search page, but are allowed to crawl the /blog/ path. Additionally, Bingbot is instructed to wait 5 seconds between requests to reduce the load on the server.
Fluid Topics cannot be held responsible for issues caused by custom directives in the robots.txt file. Proceed with caution.