In the Web search engines administration interface, a toggle is available to allow or disallow bots from crawling the content of a portal.
Bots can only access and index public content. By default, each document is public until an ADMIN or KHUB_ADMIN user configures content access rights to keep it from being exposed to unauthorized users (including bots).