Manual:Robot policy

Robot policy is used to tell web crawlers and other tools what to exclude.

Configuration settings

edit