Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. By writing a structured text file you can indicate to robots that certain parts of your server are not to be cached / accessed / crawled by some or all robots.