# Disallow crawling by all web spiders User-agent: * Disallow: /