# Blocks robots from specific folders / directories User-agent: googlebot User-agent: google User-agent: bingbot User-agent: bing User-agent: Mediapartners-Google Disallow: /cgi-bin/ Disallow: /cron/ Disallow: /log/ Crawl-delay: 15 User-agent: * Disallow: /