User-agent: Scrapy Allow: / User-Agent: * Disallow: /download/* Sitemap: https://www.showeet.com/sitemap.xml