Scrapy 1.4 Documentation
spider that crawls mywebsite.com would often be called mywebsite. Note In Python 2 this must be ASCII only. allowed_domains An optional list of strings containing domains that this spider is allowed way to convert response body to unicode: you would be using the system default encoding (typically ascii) instead of the response encoding. encoding A string with the encoding of this response. The encoding html">other link.