rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy shell Test your extraction code in an interactive environment. Items Define the data you want to scrape. Item reports. bench New in version 0.17. Syntax: scrapy bench Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts- with() or contains() are not

useful for bug reports. bench • Syntax: scrapy bench • Requires project: no Run a quick benchmark test. Benchmarking. 34 Chapter 3. Basic concepts Scrapy Documentation, Release 2.10.1 3.1.6 Custom project manipulation 60 Chapter 3. Basic concepts Scrapy Documentation, Release 2.10.1 Regular expressions The test() function, for example, can prove quite useful when XPath’s starts-with() or contains() are not sufficient getall() ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() ['link1.html', 'link2.html', 'link4.html', 'link5.html'] Warning:

useful for bug reports. bench • Syntax: scrapy bench • Requires project: no Run a quick benchmark test. Benchmarking. 34 Chapter 3. Basic concepts Scrapy Documentation, Release 2.11.1 3.1.6 Custom project org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts-with() or contains() are not sufficient getall() ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() ['link1.html', 'link2.html', 'link4.html', 'link5.html'] Warning:

useful for bug reports. bench • Syntax: scrapy bench • Requires project: no Run a quick benchmark test. Benchmarking. 34 Chapter 3. Basic concepts Scrapy Documentation, Release 2.11.1 3.1.6 Custom project org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts-with() or contains() are not sufficient getall() ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() ['link1.html', 'link2.html', 'link4.html', 'link5.html'] Warning:

useful for bug reports. bench • Syntax: scrapy bench • Requires project: no Run a quick benchmark test. Benchmarking. 34 Chapter 3. Basic concepts Scrapy Documentation, Release 2.11.1 3.1.6 Custom project org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts-with() or contains() are not sufficient getall() ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() ['link1.html', 'link2.html', 'link4.html', 'link5.html'] Warning:

useful for bug reports. bench • Syntax: scrapy bench • Requires project: no Run a quick benchmark test. Benchmarking. 34 Chapter 3. Basic concepts Scrapy Documentation, Release 2.9.0 3.1.6 Custom project org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts-with() or contains() are not sufficient getall() ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() ['link1.html', 'link2.html', 'link4.html', 'link5.html'] 60

bench New in version 0.17. • Syntax: scrapy bench • Requires project: no Run a quick benchmark test. Benchmarking. 3.1.5 Custom project commands You can also add your custom project commands by using org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts-with() or contains() are not sufficient [u'link1.html', u'link2.html', u'link3.html', u'link4.html', u'link5.html'] >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').extract() [u'link1.html', u'link2.html', u'link4.html', u'link5.html']

useful for bug reports. bench • Syntax: scrapy bench • Requires project: no Run a quick benchmark test. Benchmarking. 3.1. Command line tool 31 Scrapy Documentation, Release 2.4.1 3.1.6 Custom project org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts-with() or contains() are not sufficient getall() ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() ['link1.html', 'link2.html', 'link4.html', 'link5.html'] 3

bench New in version 0.17. • Syntax: scrapy bench • Requires project: no Run a quick benchmark test. Benchmarking. 3.1.6 Custom project commands You can also add your custom project commands by using org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts-with() or contains() are not sufficient getall() ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() ['link1.html', 'link2.html', 'link4.html', 'link5.html'] >>>

rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy shell Test your extraction code in an interactive environment. Items Define the data you want to scrape. Item useful for bug reports. bench Syntax: scrapy bench Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the org/regular-expressions regular expressions set http://exslt.org/sets set manipulation Regular expressions The test() function, for example, can prove quite useful when XPath’s starts- with() or contains() are not