argument. In the parse callback we extract the links to the question pages using a CSS Selector with a custom extension that allows to get the value for an attribute. Then we yield a few more requests to be see sites being printed in your output. Run: scrapy crawl dmoz Using our item Item objects are custom Python dicts; you can access the values of their fields (attributes of the class we defined earlier) bench • Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy com-

bench • Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy com- COMMANDS_MODULE Default: '' (empty string) A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example: 30 Chapter 3. Basic concepts from their pages (i.e. scraping items). In other words, Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site (or, in some cases, a group of sites)

bench • Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy com- COMMANDS_MODULE Default: '' (empty string) A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example: COMMANDS_MODULE = 'mybot.commands' from their pages (i.e. scraping items). In other words, Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site (or, in some cases, a group of sites)

bench • Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy com- COMMANDS_MODULE Default: '' (empty string) A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example: COMMANDS_MODULE = 'mybot.commands' from their pages (i.e. scraping items). In other words, Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site (or, in some cases, a group of sites)

Middleware Customize the input and output of your spiders. Extensions Extend Scrapy with your custom functionality Core API Use it on extensions and middlewares to extend Scrapy functionality Signals argument. In the parse callback we extract the links to the question pages using a CSS Selector with a custom extension that allows to get the value for an attribute. Then we yield a few more requests to be see sites being printed in your output. Run: scrapy crawl dmoz Using our item Item objects are custom Python dicts; you can access the values of their fields (attributes of the class we defined earlier)

• Requires project: no Run a quick benchmark test. Benchmarking. 3.1.6 Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy com- COMMANDS_MODULE Default: '' (empty string) A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example: COMMANDS_MODULE = 'mybot.commands' from their pages (i.e. scraping items). In other words, Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site (or, in some cases, a group of sites)

• Requires project: no Run a quick benchmark test. Benchmarking. 3.1.5 Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy com- COMMANDS_MODULE Default: '' (empty string) A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example: COMMANDS_MODULE = 'mybot.commands' from their pages (i.e. scraping items). In other words, Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site (or, in some cases, a group of sites)

bench • Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy com- COMMANDS_MODULE Default: '' (empty string) A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example: COMMANDS_MODULE = 'mybot.commands' from their pages (i.e. scraping items). In other words, Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site (or, in some cases, a group of sites)

Middleware Customize the input and output of your spiders. Extensions Extend Scrapy with your custom functionality Core API Use it on extensions and middlewares to extend Scrapy functionality Signals scrapy bench Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy commands COMMANDS_MODULE Default: '' (empty string) A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example: COMMANDS_MODULE = 'mybot.commands'

Middleware Customize the input and output of your spiders. Extensions Extend Scrapy with your custom functionality Core API Use it on extensions and middlewares to extend Scrapy functionality Signals scrapy bench Requires project: no Run a quick benchmark test. Benchmarking. Custom project commands You can also add your custom project commands by using the COMMANDS_MODULE setting. See the Scrapy commands COMMANDS_MODULE Default: '' (empty string) A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example: COMMANDS_MODULE = 'mybot.commands'