Architecture overview Understand the Scrapy architecture. Downloader Middleware Customize how pages get requested and downloaded. Spider Middleware Customize the input and output of your spiders. Extensions an iterable of Item objects. Parameters: response – the response to parse log(message[, level, component]) Log a message using the scrapy.log.msg() function, automatically populating the spider argument adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware and before start parsing it. It can be used used for modifying the response body before parsing

. . . . . . . . . . 205 6.2 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . objects. Parameters response (Response) – the response to parse log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware, before the spider starts parsing it. It can be used to modify the response body before parsing

. . . . . . . . . . 219 6.2 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . objects. Parameters response (Response) – the response to parse log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware, before the spider starts parsing it. It can be used to modify the response body before parsing

. . . . . . . . . . 211 6.2 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . objects. Parameters response (Response) – the response to parse log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware, before the spider starts parsing it. It can be used to modify the response body before parsing

. . . . . . . . . . 238 6.3 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 6.4 Spider Middleware . . . . . . . . . . . . . . . . . . . . Requires project: yes Start crawling using a spider. Supported options: • -h, --help: show a help message and exit • -a NAME=VALUE: set a spider argument (may be repeated) • --output FILE or -o FILE: append or None. Parameters response (Response) – the response to parse log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more

. . . . . . . . . . . 87 6.2 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . an iterable of Item objects. Parameters response – the response to parse log(message[, level, component]) Log a message using the scrapy.log.msg() function, automatically populating the spider argument adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware and before start parsing it. It can be used used for modifying the response body before parsing

. . . . . . . . . . 199 6.2 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . objects. Parameters response (Response) – the response to parse log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware, before the spider starts parsing it. It can be used to modify the response body before parsing

. . . . . . . . . . 203 6.2 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . objects. Parameters response (Response) – the response to parse log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware, before the spider starts parsing it. It can be used to modify the response body before parsing

. . . . . . . . . . 205 6.2 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . objects. Parameters response (Response) – the response to parse log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware, before the spider starts parsing it. It can be used to modify the response body before parsing

. . . . . . . . . . 197 6.2 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . objects. Parameters response (Response) – the response to parse log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware, before the spider 3.2. Spiders 39 Scrapy Documentation, Release 2.0.1 starts parsing it. It