Developer Tools for scraping . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.7 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 i 5.8 (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 i 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one

dynamically-loaded content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. 18 Chapter 2. First steps Scrapy

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 i 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 i 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one

content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 i 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one

dynamically-loaded content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one

dynamically-loaded content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one

dynamically-loaded content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 i 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.9 Downloading (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a few settings. You can do things like setting a download as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. In small projects (like the one