performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. 3.11. Settings 93 Scrapy Documentation, Release 1.2.3 CONCURRENT_REQUESTS_PER_IP Default: CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle AUTOTHROTTLE_ENABLED • AUTOTHROTTLE_MAX_DELAY • AUTOTHROTTLE_START_DELAY • AUTOTHROTTLE_TARGET_CONCURRENCY • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY • BOT_NAME • CLOSESPIDER_ERRORCOUNT • CLOSESPIDER_ITEMCOUNT

performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. CONCURRENT_REQUESTS_PER_IP Default: 0 The maximum number of concurrent (ie. simultaneous) CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle AUTOTHROTTLE_ENABLED • AUTOTHROTTLE_MAX_DELAY • AUTOTHROTTLE_START_DELAY • AUTOTHROTTLE_TARGET_CONCURRENCY • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY • BOT_NAME • CLOSESPIDER_ERRORCOUNT • CLOSESPIDER_ITEMCOUNT

performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. CONCURRENT_REQUESTS_PER_IP Default: 0 The maximum number of concurrent (ie. simultaneous) CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle AUTOTHROTTLE_ENABLED • AUTOTHROTTLE_MAX_DELAY • AUTOTHROTTLE_START_DELAY • AUTOTHROTTLE_TARGET_CONCURRENCY • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY • BOT_NAME • CLOSESPIDER_ERRORCOUNT • CLOSESPIDER_ITEMCOUNT

performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. CONCURRENT_REQUESTS_PER_IP Default: 0 The maximum number of concurrent (ie. simultaneous) CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle AUTOTHROTTLE_ENABLED • AUTOTHROTTLE_MAX_DELAY • AUTOTHROTTLE_START_DELAY • AUTOTHROTTLE_TARGET_CONCURRENCY • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY • BOT_NAME • CLOSESPIDER_ERRORCOUNT • CLOSESPIDER_ITEMCOUNT

performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. CONCURRENT_REQUESTS_PER_IP Default: 0 The maximum number of concurrent (ie. simultaneous) CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle AUTOTHROTTLE_ENABLED • AUTOTHROTTLE_MAX_DELAY • AUTOTHROTTLE_START_DELAY • AUTOTHROTTLE_TARGET_CONCURRENCY • AWS_ACCESS_KEY_ID • AWS_ENDPOINT_URL • AWS_REGION_NAME • AWS_SECRET_ACCESS_KEY • AWS_USE_SSL

performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. CONCURRENT_REQUESTS_PER_IP Default: 0 The maximum number of concurrent (ie. simultaneous) CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle AUTOTHROTTLE_ENABLED • AUTOTHROTTLE_MAX_DELAY • AUTOTHROTTLE_START_DELAY • AUTOTHROTTLE_TARGET_CONCURRENCY • AWS_ACCESS_KEY_ID • AWS_ENDPOINT_URL • AWS_REGION_NAME 120 Chapter 3. Basic concepts Scrapy

performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. 3.11. Settings 113 Scrapy Documentation, Release 1.8.4 CONCURRENT_REQUESTS_PER_IP Default: CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle AUTOTHROTTLE_ENABLED • AUTOTHROTTLE_MAX_DELAY • AUTOTHROTTLE_START_DELAY • AUTOTHROTTLE_TARGET_CONCURRENCY • AWS_ACCESS_KEY_ID • AWS_ENDPOINT_URL • AWS_REGION_NAME • AWS_SECRET_ACCESS_KEY • AWS_USE_SSL

crawl. 5.5.1 Increase concurrency Concurrency is the number of requests that are processed in parallel. There is a global limit and a per-domain limit. The default global concurrency limit in Scrapy is identifying at what concurrency your Scrapy process gets CPU bounded. For optimum performance, You should pick a concurrency where CPU usage is at 80-90%. To increase the global concurrency use: CONCURRENT_REQUESTS extension builds on that premise. 5.12.3 Throttling algorithm This adjusts download delays and concurrency based on the following rules: 1. spiders always start with one concurrent request and a download

performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. CONCURRENT_REQUESTS_PER_IP Default: 0 The maximum number of concurrent (ie. simultaneous) CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle AUTOTHROTTLE_ENABLED • AUTOTHROTTLE_MAX_DELAY • AUTOTHROTTLE_START_DELAY • AUTOTHROTTLE_TARGET_CONCURRENCY • AWS_ACCESS_KEY_ID • AWS_SECRET_ACCESS_KEY • BOT_NAME • CLOSESPIDER_ERRORCOUNT • CLOSESPIDER_ITEMCOUNT

performed to any single domain. See also: AutoThrottle extension and its AUTOTHROTTLE_TARGET_CONCURRENCY option. CONCURRENT_REQUESTS_PER_IP Default: 0 The maximum number of concurrent (i.e. simultaneous) CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY and AutoThrottle lower the effective per-domain concurrency below CONCURRENT_REQUESTS_PER_DOMAIN. If the response time of a domain is lower than DOWNLOAD_DELAY, the effective concurrency for that domain is 1. When testing