Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images Download static images associated must load all DOM in memory which could be a problem for big feeds 'xml' - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images Download static images associated must load all DOM in memory which could be a problem for big feeds 'xml' - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images Download static images associated must load all DOM in memory which could be a problem for big feeds 'xml' - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

Unregister. connected.remove(websocket) This simplistic example keeps track of connected clients in memory. This only works as long as you run a single process. In a practical application, the handler may built-in publish / subscribe for these use cases. Depending on the scale of your service, a simple in-memory implementation may do the job or you may need an external publish / subscribe component. What does call_soon_threadsafe(). 2.2.3 Memory usage In most cases, memory usage of a WebSocket server is proportional to the number of open connections. When a server handles thousands of connections, memory usage can become

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.5 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 4.2. Stats Collection

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.5 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.5 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.9 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem max_value('max_items_scraped', value) Set stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get stat value: >>> stats.get_value('pages_crawled') 8 Get all stats: >>> stats

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.9 Downloading and must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses Selector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem max_value(’max_items_scraped’, value) Set stat value only if lower than previous: stats.min_value(’min_free_memory_percent’, value) Get stat value: >>> stats.get_value(’pages_crawled’) 8 Get all stats: >>> stats

dynamically-loaded content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 5.9 Downloading as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line. 18 Chapter 2. First steps Scrapy and must load all DOM in memory which could be a problem for big feeds • 'xml' - an iterator which uses Selector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem