example: [settings] default = myproject.settings By default, Scrapy projects use a SQLite [http://en.wikipedia.org/wiki/SQLite] database to store persistent runtime data of the project, such as the spider queue queue (the list of spiders that are scheduled to run). By default, this SQLite database is stored in the project data directory which, by default, is the .scrapy directory inside the project root directory Example: SPIDER_MODULES = ['mybot.spiders_prod', 'mybot.spiders_dev'] SQLITE_DB Default: 'scrapy.db' The location of the project SQLite database, used for storing the spider queue and other persistent data

Scrapy projects use a SQLite database to store persistent runtime data of the project, such as the spider queue (the list of spiders that are scheduled to run). By default, this SQLite database is stored Example: SPIDER_MODULES = ['mybot.spiders_prod', 'mybot.spiders_dev'] SQLITE_DB Default: ’scrapy.db’ The location of the project SQLite database, used for storing the spider queue and other persistent data 142 SCHEDULER_ORDER, 142 SPIDER_MIDDLEWARES, 142 SPIDER_MIDDLEWARES_BASE, 143 SPIDER_MODULES, 143 SQLITE_DB, 143 STATS_CLASS, 143 STATS_DUMP, 143 STATS_ENABLED, 143 STATS_SDB_ASYNC, 66 STATS_SDB_DOMAIN

link extractors documentation (commit ba14f38) • removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175) 8.1.9 0.14.2 • move buffer pointing to start of file before 8.1. Release notes 177 Scrapy Documentation, Release 0.16.5 • Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781) • Real support for returning iterators on start_requests() method code and pluggable backends called “spider queues” (#220) • New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders

link extractors documentation (commit ba14f38) • removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175) 8.1.14 0.14.2 • move buffer pointing to start of file before to scrapyd: scrapy.spiderqueue -> scrapyd.spiderqueue (r2708) • Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781) • Real support for returning iterators on start_requests() method code and pluggable backends called “spider queues” (#220) • New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders

link extractors documentation (commit ba14f38) • removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175) 8.1.18 0.14.2 • move buffer pointing to start of file before to scrapyd: scrapy.spiderqueue -> scrapyd.spiderqueue (r2708) • Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781) • Real support for returning iterators on start_requests() method code and pluggable backends called “spider queues” (#220) • New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders

link extractors documentation (commit ba14f38) • removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175) 8.1.17 0.14.2 • move buffer pointing to start of file before to scrapyd: scrapy.spiderqueue -> scrapyd.spiderqueue (r2708) • Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781) • Real support for returning iterators on start_requests() method code and pluggable backends called “spider queues” (#220) • New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders

[https://github.com/scrapy/scrapy/commit/ba14f38]) removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175 [https://github.com/scrapy/scrapy/commit/0665175]) 0.14.2 move spiderqueue (r2708 [http://hg.scrapy.org/scrapy/changeset/2708]) Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781 [http://hg.scrapy.org/scrapy/changeset/2781]) Real support for base code and pluggable backends called “spider queues” (#220) New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders

[https://github.com/scrapy/scrapy/commit/ba14f38]) removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175 [https://github.com/scrapy/scrapy/commit/0665175]) 0.14.2 move spiderqueue (r2708 [http://hg.scrapy.org/scrapy/changeset/2708]) Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781 [http://hg.scrapy.org/scrapy/changeset/2781]) Real support for code and pluggable backends called “spider queues” (#220) New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders

[https://github.com/scrapy/scrapy/commit/ba14f38]) removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175 [https://github.com/scrapy/scrapy/commit/0665175]) 0.14.2 move spiderqueue (r2708 [http://hg.scrapy.org/scrapy/changeset/2708]) Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781 [http://hg.scrapy.org/scrapy/changeset/2781]) Real support for base code and pluggable backends called “spider queues” (#220) New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders

link extractors documentation (commit ba14f38) • removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175) 8.1.27 0.14.2 • move buffer pointing to start of file before to scrapyd: scrapy.spiderqueue -> scrapyd.spiderqueue (r2708) • Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781) • Real support for returning iterators on start_requests() method code and pluggable backends called “spider queues” (#220) • New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders