This is a simple instruction that trying to crawl news content from an authenticated ASP.NET website using Scrapy.
Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.
Make sure Ruby and gem are installed in the system. Bundler is required by Heroku to handle dependency hell. Check the Heroku's sample app tutorial. To install the bundle program, run the following command: