Crawling a Website that loads content using Javascript with Selenium Webdriver in Python

Selenium is a browser automation tool that is used primarily for testing web applications. You can simulate real user actions and interactions with your web applications. Selenium supports all the major browser platforms  and operating systems. There are bindings for all the popular programming languages. The power of Selenium is not just restricted to testing your web apps, one other use can be of crawling or scraping websites, in particular, the ones which don't provide an API and load content lazily using Javascript.

Today, we will be crawling an online merchant website with Selenium using its python bindings. Jabong loads more products as you scroll down a web page. We will use Selenium to simulate this user action of scrolling down a web page and then retrieving all the product titles and the corresponding links to the product detail pages.


Platform: Ubuntu 12.04

Tools Required

sudo pip install selenium

sudo pip install xvfb

sudo pip install pyvirtualdisplay

We use pyvirtualdisplay which is a wrapper around xvfb and enables you to run Firefix headlessly.

Page to Crawl:

A quick "Inspect Element" on a shoe above shows that each of the product is wrapped by a "div" element with class "hover-box" and the title and links are embedded in an "a" element within those "div" elements.


Nike Ballista Iv Msl Grey Running Shoes
U.S. Polo Assn. Navy Blue Sneakers
Nike Dewired Navy Blue Sneakers
United Colors of Benetton Black Boat Shoes
Phosphorus Brown Loafers
U.S. Polo Assn. Delta Beige Sneakers
Asics Kayano 20 Black Running Shoes
Nike Air Max 2014 Blue Running Shoes
Asics Excel 33 3 Navy Blue Running Shoes
Nike Air Relentless 3 Msl Blue Running Shoes
Asics Kayano 20 Navy Blue Running Shoes
Nike Lunarinternationalist Grey Running Shoes
Nike Free 5.0+ Blue Running Shoes
Nike Fs Lite Run Black Running Shoes
Nike Lunarinternationalist Blue Running Shoes
Nike Free Trainer 5.0 Grey Running Shoes
Andrew Hill Brown Dress Shoes
Nike Lunar Forever 3 Msl Grey Running Shoes
Asics Kayano 20 Red Running Shoes
Nike Free Trainer 5.0 Black Running Shoes
Nike Chroma Thong Iii Green Slippers
Nike Aquahype Blue Flip Flops
Z Collection Green Loafers
Nike Flex 2013 Rn Black Running Shoes
U.S. Polo Assn. Brown Sneakers
Phosphorus Black Loafers
Phosphorus Black Loafers
Nike Eliminate Ii Leather Grey Sneakers
Nike Fs Lite Trainer Blue Running Shoes
Nike Suketo 2 Leather Red Sneakers
Nike Free Flyknit+ Red Running Shoes
Nike Air Pegasus+ 30 Grey Running Shoes
Nike Flex Supreme Tr 2 Grey Running Shoes
Phosphorus Black Loafers
Phosphorus Black Loafers
Phosphorus Black Loafers
Nike Zoom Structure+ 17 Black Running Shoes
Nike Flyknit Lunar2 Black Running Shoes
Phosphorus Brown Loafers
Andrew Hill Black Dress Shoes
Phosphorus Tan Loafers
Phosphorus Black Loafers
U.S. Polo Assn. Delta Navy Blue Sneakers
Phosphorus Brown Loafers
Nike Lunar Forever 3 Msl White Running Shoes
Asics Kayano 20 White Running Shoes


  1. did you consider Scrapy?
    I'm keen to know why did you choose selenium ?

  2. Most likely because scrapy does not render javascript.

  3. Good article.It is difficult to get data from website that call data using ajax. There are web scraping softwares like content grabber or screen scraper using which you can download data from very easily.

  4. This comment has been removed by the author.

  5. It is very difficult to get data from websites that has infinite scrolling like jabong.But with some tricks one can also get data from those websites.I am web scraper and doing web scraping since last 5 years.


  6. I love looking through a post that will make people think. Also, many thanks for allowing for me to comment! apple itunes login

  7. Great article. Visit our muse website templates created on the basis of Adobe Muse CC 2017, which are easy to edit and customize. The best website with any templates you need.

  8. This comment has been removed by the author.

  9. If you are eager to learn more about the most sophisticated players in the outsourced product development industry, I'd suggest checking out this list of top product development companies.