Crawling a Website that loads content using Javascript with Selenium Webdriver in Python

Selenium is a browser automation tool that is used primarily for testing web applications. You can simulate real user actions and interactions with your web applications. Selenium supports all the major browser platforms  and operating systems. There are bindings for all the popular programming languages. The power of Selenium is not just restricted to testing your web apps, one other use can be of crawling or scraping websites, in particular, the ones which don't provide an API and load content lazily using Javascript.

Today, we will be crawling an online merchant website www.jabong.com with Selenium using its python bindings. Jabong loads more products as you scroll down a web page. We will use Selenium to simulate this user action of scrolling down a web page and then retrieving all the product titles and the corresponding links to the product detail pages.

Reference:

Platform: Ubuntu 12.04

Tools Required

sudo pip install selenium

sudo pip install xvfb

sudo pip install pyvirtualdisplay

We use pyvirtualdisplay which is a wrapper around xvfb and enables you to run Firefix headlessly.

Page to Crawl:




A quick "Inspect Element" on a shoe above shows that each of the product is wrapped by a "div" element with class "hover-box" and the title and links are embedded in an "a" element within those "div" elements.



Code:



Output
 javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Liteforce-Ii-Mid-White-Sneakers-503958.html','6')
Nike Ballista Iv Msl Grey Running Shoes http://www.jabong.com/Nike-Ballista-Iv-Msl-Grey-Running-Shoes-503907.html
U.S. Polo Assn. Navy Blue Sneakers http://www.jabong.com/Us_Polo_Assn-Navy-Blue-Sneakers-518870.html
Nike Dewired Navy Blue Sneakers http://www.jabong.com/Nike-Dewired-Navy-Blue-Sneakers-503897.html
 javascript:sendSizeFormWithSize('http://www.jabong.com/Us_Polo_Assn-Black-Dress-Shoes-518868.html','6')
 javascript:sendSizeFormWithSize('http://www.jabong.com/Us_Polo_Assn-Destiny-Navy-Blue-Sneakers-518878.html','6')
United Colors of Benetton Black Boat Shoes http://www.jabong.com/United-Colors-of-Benetton-Black-Boat-Shoes-512546.html
Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506097.html
 javascript:sendSizeFormWithSize('http://www.jabong.com/United-Colors-of-Benetton-Blue-Sneakers-512548.html','6')
U.S. Polo Assn. Delta Beige Sneakers http://www.jabong.com/Us_Polo_Assn-Delta-Beige-Sneakers-518873.html
Asics Kayano 20 Black Running Shoes http://www.jabong.com/Asics-Kayano-20-Black-Running-Shoes-507673.html
Nike Air Max 2014 Blue Running Shoes http://www.jabong.com/Nike-Air-Max-2014-Blue-Running-Shoes-503964.html
Asics Excel 33 3 Navy Blue Running Shoes http://www.jabong.com/Asics-Excel-33-3-Navy-Blue-Running-Shoes-507674.html
Nike Air Relentless 3 Msl Blue Running Shoes http://www.jabong.com/Nike-Air-Relentless-3-Msl-Blue-Running-Shoes-503954.html
Asics Kayano 20 Navy Blue Running Shoes http://www.jabong.com/Asics-Kayano-20-Navy-Blue-Running-Shoes-507672.html
Nike Lunarinternationalist Grey Running Shoes http://www.jabong.com/Nike-Lunarinternationalist-Grey-Running-Shoes-503975.html
Nike Free 5.0+ Blue Running Shoes http://www.jabong.com/Nike-Free-50-Blue-Running-Shoes-503919.html
Nike Fs Lite Run Black Running Shoes http://www.jabong.com/Nike-Fs-Lite-Run-Black-Running-Shoes-503955.html
Nike Lunarinternationalist Blue Running Shoes http://www.jabong.com/Nike-Lunarinternationalist-Blue-Running-Shoes-503976.html
Nike Free Trainer 5.0 Grey Running Shoes http://www.jabong.com/Nike-Free-Trainer-50-Grey-Running-Shoes-503886.html
 javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Zoom-Terra-Kiger-Blue-Running-Shoes-503930.html','6')
Andrew Hill Brown Dress Shoes http://www.jabong.com/andrew-hill-Brown-Dress-Shoes-506110.html
Nike Lunar Forever 3 Msl Grey Running Shoes http://www.jabong.com/Nike-Lunar-Forever-3-Msl-Grey-Running-Shoes-503970.html
Asics Kayano 20 Red Running Shoes http://www.jabong.com/Asics-Kayano-20-Red-Running-Shoes-507671.html
Nike Free Trainer 5.0 Black Running Shoes http://www.jabong.com/Nike-Free-Trainer-50-Black-Running-Shoes-503915.html
Nike Chroma Thong Iii Green Slippers http://www.jabong.com/Nike-Chroma-Thong-Iii-Green-Slippers-503977.html
Nike Aquahype Blue Flip Flops http://www.jabong.com/Nike-Aquahype-Blue-Flip-Flops-503890.html
Z Collection Green Loafers http://www.jabong.com/z-collection-Green-Loafers-517513.html
Nike Flex 2013 Rn Black Running Shoes http://www.jabong.com/Nike-Flex-2013-Rn-Black-Running-Shoes-503916.html
U.S. Polo Assn. Brown Sneakers http://www.jabong.com/Us_Polo_Assn-Brown-Sneakers-518869.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506093.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506102.html
Nike Eliminate Ii Leather Grey Sneakers http://www.jabong.com/Nike-Eliminate-Ii-Leather-Grey-Sneakers-503909.html
Nike Fs Lite Trainer Blue Running Shoes http://www.jabong.com/Nike-Fs-Lite-Trainer-Blue-Running-Shoes-503952.html
Nike Suketo 2 Leather Red Sneakers http://www.jabong.com/Nike-Suketo-2-Leather-Red-Sneakers-503972.html
Nike Free Flyknit+ Red Running Shoes http://www.jabong.com/Nike-Free-Flyknit-Red-Running-Shoes-469765.html
 javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Zoom-Wildhorse-Blue-Running-Shoes-503931.html','6')
Nike Air Pegasus+ 30 Grey Running Shoes http://www.jabong.com/Nike-Air-Pegasus-30-Grey-Running-Shoes-503934.html
Nike Flex Supreme Tr 2 Grey Running Shoes http://www.jabong.com/Nike-Flex-Supreme-Tr-2-Grey-Running-Shoes-503942.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506088.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506087.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506091.html
Nike Zoom Structure+ 17 Black Running Shoes http://www.jabong.com/Nike-Zoom-Structure-17-Black-Running-Shoes-503949.html
Nike Flyknit Lunar2 Black Running Shoes http://www.jabong.com/Nike-Flyknit-Lunar2-Black-Running-Shoes-503959.html
Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506103.html
Andrew Hill Black Dress Shoes http://www.jabong.com/andrew-hill-Black-Dress-Shoes-506109.html
Phosphorus Tan Loafers http://www.jabong.com/phosphorus-Tan-Loafers-506095.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506090.html
U.S. Polo Assn. Delta Navy Blue Sneakers http://www.jabong.com/Us_Polo_Assn-Delta-Navy-Blue-Sneakers-518872.html
Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506094.html
Nike Lunar Forever 3 Msl White Running Shoes http://www.jabong.com/Nike-Lunar-Forever-3-Msl-White-Running-Shoes-503971.html
Asics Kayano 20 White Running Shoes http://www.jabong.com/Asics-Kayano-20-White-Running-Shoes-507670.html

30 comments:

  1. did you consider Scrapy?
    I'm keen to know why did you choose selenium ?

    ReplyDelete
  2. Most likely because scrapy does not render javascript.

    ReplyDelete
  3. Good article.It is difficult to get data from website that call data using ajax. There are web scraping softwares like content grabber or screen scraper using which you can download data from jabong.com very easily.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. It is very difficult to get data from websites that has infinite scrolling like jabong.But with some tricks one can also get data from those websites.I am web scraper and doing web scraping since last 5 years.

    ReplyDelete

  6. I love looking through a post that will make people think. Also, many thanks for allowing for me to comment! apple itunes login

    ReplyDelete
  7. Great article. Visit our muse website templates created on the basis of Adobe Muse CC 2017, which are easy to edit and customize. The best website with any templates you need.

    ReplyDelete
  8. This comment has been removed by a blog administrator.

    ReplyDelete
  9. This comment has been removed by a blog administrator.

    ReplyDelete
  10. This comment has been removed by a blog administrator.

    ReplyDelete
  11. This comment has been removed by a blog administrator.

    ReplyDelete
  12. This comment has been removed by a blog administrator.

    ReplyDelete
  13. This comment has been removed by a blog administrator.

    ReplyDelete
  14. Great job for publishing such a nice article. Your article isn’t only useful but it is additionally really informative. Thank you because you have been willing to share information with us. Website Design Ireland

    ReplyDelete
  15. Secured personal loans have traditionally served as a loan option for almost everybody and anybody that has ever been in need of a loan. A homeowner can apply for a personal loan for any purpose vacation, home improvement, car finance, education. Even bad credit or adverse credit fails to materialize its consequences with secured personal loans. An all purpose loans for any person has not found a better name than secured personal loans. buymodafinilonline.reviews

    ReplyDelete
  16. Are you thinking about starting a business in 2019, but don't know how or even where to begin? This article outlines the biggest obstacles to overcome, what you need to start your business, and what to do after year one! download rhinoceros 5

    ReplyDelete
  17. I saw a lot of website but I believe this one holds something special in it in it putlocker

    ReplyDelete
  18. You're so interesting! I don't think I've read a single thing like that before. So wonderful to find another person with unique thoughts on this subject. Really.. thank you for starting this up. This web site is one thing that is required on the internet, reviews someone with a bit of originality!

    ReplyDelete
  19. Pretty! This has been a really wonderful post. Many thanks for supplying these shows details.

    ReplyDelete