Selenium is a browser automation tool that is used primarily for testing web applications. You can simulate real user actions and interactions with your web applications. Selenium supports all the major browser platforms and operating systems. There are bindings for all the popular programming languages. The power of Selenium is not just restricted to testing your web apps, one other use can be of crawling or scraping websites, in particular, the ones which don't provide an API and load content lazily using Javascript.
Today, we will be crawling an online merchant website www.jabong.com with Selenium using its python bindings. Jabong loads more products as you scroll down a web page. We will use Selenium to simulate this user action of scrolling down a web page and then retrieving all the product titles and the corresponding links to the product detail pages.
Reference:
Platform: Ubuntu 12.04
Tools Required
Page to Crawl:
A quick "Inspect Element" on a shoe above shows that each of the product is wrapped by a "div" element with class "hover-box" and the title and links are embedded in an "a" element within those "div" elements.
Code:
Output Today, we will be crawling an online merchant website www.jabong.com with Selenium using its python bindings. Jabong loads more products as you scroll down a web page. We will use Selenium to simulate this user action of scrolling down a web page and then retrieving all the product titles and the corresponding links to the product detail pages.
Reference:
Platform: Ubuntu 12.04
Tools Required
sudo pip install selenium sudo pip install xvfb sudo pip install pyvirtualdisplayWe use pyvirtualdisplay which is a wrapper around xvfb and enables you to run Firefix headlessly.
Page to Crawl:
A quick "Inspect Element" on a shoe above shows that each of the product is wrapped by a "div" element with class "hover-box" and the title and links are embedded in an "a" element within those "div" elements.
Code:
javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Liteforce-Ii-Mid-White-Sneakers-503958.html','6') Nike Ballista Iv Msl Grey Running Shoes http://www.jabong.com/Nike-Ballista-Iv-Msl-Grey-Running-Shoes-503907.html U.S. Polo Assn. Navy Blue Sneakers http://www.jabong.com/Us_Polo_Assn-Navy-Blue-Sneakers-518870.html Nike Dewired Navy Blue Sneakers http://www.jabong.com/Nike-Dewired-Navy-Blue-Sneakers-503897.html javascript:sendSizeFormWithSize('http://www.jabong.com/Us_Polo_Assn-Black-Dress-Shoes-518868.html','6') javascript:sendSizeFormWithSize('http://www.jabong.com/Us_Polo_Assn-Destiny-Navy-Blue-Sneakers-518878.html','6') United Colors of Benetton Black Boat Shoes http://www.jabong.com/United-Colors-of-Benetton-Black-Boat-Shoes-512546.html Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506097.html javascript:sendSizeFormWithSize('http://www.jabong.com/United-Colors-of-Benetton-Blue-Sneakers-512548.html','6') U.S. Polo Assn. Delta Beige Sneakers http://www.jabong.com/Us_Polo_Assn-Delta-Beige-Sneakers-518873.html Asics Kayano 20 Black Running Shoes http://www.jabong.com/Asics-Kayano-20-Black-Running-Shoes-507673.html Nike Air Max 2014 Blue Running Shoes http://www.jabong.com/Nike-Air-Max-2014-Blue-Running-Shoes-503964.html Asics Excel 33 3 Navy Blue Running Shoes http://www.jabong.com/Asics-Excel-33-3-Navy-Blue-Running-Shoes-507674.html Nike Air Relentless 3 Msl Blue Running Shoes http://www.jabong.com/Nike-Air-Relentless-3-Msl-Blue-Running-Shoes-503954.html Asics Kayano 20 Navy Blue Running Shoes http://www.jabong.com/Asics-Kayano-20-Navy-Blue-Running-Shoes-507672.html Nike Lunarinternationalist Grey Running Shoes http://www.jabong.com/Nike-Lunarinternationalist-Grey-Running-Shoes-503975.html Nike Free 5.0+ Blue Running Shoes http://www.jabong.com/Nike-Free-50-Blue-Running-Shoes-503919.html Nike Fs Lite Run Black Running Shoes http://www.jabong.com/Nike-Fs-Lite-Run-Black-Running-Shoes-503955.html Nike Lunarinternationalist Blue Running Shoes http://www.jabong.com/Nike-Lunarinternationalist-Blue-Running-Shoes-503976.html Nike Free Trainer 5.0 Grey Running Shoes http://www.jabong.com/Nike-Free-Trainer-50-Grey-Running-Shoes-503886.html javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Zoom-Terra-Kiger-Blue-Running-Shoes-503930.html','6') Andrew Hill Brown Dress Shoes http://www.jabong.com/andrew-hill-Brown-Dress-Shoes-506110.html Nike Lunar Forever 3 Msl Grey Running Shoes http://www.jabong.com/Nike-Lunar-Forever-3-Msl-Grey-Running-Shoes-503970.html Asics Kayano 20 Red Running Shoes http://www.jabong.com/Asics-Kayano-20-Red-Running-Shoes-507671.html Nike Free Trainer 5.0 Black Running Shoes http://www.jabong.com/Nike-Free-Trainer-50-Black-Running-Shoes-503915.html Nike Chroma Thong Iii Green Slippers http://www.jabong.com/Nike-Chroma-Thong-Iii-Green-Slippers-503977.html Nike Aquahype Blue Flip Flops http://www.jabong.com/Nike-Aquahype-Blue-Flip-Flops-503890.html Z Collection Green Loafers http://www.jabong.com/z-collection-Green-Loafers-517513.html Nike Flex 2013 Rn Black Running Shoes http://www.jabong.com/Nike-Flex-2013-Rn-Black-Running-Shoes-503916.html U.S. Polo Assn. Brown Sneakers http://www.jabong.com/Us_Polo_Assn-Brown-Sneakers-518869.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506093.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506102.html Nike Eliminate Ii Leather Grey Sneakers http://www.jabong.com/Nike-Eliminate-Ii-Leather-Grey-Sneakers-503909.html Nike Fs Lite Trainer Blue Running Shoes http://www.jabong.com/Nike-Fs-Lite-Trainer-Blue-Running-Shoes-503952.html Nike Suketo 2 Leather Red Sneakers http://www.jabong.com/Nike-Suketo-2-Leather-Red-Sneakers-503972.html Nike Free Flyknit+ Red Running Shoes http://www.jabong.com/Nike-Free-Flyknit-Red-Running-Shoes-469765.html javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Zoom-Wildhorse-Blue-Running-Shoes-503931.html','6') Nike Air Pegasus+ 30 Grey Running Shoes http://www.jabong.com/Nike-Air-Pegasus-30-Grey-Running-Shoes-503934.html Nike Flex Supreme Tr 2 Grey Running Shoes http://www.jabong.com/Nike-Flex-Supreme-Tr-2-Grey-Running-Shoes-503942.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506088.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506087.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506091.html Nike Zoom Structure+ 17 Black Running Shoes http://www.jabong.com/Nike-Zoom-Structure-17-Black-Running-Shoes-503949.html Nike Flyknit Lunar2 Black Running Shoes http://www.jabong.com/Nike-Flyknit-Lunar2-Black-Running-Shoes-503959.html Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506103.html Andrew Hill Black Dress Shoes http://www.jabong.com/andrew-hill-Black-Dress-Shoes-506109.html Phosphorus Tan Loafers http://www.jabong.com/phosphorus-Tan-Loafers-506095.html Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506090.html U.S. Polo Assn. Delta Navy Blue Sneakers http://www.jabong.com/Us_Polo_Assn-Delta-Navy-Blue-Sneakers-518872.html Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506094.html Nike Lunar Forever 3 Msl White Running Shoes http://www.jabong.com/Nike-Lunar-Forever-3-Msl-White-Running-Shoes-503971.html Asics Kayano 20 White Running Shoes http://www.jabong.com/Asics-Kayano-20-White-Running-Shoes-507670.html
did you consider Scrapy?
ReplyDeleteI'm keen to know why did you choose selenium ?
Most likely because scrapy does not render javascript.
ReplyDeleteGood article.It is difficult to get data from website that call data using ajax. There are web scraping softwares like content grabber or screen scraper using which you can download data from jabong.com very easily.
ReplyDeleteWow. This really made my day. Thanks a lot!
DeleteJavaScript Online Training | HTML5 Online Training
JavaScript Training Courses | Javascript Online Training | Angular 2 Online Training
Great Article
DeleteFinal Year Projects for CSE in Python
FInal Year Project Centers in Chennai
Python Training in Chennai
Python Training in Chennai
This comment has been removed by the author.
ReplyDeleteIt is very difficult to get data from websites that has infinite scrolling like jabong.But with some tricks one can also get data from those websites.I am web scraper and doing web scraping since last 5 years.
ReplyDeleteJavaScript Online Training JavaScript Online Training JQuery Online Training JQuery Online Training
ReplyDeleteJavaScript Course | HTML5 Online Training
JavaScript Online Training JavaScript Online Training JQuery Online Training JQuery Online Training
ReplyDeleteJavaScript Course | HTML5 Online Training
ReplyDeleteI love looking through a post that will make people think. Also, many thanks for allowing for me to comment! apple itunes login
Nice Article
ReplyDeleteDOT NET training in bangalore
dot net training institute in bangalore
dot net course in bangalroe
best dot net training institute in bangalore
Nice post
ReplyDeletedata science training in Marathahalli
best data science courses in Marathahalli
data science institute in Marathahalli
data science certification Marathahalli
data analytics training in Marathahalli
data science training institute in Marathahalli
good article..
ReplyDeletebest training institute for hadoop in BTM
best big data hadoop training in BTM
hadoop training in btm
hadoop training institutes in btm
hadoop course in btm
Great article. Visit our muse website templates created on the basis of Adobe Muse CC 2017, which are easy to edit and customize. The best website with any templates you need.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteGreat job for publishing such a nice article. Your article isn’t only useful but it is additionally really informative. Thank you because you have been willing to share information with us. Website Design Ireland
ReplyDeleteSecured personal loans have traditionally served as a loan option for almost everybody and anybody that has ever been in need of a loan. A homeowner can apply for a personal loan for any purpose vacation, home improvement, car finance, education. Even bad credit or adverse credit fails to materialize its consequences with secured personal loans. An all purpose loans for any person has not found a better name than secured personal loans. buymodafinilonline.reviews
ReplyDeleteAre you thinking about starting a business in 2019, but don't know how or even where to begin? This article outlines the biggest obstacles to overcome, what you need to start your business, and what to do after year one! download rhinoceros 5
ReplyDeleteI saw a lot of website but I believe this one holds something special in it in it putlocker
ReplyDeleteThis blog gives more attractive information.i am really impressed with this blog.
ReplyDeletespoken english classes in bangalore
Spoken English Classes in Chennai
Spoken English Classes near Marathahalli
Spoken English Marathahalli
English Speaking Classes in Bangalore
Spoken English Classes in BTM
PHP Training in Bangalore
AWS Training in Bangalore
Data Science Courses in Bangalore
DevOps Training in Bangalore
Amazing Post. keep update more information.
ReplyDeleteSelenium Training in Chennai
Selenium Training in Bangalore
Selenium Training in Coimbatore
Selenium Course in Bangalore
Best Selenium Training in Bangalore
Selenium training in marathahalli
Selenium training in Btm
Ethical Hacking Course in Bangalore
Tally Course in Chennai
Thanks for updating this information. Good job.
ReplyDeleteGerman Classes in Chennai
best german classes in chennai
french courses in chennai
pearson vue
Blockchain Training in Chennai
Ionic Training in Chennai
spanish institute in chennai
content writing training in chennai
German Classes in Anna Nagar
German Classes in Tambaram
You're so interesting! I don't think I've read a single thing like that before. So wonderful to find another person with unique thoughts on this subject. Really.. thank you for starting this up. This web site is one thing that is required on the internet, reviews someone with a bit of originality!
ReplyDeletePretty! This has been a really wonderful post. Many thanks for supplying these shows details.
ReplyDeleteThis Blog is really informative!! keep update more about this…
ReplyDeleteAviation Courses in Bangalore
Air Hostess Training in Bangalore
Airport Management Courses in Bangalore
Ground Staff Training in Bangalore
Aviation Institute in Bangalore
Air Hostess Academy Bangalore
Airport Management in Bangalore
Excellent post! This post was delivered a lot of information. Thanks for the effort to share this with us. Keep updating.
ReplyDeletePrimavera Training in Chennai
Primavera Coaching in Chennai
Corporate Training in Chennai
Pega Training in Chennai
Spark Training in Chennai
Embedded System Course Chennai
Linux Training in Chennai
Graphic Design Courses in Chennai
Social Media Marketing Courses in Chennai
Soft Skills Training in Chennai
JMeter Training in Chennai
Great explanation to given on this post and i read our full content was really amazing,then the this more important in my part of life. The given information very impressed for me really so nice content.
ReplyDeleteAWS training in chennai | AWS training in anna nagar | AWS training in omr | AWS training in porur | AWS training in tambaram | AWS training in velachery
I was very pleased to find this site. I wanted to thank you for this great read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post.
ReplyDeleteData Science Course
I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
ReplyDeleteData Science Training Institute in Bangalore
ReplyDeleteSuch a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.
Best Data Science Courses in Bangalore
You re in point of fact a just right webmaster. The website loading speed is amazing. It kind of feels that you're doing any distinctive trick. Moreover, The contents are masterpiece. you have done a fantastic activity on this subject!
ReplyDeleteData Science Course in Bangalore
Regular visits listed here are the easiest method to appreciate your energy, which is why why I am going to the website everyday, searching for new, interesting info. Many, thank you!
ReplyDeleteData Science Training in Bangalore
Two full thumbs up for this magneficant article of yours. I've really enjoyed reading this article today and I think this might be one of the best article that I've read yet. Please, keep this work going on in the same quality.
ReplyDeleteData Science Training
Great Article
ReplyDeleteArtificial Intelligence Projects
Project Center in Chennai
JavaScript Training in Chennai
JavaScript Training in Chennai
Wonderful blog found to be very impressive to come across such an awesome blog. I should really appreciate the blogger for the efforts they have put in to develop such an amazing content for all the curious readers who are very keen of being updated across every corner. Ultimately, this is an awesome experience for the readers. Anyways, thanks a lot and keep sharing the content in future too.
ReplyDelete360DigiTMG Tableau Course
Highly appreciable regarding the uniqueness of the content. This perhaps makes the readers feels excited to get stick to the subject. Certainly, the learners would thank the blogger to come up with the innovative content which keeps the readers to be up to date to stand by the competition. Once again nice blog keep it up and keep sharing the content as always.
ReplyDelete360DigiTMG Business Analytics Course
Clipping Xpert
ReplyDeleteClipping Xpert India
Paragon Clipping Path
Clipping Path Service
Image Editing Company
Ecommerce Image Editing Service
Clipping path company
Clipping Path Service
Image Editing Company
Ecommerce Image Editing Service
Clipping path company
Clipping Path Service
Image Editing Company
Ecommerce Image Editing Service
Clipping path company
Aivivu chuyên cung cấp
ReplyDeletevé máy bay giá rẻ tết 2021
vé máy bay đi Mỹ Vietnam Airline
vé máy bay Việt Nam đi Pháp
đặt vé máy bay hàn việt
bay nhật bản
vé máy bay đi Anh quốc
Bad Data can take you downhill and with no relevant report. data science course syllabus
ReplyDeleteThanks for the valuable post.
ReplyDeletephp latest version
digital marketing tips and tricks
big data analytics examples
salesforce communities
android interview questions for senior developer
This calculator is provided for general information purposes only. mortgage calculator Biweekly payment schedules can be arranged together with your bank, with some banks providing this service for free. canada mortgage calculator
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteI was really impressed to see this blog, it was very interesting and it is very useful for all.
ReplyDeletewhich software is used for python
scope in networking after doing ccna
python vs php performance
machine learning jobs
data scientist interview questions and answers pdf
seo analyst interview questions
SAP stands for Systems Applications and Products in Data Processing. SAP, by definition, is also the name of the ERP (Enterprise Resource Planning) software as well as the name of the company.
ReplyDeletetally training in chennai
hadoop training in chennai
sap training in chennai
oracle training in chennai
angular js training in chennai