Breaking News: Grepper is joining You.com. Read the official announcement!

scrapy with selenium

Srey Sitharo answered on September 12, 2022 Popularity 9/10 Helpfulness 2/10

answer scrapy with selenium

related scrapy with selenium

scrapy with selenium

Comment

1. First, install the Scrapy framework with the following command: pip install scrapy
2. Next, add a folder and use the command prompt character to switch to the directory of that folder to create the Scrapy project, as follows: scrapy startproject any_name
3. Once you have the project, you can build a Scrapy web crawler with the following commands: scrapy genspider any_name any_name.com
4. Install scrapy-selenium following command: pip install scrapy-selenium
5. Next, go to download the browser driver for Selenium and place it in the Scrapy project folder
6. Open the settings.py file and add scrapy-selenium Middleware settings, as shown below: 
    DOWNLOADER_MIDDLEWARES = {
    'scrapy_selenium.SeleniumMiddleware': 800
   }
 
   SELENIUM_DRIVER_NAME = 'chrome'
   SELENIUM_DRIVER_EXECUTABLE_PATH = 'chromedriver.exe'
   SELENIUM_DRIVER_ARGUMENTS = ['-headless']
7. go to your spider file and write code that u want, and the code below is a sample:

import scrapy
from scrapy_selenium import SeleniumRequest


class TestSpider(scrapy.Spider):
    name = 'test'

    def start_requests(self):
        yield SeleniumRequest(url='https://www.example.com/', callback=self.parse)

    def parse(self, response):
        titles = response.xpath("//div[@class='image']").getall()  # Crawl all event titles

        for title in titles:
            print(title)  # Print the event title

Popularity 9/10 Helpfulness 2/10 Language python

Source: Grepper

Tags: python scrapy selenium

Link to this answer
Share Copy Link

Contributed on Sep 26 2023

Srey Sitharo

0 Answers Avg Quality 2/10

Closely Related Answers

scrapy with selenium

Comment

-1

import scrapy
from selenium import webdriver

class ProductSpider(scrapy.Spider):
    name = "product_spider"
    allowed_domains = ['ebay.com']
    start_urls = ['http://www.ebay.com/sch/i.html?_odkw=books&_osacat=0&_trksid=p2045573.m570.l1313.TR0.TRC0.Xpython&_nkw=python&_sacat=0&_from=R40']

    def __init__(self):
        self.driver = webdriver.Firefox()

    def parse(self, response):
        self.driver.get(response.url)

        while True:
            next = self.driver.find_element_by_xpath('//td[@class="pagn-next"]/a')

            try:
                next.click()

                # get the data and write it to scrapy items
            except:
                break

        self.driver.close()

Popularity 10/10 Helpfulness 1/10 Language python

Source: stackoverflow.com

Tags: python py

Link to this answer
Share Copy Link

Contributed on Sep 12 2022

Ugliest Unicorn

0 Answers Avg Quality 2/10

scrapy with selenium

Contents

More Related Answers

scrapy with selenium

Closely Related Answers

scrapy with selenium

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.