Breaking News: Grepper is joining You.com. Read the official announcement!

load sitemap from cli scrapy

Pleasant Pony answered on March 6, 2023 Popularity 3/10 Helpfulness 7/10

answer load sitemap from cli scrapy

load sitemap from cli scrapy

Comment

add updated sitemap_urls to your SitemapSpider class

from scrapy.spiders import SitemapSpider
class my_spider(SitemapSpider):
  name = 'my_spider'
  allowed_domains = ['anything.com']
  sitemap_urls = ['']
  
  def __init__(self, sitemap_urls=None, **kwargs):
    super().__init__(**kwargs)
    if sitemap_urls:
      self.sitemap_urls = sitemap_urls.split(',')
   
  def parse(self, response):
    data = response.css('element_you_want_to_parse').get()
    yield {'data': data}

> cli
scrapy crawl my_spider -a sitemap_urls='sitemap_url','sitemap_url'

Popularity 3/10 Helpfulness 7/10 Language python

Source: Grepper

Tags: load python scrapy sitemap

Link to this answer
Share Copy Link

Contributed on Mar 06 2023

Pleasant Pony

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

load sitemap from cli scrapy

Contents

More Related Answers

load sitemap from cli scrapy

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.