Breaking News: Grepper is joining You.com. Read the official announcement!

while loop to click next

sulav raj sharma answered on October 31, 2022 Popularity 1/10 Helpfulness 1/10

answer while loop to click next

while loop to click next

Comment

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time
import numpy as np
import pandas as pd


driver =  webdriver.Firefox()
driver.maximize_window()
url = "https://profiles.ucr.edu/app/home/search;name=;org=Physics%20and%20Astronomy;title=;phone=;affiliation=Faculty"
driver.get(url)



timeout = 20

# This container collect data from first page 
containers = WebDriverWait(driver, timeout).until(EC.visibility_of_all_elements_located((By.XPATH,'//div[@class="column ng-star-inserted"]' )))
for container in containers:
    name = container.find_element_by_css_selector('.header-details h5') #we are srcaping name from each page
    print(name.text)

i = 1
try:
    while i < 2: #Now it will look for “next page button” in every page and continuing click on “next page button”  until it will reach the last page.   
        next_page_button = driver.find_element_by_xpath("//li[@class='pagination-next ng-star-inserted']")
        next_page_button.click()

        #our this container2 start collect data from second page to last page
        containers = WebDriverWait(driver, timeout).until(EC.visibility_of_all_elements_located((By.XPATH,'//div[@class="column ng-star-inserted"]' )))
        for container in containers:
            name = container.find_element_by_css_selector('.header-details h5') #we are srcaping name from each page
            print(name.text)
            time.sleep(3)
except:
    pass #if any page don't have “next page button” then our code will be end without any error.

Popularity 1/10 Helpfulness 1/10 Language whatever

Source: stackoverflow.com

Tags: click next whatever while-loop

Link to this answer
Share Copy Link

Contributed on Oct 31 2022

sulav raj sharma

0 Answers Avg Quality 2/10

while loop to click next

Contents

More Related Answers

while loop to click next

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.