Web Scraping with Selenium and BeautifulSoup (YouTube Videos)

Last time I tried to extract all the information with Selenium, but it’s not stable. So I decided to combine Selenium and BeautifulSoup to get the data. I search “FFXIV” and the Selenium will keep scrolling down and open all the videos on the page. On the video page, the title, published date, description, views and likes will be extracted by BeautifulSoup.

1. Check the Selenium setup here
2. Extract video information, and save it as a CSV file
3. Add the keep scrolling function to get more video information (Not finished…)

1. Previous code

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://www.youtube.com/results?search_query=ffxiv")
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
video_list = driver.find_elements("xpath", '//*[@id="video-title"]')
links =…