BeautifulSoup and Selenium both work with web pages, but they serve different purposes.
Selection Rule
If the target data already exists in the HTML, use BeautifulSoup. It is fast and lightweight.
If the page requires JavaScript execution, login, clicking, scrolling, or dynamic content, use Selenium.
BeautifulSoup Setup
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, "html.parser")Finding Elements
first_p = soup.find("p")
all_links = soup.find_all("a")
title = soup.find("p", class_="title")CSS selectors:
links = soup.select("a.sister")
id_link = soup.select("#link1")
nested = soup.select("p.story a")Text and attributes:
text = soup.find("p").get_text()
href = soup.find("a").get("href")Modify and delete:
tag = soup.find("b")
tag.string = "New Title"
link = soup.find("a", id="link1")
link.decompose()Selenium Setup
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://example.com")Browser operations:
driver.maximize_window()
driver.refresh()
driver.back()
driver.forward()
print(driver.current_url)
print(driver.title)Locating Elements
from selenium.webdriver.common.by import By
driver.find_element(By.ID, "username")
driver.find_element(By.NAME, "email")
driver.find_element(By.CSS_SELECTOR, "button.submit")
driver.find_element(By.XPATH, "//div[@id='content']")Element Actions
input_box.send_keys("my_username")
input_box.clear()
button.click()
form.submit()Explicit Waits
Dynamic pages need condition-based waits, not blind sleeps.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.ID, "username"))
)Advanced Page Operations
Scroll:
driver.execute_script("arguments[0].scrollIntoView();", element)Alerts:
alert = driver.switch_to.alert
alert.accept()iframes:
driver.switch_to.frame("iframe_name")
driver.switch_to.default_content()Screenshots:
driver.save_screenshot("page.png")
element.screenshot("element.png")In practice, parse static HTML first, then page JSON, and only then start browser automation.
Deeper Notes
When reviewing this topic, do not memorize names only. Focus on static HTML parsing, Selenium dynamic pages, waits, locators, and resilient scraping workflows. If this stays at the definition level, it becomes hard to explain in interviews or apply in projects. A stronger way to study it is to place it in a concrete scenario: who calls it, where the input comes from, what happens on failure, and whether data or state can be processed twice.
- Scraping is difficult because of page changes, wait strategy, deduplication, rate limiting, recovery, and data quality.
- Choose BeautifulSoup, Selenium, or Scrapy based on dynamic behavior, data volume, and downstream cleaning needs.
- Reliable crawlers need logs, resumability, retries, and field-level validation, not just one successful run.
In a real project, use it as a decision framework: identify inputs, constraints, failure modes, and observability before choosing a specific tool or pattern. If a solution looks simple, keep asking whether it still works when scale grows, permissions change, recovery matters, and more people collaborate on it.
Practical Checklist
- Identify where this concept sits in the system: development-time constraint, runtime behavior, infrastructure capability, or collaboration workflow.
- Write one minimal working example and one failure example; only knowing the happy path is usually not enough.
- Record common misuses: edge cases, permission assumptions, performance assumptions, sync/async differences, or environment differences.
- Connect the concept to a project experience so that an interview answer can be grounded in real tradeoffs.
- End with one sentence about tradeoff: what it gives up and what it buys.
Self-Check Questions
- What core problem does this topic solve?
- What alternatives exist, and what are their costs?
- Where are the most likely edge cases?
- How would code, tests, or monitoring prove that it is reliable?
Applied Scenario
A job or product crawler is a useful scenario. First decide whether the page is static HTML or dynamically rendered, then choose BeautifulSoup, Selenium, or Scrapy. After extraction, the system still needs cleaning, deduplication, retries, and persistence. A robust crawler is not one that succeeds once; it is one that can recover when page structure changes, the network times out, anti-bot rules appear, or some fields are missing.
Common Pitfalls:
- Hard-coding fragile selectors without fallback.
- Missing rate limits and retries, causing blocks or data loss.
- Saving raw data without field-level quality checks.