Scrape scholarships from university websites

Scraping scholarship information from university websites involves programmatically extracting data from web pages. This can be achieved using Python with libraries like requests, BeautifulSoup, or browser automation tools like Selenium. Below is a simple yet robust step-by-step guide to scrape scholarships from university websites.

⚠️ Important Note on Ethics and Legality

Before scraping any website, check its robots.txt file (e.g., https://example.edu/robots.txt) and terms of service. Only scrape publicly available data and avoid overloading servers. Use appropriate time delays between requests.

1. Basic Setup

Install required Python packages:

bash
CopyEdit
pip install requests beautifulsoup4 lxml

For dynamic sites (JavaScript-rendered), you might need Selenium:

bash
CopyEdit
pip install selenium

Also, install the browser driver (e.g., ChromeDriver) if using Selenium.

2. Sample Code Using BeautifulSoup (Static Pages)

python
CopyEdit
import requests
from bs4 import BeautifulSoup

def scrape_scholarships(url):
    response = requests.get(url)
    if response.status_code != 200:
        print(f"Failed to retrieve {url}")
        return []

    soup = BeautifulSoup(response.text, 'lxml')
    scholarships = []

    # Example: looking for links or headings with keyword "scholarship"
    for link in soup.find_all(['a', 'h2', 'h3'], string=lambda text: text and "scholarship" in text.lower()):
        scholarships.append(link.get_text(strip=True))

    return scholarships

# Example university scholarship page
url = 'https://www.exampleuniversity.edu/financial-aid/scholarships'
data = scrape_scholarships(url)

for i, item in enumerate(data, 1):
    print(f"{i}. {item}")

3. Handling JavaScript-Rendered Pages with Selenium

python
CopyEdit
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import time

def scrape_scholarships_selenium(url):
    service = Service('/path/to/chromedriver')
    driver = webdriver.Chrome(service=service)

    driver.get(url)
    time.sleep(3)  # wait for JavaScript to render

    soup = BeautifulSoup(driver.page_source, 'lxml')
    driver.quit()

    scholarships = []

    for element in soup.find_all(['a', 'h2', 'h3'], string=lambda text: text and "scholarship" in text.lower()):
        scholarships.append(element.get_text(strip=True))

    return scholarships

# Use this for JS-heavy pages
# url = "https://www.dynamicuniversity.edu/scholarships"
# print(scrape_scholarships_selenium(url))

4. Crawling Multiple University Sites (with Predefined List)

You can loop through a list of scholarship page URLs:

python
CopyEdit
university_urls = [
    'https://www.harvard.edu/financial-aid/scholarships/',
    'https://www.mit.edu/financial-aid/scholarships/',
    'https://www.stanford.edu/scholarships/'
]

for uni_url in university_urls:
    print(f"nScraping {uni_url}")
    scholarships = scrape_scholarships(uni_url)
    for scholarship in scholarships:
        print(f" - {scholarship}")

5. Tips for Better Accuracy

• Scrape structured sections like tables or lists with specific CSS classes.

• Use soup.select() for complex CSS selectors.

• Combine keyword filtering (e.g., “merit-based”, “international”, “undergraduate”) for richer data.

• Consider NLP tools for extracting context if you’re scraping entire paragraphs.

6. Storing Results

You can save data to a CSV:

python
CopyEdit
import csv

with open('scholarships.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow(['University', 'Scholarship Title'])

    for url in university_urls:
        scholarships = scrape_scholarships(url)
        for title in scholarships:
            writer.writerow([url, title])

If you want help scraping a specific list of university sites or building a large-scale scholarship scraper with categories and filters, I can help build that as well.