Scrape course discounts from e-learning sites

Scraping course discounts from e-learning sites involves extracting publicly available information about promotions, deals, or reduced prices from platforms offering online courses. Here’s a detailed guide on how to approach this task effectively and ethically:

Understanding the Scope and Legality

• Public Data Only: Focus on publicly available discount info (e.g., course pages, promo banners).

• Terms of Service: Review each site’s terms of service to ensure scraping is allowed.

• Respect Robots.txt: Check the robots.txt file of the websites to confirm which parts are open to crawlers.

• Avoid Overloading Servers: Use rate limiting and caching to reduce server strain.

Target E-learning Sites Common for Discounts

• Udemy

• Coursera

• Skillshare

• LinkedIn Learning

• edX

• Pluralsight

• Teachable-based platforms

Data Points to Extract

• Course title

• Original price

• Discounted price

• Discount percentage

• Course URL

• Course category or subject

• Expiry date of the discount (if available)

Tools and Technologies

• Python for scripting

• Libraries like:

  • requests for HTTP requests

  • BeautifulSoup or lxml for parsing HTML

  • Selenium for dynamic pages (JS-rendered)

  • Scrapy for structured scraping workflows

  • pandas for data handling

Step-by-Step Scraping Approach

1. Identify Discount URLs or Pages

   • Many sites have a dedicated page for sales or discounts (e.g., Udemy’s “Deals” page).

   • Use search filters to narrow down discounted courses.

2. Send Requests and Parse HTML

   • Use requests.get() to fetch page content.

   • Parse HTML to locate course titles, prices, and discounts. Look for elements like <span>, <div>, or classes/id that mention price or discount.

3. Handle Dynamic Content

   • If discounts are loaded via JavaScript, use Selenium to automate browser interactions and extract rendered HTML.

4. Extract Price and Discount Info

   • Original price often appears with a strikethrough style.

   • Discounted price usually highlighted or next to the original price.

   • Percentage discount can be calculated if not explicitly provided.

5. Pagination and Multiple Pages

   • Many sites show discounts over multiple pages.

   • Automate requests to iterate through pages, stopping when no more courses appear.

6. Store Data

   • Store extracted data into CSV, JSON, or databases for further analysis.

Sample Python Snippet for Udemy Discount Scraping

python
CopyEdit
import requests
from bs4 import BeautifulSoup

url = 'https://www.udemy.com/courses/search/?price=price-free%2Cprice-paid&sort=discount'

headers = {
    "User-Agent": "Mozilla/5.0"
}

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

courses = soup.find_all('div', class_='course-card--container--3w8Zm')

for course in courses:
    title = course.find('div', class_='course-card--course-title--2f7tE').text.strip()
    try:
        original_price = course.find('div', class_='price-text--original-price--2e-F-').text.strip()
    except:
        original_price = "N/A"
    try:
        discounted_price = course.find('div', class_='price-text--price-part--Tu6MH').text.strip()
    except:
        discounted_price = "N/A"
    print(f"Course: {title}nOriginal Price: {original_price}nDiscounted Price: {discounted_price}n")

Challenges and Considerations

• Anti-scraping measures: Some sites use CAPTCHAs, IP blocking, or AJAX-loaded content.

• Data freshness: Discounts change frequently, so schedule scraping runs accordingly.

• Data normalization: Prices may vary by currency and locale.

• API availability: Some platforms may have official APIs offering this data legally and easily.

Conclusion

Scraping course discounts is feasible by targeting key pages, carefully parsing pricing data, and handling dynamic content. Always prioritize ethical scraping, respect website policies, and consider using official APIs where available to obtain discount information efficiently.