Scrape training courses by category

To “scrape training courses by category,” you typically need to extract data from websites that list training or educational content (e.g., Coursera, Udemy, edX). This can be done using web scraping tools or scripts, usually built with Python libraries like BeautifulSoup or Scrapy. Below is a sample Python script to scrape training courses by category from a hypothetical site (as most real platforms have protections like CAPTCHA, authentication, or anti-bot mechanisms):

Python Script to Scrape Training Courses by Category (Example)

python
CopyEdit
import requests
from bs4 import BeautifulSoup

# Example: Hypothetical training site with categories in the URL
base_url = "https://www.example-training-site.com/category/"
categories = ['data-science', 'web-development', 'business', 'design']

# Storage for course data
courses_data = []

for category in categories:
    url = f"{base_url}{category}"
    response = requests.get(url)
    
    if response.status_code == 200:
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Update selector based on actual HTML structure of the site
        course_cards = soup.select('.course-card')
        
        for card in course_cards:
            title = card.select_one('.course-title').get_text(strip=True)
            description = card.select_one('.course-description').get_text(strip=True)
            link = card.select_one('a')['href']
            
            courses_data.append({
                'category': category,
                'title': title,
                'description': description,
                'url': link
            })
    else:
        print(f"Failed to fetch category: {category}")

# Print scraped course data
for course in courses_data:
    print(f"{course['category']} | {course['title']} | {course['url']}")

Key Notes

• Legality & Ethics: Always check the site’s robots.txt and Terms of Service. Some prohibit scraping.

• Performance: For large-scale scraping, use throttling, user-agent rotation, and proxies.

• Dynamic Content: For JavaScript-heavy sites, use Selenium or Playwright instead of requests and BeautifulSoup.

• Export: You can modify the script to save results to CSV or a database.

If you have a specific site or format in mind, I can tailor the code accordingly.