Scrape headlines from news aggregators

To scrape headlines from news aggregators, you’ll need to use a combination of web scraping tools and techniques. Here’s a step-by-step guide using Python and libraries like requests, BeautifulSoup, or even newspaper3k for structured news extraction. Below is a basic example using BeautifulSoup:

Example: Scraping Headlines from Google News

python
CopyEdit
import requests
from bs4 import BeautifulSoup

def scrape_google_news_headlines():
    url = 'https://news.google.com/topstories?hl=en-US&gl=US&ceid=US:en'
    headers = {
        "User-Agent": "Mozilla/5.0"
    }

    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, 'html.parser')

    headlines = []
    for item in soup.select('h3'):
        headline = item.get_text()
        link_tag = item.find('a')
        if link_tag:
            link = 'https://news.google.com' + link_tag['href'][1:]  # remove leading '.' and prepend base URL
        else:
            link = None
        headlines.append({'headline': headline, 'url': link})
    
    return headlines

# Example usage
headlines = scrape_google_news_headlines()
for i, h in enumerate(headlines[:10], 1):
    print(f"{i}. {h['headline']} - {h['url']}")

Alternative Aggregators You Can Scrape

• Yahoo News (https://news.yahoo.com)

• Bing News (https://www.bing.com/news)

• Reddit r/news or r/worldnews (https://www.reddit.com/r/news/)

• News API services like:

  • NewsAPI.org

  • GNews.io

  • ContextualWeb

Tips for Effective Scraping

• Use rotating proxies or time.sleep() to avoid rate-limiting.

• Respect robots.txt and site terms of service.

• Use APIs when available—they’re faster, structured, and more reliable.

Would you like the scraper tailored to a specific news aggregator or format (e.g., CSV, JSON output)?