Scrape medium articles by tag

To scrape Medium articles by tag, you can use the following Python script that leverages Medium’s tag-based RSS feeds. Medium does not provide a public API for scraping, so this method uses RSS feeds, which are the most stable and officially provided interface.

✅ Python Script to Scrape Medium Articles by Tag

python
CopyEdit
import feedparser

def scrape_medium_articles_by_tag(tag, max_articles=10):
    url = f"https://medium.com/feed/tag/{tag}"
    feed = feedparser.parse(url)

    articles = []
    for entry in feed.entries[:max_articles]:
        article = {
            "title": entry.title,
            "link": entry.link,
            "published": entry.published,
            "summary": entry.summary
        }
        articles.append(article)
    
    return articles

# Example usage
tag = "artificial-intelligence"
articles = scrape_medium_articles_by_tag(tag, max_articles=5)

for i, article in enumerate(articles, 1):
    print(f"{i}. {article['title']}")
    print(f"Link: {article['link']}")
    print(f"Published: {article['published']}")
    print(f"Summary: {article['summary']}n")

🔍 How It Works

• feedparser is a Python library for parsing RSS feeds.

• Medium provides tag-based RSS feeds at https://medium.com/feed/tag/{tag}.

• The script extracts the article’s title, URL, published date, and summary.

📦 Installation

If you haven’t installed feedparser yet, run:

bash
CopyEdit
pip install feedparser

⚠️ Notes

• Medium often uses redirects or JavaScript-rendered content for full posts. This script gives you metadata, not full article text.

• To scrape full content, you would need an HTML parser like BeautifulSoup and handle potential anti-bot protections.

Let me know if you need a version that saves results to a CSV or scrapes full article content.