Scrape new research articles by keyword

To scrape new research articles by keyword, you can use a Python script that interacts with sources like:

• Google Scholar (limited due to restrictions)

• PubMed (for biomedical topics)

• arXiv (for physics, CS, math, etc.)

• CrossRef API (DOI metadata across journals)

• Semantic Scholar API

Here’s a simple example using the Semantic Scholar API:

🔍 Python Script: Scrape New Research Articles by Keyword

python
CopyEdit
import requests

def get_research_articles(keyword, limit=10):
    url = "https://api.semanticscholar.org/graph/v1/paper/search"
    params = {
        "query": keyword,
        "limit": limit,
        "fields": "title,authors,year,venue,url"
    }
    response = requests.get(url, params=params)
    if response.status_code == 200:
        data = response.json().get("data", [])
        for i, paper in enumerate(data, 1):
            title = paper.get("title")
            authors = ", ".join([author["name"] for author in paper.get("authors", [])])
            year = paper.get("year")
            venue = paper.get("venue", "Unknown venue")
            paper_url = paper.get("url", "No URL")
            print(f"{i}. {title}n   Authors: {authors}n   Year: {year} | Venue: {venue}n   Link: {paper_url}n")
    else:
        print(f"Error {response.status_code}: Unable to fetch articles.")

# Example usage
get_research_articles("machine learning", limit=5)

📌 Notes:

• This script fetches the latest papers related to the keyword.

• You can automate this or export results to a file if needed.

• API limits apply (use API key if available for higher rate limits).

Would you like a version that stores the results in a CSV or automates daily scraping?