Scrape new entries from a blog’s RSS feed

To scrape new entries from a blog’s RSS feed, you can follow a basic approach using Python with libraries like feedparser or requests. Here’s how you can do it:

Steps:

1. Install Dependencies:
   First, you’ll need to install the necessary Python libraries. You can use pip for this:

   bash
   CopyEdit
   pip install feedparser requests
   

2. Fetch RSS Feed:
   Use the feedparser library to parse the RSS feed and extract entries.

3. Check for New Entries:
   You can either store the last fetched entry’s publication date or maintain a list of the most recent entries, comparing them to the new ones fetched in subsequent runs.

4. Example Code:

   python
   CopyEdit
   import feedparser
   
   def fetch_rss_feed(rss_url):
       # Parse the RSS feed
       feed = feedparser.parse(rss_url)
       return feed.entries
   
   def get_new_entries(feed_url, last_checked):
       # Get the RSS feed entries
       entries = fetch_rss_feed(feed_url)
   
       new_entries = []
       for entry in entries:
           # Check if the entry's published date is newer than last_checked
           if entry.published_parsed > last_checked:
               new_entries.append(entry)
       
       return new_entries
   
   # Example Usage
   if __name__ == "__main__":
       rss_url = 'https://example.com/rss'  # Replace with the actual RSS feed URL
       
       # Assume last_checked is a time.struct_time or similar that you tracked previously
       last_checked = time.strptime('2023-05-10 10:00:00', '%Y-%m-%d %H:%M:%S')
       
       new_entries = get_new_entries(rss_url, last_checked)
       
       # Process the new entries
       for entry in new_entries:
           print(f"Title: {entry.title}")
           print(f"Link: {entry.link}")
           print(f"Published: {entry.published}")
           print('---')
   

Explanation:

• fetch_rss_feed(rss_url): This function parses the RSS feed from the provided URL and returns the feed entries.

• get_new_entries(feed_url, last_checked): Compares each entry’s published date to last_checked and returns only those entries that are newer than the specified time.

Additional Considerations:

• You can use the last_checked timestamp to store the last fetch time. For the first run, you might set it to a very old date, like 1970-01-01.

• To avoid repetitive calls, you could store the timestamp of the last fetched entry in a file or database, so it persists across script runs.

• This is a basic example; you might need to handle exceptions, invalid RSS formats, or different feed structures based on the blog you’re targeting.

Let me know if you need more detailed steps or specific features!