Scrape popular questions from Reddit

To scrape popular questions from Reddit, here’s a brief overview of how you can do it:

Method 1: Use Reddit’s API

Reddit offers a free API for accessing posts and comments.

Steps:

1. Create a Reddit App:

   • Go to https://www.reddit.com/prefs/apps

   • Create a script-type app to get a client_id and client_secret.

2. Use Python and PRAW (Python Reddit API Wrapper):

   python
   CopyEdit
   import praw
   
   reddit = praw.Reddit(
       client_id='YOUR_CLIENT_ID',
       client_secret='YOUR_CLIENT_SECRET',
       user_agent='your_app_name',
   )
   
   subreddit = reddit.subreddit("AskReddit")  # Example subreddit
   
   for post in subreddit.hot(limit=50):  # Get 50 hot posts
       if post.title.endswith("?"):
           print(post.title)
   

3. Filter for Questions Only:

   • Many popular subreddits like r/AskReddit, r/TooAfraidToAsk, or r/NoStupidQuestions are filled with question-style titles.

Method 2: Use Pushshift API (Unofficial)

Pushshift.io offers more advanced querying, including historical data.

Example:

url
CopyEdit
https://api.pushshift.io/reddit/search/submission/?subreddit=AskReddit&sort=desc&sort_type=score&size=100

• Use this endpoint with filters to grab popular questions.

• Note: Pushshift is less reliable for real-time data as it may lag behind Reddit’s official API.

Method 3: Use Web Scraping (If API is not an option)

Use libraries like BeautifulSoup with requests, or automation tools like Selenium.

Example with Python:

python
CopyEdit
import requests
from bs4 import BeautifulSoup

url = "https://www.reddit.com/r/AskReddit/hot/"
headers = {'User-Agent': 'Mozilla/5.0'}

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, "html.parser")

for post in soup.select("h3"):
    title = post.get_text()
    if title.endswith("?"):
        print(title)

Note: Reddit may block automated scrapers. Use API if possible and respect their terms of service.

Would you like a ready-to-use script or list of currently trending questions pulled live?