Scrape eBay auctions for niche products

To scrape eBay auctions for niche products, you can use Python with libraries such as requests, BeautifulSoup, or Selenium (for dynamic content). eBay also provides an official API, which is the most reliable and scalable method for accessing auction data.

Below is a basic guide using eBay’s Finding API (recommended) and a scraping alternative with BeautifulSoup (not recommended for production due to ToS issues and bot detection).

Option 1: Using eBay Finding API (Recommended)

Step 1: Get eBay API Key

1. Sign up at eBay Developer Program

2. Create an application to get your App ID (aka Client ID)

Step 2: Sample Python Code

python
CopyEdit
import requests

def search_ebay_auctions(query, category_id=None, entries_per_page=10):
    url = "https://svcs.ebay.com/services/search/FindingService/v1"
    params = {
        'OPERATION-NAME': 'findItemsAdvanced',
        'SERVICE-VERSION': '1.0.0',
        'SECURITY-APPNAME': 'YourAppIDHere',  # Replace with your App ID
        'RESPONSE-DATA-FORMAT': 'JSON',
        'REST-PAYLOAD': '',
        'keywords': query,
        'itemFilter(0).name': 'ListingType',
        'itemFilter(0).value': 'Auction',
        'paginationInput.entriesPerPage': entries_per_page
    }
    if category_id:
        params['categoryId'] = category_id

    response = requests.get(url, params=params)
    data = response.json()
    items = data['findItemsAdvancedResponse'][0]['searchResult'][0].get('item', [])
    return [{
        'title': item['title'][0],
        'price': item['sellingStatus'][0]['currentPrice'][0]['__value__'],
        'currency': item['sellingStatus'][0]['currentPrice'][0]['@currencyId'],
        'url': item['viewItemURL'][0]
    } for item in items]

# Example usage:
results = search_ebay_auctions("vintage pocket watch")
for r in results:
    print(f"{r['title']} - {r['price']} {r['currency']} - {r['url']}")

Option 2: Scraping with BeautifulSoup (Use Caution)

eBay uses dynamic loading and bot protection. This method is less reliable and violates eBay’s Terms of Service.

Sample (For Educational Use Only)

python
CopyEdit
import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0"
}

def scrape_ebay(query):
    query = query.replace(" ", "+")
    url = f"https://www.ebay.com/sch/i.html?_nkw={query}&_sop=1&rt=nc&LH_Auction=1"
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')

    items = []
    for item in soup.select('.s-item'):
        title = item.select_one('.s-item__title')
        price = item.select_one('.s-item__price')
        link = item.select_one('.s-item__link')
        if title and price and link:
            items.append({
                'title': title.text,
                'price': price.text,
                'url': link['href']
            })
    return items

# Example usage:
results = scrape_ebay("retro gaming console")
for item in results:
    print(f"{item['title']} - {item['price']} - {item['url']}")

Best Practices

• Use the eBay API for reliability and compliance.

• If scraping, implement delays and rotate headers/IPs to avoid being blocked.

• Always check eBay’s API Usage Policy and Terms of Service.

Let me know if you want a complete scraper with filtering options, scheduling, or database storage.