The Palos Publishing Company

Follow Us On The X Platform @PalosPublishing
Categories We Write About

Scrape SEO metadata from websites

Written by

Bernardo Palos

in

Scraping SEO metadata from websites involves extracting HTML meta tags and other relevant SEO-related content from the <head> section of a webpage. The most common metadata includes:

  • <title> tag

  • <meta name="description">

  • <meta name="keywords">

  • <meta name="robots">

  • <meta property="og:..."> (Open Graph for social sharing)

  • <meta name="twitter:..."> (Twitter cards)

Here’s a basic Python script using requests and BeautifulSoup to scrape SEO metadata from a given URL:

python
import requests
from bs4 import BeautifulSoup

def scrape_seo_metadata(url):
    headers = {
        "User-Agent": "Mozilla/5.0"
    }
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")
        
        metadata = {
            "title": soup.title.string if soup.title else None,
            "meta_description": None,
            "meta_keywords": None,
            "meta_robots": None,
            "og_tags": {},
            "twitter_tags": {}
        }

        for tag in soup.find_all("meta"):
            name = tag.get("name", "").lower()
            prop = tag.get("property", "").lower()
            content = tag.get("content", "")
            
            if name == "description":
                metadata["meta_description"] = content
            elif name == "keywords":
                metadata["meta_keywords"] = content
            elif name == "robots":
                metadata["meta_robots"] = content
            elif prop.startswith("og:"):
                metadata["og_tags"][prop] = content
            elif name.startswith("twitter:"):
                metadata["twitter_tags"][name] = content

        return metadata
    
    except requests.RequestException as e:
        return {"error": str(e)}

# Example usage:
url = "https://example.com"
seo_data = scrape_seo_metadata(url)
print(seo_data)

Notes:

  • Install required libraries:

    bash
    pip install requests beautifulsoup4

  • You can enhance the script to scrape multiple URLs, save results to a CSV, or handle JavaScript-rendered pages using tools like Selenium or Playwright.

Let me know if you want this as a tool, API, or browser extension.

Share this Page your favorite way: Click any app below to share.
See all the ways to share this page

Enter your email below to join The Palos Publishing Company Email List

We respect your email privacy

Check Out Our Newest Posts we wrote about

Categories We Write About