Detect broken links in a website

Detecting broken links on a website involves checking each link on your site to verify whether it leads to a valid destination. Here are various methods and tools—manual and automated—that you can use to detect broken links:

---

1. Manual Checking (Not Scalable)

You can manually click on each link on your website and see if it leads to a 404 or an error page. This approach is only suitable for small websites.

---

2. Browser Extensions

• Check My Links (Chrome Extension)
  Highlights valid and broken links directly on the webpage. Green = valid, red = broken.
  Ideal for quick testing.

• Link Checker (Firefox Extension)
  Scans the current page and identifies broken or redirected links.

---

3. Online Broken Link Checker Tools

• Dead Link Checker
  URL: https://www.deadlinkchecker.com/
  Allows scanning of single web pages or entire sites.

• Broken Link Checker by Ahrefs
  URL: https://ahrefs.com/broken-link-checker
  More comprehensive. Requires Ahrefs account for full access.

• W3C Link Checker
  URL: https://validator.w3.org/checklink
  Free and precise. Analyzes a single page at a time.

---

4. Use Website Crawling Tools

Screaming Frog SEO Spider

• Available on Windows, macOS, Linux

• Scans up to 500 URLs for free

• Identifies 404 errors, server errors, and redirect loops

• Exports broken link reports in CSV/Excel

Xenu’s Link Sleuth (Windows Only)

• Free, lightweight, and effective

• Useful for scanning entire domains and exporting reports

---

5. Google Search Console

• Go to Coverage > Excluded

• Look for entries like “Not found (404)” or “Soft 404”

• This includes URLs that are either broken internally or externally

---

6. Use Python Scripts for Custom Checking

python
CopyEdit
import requests
from bs4 import BeautifulSoup

def check_links(url):
    broken_links = []
    try:
        response = requests.get(url)
        soup = BeautifulSoup(response.text, "html.parser")
        links = [a.get("href") for a in soup.find_all("a", href=True)]
        
        for link in links:
            try:
                res = requests.head(link, allow_redirects=True, timeout=5)
                if res.status_code >= 400:
                    broken_links.append((link, res.status_code))
            except Exception as e:
                broken_links.append((link, str(e)))
        
        return broken_links
    except Exception as e:
        return [("Main Page Error", str(e))]

url = "https://example.com"
for link, error in check_links(url):
    print(f"Broken Link: {link}, Error: {error}")

This code scans all anchor links on a webpage and tests their HTTP status.

---

7. CMS-Based Plugins

If you’re using a CMS like WordPress:

• Broken Link Checker Plugin
  Automatically scans and notifies you of broken links
  Provides options to unlink or update links directly in the dashboard

• SEMrush Site Audit Tool (for WordPress and others)
  Provides in-depth link error reports including broken internal/external links

---

8. Command Line Tools

• wget

bash
CopyEdit
wget --spider -r -nd -nv -l 5 https://example.com 2>&1 | grep -B 2 "404 Not Found"

This recursively crawls and checks links. Ideal for tech-savvy users.

• Broken Link Checker (Node.js)

bash
CopyEdit
npx broken-link-checker https://example.com --recursive

Requires Node.js and provides a detailed broken link report.

---

Tips to Prevent Broken Links

• Use relative URLs when appropriate to reduce dependency on full domain structures.

• Set up 301 redirects for moved or deleted content.

• Regularly audit your website—monthly or quarterly.

• Use a 404 page with helpful navigation or search options.

---

By using a combination of automated tools and scheduled audits, you can maintain a healthy internal and external link structure, improving both user experience and SEO rankings.