Scrape job boards and filter results

To scrape job boards and filter results effectively, you can build or use a script or application that:

1. Scrapes multiple job boards (like Indeed, Glassdoor, LinkedIn, etc.)

2. Parses the relevant job data

3. Filters results based on keywords, location, salary, remote options, company, and more.

Here’s a high-level breakdown of how to do it using Python:

Step 1: Choose Job Boards to Scrape

Some common job boards include:

• Indeed

• LinkedIn Jobs

• Glassdoor

• RemoteOK

• We Work Remotely

Some of these sites have protection mechanisms like CAPTCHAs or JavaScript rendering, so consider using APIs or scraping-friendly platforms when available.

Step 2: Set Up the Environment

You’ll need:

bash
CopyEdit
pip install requests beautifulsoup4 pandas

For dynamic pages:

bash
CopyEdit
pip install selenium

Step 3: Basic Scraper Example (Indeed)

python
CopyEdit
import requests
from bs4 import BeautifulSoup
import pandas as pd

def scrape_indeed(keyword, location, max_results=50):
    jobs = []
    base_url = "https://www.indeed.com/jobs"
    for start in range(0, max_results, 10):
        params = {
            'q': keyword,
            'l': location,
            'start': start
        }
        response = requests.get(base_url, params=params)
        soup = BeautifulSoup(response.text, 'html.parser')

        for card in soup.find_all('div', class_='job_seen_beacon'):
            title = card.find('h2', class_='jobTitle')
            company = card.find('span', class_='companyName')
            location = card.find('div', class_='companyLocation')
            summary = card.find('div', class_='job-snippet')
            if title and company:
                jobs.append({
                    'Title': title.text.strip(),
                    'Company': company.text.strip(),
                    'Location': location.text.strip() if location else '',
                    'Summary': summary.text.strip() if summary else ''
                })

    return pd.DataFrame(jobs)

Step 4: Filter Results

python
CopyEdit
def filter_jobs(jobs_df, include_keywords=None, exclude_keywords=None, remote_only=False):
    if include_keywords:
        jobs_df = jobs_df[jobs_df['Summary'].str.contains('|'.join(include_keywords), case=False)]
    if exclude_keywords:
        jobs_df = jobs_df[~jobs_df['Summary'].str.contains('|'.join(exclude_keywords), case=False)]
    if remote_only:
        jobs_df = jobs_df[jobs_df['Location'].str.contains('remote', case=False)]

    return jobs_df

Step 5: Example Usage

python
CopyEdit
df = scrape_indeed("Python Developer", "Remote", max_results=30)
filtered = filter_jobs(df, include_keywords=["Django", "API"], exclude_keywords=["senior", "manager"], remote_only=True)

print(filtered.head())

Optional Enhancements

• Use Selenium for JavaScript-heavy websites.

• Store results in CSV for future analysis.

• Schedule scraping with cron jobs or use services like Airflow.

• Respect robots.txt and terms of service of job boards.

Alternatives

If scraping is too complex or legally risky for some platforms, consider using these:

• Google Jobs API via CSE

• LinkedIn Jobs API (paid access)

• JSearch API – Free job data API on RapidAPI

Let me know your target filters and job sources, and I can tailor a complete script for your case.