The Palos Publishing Company

Follow Us On The X Platform @PalosPublishing
Categories We Write About

Build a personal document search engine

Written by

Bernardo Palos

in

Building a personal document search engine involves creating a system that can index your personal files (PDFs, DOCX, TXT, etc.), extract their content, and enable fast, accurate searching through a user-friendly interface. Here’s a step-by-step guide to build one:

1. Define Requirements

  • Document types: PDF, Word (DOCX), Text files, etc.

  • Search capabilities: Keyword search, full-text search, semantic search (optional).

  • Interface: Command-line, web-based, or desktop GUI.

  • Scope: Local-only or remote access.

  • Storage size: How many documents and total size?

2. Set Up the Environment

Use Python for ease of development and strong library support.

Dependencies:

bash
pip install flask whoosh python-docx PyMuPDF pdfminer.six

3. Document Processing

Extract text content from documents.

python
import os
from docx import Document
import fitz  # PyMuPDF for PDFs

def extract_text(file_path):
    ext = os.path.splitext(file_path)[1].lower()
    if ext == '.pdf':
        return extract_pdf(file_path)
    elif ext == '.docx':
        return extract_docx(file_path)
    elif ext == '.txt':
        return open(file_path, 'r', encoding='utf-8').read()
    return ''

def extract_docx(path):
    doc = Document(path)
    return 'n'.join([para.text for para in doc.paragraphs])

def extract_pdf(path):
    doc = fitz.open(path)
    return 'n'.join([page.get_text() for page in doc])

4. Indexing with Whoosh

Use the Whoosh library to create an index for fast search.

python
from whoosh.index import create_in
from whoosh.fields import Schema, TEXT, ID
import os

def create_search_index(doc_folder, index_folder='indexdir'):
    if not os.path.exists(index_folder):
        os.mkdir(index_folder)

    schema = Schema(title=ID(stored=True), path=ID(stored=True), content=TEXT)
    ix = create_in(index_folder, schema)
    writer = ix.writer()

    for root, _, files in os.walk(doc_folder):
        for file in files:
            full_path = os.path.join(root, file)
            text = extract_text(full_path)
            writer.add_document(title=file, path=full_path, content=text)

    writer.commit()

5. Search Functionality

Query the index to find matching documents.

python
from whoosh.qparser import QueryParser
from whoosh.index import open_dir

def search_documents(query, index_folder='indexdir'):
    ix = open_dir(index_folder)
    results_list = []

    with ix.searcher() as searcher:
        parser = QueryParser("content", ix.schema)
        parsed_query = parser.parse(query)
        results = searcher.search(parsed_query, limit=10)
        for result in results:
            results_list.append({
                "title": result['title'],
                "path": result['path']
            })
    return results_list

6. Web Interface (Flask App)

Create a simple search frontend using Flask.

python
from flask import Flask, request, render_template_string

app = Flask(__name__)

TEMPLATE = '''
<!doctype html>
<title>Document Search</title>
<h2>Search Documents</h2>
<form method="post">
    <input name="query" placeholder="Search..." style="width: 300px;">
    <input type="submit">
</form>
{% if results %}
    <ul>
    {% for r in results %}
        <li><b>{{ r.title }}</b> — {{ r.path }}</li>
    {% endfor %}
    </ul>
{% endif %}
'''

@app.route('/', methods=['GET', 'POST'])
def search():
    results = []
    if request.method == 'POST':
        query = request.form['query']
        results = search_documents(query)
    return render_template_string(TEMPLATE, results=results)

if __name__ == '__main__':
    # First index your documents once
    # create_search_index('/path/to/your/documents')
    app.run(debug=True)

7. Optional Enhancements

  • Semantic Search: Integrate with embeddings and vector databases like FAISS or use OpenAI/transformer-based models for smarter search.

  • OCR Support: Use pytesseract for scanned images/PDFs.

  • Desktop GUI: Use Tkinter or PyQt5 instead of Flask.

  • Authentication: Add login for privacy if hosting online.

8. Deployment Tips

  • Deploy Flask app using gunicorn or uvicorn.

  • Use a lightweight web server (e.g., Nginx) for production.

  • Secure access if documents are sensitive.

Would you like this search engine to support semantic search or file upload features as well?

Share this Page your favorite way: Click any app below to share.
See all the ways to share this page

Enter your email below to join The Palos Publishing Company Email List

We respect your email privacy

Check Out Our Newest Posts we wrote about

Categories We Write About