Summarize text with Python

Summarizing text with Python can be done using various libraries and approaches, depending on whether you want an extractive or abstractive summary. Here’s a concise guide showing simple ways to summarize text in Python:

1. Using nltk for Extractive Summarization

nltk (Natural Language Toolkit) can be used to tokenize sentences and rank them by importance:

python
CopyEdit
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, sent_tokenize
import heapq

# Sample text
text = """Your long text goes here."""

# Download required data
nltk.download('punkt')
nltk.download('stopwords')

stop_words = set(stopwords.words("english"))
words = word_tokenize(text.lower())

# Calculate word frequency excluding stopwords
freq_table = {}
for word in words:
    if word not in stop_words and word.isalpha():
        freq_table[word] = freq_table.get(word, 0) + 1

# Score sentences based on word frequency
sentences = sent_tokenize(text)
sentence_scores = {}
for sent in sentences:
    for word in word_tokenize(sent.lower()):
        if word in freq_table:
            sentence_scores[sent] = sentence_scores.get(sent, 0) + freq_table[word]

# Select top 30% sentences
summary_length = int(len(sentences) * 0.3)
summary_sentences = heapq.nlargest(summary_length, sentence_scores, key=sentence_scores.get)

summary = ' '.join(summary_sentences)
print(summary)

2. Using Hugging Face Transformers for Abstractive Summarization

You can use pretrained models like BART or T5 for abstractive summaries:

python
CopyEdit
from transformers import pipeline

# Load summarization pipeline
summarizer = pipeline("summarization")

text = """Your long text goes here."""

summary = summarizer(text, max_length=130, min_length=30, do_sample=False)
print(summary[0]['summary_text'])

Make sure to install transformers and torch first:

bash
CopyEdit
pip install transformers torch

3. Using sumy Library for Extractive Summarization

sumy provides multiple summarization algorithms:

python
CopyEdit
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.lsa import LsaSummarizer

text = """Your long text goes here."""
parser = PlaintextParser.from_string(text, Tokenizer("english"))
summarizer = LsaSummarizer()

summary = summarizer(parser.document, 3)  # Summarize to 3 sentences
for sentence in summary:
    print(sentence)

Install sumy via:

bash
CopyEdit
pip install sumy

These are straightforward methods to summarize text using Python depending on your needs and environment.