How to Apply EDA to Text Data for Sentiment Analysis

Exploratory Data Analysis (EDA) plays a crucial role in preparing and understanding textual data before building a sentiment analysis model. Applying EDA to text data requires specialized techniques since traditional statistical methods aren’t directly applicable to unstructured text. Here’s a comprehensive breakdown of how to apply EDA to text data for sentiment analysis.

---

Understanding the Dataset

The first step in any EDA process is understanding the structure of the dataset. In sentiment analysis, datasets typically consist of textual data (such as reviews, tweets, or comments) and corresponding sentiment labels (positive, negative, or neutral).

Initial Steps:

• Load the dataset using libraries such as pandas.

• Check for null values and handle them.

• Inspect the balance of sentiment classes to identify any class imbalance.

python
CopyEdit
import pandas as pd

df = pd.read_csv('sentiment_data.csv')
print(df.info())
print(df['sentiment'].value_counts())

---

Basic Text Statistics

Once the data is loaded, analyze basic text properties to understand the distribution and structure of the content.

Key Statistics to Compute:

• Number of words per document

• Number of characters per document

• Average word length

• Number of unique words

• Frequency distribution of sentiments

python
CopyEdit
df['word_count'] = df['text'].apply(lambda x: len(str(x).split()))
df['char_count'] = df['text'].apply(lambda x: len(str(x)))
df['avg_word_length'] = df['char_count'] / df['word_count']

Use histograms and boxplots to visualize these statistics and detect outliers or anomalies.

---

Text Cleaning and Preprocessing

Before diving deeper into EDA, clean the text to remove noise and standardize the format.

Common Cleaning Steps:

• Lowercasing

• Removing punctuation and special characters

• Removing numbers

• Removing stopwords

• Lemmatization or stemming

python
CopyEdit
import string
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
import re

lemmatizer = WordNetLemmatizer()
stop_words = set(stopwords.words('english'))

def clean_text(text):
    text = text.lower()
    text = re.sub(r'd+', '', text)
    text = text.translate(str.maketrans('', '', string.punctuation))
    words = text.split()
    words = [lemmatizer.lemmatize(word) for word in words if word not in stop_words]
    return " ".join(words)

df['clean_text'] = df['text'].apply(clean_text)

---

Frequency Distribution of Words

Analyzing the most common words in each sentiment class helps in understanding sentiment-specific vocabulary.

Steps:

• Tokenize the text

• Count word frequencies

• Plot top N frequent words using bar plots or word clouds

python
CopyEdit
from collections import Counter
from wordcloud import WordCloud
import matplotlib.pyplot as plt

def plot_wordcloud(data, title):
    text = " ".join(data)
    wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)
    plt.figure(figsize=(10, 5))
    plt.imshow(wordcloud, interpolation='bilinear')
    plt.axis('off')
    plt.title(title)
    plt.show()

plot_wordcloud(df[df['sentiment'] == 'positive']['clean_text'], 'Positive Sentiment WordCloud')
plot_wordcloud(df[df['sentiment'] == 'negative']['clean_text'], 'Negative Sentiment WordCloud')

---

N-gram Analysis

Unigram, bigram, and trigram analysis reveal common patterns or phrases that are useful for sentiment detection.

python
CopyEdit
from sklearn.feature_extraction.text import CountVectorizer

def display_ngrams(corpus, n=2, top_k=20):
    vectorizer = CountVectorizer(ngram_range=(n, n))
    X = vectorizer.fit_transform(corpus)
    ngram_counts = X.sum(axis=0).tolist()[0]
    vocab = vectorizer.get_feature_names_out()
    freq_dist = sorted(zip(vocab, ngram_counts), key=lambda x: x[1], reverse=True)[:top_k]
    for phrase, freq in freq_dist:
        print(f'{phrase}: {freq}')

print("Top Bigrams in Positive Sentiments:")
display_ngrams(df[df['sentiment'] == 'positive']['clean_text'], n=2)

---

Sentiment Distribution and Imbalance Handling

Visualize the distribution of sentiment classes to identify imbalance issues that could affect model performance.

python
CopyEdit
import seaborn as sns

sns.countplot(x='sentiment', data=df)

If there is a significant imbalance, consider techniques like:

• Oversampling (e.g., SMOTE)

• Undersampling

• Stratified sampling during train-test split

---

TF-IDF Analysis

Term Frequency-Inverse Document Frequency (TF-IDF) scores help identify words that are significant across the corpus.

python
CopyEdit
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(max_features=100)
tfidf_matrix = tfidf.fit_transform(df['clean_text'])
tfidf_df = pd.DataFrame(tfidf_matrix.toarray(), columns=tfidf.get_feature_names_out())

Analyze TF-IDF scores to extract high-weight features relevant for classification.

---

Sentiment Lexicon and Polarity Score

Use sentiment lexicons like VADER or TextBlob to assign sentiment scores and validate labels.

python
CopyEdit
from textblob import TextBlob

df['polarity'] = df['clean_text'].apply(lambda x: TextBlob(x).sentiment.polarity)
sns.histplot(data=df, x='polarity', hue='sentiment', kde=True)

This polarity score can help in:

• Verifying mislabeled data

• Setting threshold-based sentiment classification

• Feature engineering for supervised learning

---

Topic Modeling (Optional)

Latent Dirichlet Allocation (LDA) can uncover latent topics in the text that correspond with sentiment tendencies.

python
CopyEdit
import gensim
from gensim import corpora

tokens = [text.split() for text in df['clean_text']]
dictionary = corpora.Dictionary(tokens)
corpus = [dictionary.doc2bow(text) for text in tokens]

lda_model = gensim.models.ldamodel.LdaModel(corpus, num_topics=5, id2word=dictionary, passes=15)

topics = lda_model.print_topics()
for topic in topics:
    print(topic)

---

Co-occurrence and Correlation Analysis

Analyze which words frequently occur together in the same documents. This helps in identifying contextual associations that may influence sentiment.

python
CopyEdit
import numpy as np

co_matrix = (tfidf_matrix.T * tfidf_matrix)
co_matrix.setdiag(0)
co_occur_df = pd.DataFrame(co_matrix.toarray(), index=tfidf.get_feature_names_out(), columns=tfidf.get_feature_names_out())

---

Conclusion

EDA in text data for sentiment analysis involves a mix of preprocessing, statistical analysis, and visualization tailored to unstructured textual content. It helps identify patterns, class imbalances, and noise that can impact the effectiveness of machine learning models. By applying these EDA techniques thoroughly, you gain critical insights into the data that drive informed decisions in the model development phase.