Sentiment Analysis Using RNNs with and without Attention
Comparing two RNN architectures for sentiment analysis on financial news data.
Abstract
This project explores the application of Recurrent Neural Networks (RNNs) to perform sentiment analysis on financial news data. The primary objective is to classify sentences from the Financial Phrase Bank into Positive/Neutral or Negative sentiment categories. The project compares two RNN architectures:
- RNN without Attention
- RNN with Attention Mechanism
Both models were evaluated using the ROC Curve and AUC Score to measure their performance. Additionally, an MLP baseline model using mean embeddings was included to provide a reference point.
Model Descriptions
Model 1: RNN without Attention
The first model uses a basic RNN with LSTM cells to process sequences of word embeddings and predict sentiment labels based on the final state of the LSTM.
Key Layers:
- LSTM Layer: Processes the input sequences and generates hidden states.
- Dropout Layer: Reduces overfitting by randomly dropping units during training.
- Fully Connected Layer: Converts the final hidden state to output logits.
- LogSoftmax Activation: Converts logits to log probabilities for classification.
Results
The RoC curve obtained is shown below.
The AUC for RNN without Attention is 0.92. This is a very good value at first.
Optional Experiment: MLP Classifier
In addition to the RNN, the project includes an optional experiment using a Multi-Layer Perceptron (MLP) classifier to predict sentiment labels based on the mean of word embeddings.
| Layer Type | Neurons | Activation Function |
|---|---|---|
| Fully Connected 1 | 10 | ReLU |
| Fully Connected 2 | 5 | ReLU |
| Output Layer | 2 | LogSoftmax |
Results
The RoC curve obtained is shown below.
The AUC for RNN without Attention is 0.76 which is shows a much poorer performance than the RNN model, which was expected.
Model 2: RNN with Attention
The second model incorporates an attention mechanism, which allows the network to focus on specific parts of the input sequence when making predictions. This mechanism improves the model’s interpretability and overall performance.
Key Layers:
- LSTM Layer: Generates hidden states for each token in the input sequence.
- Attention Mechanism: Calculates attention weights to identify important tokens.
- Dropout Layer: Reduces overfitting.
- Fully Connected Layer: Converts the context vector to output logits.
Results
The RoC Curve obtained is shown below.
The AUC for RNN without Attention is 0.95, which has improved from the model without Attention.
Dataset Details
The dataset used in this project is the Financial Phrase Bank, which contains 4,840 sentences from financial news articles labeled as Positive, Negative, or Neutral. The labels were adjusted for this project to combine Positive and Neutral into a single category.
Dataset Summary
| Attribute | Description |
|---|---|
| Dataset Name | Financial Phrase Bank |
| Number of Sentences | 4,840 |
| Categories | Positive/Neutral, Negative |
| Source | Financial Phrase Bank |
Results and Comparison
The performance of all models was evaluated using the ROC Curve and AUC Score.
| Model | AUC Score |
|---|---|
| RNN without Attention | 0.92 |
| MLP Model | 0.76 |
| RNN with Attention | 0.95 |
Observing these results, we can conclude:
- The MLP classifier does not perform well compared to RNN-based models.
- The RNN with Attention significantly outperforms the RNN without Attention, demonstrating the effectiveness of attention mechanisms in improving sentiment analysis.
This project was carried out in collaboration with Elena Almagro and Juan Muñoz.
GitHub Repository
You can find the Python notebooks and pre-trained models for this project on GitHub.