Calibration of Convolutional Neural Networks (CNNs) using Temperature Scaling
Exploring the calibration of CNNs through temperature scaling to improve model reliability in binary classification tasks.
Abstract
This project investigates the calibration of Convolutional Neural Networks (CNNs) using temperature scaling to enhance their reliability in binary classification tasks. The project primarily focuses on understanding how well a CNN’s predicted probabilities align with actual outcomes through calibration metrics like Expected Calibration Error (ECE) and reliability diagrams.
Two models were used to achieve these objectives:
- LeNet-5: A simple CNN architecture trained from scratch.
- DenseNet-121: A more complex pre-trained model, fine-tuned for binary classification.
Both models were trained on the CIFAR-10 dataset, filtered to include only images of birds and cats. The calibration process was evaluated through reliability diagrams and ECE calculations, showing significant improvements after applying temperature scaling.
Model Descriptions
Model 1: LeNet-5
The LeNet-5 model was adapted for binary classification to distinguish between cats and birds in the CIFAR-10 dataset. The architecture was modified to include dropout layers and batch normalization for better performance.
Key Layers:
- Two convolutional layers with ReLU activation.
- Two max-pooling layers.
- Three fully connected layers with a sigmoid activation for binary classification.
| Layer Type | Output Shape | Activation Function |
|---|---|---|
| Conv Layer 1 | (6, 28, 28) | ReLU |
| Max Pooling Layer 1 | (6, 14, 14) | - |
| Conv Layer 2 | (16, 10, 10) | ReLU |
| Max Pooling Layer 2 | (16, 5, 5) | - |
| Fully Connected 1 | 120 | ReLU |
| Fully Connected 2 | 84 | ReLU |
| Output Layer | 1 | Sigmoid |
The model’s performance was evaluated using:
- Accuracy
- Reliability Diagrams
- Expected Calibration Error (ECE)
Model 2: DenseNet-121
In addition to LeNet-5, the project also explores DenseNet-121, a pre-trained model from the torchvision library, fine-tuned for binary classification.
Key modifications:
- Replaced the last classification layer to adapt the model for binary classification.
- Applied temperature scaling to calibrate the model’s predictions.
| Model Name | Layers | Parameters |
|---|---|---|
| DenseNet-121 | 121 | ~7 million |
Calibration Techniques
The project focuses on improving model calibration using temperature scaling, a post-processing method that adjusts the predicted probabilities by a temperature parameter T. Lowering or increasing this parameter helps to smooth the probabilities and reduce calibration errors.
Steps to Apply Temperature Scaling:
- Train the model and obtain logits.
- Adjust the logits with a temperature parameter.
- Recalculate the probabilities and evaluate the Expected Calibration Error (ECE).
Results
The project shows a clear improvement in model calibration after applying temperature scaling.
Reliability Diagram Comparison
Below are the reliability diagrams comparing the initial model predictions and the calibrated predictions using the optimal temperature factor for both models.
The reliability diagram highlights the difference between the predicted probabilities and actual outcomes. The red line shows the calibrated model, which significantly reduces the calibration error compared to the initial predictions highlighted in blue.
Dataset
The project uses a filtered version of the CIFAR-10 dataset, containing images of birds and cats. The dataset is preprocessed to include only these two categories, and the images are normalized for training.
| Class | Label |
|---|---|
| Bird | 0 |
| Cat | 1 |
The dataset can be downloaded from the following link:
This project was carried out in collaboration with Elena Almagro and Juan Muñoz.
Full Report
You can download the complete report document via the following link:
GitHub Repository
You can find the Python notebook for this project, with its results and the whole process on GitHub