> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lovishchopra/NL2FOL/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluation Metrics
> Calculate performance metrics for logical fallacy detection
## Overview
The `get_metrics.py` script calculates standard classification metrics to evaluate the performance of the NL2FOL system on logical fallacy detection tasks.
## How It Works
Reads the results file generated by `fol_to_cvc.py` containing predictions.
* Ground truth labels: `label` column (0=fallacy, 1=valid)
* Predictions: `result` column ("LF"=fallacy, "Valid"=valid)
Maps categorical results to binary format:
* "LF" → 0 (fallacy)
* "Valid" → 1 (valid)
* Empty/Error → 1 (conservative fallback)
Computes accuracy, precision, recall, and F1 score using scikit-learn.
Prints the metrics as a tuple: (accuracy, precision, recall, f1)
## Command Usage
### Basic Command
```bash theme={null}
python3 eval/get_metrics.py
```
### Parameters
Path to the CSV file containing results from `fol_to_cvc.py`.
Expected format: `results/_results.csv`
Required columns:
* `label` - Ground truth (0 or 1)
* `result` - Prediction ("LF", "Valid", or empty)
## Example Usage
```bash Basic Usage theme={null}
python3 eval/get_metrics.py results/llama_experiment_1_results.csv
```
```bash Multiple Experiments theme={null}
# Compare different runs
for run in experiment_1 experiment_2 experiment_3; do
echo "Results for $run:"
python3 eval/get_metrics.py results/${run}_results.csv
echo ""
done
```
```bash Save to File theme={null}
python3 eval/get_metrics.py results/experiment_1_results.csv > metrics.txt
```
## Output Format
The script outputs a tuple of four metrics:
```
(accuracy, precision, recall, f1_score)
```
**Example Output:**
```
(0.8421052631578947, 0.8333333333333334, 0.8695652173913043, 0.8510638297872342)
```
Interpreted as:
* **Accuracy**: 84.21%
* **Precision**: 83.33%
* **Recall**: 86.96%
* **F1 Score**: 85.11%
## Metrics Explained
### Accuracy
```python theme={null}
accuracy = (TP + TN) / (TP + TN + FP + FN)
```
Percentage of correct predictions (both valid arguments and fallacies).
**Interpretation**: Overall correctness of the system.
### Precision
```python theme={null}
precision = TP / (TP + FP)
```
Of all arguments marked as fallacies, what percentage are actually fallacies?
**Interpretation**: How reliable are fallacy detections? High precision means few false alarms.
### Recall (Sensitivity)
```python theme={null}
recall = TP / (TP + FN)
```
Of all actual fallacies, what percentage are detected?
**Interpretation**: How many fallacies are caught? High recall means few fallacies slip through.
### F1 Score
```python theme={null}
f1 = 2 * (precision * recall) / (precision + recall)
```
Harmonic mean of precision and recall.
**Interpretation**: Balanced measure of overall performance.
F1 score is particularly useful when classes are imbalanced or when both false positives and false negatives are equally important.
## Label Encoding
The script inverts labels because it evaluates **fallacy detection** (not validity detection):
```python theme={null}
# Original labels
label = df['label'] # 0=fallacy, 1=valid
# Predictions
preds = pd.Categorical(
df['result'],
categories=['LF', 'Valid']
).codes # 0=LF (fallacy), 1=Valid
preds = np.where(preds == -1, 1, preds) # Handle missing values
# Invert for fallacy detection
get_results(1 - label, 1 - preds)
```
After inversion:
* **1** = Fallacy
* **0** = Valid argument
## Implementation Details
### Core Function
```python theme={null}
def get_results(label, preds):
acc = accuracy_score(label, preds)
prec = precision_score(label, preds)
rec = recall_score(label, preds)
f1 = f1_score(label, preds)
return acc, prec, rec, f1
```
### Full Script Structure
```python theme={null}
import numpy as np
import pandas as pd
import argparse
from sklearn.metrics import (
accuracy_score,
f1_score,
precision_score,
recall_score
)
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='Evaluate model performance.'
)
parser.add_argument(
'filename',
type=str,
help='Path to the CSV file'
)
args = parser.parse_args()
df = pd.read_csv(args.filename)
label = df['label']
preds = pd.Categorical(
df['result'],
categories=['LF','Valid']
).codes
preds = np.where(preds == -1, 1, preds)
print(get_results(1 - label, 1 - preds))
```
## Understanding Results
### High Accuracy, Low Precision
**Scenario**: System predicts "valid" for most arguments.
```
Accuracy: 90%
Precision: 50%
Recall: 95%
```
**Interpretation**: Catches most fallacies but has many false positives.
### High Precision, Low Recall
**Scenario**: System only flags obvious fallacies.
```
Accuracy: 85%
Precision: 95%
Recall: 60%
```
**Interpretation**: When it flags a fallacy, it's usually correct, but misses many fallacies.
### Balanced Performance
```
Accuracy: 85%
Precision: 84%
Recall: 86%
F1: 85%
```
**Interpretation**: Well-balanced system with consistent performance.
For fallacy detection in educational contexts, prioritize **recall** (don't miss fallacies).
For automated content moderation, prioritize **precision** (avoid false accusations).
## Handling Missing Values
Empty predictions (from processing errors) are treated as "Valid":
```python theme={null}
preds = np.where(preds == -1, 1, preds)
```
This conservative approach:
* Avoids false fallacy accusations
* Penalizes recall (increases false negatives)
* Reflects real-world system behavior
**Alternative**: Filter out missing values before evaluation:
```python theme={null}
# Remove rows with missing predictions
df_clean = df[df['result'].notna() & (df['result'] != '')]
label = df_clean['label']
preds = pd.Categorical(df_clean['result'], categories=['LF','Valid']).codes
```
## Confusion Matrix Analysis
Extend the script for detailed analysis:
```python theme={null}
from sklearn.metrics import confusion_matrix, classification_report
# Confusion matrix
cm = confusion_matrix(1 - label, 1 - preds)
print("\nConfusion Matrix:")
print(" Predicted")
print(" Valid Fallacy")
print(f"Actual Valid {cm[0,0]:5d} {cm[0,1]:5d}")
print(f"Actual Fallacy {cm[1,0]:5d} {cm[1,1]:5d}")
# Detailed report
print("\nClassification Report:")
print(classification_report(
1 - label,
1 - preds,
target_names=['Valid', 'Fallacy']
))
```
**Example Output:**
```
Confusion Matrix:
Predicted
Valid Fallacy
Actual Valid 42 8
Actual Fallacy 6 44
Classification Report:
precision recall f1-score support
Valid 0.88 0.84 0.86 50
Fallacy 0.85 0.88 0.86 50
accuracy 0.86 100
macro avg 0.86 0.86 0.86 100
weighted avg 0.86 0.86 0.86 100
```
## Per-Dataset Analysis
Compare performance across datasets:
```python theme={null}
import pandas as pd
from get_metrics import get_results
datasets = ['logic', 'logicclimate', 'folio']
results = []
for dataset in datasets:
df = pd.read_csv(f'results/{dataset}_results.csv')
label = df['label']
preds = pd.Categorical(df['result'], categories=['LF','Valid']).codes
preds = np.where(preds == -1, 1, preds)
metrics = get_results(1 - label, 1 - preds)
results.append({
'Dataset': dataset,
'Accuracy': metrics[0],
'Precision': metrics[1],
'Recall': metrics[2],
'F1': metrics[3]
})
results_df = pd.DataFrame(results)
print(results_df.to_string(index=False))
```
## Statistical Significance
For comparing models, use bootstrap confidence intervals:
```python theme={null}
from sklearn.utils import resample
import numpy as np
def bootstrap_ci(y_true, y_pred, metric_func, n_iterations=1000):
scores = []
for _ in range(n_iterations):
indices = resample(range(len(y_true)), n_samples=len(y_true))
y_true_boot = y_true[indices]
y_pred_boot = y_pred[indices]
scores.append(metric_func(y_true_boot, y_pred_boot))
lower = np.percentile(scores, 2.5)
upper = np.percentile(scores, 97.5)
return lower, upper
# Calculate 95% CI for F1 score
lower, upper = bootstrap_ci(
1 - label,
1 - preds,
f1_score
)
print(f"F1 Score: {f1_score(1-label, 1-preds):.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
```
## Visualization
Create visual reports:
```python theme={null}
import matplotlib.pyplot as plt
import seaborn as sns
# Bar chart of metrics
metrics = get_results(1 - label, 1 - preds)
metric_names = ['Accuracy', 'Precision', 'Recall', 'F1']
plt.figure(figsize=(10, 6))
plt.bar(metric_names, metrics)
plt.ylim(0, 1)
plt.ylabel('Score')
plt.title('Model Performance Metrics')
plt.grid(axis='y', alpha=0.3)
plt.savefig('metrics.png')
plt.show()
# Confusion matrix heatmap
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(1 - label, 1 - preds)
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
xticklabels=['Valid', 'Fallacy'],
yticklabels=['Valid', 'Fallacy'])
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.savefig('confusion_matrix.png')
plt.show()
```
## Export Results
Save metrics to file:
```python theme={null}
import json
metrics = get_results(1 - label, 1 - preds)
results_dict = {
'accuracy': float(metrics[0]),
'precision': float(metrics[1]),
'recall': float(metrics[2]),
'f1_score': float(metrics[3])
}
with open('metrics.json', 'w') as f:
json.dump(results_dict, f, indent=2)
```
## Benchmarking
Compare against baseline methods:
```python theme={null}
# Random baseline
from sklearn.dummy import DummyClassifier
dummy = DummyClassifier(strategy='stratified', random_state=42)
dummy.fit(np.zeros((len(label), 1)), label)
baseline_preds = dummy.predict(np.zeros((len(label), 1)))
print("Baseline (Random):")
print(get_results(1 - label, 1 - baseline_preds))
print("\nNL2FOL System:")
print(get_results(1 - label, 1 - preds))
```
## Error Analysis
Identify problem cases:
```python theme={null}
# Find false positives and false negatives
df['pred_binary'] = preds
df['label_inverted'] = 1 - label
false_positives = df[
(df['pred_binary'] == 1) & (df['label_inverted'] == 0)
]
false_negatives = df[
(df['pred_binary'] == 0) & (df['label_inverted'] == 1)
]
print(f"False Positives: {len(false_positives)}")
print(f"False Negatives: {len(false_negatives)}")
# Export for analysis
false_positives.to_csv('false_positives.csv', index=False)
false_negatives.to_csv('false_negatives.csv', index=False)
```
**Common Pitfalls**
1. **Label Confusion**: Ensure consistent encoding (0/1 vs LF/Valid)
2. **Missing Values**: Decide how to handle processing errors
3. **Class Imbalance**: Consider using balanced accuracy or weighted F1
4. **Data Leakage**: Never evaluate on training data
## Next Steps
After evaluating performance:
1. **Error Analysis**: Review false positives and false negatives
2. **Prompt Refinement**: Adjust prompts based on error patterns
3. **Model Comparison**: Test different LLMs or NLI models
4. **Dataset Expansion**: Evaluate on diverse fallacy types
5. **Ablation Studies**: Test impact of each pipeline component
See [Development Guide](/development) for tips on improving the system.