> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lovishchopra/NL2FOL/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Datasets

> Learn how to process your own datasets for logical fallacy detection

## Overview

NL2FOL provides utilities to process datasets of natural language statements for logical fallacy detection. This guide shows you how to work with built-in datasets and create your own.

## Dataset Structure

Datasets in NL2FOL use a simple CSV format with the following columns:

<ParamField path="articles" type="string" required>
  The natural language statement or argument to analyze
</ParamField>

<ParamField path="label" type="integer" required>
  Binary label: `0` for logical fallacy, `1` for valid logical reasoning
</ParamField>

<ParamField path="source_article" type="string">
  Alternative field name for the text content (merged with `articles`)
</ParamField>

<ParamField path="sentence" type="string">
  Alternative field name for the text content (merged with `articles`)
</ParamField>

## Built-in Datasets

NL2FOL includes several pre-configured datasets in the `data/` directory:

<CardGroup cols={2}>
  <Card title="Logic Fallacies" icon="circle-exclamation">
    **File:** `data/fallacies.csv`

    General logical fallacies from various domains with labeled fallacy types.
  </Card>

  <Card title="Climate Fallacies" icon="temperature-high">
    **File:** `data/fallacies_climate.csv`

    Climate change-related arguments with identified logical fallacies.
  </Card>

  <Card title="NLI Fallacies" icon="triangle-exclamation">
    **File:** `data/nli_fallacies_test.csv`

    Natural Language Inference-based fallacious reasoning examples.
  </Card>

  <Card title="NLI Entailments" icon="check-circle">
    **File:** `data/nli_entailments_test.csv`

    Valid logical entailments for comparison and evaluation.
  </Card>
</CardGroup>

## Using Built-in Datasets

The `setup_dataset()` function (defined in `src/nl_to_fol.py:389`) loads and prepares datasets:

```python theme={null}
import pandas as pd
from nl_to_fol import setup_dataset

# Load a balanced dataset of fallacies and valid arguments
df = setup_dataset(fallacy_set='logic', length=100)

print(df.head())
print(f"Dataset shape: {df.shape}")
print(f"Label distribution:\n{df['label'].value_counts()}")
```

### Available Dataset Types

<Tabs>
  <Tab title="logic">
    **General logical fallacies dataset**

    ```python theme={null}
    df = setup_dataset(fallacy_set='logic', length=100)
    ```

    Loads from `data/fallacies.csv` and `data/nli_entailments_test.csv`, creating a balanced dataset with:

    * 100 fallacious arguments (label=0)
    * 100 valid arguments (label=1)
  </Tab>

  <Tab title="logicclimate">
    **Climate change fallacies dataset**

    ```python theme={null}
    df = setup_dataset(fallacy_set='logicclimate', length=50)
    ```

    Loads climate-related arguments from `data/fallacies_climate.csv`:

    * 50 climate fallacies (label=0)
    * 50 valid climate arguments (label=1)
  </Tab>

  <Tab title="nli">
    **NLI-based fallacies dataset**

    ```python theme={null}
    df = setup_dataset(fallacy_set='nli', length=200)
    ```

    Loads from `data/nli_fallacies_test.csv` and `data/nli_entailments_test.csv`:

    * 200 NLI fallacies (label=0)
    * 200 valid NLI entailments (label=1)
  </Tab>

  <Tab title="folio">
    **FOLIO dataset**

    ```python theme={null}
    df = setup_dataset(fallacy_set='folio', length=100)
    ```

    Loads from `data/folio.csv` for first-order logic inference tasks.
  </Tab>
</Tabs>

## Processing a Dataset

Here's a complete example of processing a dataset:

```python process_dataset.py theme={null}
import pandas as pd
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from nl_to_fol import NL2FOL, setup_dataset

def process_custom_dataset():
    # Initialize models (GPT-4 example)
    model_type = 'gpt'
    pipeline = None
    tokenizer = None
    
    nli_model_name = "microsoft/deberta-large-mnli"
    nli_tokenizer = AutoTokenizer.from_pretrained(nli_model_name)
    nli_model = AutoModelForSequenceClassification.from_pretrained(nli_model_name)
    
    # Load dataset
    df = setup_dataset(fallacy_set='logic', length=10)
    
    # Storage for results
    claims = []
    implications = []
    final_lfs = []
    final_lfs2 = []
    
    # Process each row
    for i, row in df.iterrows():
        print(f"Processing {i+1}/{len(df)}...")
        
        nl2fol = NL2FOL(
            sentence=row['articles'],
            model_type=model_type,
            pipeline=pipeline,
            tokenizer=tokenizer,
            nli_model=nli_model,
            nli_tokenizer=nli_tokenizer,
            debug=False  # Set to True for detailed output
        )
        
        lf1, lf2 = nl2fol.convert_to_first_order_logic()
        
        claims.append(nl2fol.claim)
        implications.append(nl2fol.implication)
        final_lfs.append(lf1)
        final_lfs2.append(lf2)
    
    # Add results to dataframe
    df['Claim'] = claims
    df['Implication'] = implications
    df['Logical Form 1'] = final_lfs
    df['Logical Form 2'] = final_lfs2
    
    # Save results
    df.to_csv('results/processed_dataset.csv', index=False)
    print("\nProcessing complete! Results saved to results/processed_dataset.csv")
    
    return df

if __name__ == "__main__":
    results = process_custom_dataset()
    print(results[['articles', 'label', 'Claim', 'Implication']].head())
```

<Note>
  The `setup_dataset()` function automatically balances the dataset by sampling an equal number of fallacies and valid arguments.
</Note>

## Creating Custom Datasets

<Steps>
  <Step title="Prepare your CSV file">
    Create a CSV file with the required columns:

    ```csv custom_fallacies.csv theme={null}
    articles,label,fallacy_type
    "All politicians lie. Sarah is a politician. Therefore Sarah lies.",0,hasty_generalization
    "If it rains, the ground is wet. The ground is wet. Therefore it rained.",0,affirming_consequent
    "All mammals have lungs. Whales are mammals. Therefore whales have lungs.",1,valid_syllogism
    "Either we ban cars or pollution will kill us all.",0,false_dilemma
    ```
  </Step>

  <Step title="Load your dataset">
    Use pandas to load and prepare your data:

    ```python theme={null}
    import pandas as pd

    # Load custom dataset
    df = pd.read_csv('custom_fallacies.csv')

    # Ensure required columns exist
    if 'articles' not in df.columns:
        df['articles'] = df['sentence']  # or other text column

    if 'label' not in df.columns:
        df['label'] = 0  # default to fallacy if not specified

    print(f"Loaded {len(df)} examples")
    ```
  </Step>

  <Step title="Process your dataset">
    Use the same processing loop as shown above:

    ```python theme={null}
    for i, row in df.iterrows():
        nl2fol = NL2FOL(
            sentence=row['articles'],
            model_type='gpt',
            pipeline=None,
            tokenizer=None,
            nli_model=nli_model,
            nli_tokenizer=nli_tokenizer
        )
        
        lf1, lf2 = nl2fol.convert_to_first_order_logic()
        # Store results...
    ```
  </Step>
</Steps>

## Batch Processing with Command Line

For large datasets, use the command-line interface:

```bash theme={null}
python src/nl_to_fol.py \
  --model_name gpt-4o \
  --nli_model_name microsoft/deberta-large-mnli \
  --run_name my_experiment \
  --length 500 \
  --dataset logic
```

### Command-Line Arguments

<ParamField path="--model_name" type="string" required>
  Model name for text generation (`gpt-4o`, `meta-llama/Llama-2-13b-hf`, etc.)
</ParamField>

<ParamField path="--nli_model_name" type="string" required>
  HuggingFace model name for NLI (e.g., `microsoft/deberta-large-mnli`)
</ParamField>

<ParamField path="--run_name" type="string" required>
  Name for the output CSV file (saved to `results/{run_name}.csv`)
</ParamField>

<ParamField path="--length" type="integer" required>
  Number of examples to process from each class (total will be 2x this)
</ParamField>

<ParamField path="--dataset" type="string" required>
  Dataset type: `logic`, `logicclimate`, `nli`, or `folio`
</ParamField>

## Output Format

Processed datasets are saved with the following additional columns:

```python theme={null}
Output Columns:
- Claim: Extracted claim from the sentence
- Implication: Extracted implication
- Referring Expressions - Claim: Entities in the claim
- Referring Expressions - Implication: Entities in the implication
- Property Implications: Property relationships
- Equal Entities: Entity equivalences
- Subset Entities: Subset relationships
- Claim Lfs: Logical form of the claim
- Implication Lfs: Logical form of the implication
- Logical Form: Final first-order logic formula (method 1)
- Logical Form 2: Final first-order logic formula (method 2)
```

## Example Output

<CodeGroup>
  ```csv Input theme={null}
  articles,label
  "All birds fly. Penguins are birds. Thus penguins fly.",0
  ```

  ```csv Output theme={null}
  articles,label,Claim,Implication,Logical Form
  "All birds fly. Penguins are birds. Thus penguins fly.",0,"All birds fly","Penguins fly","(forall a (Bird(a) and Fly(a))) -> (exists b (Fly(b)))"
  ```
</CodeGroup>

## Dataset Statistics

Analyze your processed dataset:

```python theme={null}
import pandas as pd

df = pd.read_csv('results/my_experiment.csv')

print("Dataset Statistics:")
print(f"Total examples: {len(df)}")
print(f"Fallacies: {(df['label'] == 0).sum()}")
print(f"Valid arguments: {(df['label'] == 1).sum()}")
print(f"\nSuccessful conversions: {df['Logical Form'].notna().sum()}")
print(f"Failed conversions: {df['Logical Form'].isna().sum()}")

# Average formula complexity
df['formula_length'] = df['Logical Form'].str.len()
print(f"\nAverage formula length: {df['formula_length'].mean():.1f} characters")
```

## Working with Multiple Datasets

Combine multiple datasets for comprehensive analysis:

```python theme={null}
import pandas as pd
from nl_to_fol import setup_dataset

# Load multiple dataset types
logic_df = setup_dataset(fallacy_set='logic', length=100)
climate_df = setup_dataset(fallacy_set='logicclimate', length=50)
nli_df = setup_dataset(fallacy_set='nli', length=150)

# Add source labels
logic_df['source'] = 'logic'
climate_df['source'] = 'climate'
nli_df['source'] = 'nli'

# Combine
combined_df = pd.concat([logic_df, climate_df, nli_df], ignore_index=True)

print(f"Combined dataset size: {len(combined_df)}")
print(combined_df['source'].value_counts())
```

## Performance Considerations

<Warning>
  Processing large datasets can be time-consuming:

  * GPT-4: \~15-20 seconds per example (API rate limits apply)
  * Llama 13B: \~5-10 seconds per example (GPU-dependent)

  For a 1000-example dataset:

  * GPT-4: \~4-6 hours
  * Llama 13B: \~2-3 hours
</Warning>

### Optimization Tips

<Steps>
  <Step title="Batch processing">
    Process datasets in smaller batches and save intermediate results:

    ```python theme={null}
    batch_size = 50
    for i in range(0, len(df), batch_size):
        batch_df = df.iloc[i:i+batch_size]
        # Process batch...
        batch_df.to_csv(f'results/batch_{i}.csv', index=False)
    ```
  </Step>

  <Step title="Enable multiprocessing">
    For Llama models, process multiple examples in parallel if you have multiple GPUs.
  </Step>

  <Step title="Cache results">
    Store intermediate results to avoid reprocessing on failures:

    ```python theme={null}
    if os.path.exists('cache/intermediate.csv'):
        df = pd.read_csv('cache/intermediate.csv')
    ```
  </Step>

  <Step title="Use debug mode selectively">
    Only enable `debug=True` for troubleshooting, not production runs.
  </Step>
</Steps>

## Next Steps

<CardGroup cols={2}>
  <Card title="SMT Solving" icon="gears" href="/usage/fol-to-smt">
    Convert logical forms to SMT and verify with CVC5
  </Card>

  <Card title="Evaluation" icon="chart-line" href="/usage/evaluation">
    Measure accuracy and performance metrics
  </Card>

  <Card title="Model Backends" icon="microchip" href="/examples/model-backends">
    Choose the right model for your dataset
  </Card>

  <Card title="API Reference" icon="code" href="/api/nl2fol">
    Explore advanced configuration options
  </Card>
</CardGroup>