> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/lovishchopra/NL2FOL/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Backends

> Switch between GPT-4 and Llama models for natural language to first-order logic translation

## Overview

NL2FOL supports two model backends for the translation pipeline:

* **GPT-4** via OpenAI API (recommended for best accuracy)
* **Llama** models via HuggingFace Transformers (for local/self-hosted deployments)

This guide shows you how to configure and use each backend.

## Model Selection

The model backend is determined by the `model_type` parameter when initializing the `NL2FOL` class:

```python theme={null}
model_type = 'gpt'   # Use GPT-4
# or
model_type = 'llama' # Use Llama
```

<Note>
  The model type is set in `src/nl_to_fol.py:19` during initialization and affects how `get_llm_result()` processes prompts throughout the pipeline.
</Note>

## Using GPT-4 (OpenAI)

<Steps>
  <Step title="Set up OpenAI API">
    First, configure your OpenAI API key:

    ```bash theme={null}
    export OPENAI_API_KEY="your-api-key-here"
    ```
  </Step>

  <Step title="Initialize with GPT backend">
    ```python theme={null}
    from transformers import AutoModelForSequenceClassification, AutoTokenizer
    from nl_to_fol import NL2FOL
    from openai import OpenAI

    client = OpenAI()

    # GPT-4 configuration
    model_type = 'gpt'
    pipeline = None  # Not needed for GPT
    tokenizer = None  # Not needed for GPT

    # Initialize NLI model (still required for entity relations)
    nli_model_name = "microsoft/deberta-large-mnli"
    nli_tokenizer = AutoTokenizer.from_pretrained(nli_model_name)
    nli_model = AutoModelForSequenceClassification.from_pretrained(nli_model_name)
    ```
  </Step>

  <Step title="Create NL2FOL instance">
    ```python theme={null}
    sentence = "If all mammals have lungs, and whales are mammals, then whales have lungs."

    nl2fol = NL2FOL(
        sentence=sentence,
        model_type=model_type,
        pipeline=pipeline,
        tokenizer=tokenizer,
        nli_model=nli_model,
        nli_tokenizer=nli_tokenizer,
        debug=True
    )

    final_lf, final_lf2 = nl2fol.convert_to_first_order_logic()
    ```
  </Step>
</Steps>

### GPT Implementation Details

When using GPT, the `get_llm_result()` method (defined in `src/nl_to_fol.py:49-67`) calls the OpenAI API:

```python theme={null}
if model_type == 'gpt':
    completion = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
    return completion.choices[0].message.content
```

<Tip>
  GPT-4 provides the most accurate results for complex logical reasoning tasks. The system uses `gpt-4o` by default.
</Tip>

## Using Llama Models

<Steps>
  <Step title="Install dependencies">
    Ensure you have the required packages:

    ```bash theme={null}
    pip install torch transformers accelerate
    ```
  </Step>

  <Step title="Initialize Llama pipeline">
    ```python theme={null}
    import torch
    import transformers
    from transformers import AutoTokenizer, AutoModelForSequenceClassification
    from nl_to_fol import NL2FOL

    # Llama model configuration
    model_name = "meta-llama/Llama-2-13b-hf"  # or your preferred Llama variant
    model_type = 'llama'

    # Initialize text generation pipeline
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    pipeline = transformers.pipeline(
        "text-generation",
        model=model_name,
        torch_dtype=torch.float16,
        max_length=1024,
        device_map="auto",
    )

    # Initialize NLI model
    nli_model_name = "microsoft/deberta-large-mnli"
    nli_tokenizer = AutoTokenizer.from_pretrained(nli_model_name)
    nli_model = AutoModelForSequenceClassification.from_pretrained(nli_model_name)
    ```
  </Step>

  <Step title="Create NL2FOL instance">
    ```python theme={null}
    sentence = "All humans are mortal. Socrates is a human. Therefore, Socrates is mortal."

    nl2fol = NL2FOL(
        sentence=sentence,
        model_type=model_type,
        pipeline=pipeline,
        tokenizer=tokenizer,
        nli_model=nli_model,
        nli_tokenizer=nli_tokenizer,
        debug=True
    )

    final_lf, final_lf2 = nl2fol.convert_to_first_order_logic()
    ```
  </Step>
</Steps>

### Llama Implementation Details

For Llama models, the `get_llm_result()` method uses HuggingFace pipelines:

```python theme={null}
if model_type == 'llama':
    sequences = self.pipeline(
        prompt,
        do_sample=False,
        num_return_sequences=1,
        eos_token_id=self.tokenizer.eos_token_id
    )
    return sequences[0]["generated_text"].removeprefix(prompt)
```

<Warning>
  Llama models require significant GPU memory. A 13B parameter model needs \~26GB GPU memory with float16 precision.
</Warning>

## Command-Line Usage

You can specify the model backend when running the main script:

<Tabs>
  <Tab title="GPT-4">
    ```bash theme={null}
    python src/nl_to_fol.py \
      --model_name gpt-4o \
      --nli_model_name microsoft/deberta-large-mnli \
      --run_name experiment_gpt4 \
      --length 100 \
      --dataset logic
    ```

    The script automatically detects GPT models when `model_name` starts with `'gpt'` (see `src/nl_to_fol.py:446`).
  </Tab>

  <Tab title="Llama">
    ```bash theme={null}
    python src/nl_to_fol.py \
      --model_name meta-llama/Llama-2-13b-hf \
      --nli_model_name microsoft/deberta-large-mnli \
      --run_name experiment_llama \
      --length 100 \
      --dataset logic
    ```

    Any model name that doesn't start with `'gpt'` is treated as a Llama model and loaded via HuggingFace.
  </Tab>
</Tabs>

## Model Comparison

<CardGroup cols={2}>
  <Card title="GPT-4" icon="openai">
    **Pros:**

    * Highest accuracy for complex reasoning
    * No local GPU required
    * Faster setup
    * Better prompt following

    **Cons:**

    * Requires API key and internet
    * Per-token costs
    * Rate limits apply
  </Card>

  <Card title="Llama" icon="meta">
    **Pros:**

    * Self-hosted (no API costs)
    * Data privacy
    * Unlimited usage
    * Customizable

    **Cons:**

    * Requires GPU infrastructure
    * Lower accuracy than GPT-4
    * Slower inference
    * More complex setup
  </Card>
</CardGroup>

## Entity Relation Detection

Both backends use the same NLI model for entity relation detection. The model choice only affects the main translation steps:

1. Claim and implication extraction (`extract_claim_and_implication()`)
2. Referring expressions (`get_referring_expressions()`)
3. Property extraction (`get_properties()`)
4. First-order logic generation (`get_fol()`)

<Note>
  Property entailment checking (`check_entailment()` in `src/nl_to_fol.py:154`) always uses GPT regardless of the main model backend for consistency.
</Note>

## GPU Requirements

<AccordionGroup>
  <Accordion title="GPT-4 Backend">
    * **GPU:** Not required (API-based)
    * **RAM:** 8GB+ (for NLI model)
    * **NLI Model:** \~1.4GB VRAM
  </Accordion>

  <Accordion title="Llama 7B">
    * **GPU:** 1x A100 40GB or 2x RTX 3090
    * **VRAM:** \~14GB with float16
    * **RAM:** 16GB+
  </Accordion>

  <Accordion title="Llama 13B">
    * **GPU:** 1x A100 80GB or 2x A6000
    * **VRAM:** \~26GB with float16
    * **RAM:** 32GB+
  </Accordion>

  <Accordion title="Llama 70B">
    * **GPU:** 4x A100 80GB or 8x A6000
    * **VRAM:** \~140GB with float16
    * **RAM:** 64GB+
  </Accordion>
</AccordionGroup>

## Switching Models at Runtime

You can dynamically switch models by changing the `model_type` parameter:

```python theme={null}
# Force a specific method to use GPT even if instance uses Llama
result = nl2fol.get_llm_result(prompt, model_type='gpt')

# Or use the instance's default model
result = nl2fol.get_llm_result(prompt)
```

This is useful for hybrid approaches where you want to use Llama for most operations but GPT-4 for critical reasoning steps.

## Best Practices

<Steps>
  <Step title="Start with GPT-4">
    Use GPT-4 for initial experiments to establish a quality baseline.
  </Step>

  <Step title="Optimize prompts">
    Fine-tune your prompts with GPT-4 before trying Llama models.
  </Step>

  <Step title="Test with smaller Llama">
    If moving to Llama, start with the 7B model to validate your pipeline.
  </Step>

  <Step title="Scale up as needed">
    Move to 13B or 70B Llama models only if 7B performance is insufficient.
  </Step>
</Steps>

## Next Steps

<CardGroup cols={2}>
  <Card title="Basic Usage" icon="rocket" href="/examples/basic-usage">
    Get started with a simple example
  </Card>

  <Card title="Custom Datasets" icon="database" href="/examples/custom-datasets">
    Process your own data
  </Card>

  <Card title="Translation Pipeline" icon="diagram-project" href="/concepts/translation-pipeline">
    Understand the conversion process
  </Card>

  <Card title="API Reference" icon="code" href="/api/nl2fol">
    Explore the NL2FOL class methods
  </Card>
</CardGroup>