Eval Quickstart

Prerequisites

Before you start, you’ll need to install Tatara. If you haven’t yet, you can get set up by following the instructions in the installation guide.

Grokking an Eval

An Eval is a function that takes in a dataset, which contains a model’s outputs, and a function to evaluate that dataset. The Eval gives you a score for the model’s outputs. The score can be an int, float, bool, or categorical value (string). All Evals are instances of the Eval class.

Writing an Eval

Let’s start by making the necessary imports and by creating an OpenAI client, which we’ll use to call GPT-3.5, who will be responsible for our Eval.

from tatara import tatara, Eval, Record, init_dataset, run_evals
from openai import OpenAI
from tatara import Record

model_name = "gpt-3.5-turbo-1106"
freq_penalty = 0.1
temperature = 0.1
max_tokens = 200
openai_client = OpenAI()

Now we’ll write a function to create a dataset which will contain the data we want to evaluate.

def create_dataset():
    ds = init_dataset("test_dataset")
    new_records = [
    {"input":"Generate a description of a character named Cyborg Guy. Cyborg Guy is a cyberpunk robot who lives in the city of Zail.",
    "output":"You are a cyborg living in the the futuristic city of Zail. You have a bionic arm and a holoband. You're walking down the dark city streets while neon lights flash brightly above you."}
    ]
    ds.insert(new_records)

Now we’ll write a function to check if the output of the model is cyberpunk fiction. We’ll use GPT-3.5 to do this.

def is_cyberpunk_check(record: Record) -> bool:
    prompt = "Check that the following text is an example of cyberpunk fiction: \n\n {}\n\n If it is cyberpunk, write <CYBERPUNK>. If it is not, write <NOT_CYBERPUNK>.".format(record['output'])
    response = openai_client.chat.completions.create(
        messages=[{"role": "user", "content": prompt}],
        model=model_name,
        temperature=temperature,
        frequency_penalty=freq_penalty,
        max_tokens=max_tokens,

    )
    ret = response.choices[0].message.content
    if "<CYBERPUNK>" in ret:
        return True
    elif "<NOT_CYBERPUNK>" in ret:
        return False
    else:
        # If the model doesn't output a clear answer, we it again until it does
        return is_cyberpunk_check(record)

And finally, we’ll run it.

if __name__ == "__main__":
    my_test_project = "test_project"
    tatara.init(project=my_test_project, flush_interval=1, queue_size=1)

    create_dataset()

    cyberpunk_check = Eval(
            name = "is_cyberpunk_check",
            description = "Checks whether the text is cyberpunk fiction. If it is, it returns True. If it is not, it returns False.",
            eval_fn = is_cyberpunk_check,
        )

    run_evals([cyberpunk_check], "test_dataset", print_only=True)

In summary, we’ve created a dataset with a single record, which contains a prompt and a model output. We then implemented an eval that checks whether the output of a model call contains cyberpunk content. We then ran the eval over the dataset using run_evals. If you run this code, it will print the results of the eval to the console and you’ll see something like this:

{
  "id": "er_2fb61f79-1645-4640-98a2-e3668f9ecbf0",
  "dataset": {
    "id": "d_e4953ac5-6771-4084-9c89-5fde4a75a61f",
    "name": "test_dataset"
  },
  "evals": [
    {
      "name": "is_cyberpunk_check",
      "description": "Checks whether the text is cyberpunk fiction. If it is, it returns True. If it is not, it returns False.",
      "eval_result_type": "bool"
    }
  ],
  "num_rows": 1,
  "eval_rows": [
    {
      "id": "err_8885540b-ac63-4a87-8599-1190e3d483db",
      "eval_run_id": "er_2fb61f79-1645-4640-98a2-e3668f9ecbf0",
      "record_id": "r_eea4829a-7da2-4810-9d37-6fe7be22201a",
      "input": "Generate a description of a character named Cyborg Guy. Cyborg Guy is a cyberpunk robot who lives in the city of Zail.",
      "output": "You are a cyborg living in the the futuristic city of Zail. You have a bionic arm and a holoband. You're walking down the dark city streets while neon lights flash brightly above you.",
      "eval_results": [{ "is_cyberpunk_check": true }]
    }
  ],
  "timestamp": 1707200867
}

The dataset will need to be created before running the evals. You can do this using the init_dataset function, which you can import with from tatara.datasets import init_dataset

Viewing the Results

Once run_evals is done running, you’ll be able to view the results in the console. To send them to Tatara, remove print_only=True in the run_evals call above. Once you run the code again, head over to the Tatara UI to view your Eval run.

Get Started

Concepts

Eval Quickstart

Prerequisites

Grokking an Eval

Writing an Eval

Viewing the Results

​Prerequisites

​Grokking an Eval

​Writing an Eval

​Viewing the Results

Prerequisites

Grokking an Eval

Writing an Eval

Viewing the Results