Claude 3 Suite - Casual Prompting Performance

Introduction

Let’s find out how the Anthropic Claude 3 Models handle a Casual Prompt.

This is a prompt that a well-informed, casual user of an AI front-end might reasonably use, expecting it to be interpreted as another human would. These types of prompts are typically used for tasks such as market analysis, product concepts and summarization.

Understanding how LLMs handle this type of prompt is crucial for assessing their real-world applicability and reliability.

Our prompt will introduce two personas - Alice and Bob, provide an article, and ask the model to produce a “suitability score” out of 10 for each of the personas.

We’ll supply a dry, technical and factual article on the topic of Internet Security Regulation.

Looking at the personas, and the article, I’d guess a suitability score of between 7 and 8 for Alice, and between 3 and 4 for Bob. It’s not perfect for Alice - neither is it completely useless for Bob.

I’d expect that:

• The score for Alice should always be higher than for Bob.
• The difference between the scores should be somewhere between 3 and 5.

We’ll ask the LLM to score the article for Alice and Bob, and return the scores in an easy to extract format:

The final prompt for testing therefore looks like this:

With our prompt defined, let’s explore how different LLMs interpret and respond to this casual everyday query.

Initial Testing

For our first test, we’ll compare the performance of the most recent Anthropic suite of Models: Haiku 3, Sonnet 3 and Opus 3. The details, including the price at the time of testing (June 2024) are below:

Note that Opus is exactly 60 times more expensive than Haiku 3 for both input and output.

Setup

To start, we will run with default settings (temperature set to 1) and an empty system prompt. For each model, we’ll run the prompt 50 times, and calculate a “Score Difference” by subtracting Bobs score from Alices score.

There is no “correct” score for this test, but we expect all score differences to be positive, and that scores closer to 5 match the intent of the prompt.

Results

Here are the results with No System Prompt set. The X axis shows the frequency of the calculated Score Differences (Alices Score - Bobs Score) across the 50 runs.

The results show a clear trend: the more expensive models produce score differences closer to our expected range of 3-5. Haiku 3, the cheapest model, mostly produced smaller differences, while Opus 3, the most expensive, consistently gave differences of 4 or 5.

Straight away we can see a direct correlation between model price and performance at this task.

Initial Observations

These results not only highlight performance differences between models but also validate our initial test assumptions in a delightfully circular manner. We designed the test expecting more sophisticated models to produce score differences closer to our 3-5 range, and that is what we found.

Nevertheless, the clear correlation between model sophistication, performance, and consistency provides valuable insights for those navigating the trade-offs between cost and capability in real-world applications.

With the baseline results established, lets see what effect tweaking the system prompt has…

A Helpful AI assistant

Lets run the same test, but this time with a minimal system prompt : “You are a helpful AI assistant”.

Results

Here are the results of that test:

Click the buttons below to compare models with and without system prompt:

We will label the conditions “Empty” for No System Prompt and “Helpful” for “You are a helpful AI assistant” for brevity.

Persona Scoring

Here are the mean scores differences and changes in tabular format:

Both Haiku 3 and Opus 3 show a clear benefit from having a system prompt, with improvements in their ability to differentiate article suitability between personas. Surprisingly, Sonnet 3 displays a slight negative effect when given a system prompt.

This is unexpected - especially as the other models in the same suite behave differently. We’ll double check the API request to make sure the message is sent:

  {"model":"claude-3-sonnet-20240229",
    "system":"You are a helpful AI assistant","max_tokens":1024,
    "messages":[
        {"role":"user","content":[
        {"type":"text","text":"Alice is an extremely technical person..."}]}
    ],
  "stream":false,"temperature":1}

Here are the mean scores for Alice and Bob across the 50 runs:

The addition of a System Prompt affects models differently. For Haiku and Sonnet, it increases suitability scores for both personas, though Haiku’s improvement in score differentiation comes at the cost of an inflated baseline for Bob. Opus, however, behaves distinctly: it maintains Alice’s high score while further lowering Bob’s score, which is more aligned with our expectations and thus more useful.

Sonnet and Opus agree on Alice’s score of 8, with minimal impact from the prompt. The key difference lies in their scoring for Bob, for whom the article is less suitable, with Opus showing the most pronounced and beneficial differentiation.

Notably, Opus 3 demonstrates higher consistency in its scoring, showing less variance across both the “No System Prompt” and “Helpful AI Assistant” conditions.

Verbosity

The model, and addition of a System Prompt also show some marked differences to the verbosity (Completion Token usage) of the response. The below chart summarises the data from the performance runs.

Click the buttons below to compare models with and without system prompt:

Here’s a summary of the token output statistics:

Opus earns its reputation of producing verbose output, and remains consistent across the scenarios. Its responses cluster predominantly in the 151-200 token range, rarely dipping below 100 tokens.

Haiku is particularly sensitive to the addition of a System Prompt, improving score difference performance and shifting from extremely terse to relatively chatty. This sensitivity could be crucial for fine-tuning the more economical models.

Sonnet, intriguingly, shows an opposite trend. The “Helpful AI Assistant” prompt actually decreases its verbosity, with the median token count remaining low at 17.

Testing Costs

The table below summarizes the input token usage for our experiments, comparing scenarios with and without a system prompt across 50 runs:

Adding the output tokens then allows us to calculate our overall cost per scenario:

While Haiku’s list price is 1/60th of Opus’s, with no System Prompt the difference is actually 86 times due to the difference in verbosity. The addition of the system prompt closes the gap to 64 times - still higher than the list price would indicate.

Summary and Implications

These results underscore the importance of testing prompts with specific models to optimize performance in real-world applications. They also highlight that even within a family of models, individual AIs may respond differently to the same prompting strategies.

Opus performs well at this task, and is the clear winner - it appears to understand the user intent, and the addition of a System Prompt tunes it to further differentiate the personas. Notably this superior performance comes with increased verbosity, leading to higher token usage and costs compared to the other models.

This test reinforces that Anthropic’s Suite has a clear price/performance distinction for their models, particularly for certain creative tasks like content creation and product management. While Opus’s cost may seem high, it’s negligible compared to manual content creation expenses. Opting for lower-priced models could lead to increased human effort or subtly lower quality outputs, potentially negating any perceived savings. For high-stakes content creation or analysis tasks, the relative cost difference between models is often insignificant compared to the potential impact of the output quality.

Both the User and System prompt can be tuned to improve performance of the lower models, but our goal here was to understand the type of performance a Casual Prompter would experience when interacting with a Large Language Model. This approach provides valuable insights into real-world applicability and the trade-offs between model capabilities and costs.