You're an expert at... using Claude's Workbench

Introduction

“You are an expert in” is a well known prompting ritual, and a standard piece of advice - but is it effective? Recent research describes it as “not working”.

At the same time, the Anthropic API documentation says that “With a role, Claude catches critical issues that could cost millions”… so it’s a potentially high stakes question.

This is a good excuse for us to try out the new Anthropic Developer Console that lets us set up Prompts and Evaulations for testing. So, let’s delve right in.

Testing

1. Review the API documentation

There are a couple of examples of Role Prompting supplied in the API Documentation: Legal contract analysis and Financial analysis. The examples demonstrate a strong difference in outputs.

We’ll use the Financial analysis examples for our test.

2. Setup and Run a Test

Lets put the Financial analysis prompt straight in to the new Workbench (Anthropic Console account required).

We’ll use Sonnet 3.5, and set the Temperature to the API Default of 1. Although the documentation instructs us to supply the role as the “System” parameter, the examples show “User” interactions, so that’s how we’ll set it up. The System Prompt is left blank.

We don’t have access to the “dataset” referenced in the examples, but fortunately one of the scenarios in our Advanced Product Management course is for a SaaS company.

We’ll use ChatGPT’s data analysis tools to produce a financial statement for Claude to analyse.

Now that we’ve got some financial data, we enter it in to the {{FINANCIALS}} variable and run the prompt. The “Evaluate” option is then enabled:

In this view, we have the {{FINANCIALS}} variable in the first column, with the response from Sonnet in the Second column. Tests are arranged in rows - each row can have different variables so we can compare outputs based on different scenarios.

There are a couple of cool features here. The “Generate Test Case” button populates the new row with a new financial scenario. In this case, it’s a company called “GreenGrow Agritech Solutions” who are in a much better financial position than “Anthropic Overload Systems”. It didnt’ just copy the format directly either, the new sample includes COGS and Debt-to-Equity ratios which the original financial analysis didn’t. There’s also a bulk import function if we already have data we want to load and try.

We can also manually rate the quality of each response:

3. Adjust the Roles

Of course, it’s actually the instructions in the prompt we want to change - so we’ll need to adjust our prompt template to make the Preamble and Summarisation parts of the prompt variables too:

{{ROLE-PREAMBLE}}
<data>
{{FINANCIALS}}
</data>
{{ROLE-SUMMARY}}

Variables that haven’t been supplied yet are indicated in Red in the Prompt Tool.

We’ll supply the sections from the “Without Role” Financial analysis example prompt, and run the test:

Going to the results (evals) table, we can see that the Console has applied a version number to our prompt. Next to “v1” is an average of the ratings we had given the outputs. It’s also stored the result of our “previous Without role” test run.

We can now enter a new row, and supply the variables to use the “Financial analysis with role prompting” preamble and summary from the API example.

That’s produced 2 outputs, but which is better?

4. Assess the Outputs

By eye, I prefer the second one.

But I’d prefer not to do manual analysis, so let’s produce a scoring rubric.

In our initial “Financial Analyst” prompts, there were directions to produce Actions and identify Trends in both examples, so I’ve given that as a hint.

To get as many ideas and perspective as possible, we’ll use Big-AGI to combine the answers from 4 different models (Sonnet 3.5, GPT-4o, Opus 3 and Qwen2-7b).

The ensembling produces us the following scoring rubric. The parts in bold I’ve added in preparation for the next step - running the scoring.

I’m sure that can be refined a bit further, but it’ll do for now. We’ll run it a few times to see if it gives consistent results.

Let’s score the outputs to see if there is a winner…

5. Score the Outputs

I don’t fancy pasting all this in to Claude.ai, so we’ll set the scoring prompt up in the Workbench as well.

I’ll use the above “Scoring Rubric” prompt as the System Prompt, and add a User Prompt: Score this analysis {{ANALYSIS}}. That makes it easy to run the scoring a few times, see how stable the scores are, and what differences there are between the two scenarios.

This will also give us a fresh context each time, ensuring that previous scoring and outputs won’t influence future ones.

Let’s start by scoring the “Without a role” outputs:

We’ll run it three times, at both Temperature 0 (Console Default) and Temperature 1 (API Default). A very nice touch in the Anthropic Workbench is that it treats changing the temperature setting as a new version of the prompt.

The rubric is stable at producing scores for the input, and the temperature adjustment doesn’t cause too much fluctuation either.

Let’s now score the outputs produced by the “With a role” prompt:

Again, the rubric gives us stable scores, and it’s sensitive to the input it’s assessing, which is a good sign.

OK - maybe we will paste it in to Claude.ai after all as a quick cross-check (82/100):

Taking the mean of those 6 evaluations, “Without a role” scored 70.1, “With a role” scored 84.1.

The Rubric creation prompt was careful not to match the initial prompts - there are no words in the Rubric creation prompt that aren’t in both the Example prompts, so the rubric shouldn’t be primed to fit one or the other outputs unfairly.

Against our generated rubric, “With a role” wins.

Conclusion

Anthropic Console Workbench

The Anthropic Workbench was fun to work with, and made the tests easy to set up. The inbuilt versioning, test data generation, and ability to store test results to avoid token-sapping re-runs is welcome.

Anything which moves the field of using LLMs from ritual to empiricism is a good thing, and this is an accessible starting point available to everyone with an Anthropic Developer account. Of course, this is another potential source of lock-in for Anthropic customers, but that’s a discussion for a different day.

There are 2 features missing that will improve the utility of the workbench, and surely they are coming soon:

• Model Grading of Outputs: The ability to grade outputs by supplying a snippet of code or rubric would be very convenient.
• Crowdsourced User Grading: The ability to crowdsource qualitative scoring over last large Test Runs would also be very helpful.

Role Prompting (or Scenario Building) - Overrated?

Is it time to declare the end of Role Prompting? For straightforward prompts and queries, I think this has been redundant against newer models (since GPT-4). The example used in this article was conducting a fairly complex open-ended task, but specifying a role did improve performance. The risk here is that for this type of task, evaluating the difference over a few runs by eye is hard, and the results “Without role” already look good. Incremental imrpovements from there are not going to be immediately obvious - and I fear ritualistically avoiding Role Prompting “because it doesn’t work” could end up being harmful.

The example here did more than just change the Role in the prompt though: it incorporated a scenario. The LLM is always predicting the next token ahead based on the entire context, so I think of it like constructing a novel. The scenario of the board meeting and wary investors describes a scene - that the LLM then duly narrates. If the scenario was changed to “You’re a CFO relaxing on the beach with a cocktail”, we’d probably get a different quality of analysis. Getting the context right so the LLM can author the correct scene is perhaps a more useful model than having it pretend it’s a person. I hypothesize that we could change or add roles to the original prompt, and it wouldn’t change the scores much, if at all. However, we have previously demonstrated that Claude models are sensitive to the presence of even a simple “You are a helpful AI Assistant” System Prompt.

These are all worthy experiments worth trying as we continue to increase the value received from LLMs.