How to use Promposer

A practical guide to prompt engineering, optimization, and evaluation

Prompts and Revisions

Prompts are the instructions you give to an AI model to perform a specific task. In Promposer, prompts are the foundation of your workflow.

On the home page, you'll find the prompt input with three modes:

Optimize and follow up: Optimize your prompt using best practices.
Follow up: Save your prompt as-is, without AI optimization.
Evaluate: Add your prompt and immediately proceed to set up test cases for evaluation.

You can switch between Follow up and Evaluate modes at any time.

What it does: Saves your prompt and creates an optimized revision using AI.
Revision Navigation: Browse between different revisions of your prompt.
Manual Revision: Edit or delete any revision. When editing, you can either update the current revision or create a new one based on your changes.
Copy Button: Instantly copy the current prompt text to your clipboard.
AI Revision: Use the "AI Revision" section to describe issues or desired improvements. The AI will generate a new, improved revision based on your feedback.

What it does: Saves your prompt exactly as you entered it, without any AI optimization.
Other features: All revision management options (edit, delete, copy, AI revision) are available, just like in the "Optimize and follow up" mode.
Switching modes: You can switch to "Evaluate" at any time to start testing your prompt.

Evaluations allow you to systematically test and measure the effectiveness of your prompts across different scenarios, tools, and models.

When you choose Evaluate, you'll define an evaluation template with the following sections:

Each test case defines a scenario to test your prompt. A test case includes:

Name: A descriptive name for the case.
Scenario: The situation or persona the AI should follow.
Expected Result: The outcome that determines if the test is successful.
Response Threshold: The maximum number of responses allowed to reach the expected result. If the threshold is reached before success, the case is considered failed.
Use Tools: Choose to use all available tools or select specific ones.
Use Context: Choose to use all available contexts or select specific ones.

Test cases can be added manually or generated automatically with AI.

Tools simulate external functions or APIs the AI can use during evaluation.

Name: Function-style identifier.
Description: Explains the tool's purpose to the AI.
Parameters: List of parameters (each with name, description, and "required" checkbox).
Response Mode:
- Realtime AI: The tool generates a sample response using AI in real time (good for simulating helpful tools).
- Manual: You define a list of possible responses for specific conditions (good for simulating errors or edge cases).
Responses (Manual mode): Each response includes:
- Conditions: Arguments, values, and matching operators.
- Return: The response content to be returned if conditions are met.

Tools can be added manually or generated automatically with AI.

Add extra context to be sent with the prompt. This can be any relevant information that helps the AI perform better in the scenario.

Contexts can be added manually or generated automatically with AI.

Select which large language models (LLMs) to use for evaluation. Some models are "premium" and available only to PRO accounts.

Access Mode: Choose between:
- Managed API access: Use platform-hosted API keys (uses your credits).
- Own API access: Use your own API keys (no extra charge, available to premium users).

Define your prompt (using any of the three modes).
Set up test cases to simulate real-world scenarios.
Configure tools, context, and models as needed.
Run the evaluation. Promposer will simulate conversations between an "assistant" (your prompt) and an "evaluator" (the test scenario).
Review results. Each test case will show if the expected result was achieved within the response threshold.

Prompt: The instructions for the AI.
Revision: A version of your prompt, either manually edited or AI-generated.
Test Case: A scenario to test your prompt.
Tool: A simulated function/API the AI can use.
Context: Additional information provided to the AI.
Model: The LLM used for evaluation.
Response Threshold: Max number of responses allowed to achieve the expected result.
Access Mode: Determines which API keys are used for evaluation.

To install the Promposer VSIX extension, follow these steps: