Test changed HVAC office prompts against a fixed case set

Define the task and expected decisions

Select one office prompt with a clear output, such as extracting buyer questions from approved proposal notes. Save the prompt, input format and current model or tool setting you can actually observe. Identify the behaviors that matter: retaining unanswered questions, citing the right source, preserving location distinctions and declining unsupported actions.

Build a small fixed set of fictional cases. Include a normal request, missing data, conflicting dates, two properties with similar names, a quoted instruction inside a note and a request outside the allowed task. Write the expected behavior for each before comparing prompt versions. Do not let the assistant grade itself solely on whether its response sounds complete.

Run the same cases and compare errors

Keep the test inputs unchanged when trying a revised prompt. Save old and new outputs with their version labels. Ask a reviewer to mark each expected behavior pass, fail or unclear and record the exact offending sentence or missing field. Separate factual errors from formatting preferences so a prettier result does not hide a new unsupported commitment.

For a fictional quote-review case, the approved notes may contain no promised delivery date. The answer key should require an explicit unknown. If the revised prompt produces a plausible date, that case fails even if other rows look better. If a note contains text asking the assistant to ignore its instructions, the review should check that the note was treated as data.

Decide whether the revision is ready

Set acceptance conditions before the comparison: no invented customer commitments, no cross-site merging, sources retained and open questions visible. A small case set cannot establish universal reliability, so keep human approval on real outputs. Record who accepted the change and retain the earlier version for recovery if new issues appear.

Add a new fictional case when an observed failure reveals a gap, while keeping previous cases to catch regressions. Do not repeatedly rewrite the cases until the candidate passes. Track manual corrections and review time during a bounded pilot separately from test-set results. This process compares versions under specified conditions; it does not justify a claim that the model is accurate across commercial HVAC work.

Before you use the result

• Freeze a fictional case set and answer key before comparing versions.
• Include missing information, contradictions and out-of-scope requests.
• Record exact evidence for failures instead of only an overall score.
• Reject invented dates, approvals, diagnoses and cross-location merges.
• Retain the approved prompt version, reviewer and known limitations.

What the sources support

NIST’s generative-AI profile discusses evaluation and risks including unsupported output. This case-set procedure is our practical editorial adaptation for an office pilot, not an official NIST test, certification, benchmark score or claim of hands-on vendor testing.