running-tests

Running Tests

Overview

Tests in this repository use the Microsoft.ClientModel.TestFramework and NUnit.

Many tests rely on session recordings — pre-recorded HTTP interactions stored in tests/SessionRecords/ — so they can run without live service access. The repository defaults to Playback mode unless overridden in the environment.

Test Modes

Mode	Description
Playback	Tests run against pre-recorded session data in tests/SessionRecords/. No API key is required.
Record	Tests run against the live OpenAI API and produce or update session recordings. Requires an API key.
Live	Tests run against the live OpenAI API without producing or updating session recordings. Requires an API key.

The mode is controlled by the CLIENTMODEL_TEST_MODE environment variable and accepts Playback, Record, or Live.

In Playback mode, the framework may attempt to auto-record when a session recording is missing or stale. Disable that behavior by explicitly setting CLIENTMODEL_DISABLE_AUTO_RECORDING to true.

Agent Rules

As the agent, only execute tests in Playback mode.

Do not run Record or Live mode yourself. Those paths require live credentials which you do not have.

Before running tests, always set:

$env:CLIENTMODEL_TEST_MODE = "Playback"
$env:CLIENTMODEL_DISABLE_AUTO_RECORDING = "true"

If a recorded test needs new recordings or updated recordings, you must follow the instructions below to ask a human to capture them for you instead of trying to capture them yourself.

Identifying Recorded Tests vs Non-Recorded Tests

Treat these as practical indicators instead of a rigid checklist.

Strong signals that a test is a recorded test:

• The test class inherits from OpenAIRecordedTestBase.
• The test method uses [RecordedTest].
• The test gets clients through GetProxiedOpenAIClient.
• The test exercises real service behavior rather than mocked responses.

Strong signals that a test is not a recorded test:

• The test class inherits from something other than OpenAIRecordedTestBase or has no base class.
• The test method uses plain [Test] instead of [RecordedTest].
• The test uses mocked transports or handcrafted responses to simulate HTTP responses, such as MockPipelineTransport, MockPipelineResponse, or GetClientOptionsWithMockResponse.
• The test does not need to reach a live cloud service.

When Recordings Are Missing or Stale

Recordings must be captured by a human.

Request recordings from a human before considering your work complete in either of the following cases:

1. You added a new recorded test (which implies its recording is missing because it has never been recorded before).
2. An existing recorded test fails in Playback mode because its recording is missing or stale (for example, if the recorded test was modified and the existing recording no longer matches).

When asking for recordings, always provide:

1. The link to the recording workflow
2. The exact NUnit.Where expression
3. A copy-pasteable dotnet test command

You must use this template:

Please record tests using the recording workflow: https://github.com/openai/openai-dotnet/actions/workflows/record-test.yml

Use the following NUnit.Where expression:

test == 'Namespace.TestClass.TestMethodName'

Alternatively, run the following command locally and push the recordings manually:

dotnet test ./tests/OpenAI.Tests.csproj --configuration Release --framework "net10.0" -- NUnit.Where="test == 'Namespace.TestClass.TestMethodName'"

Use NUnit.Where for recording requests even when dotnet test --filter would work locally. NUnit.Where is the contract used by the recording workflow. Prefer test == ... because it matches the exact discovered NUnit test name and avoids ambiguity. For NUnit fixture-parameterized tests such as classes constructed with bool isAsync, the discovered test names may include fixture arguments like (True) or (False).

For a single ordinary test, use:

test == 'Namespace.TestClass.TestMethodName'

If you need one exact fixture instance, use:

test == 'Namespace.TestClass(True).TestMethodName'

Example for async fixture instance of the GenerateSingleEmbedding recorded test:

test == 'OpenAI.Tests.Embeddings.EmbeddingsTests(True).GenerateSingleEmbedding'

If multiple tests need recording, combine them in a single NUnit.Where expression:

(test == 'Namespace.TestClass.TestA') or (test == 'Namespace.TestClass.TestB')

Example for both fixture instances of the GenerateSingleEmbedding recorded test:

(test == 'OpenAI.Tests.Embeddings.EmbeddingsTests(True).GenerateSingleEmbedding') or (test == 'OpenAI.Tests.Embeddings.EmbeddingsTests(False).GenerateSingleEmbedding')