csharp-spector-coverage-gaps

Discovering and implementing Spector coverage gaps for http-client-csharp

Overview

This skill discovers which Spector scenarios the C# emitter (@typespec/http-client-csharp) does not yet cover, then implements the missing test(s). Spector scenarios are defined in @typespec/http-specs and @azure-tools/azure-http-specs. The coverage dashboard is at https://typespec.io/can-i-use/http/.

Note: {PKG} refers to <repo-root>/packages/http-client-csharp throughout this document.

Inputs

You may receive one of:

"Find coverage gaps" — discover all missing scenarios and present them.
Spector spec link — a link to a specific spec under packages/http-specs/specs/... or packages/azure-http-specs/specs/....
Spec name — e.g., http/encode/duration, http/type/model/flatten, http/type/union/discriminated.

Output

A report of coverage gaps (when discovering).
New or updated C# NUnit test file(s) under {PKG}/generator/TestProjects/Spector.Tests/Http/.
Updated {PKG}/generator/TestProjects/Spector.Tests/TestProjects.Spector.Tests.csproj if a new project reference is needed.
Regenerated client code (via Generate.ps1).

Workflow

[ ] Ensure prerequisites are met (npm ci, npm run build)
[ ] Discover coverage gaps or identify the target spec
[ ] Verify the spec is not in the failing specs list
[ ] Generate the C# client for the target spec (unstubbed with -Stubbed $false, or use Test-Spector.ps1)
[ ] Read the generated client to understand the API surface
[ ] Read the TypeSpec scenario file to understand expected behavior
[ ] Find or create the test file
[ ] Implement the test(s) following existing conventions
[ ] Run the tests using Test-Spector.ps1 -filter "<spec-path>"
[ ] Validate all tests pass

Prerequisites — Environment setup

Before starting, ensure the build environment is ready. These instructions supersede the repo-root pnpm instructions.

Install dependencies (from {PKG}):
```
cd {PKG}
npm ci
```
Build the package:
```
npm run build
```

⚠️ Do NOT run pnpm install or pnpm build at the repo root — only the http-client-csharp package build is needed.

Step 1 — Discover coverage gaps

How specs are filtered

The file {PKG}/eng/scripts/Spector-Helper.psm1 defines which specs are included/excluded:

Failing specs are defined in {PKG}/eng/scripts/Spector-Helper.psm1 in the $failingSpecs array. Always check that file for the current list — do not hardcode specs here.

Azure allow-list (only these Azure specs are tested):

http/client/structure/client-operation-group
http/client/structure/default
http/client/structure/multi-client
http/client/structure/renamed-operation
http/client/structure/two-operation-group
http/resiliency/srv-driven

All other specs from @typespec/http-specs are included by default.

Discovering gaps programmatically

Compare available specs against existing tests. Run this from {PKG}:

Import-Module "{PKG}/eng/scripts/Spector-Helper.psm1" -DisableNameChecking -Force

# Get all valid specs
$specs = Get-Sorted-Specs | ForEach-Object { Get-SubPath $_ }

# Get all test files
$testFiles = Get-ChildItem -Path "{PKG}/generator/TestProjects/Spector.Tests/Http" -Recurse -Filter "*Tests.cs" |
    ForEach-Object { $_.FullName }

# For each spec, check if a corresponding test directory/file exists
foreach ($spec in $specs) {
    $specParts = $spec -replace '/', '\' -split '\\'
    # Convert to expected test path segments
    $testPath = "{PKG}/generator/TestProjects/Spector.Tests/Http"
    # ... check if test exists
}

Alternatively, manually compare the list of specs against existing test files:

Existing test coverage (test directories under Spector.Tests/Http/):

Authentication/ApiKey, Authentication/Http/Custom, Authentication/OAuth2, Authentication/Union
Client/Naming, Client/Structure/{ClientOperationGroup,Default,MultiClient,RenamedOperation,TwoOperationGroup}
Documentation
Encode/{Array,Bytes,DateTime,Numeric}
Parameters/{Basic,BodyOptionality,CollectionFormat,Path,Query,Spread}
Payload/{ContentNegotiation,JsonMergePatch,MediaType,Multipart,Pageable,Xml}
Resiliency/SrvDriven/{V1,V2}
Response/StatusCodeRange
Routes
Serialization/EncodedName/Json
Server/{Endpoint/NotDefined,Path/Multiple,Path/Single,Versions/NotVersioned,Versions/Versioned}
SpecialHeaders/{ConditionalRequest,Repeatability}
Versioning/{Added,MadeOptional,Removed,RenamedFrom,ReturnTypeChangedFrom,TypeChangedFrom}
_Type/{Dictionary,Model/Empty,Model/Inheritance/*,Model/Usage,Model/Visibility,Property/*,Scalar,Union,_Array,_Enum/*}

Accurate gap detection

The naive approach of matching spec paths to test directory names can produce false positives because:

C# reserved words get underscore-prefixed in test dirs (e.g., type/ → _Type/, array → _Array, enum → _Enum)
kebab-case gets converted to PascalCase (e.g., content-negotiation → ContentNegotiation)
Some test files cover a parent spec but not sub-specs (e.g., UnionTests.cs covers type/union but NOT type/union/discriminated)

To find real gaps, verify each candidate by checking whether a test directory exists for the spec's exact path, including sub-paths. A test file at a parent level does NOT cover child specs.

Known failing specs are tracked in $failingSpecs in {PKG}/eng/scripts/Spector-Helper.psm1. Always read that file for the current list.

Important: The gap list evolves over time. Always re-run the comparison to get current gaps. All committed Spector libraries are stubbed — [SpectorTest] will auto-skip tests unless you regenerate with -Stubbed $false or use Test-Spector.ps1.

Understanding stubbed vs unstubbed generation

All Spector libraries committed to the repository are stubbed. Generate.ps1 defaults to $Stubbed = $true, which passes --option @typespec/http-client-csharp.generator-name=StubLibraryGenerator to the emitter. Stubbed clients use expression-bodied constructors (=>) instead of block bodies ({ }), and the [SpectorTest] attribute automatically skips tests for stubbed clients.

To generate unstubbed code (for local testing), pass -Stubbed $false:

pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" -Stubbed $false

However, the recommended way to test is via Test-Spector.ps1 (see Step 7), which handles the unstubbed regeneration, test execution, and directory restoration automatically.

Step 2 — Read the spec to understand expected behavior

Spec files live in {PKG}/node_modules/@typespec/http-specs/specs/ (or @azure-tools/azure-http-specs/specs/ for Azure specs).

If node_modules is not installed, read from the source at <repo-root>/packages/http-specs/specs/.

Each spec contains:

main.tsp — the TypeSpec definition with @scenario and @scenarioDoc decorators
client.tsp (optional) — client-level customizations; takes priority over main.tsp during generation
tspconfig.yaml (optional in the Spector test project) — C#-specific generation options

Read the @scenarioDoc decorators to understand:

The HTTP method, path, and expected parameters
The expected request body shape
The expected response status code and body
Any special behavior (e.g., "should return 204", "should send header X")

Step 3 — Generate the C# client for the target spec

Use Generate.ps1 with a filter to generate only the specific spec. By default, Generate.ps1 generates stubbed code (same as what is committed to the repo). To generate unstubbed code for local testing, pass -Stubbed $false:

cd {PKG}
# Generate unstubbed (for local testing/development)
pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" -Stubbed $false

# Generate stubbed (default, matches what is committed to repo)
pwsh eng/scripts/Generate.ps1 -filter "<spec-path>"

Examples:

# Single spec (unstubbed for testing)
pwsh eng/scripts/Generate.ps1 -filter "http/encode/duration" -Stubbed $false

# Versioning spec (generates v1 + v2 automatically)
pwsh eng/scripts/Generate.ps1 -filter "http/versioning/added" -Stubbed $false

The generated code lands in {PKG}/generator/TestProjects/Spector/<spec-path>/src/Generated/.

Verify generation succeeded

# Check that client code was generated
Get-ChildItem "{PKG}/generator/TestProjects/Spector/<spec-path>/src/Generated/" -Filter "*Client.cs"

Note: All committed Spector libraries are stubbed. The [SpectorTest] attribute automatically skips tests for stubbed clients. Use Test-Spector.ps1 (Step 7) to regenerate unstubbed, run tests, and restore automatically.

Step 4 — Read the generated client API surface

Browse the generated code to understand:

Client class(es): *Client.cs — the entry point(s)
Sub-clients: accessed via Get*Client() methods
Operations: async methods like GetAsync(), PutAsync(body), SendAsync()
Models: under Models/ — request/response shapes
Constructor signature: new XClient(Uri endpoint, XClientOptions options) or new XClient(Uri endpoint, KeyCredential credential, XClientOptions options)

Pay attention to:

Method names (they map to TypeSpec operation names)
Parameter types (models, primitives, BinaryData)
Return types (ClientResult, ClientResult<T>, AsyncPageable<T>)

Step 5 — Create or extend the test file

Directory and namespace conventions

The test directory structure mirrors the spec path with these transformations:

http/ → Http/
kebab-case → PascalCase (e.g., content-negotiation → ContentNegotiation)
type/ → _Type/ (leading underscore because Type is a C# keyword)
array → _Array (same reason)
enum → _Enum (same reason)

Namespace pattern: TestProjects.Spector.Tests.Http.<PascalCasePath>

Example mappings:

Spec path	Test directory	Namespace
`http/encode/duration`	`Http/Encode/Duration/`	`TestProjects.Spector.Tests.Http.Encode.Duration`
`http/type/union/discriminated`	`Http/_Type/Union/Discriminated/`	`TestProjects.Spector.Tests.Http._Type.Union.Discriminated`
`http/special-words`	`Http/SpecialWords/`	`TestProjects.Spector.Tests.Http.SpecialWords`

Test file template

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System.Threading.Tasks;
using <GeneratedNamespace>;
using NUnit.Framework;

namespace TestProjects.Spector.Tests.Http.<Category>.<SubCategory>
{
    public class <Name>Tests : SpectorTestBase
    {
        [SpectorTest]
        public Task <ScenarioName>() => Test(async (host) =>
        {
            var response = await new <Client>(host, null).<Method>Async(<args>);
            Assert.AreEqual(<expectedStatusCode>, response.GetRawResponse().Status);
        });
    }
}

Common test patterns

Simple void operation (204 response):

[SpectorTest]
public Task SimpleOp() => Test(async (host) =>
{
    var response = await new MyClient(host, null).DoThingAsync();
    Assert.AreEqual(204, response.GetRawResponse().Status);
});

GET with typed response:

[SpectorTest]
public Task GetValue() => Test(async (host) =>
{
    var response = await new MyClient(host, null).GetValueAsync();
    Assert.AreEqual("expected", response.Value);
});

GET with model response:

[SpectorTest]
public Task GetModel() => Test(async (host) =>
{
    var response = await new MyClient(host, null).GetModelAsync();
    Assert.AreEqual("name", response.Value.Name);
    Assert.AreEqual(42, response.Value.Age);
});

PUT/POST with body:

[SpectorTest]
public Task SendModel() => Test(async (host) =>
{
    var body = new MyModel("name", 42);
    var response = await new MyClient(host, null).SendAsync(body);
    Assert.AreEqual(204, response.GetRawResponse().Status);
});

Round-trip (GET then PUT):

[SpectorTest]
public Task RoundTrip() => Test(async (host) =>
{
    var client = new MyClient(host, null);
    var getResult = await client.GetAsync();
    var response = await client.PutAsync(getResult.Value);
    Assert.AreEqual(204, response.GetRawResponse().Status);
});

Error assertion:

[SpectorTest]
public Task InvalidKey() => Test((host) =>
{
    var exception = Assert.ThrowsAsync<ClientResultException>(
        () => new MyClient(host, new ApiKeyCredential("invalid"), null).CallAsync());
    Assert.AreEqual(403, exception!.Status);
    return Task.CompletedTask;
});

Sub-client access:

[SpectorTest]
public Task SubClientOp() => Test(async (host) =>
{
    var response = await new MyClient(host, null)
        .GetSubClient()
        .DoThingAsync();
    Assert.AreEqual(204, response.GetRawResponse().Status);
});

Pagination:

[SpectorTest]
public Task ListItems() => Test(async (host) =>
{
    var items = new MyClient(host, null).GetItemsAsync();
    int count = 0;
    await foreach (var item in items)
    {
        count++;
    }
    Assert.Greater(count, 0);
});

Additional imports commonly needed

using System;
using System.ClientModel;           // ClientResult, ClientResultException, ApiKeyCredential
using System.IO;                     // For file/stream scenarios
using NUnit.Framework;               // Test framework

Step 6 — Update the .csproj if needed

If a new Spector test project directory was created (new spec), a project reference may be needed in TestProjects.Spector.Tests.csproj:

<ProjectReference Include="..\Spector\http\<spec-path>\src\<ProjectName>.csproj" />

Check existing references to match the pattern. The project name typically matches the package-name from tspconfig.yaml or is derived from the namespace.

Only add a project reference if one doesn't already exist for the spec.

Step 7 — Build and run the tests

Recommended: Use Test-Spector.ps1

The Test-Spector.ps1 script is the recommended way to test Spector specs. It automatically:

Regenerates the spec as unstubbed (so tests are not auto-skipped)
Runs the tests for that spec
Restores the directory to its original stubbed state

cd {PKG}
# Test a specific spec
pwsh eng/scripts/Test-Spector.ps1 -filter "<spec-path>"

Example:

pwsh eng/scripts/Test-Spector.ps1 -filter "http/encode/duration"

Run full Spector test suite

pwsh eng/scripts/Get-Spector-Coverage.ps1

Manual approach (if needed)

If you need more control, you can manually generate unstubbed, build, and run tests:

cd {PKG}

# Generate unstubbed
pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" -Stubbed $false

# Build
dotnet build generator

# Run only your new tests
dotnet test generator/TestProjects/Spector.Tests/TestProjects.Spector.Tests.csproj `
    --filter "FullyQualifiedName~TestProjects.Spector.Tests.Http.<YourNamespace>"

# Restore the directory to stubbed state when done
git clean -xfd generator/TestProjects/Spector/<spec-path>
git restore generator/TestProjects/Spector/<spec-path>

Step 8 — Handle special cases

Versioning specs

Versioning specs generate two clients (v1 and v2). Tests go in separate subdirectories:

Http/Versioning/<Name>/V1/<Name>V1Tests.cs
Http/Versioning/<Name>/V2/<Name>V2Tests.cs

Generation is handled automatically by Generate.ps1 when the path contains versioning.

Srv-driven (resiliency) specs

Similar to versioning — generates v1 and v2 clients from old.tsp and main.tsp.

Specs with tspconfig.yaml

Some specs have a tspconfig.yaml in the Spector test project that overrides the package-name. Check {PKG}/generator/TestProjects/Spector/<spec-path>/tspconfig.yaml before importing the generated namespace.

Notes

Do not modify Spector-Helper.psm1 to remove items from the failing list unless you're sure the generator now supports them.
All committed Spector libraries are stubbed — Generate.ps1 defaults to $Stubbed = $true. Use Test-Spector.ps1 to temporarily regenerate unstubbed and run tests.
Tests auto-skip when the generated client is stubbed — it's safe to write tests for stubbed specs. They'll activate when Test-Spector.ps1 regenerates them unstubbed.
Only commit: test files (.cs), .csproj changes, and tspconfig.yaml if needed.
Follow existing test naming conventions: <Feature>Tests.cs in the matching directory.
Use [SpectorTest] attribute (not [Test]) for all Spector tests — it enables auto-skip for stubbed implementations.
All test classes must inherit from SpectorTestBase.
Do not add change logs — the http-client-csharp instructions say they are not needed.
Do not comment out or delete existing tests.

01 前言 Front Matter

03 SKILL.md 正文