Add --show-all-tests option to GenerateTestSummary tool #13341

Copilot · 2025-12-05T01:34:09Z

Description

Adds --show-all-tests command line option to GenerateTestSummary that outputs all tests in a single markdown list ordered by full test name:

❌ Failing tests: Expanded details with error info (existing format)
✅ Passing tests: Test name only
⏭️ Skipped tests: Test name only

The --show-all-tests option can be used with --combined to include an "All Test Results" section in the combined summary report.

Changes

Program.cs

New --show-all-tests (-a) option
Can be combined with --combined to show all tests in combined reports

TestSummaryGenerator.cs

Modified CreateSingleTestSummaryReport() to accept bool showAllTests = false parameter
Modified CreateCombinedTestSummaryReport() to accept bool showAllTests = false parameter
Added GenerateAllTestsSection() method to generate all test results for combined reports
Added AppendFailedTestDetails() helper method to reduce code duplication

Workflows

specialized-test-runner.yml: New showOnlyFailedTestsInSummary input (defaults to true)
- When false, passes --show-all-tests along with --combined
tests-quarantine.yml / tests-outerloop.yml: Set showOnlyFailedTestsInSummary: false to show all tests

Example output (with --combined --show-all-tests)

# Test Summary

## Overall Summary
| Passed | Failed | Skipped | Total |
...

## All Test Results

- ✅ Aspire.Test.AnotherPassingTest
<details><summary>❌ <b>Aspire.Test.FailingTest</b></summary>
...error details...
</details>
- ✅ Aspire.Test.PassingTest1
- ⏭️ Aspire.Test.SkippedTest

Checklist

Is this feature complete?
- Yes. Ready to ship.
- No. Follow-up changes expected.
Are you including unit tests for the changes and scenario tests if relevant?
- Yes
- No
Did you add public API?
- Yes
  - If yes, did you have an API Review for it?
    - Yes
    - No
  - Did you add <remarks /> and <code /> elements on your triple slash comments?
    - Yes
    - No
- No
Does the change make any security assumptions or guarantees?
- Yes
  - If yes, have you done a threat model and had a security review?
    - Yes
    - No
- No
Does the change require an update in our Aspire docs?
- Yes
  - Link to aspire-docs issue (consider using one of the following templates):
- No

Original prompt

Update tools/GenerateTestSummary/Program.cs:

Add a new command line argument: --show-all-tests.

By default (when --show-all-tests is not present), only failing tests are included in the summary output.

When --show-all-tests is provided, the summary should be a single markdown list, ordered by full name of the test.

Each test gets one line:

Failing tests: Expanded details (as current), marked with a ❌ red cross.

Passing tests: Only full test name, with a ✅ green check.

Skipped tests: Only full test name, with a ⏭️ skip emoji.

Update .github/workflows/tests-quarantine.yml and .github/workflows/tests-outerloop.yml:

Ensure GenerateTestSummary is invoked with --show-all-tests for both workflows.

Emoji summary:

Failing: ❌

Passing: ✅

Skipped: ⏭️

All test result entries should appear in a single list, ordered by full name.

Reference code links for affected files:

Program.cs: https://github.com/dotnet/aspire/blob/main/tools/GenerateTestSummary/Program.cs

Quarantine workflow: https://github.com/dotnet/aspire/blob/main/.github/workflows/tests-quarantine.yml

Outerloop workflow: https://github.com/dotnet/aspire/blob/main/.github/workflows/tests-outerloop.yml

See chat for requirements and clarifications.

This pull request was created as a result of the following prompt from Copilot chat.

Update tools/GenerateTestSummary/Program.cs:

Add a new command line argument: --show-all-tests.

By default (when --show-all-tests is not present), only failing tests are included in the summary output.

When --show-all-tests is provided, the summary should be a single markdown list, ordered by full name of the test.

Each test gets one line:

Failing tests: Expanded details (as current), marked with a ❌ red cross.

Passing tests: Only full test name, with a ✅ green check.

Skipped tests: Only full test name, with a ⏭️ skip emoji.

Update .github/workflows/tests-quarantine.yml and .github/workflows/tests-outerloop.yml:

Ensure GenerateTestSummary is invoked with --show-all-tests for both workflows.

Emoji summary:

Failing: ❌

Passing: ✅

Skipped: ⏭️

All test result entries should appear in a single list, ordered by full name.

Reference code links for affected files:

Program.cs: https://github.com/dotnet/aspire/blob/main/tools/GenerateTestSummary/Program.cs

Quarantine workflow: https://github.com/dotnet/aspire/blob/main/.github/workflows/tests-quarantine.yml

Outerloop workflow: https://github.com/dotnet/aspire/blob/main/.github/workflows/tests-outerloop.yml

See chat for requirements and clarifications.

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Co-authored-by: radical <[email protected]>

github-actions · 2025-12-05T01:44:25Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 13341

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 13341"

radical · 2025-12-05T01:53:06Z

tools/GenerateTestSummary/TestSummaryGenerator.cs

        return overallTableBuilder.ToString();
    }

+    public static string CreateAllTestsSummaryReport(string basePath)


@copilot This is not correct. Modify CreateSingleTestSummaryReport:

this will take an additional parameter like onlyFailedTests = true

The method can then iterate over all the test results, and depending on the param it can show either only the failed ones or all the tests

Refactored as requested. Modified CreateSingleTestSummaryReport to accept bool showAllTests = false parameter and removed the separate CreateAllTestsSummaryReport method. The method now iterates over all test results when showAllTests is true, otherwise only shows failed tests. (850046d)

… instead of separate method Co-authored-by: radical <[email protected]>

radical · 2025-12-05T02:07:30Z

.github/workflows/specialized-test-runner.yml

          --
          ${{ github.workspace }}/testresults
-          --combined
+          ${{ inputs.extraSummaryArgs || '--combined' }}


@copilot instead of this:

let's add a new template parameter:

showOnlyFailedTestsInSummary defaulting to true

And if that is false then it should pass --show-all-tests as extraSummaryArgs to run-tests.yml

Also, if it is false then it we should be passing --show-all-tests in addition to --combined

In the test summary generator support show-all-tests for the combined summary too.

Implemented as requested:

Added showOnlyFailedTestsInSummary parameter (default true) to specialized-test-runner.yml

When showOnlyFailedTestsInSummary is false, passes --show-all-tests along with --combined

Updated CreateCombinedTestSummaryReport to accept showAllTests parameter and added GenerateAllTestsSection method

Updated both workflows to use showOnlyFailedTestsInSummary: false

(e6d1420)

…ts with --combined Co-authored-by: radical <[email protected]>

Initial plan

fb4bdb0

Copilot AI assigned Copilot and radical Dec 5, 2025

Copilot started work on behalf of radical December 5, 2025 01:34 View session

Add --show-all-tests option to GenerateTestSummary and update workflows

4766bf0

Co-authored-by: radical <[email protected]>

Copilot AI changed the title ~~[WIP] Add --show-all-tests command line argument for test summary~~ Add --show-all-tests option to GenerateTestSummary tool Dec 5, 2025

Copilot AI requested a review from radical December 5, 2025 01:48

Copilot finished work on behalf of radical December 5, 2025 01:48

radical reviewed Dec 5, 2025

View reviewed changes

Copilot started work on behalf of radical December 5, 2025 01:53 View session

Refactor: Use showAllTests parameter in CreateSingleTestSummaryReport…

850046d

… instead of separate method Co-authored-by: radical <[email protected]>

Copilot AI requested a review from radical December 5, 2025 02:02

Copilot finished work on behalf of radical December 5, 2025 02:02

radical reviewed Dec 5, 2025

View reviewed changes

Copilot started work on behalf of radical December 5, 2025 02:07 View session

Add showOnlyFailedTestsInSummary parameter and support --show-all-tes…

e6d1420

…ts with --combined Co-authored-by: radical <[email protected]>

Copilot AI requested a review from radical December 5, 2025 02:19

Copilot finished work on behalf of radical December 5, 2025 02:19

radical closed this Dec 5, 2025

radical reopened this Dec 5, 2025

dotnet-policy-service bot added this to the 13.1 milestone Dec 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add --show-all-tests option to GenerateTestSummary tool #13341

Add --show-all-tests option to GenerateTestSummary tool #13341

Copilot AI commented Dec 5, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 5, 2025 •

edited

Loading

Uh oh!

radical Dec 5, 2025

Uh oh!

Copilot AI Dec 5, 2025

Uh oh!

radical Dec 5, 2025

Uh oh!

Copilot AI Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add --show-all-tests option to GenerateTestSummary tool #13341

Are you sure you want to change the base?

Add --show-all-tests option to GenerateTestSummary tool #13341

Conversation

Copilot AI commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Example output (with --combined --show-all-tests)

Checklist

Uh oh!

github-actions bot commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radical Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

radical Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Dec 5, 2025 •

edited

Loading

github-actions bot commented Dec 5, 2025 •

edited

Loading