{"id":4760,"date":"2026-08-06T18:00:54","date_gmt":"2026-08-06T18:00:54","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/08\/06\/test-reporting-in-microsoft-testing-platform-from-red-build-to-root-cause\/"},"modified":"2026-08-06T18:00:54","modified_gmt":"2026-08-06T18:00:54","slug":"test-reporting-in-microsoft-testing-platform-from-red-build-to-root-cause","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/08\/06\/test-reporting-in-microsoft-testing-platform-from-red-build-to-root-cause\/","title":{"rendered":"Test reporting in Microsoft.Testing.Platform: from red build to root cause"},"content":{"rendered":"

When a test fails in CI, \u201cthe build is red\u201d is only the starting point. The useful questions are whether this
\nchange caused the failure, whether the test has failed before, and where to find the evidence. If answering
\nthem means searching a job log, downloading an artifact, and finding the teammate who remembers how that test
\nbehaved last month, the report has not done enough.<\/p>\n

MTP reporting shortens the path from a failed build to the decision a developer or reviewer needs to make.<\/p>\n

Microsoft.Testing.Platform (MTP)<\/a> powers
\ndotnet test<\/code><\/a>, the Test Explorer
\nin Visual Studio and Visual Studio Code, and test runs in CI. Over the past year its reporting has been closing
\nthat gap \u2014 putting failures where reviewers already work, and preserving the evidence when a run falls over.<\/p>\n

This works on whichever test framework you already use: MTP is supported by
\nMSTest, NUnit, xUnit.net, TUnit and Expecto<\/a>.
\nMost capabilities in this post require MTP 2.3.0 or later. The examples were validated with MSTest.Sdk<\/code> 4.3.3
\nand MTP 2.3.3; the GitHub Actions, JUnit and CTRF reporters currently ship as preview packages.
\nMost of what follows is provider-independent, but two of the strongest features read your build history, and
\nthat part is Azure DevOps only today:<\/p>\n\n\n\n\n\n\n\n\n
Capability<\/th>\nGitHub Actions<\/th>\nAzure DevOps<\/th>\n<\/tr>\n<\/thead>\n
Inline failure annotations and job summary<\/td>\nYes<\/td>\nYes<\/td>\n<\/tr>\n
Live result publishing while the run is going<\/td>\nNot yet<\/td>\nYes<\/td>\n<\/tr>\n
Flaky-vs-regression history, quarantine, slow-test history<\/td>\nNot yet<\/td>\nYes<\/td>\n<\/tr>\n
Crash-resilient reports, report formats, JSON discovery<\/td>\nProvider-independent<\/td>\nProvider-independent<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Start with inline reporting on either provider: it moves the next failure out of the raw log. If you use Azure
\nDevOps, add history-based triage next so reviewers can distinguish a new regression from a recurring failure.<\/p>\n

Put the failure where the author is already looking<\/h2>\n

A report that lands in an artifact is a report someone has to go and fetch. Both providers can surface results
\ninline instead, and MTP now emits the annotation format each one understands natively.<\/p>\n

Enable --report-gh<\/code><\/a>
\nin a GitHub Actions job and a failing test becomes an annotation on the line of code that failed. Skipped tests
\nappear as warnings, each assembly is collapsed into its own log group, and the same run writes a summary straight
\nto the workflow page:<\/p>\n

\"A<\/p>\n

The failure is visible from the job page and the pull request without downloading a report artifact.
\nAnnotations, log groups, the summary and slow-test notices can each be configured independently.<\/p>\n

Azure DevOps has the same shape behind
\n--report-azdo<\/code><\/a>,
\nand it offers something GitHub Actions doesn\u2019t have yet. With --publish-azdo-test-results<\/code>, results stream into
\nthe Tests tab while the run is still going<\/em> \u2014 instead of arriving from a separate publish step that only runs if
\nthe job survives to reach it. The two are independent: --report-azdo<\/code> handles annotations and logging,
\n--publish-azdo-test-results<\/code> handles the Tests tab, and either works alone.<\/p>\n

Azure DevOps: tell a genuine regression from a known flake<\/h2>\n

This is the central decision that build history unlocks: not only which<\/em> test failed, but whether the current
\nchange is likely to have caused it.<\/p>\n

In a large suite, intermittently failing tests train reviewers to read red as background noise. Once \u201cprobably
\njust flaky\u201d is the reasonable first guess, real regressions inherit the same shrug \u2014 and the cost isn\u2019t the
\nflaky tests, it\u2019s the response time on everything else.<\/p>\n\n

\n

<\/i>Azure DevOps only<\/strong><\/p>\n

The build-history annotations, known-flaky demotion, quarantine and history-based slow-test detection in this section are Azure DevOps-only. The GitHub Actions reporter doesn\u2019t offer them yet.<\/p><\/div>\n

Azure DevOps already exposes this evidence in the Tests tab. In this testfx run, the current build reported no
\nfailures across 86,406 tests, while the 14-day history still called out one unique failing test:<\/p>\n

\"Azure<\/p>\n

For the testfx team, that context narrows the first investigation step: a failure with no history gets immediate
\nregression attention, while a recurring failure starts with its existing flaky record.<\/p>\n

Pass that same 14-day window to the reporter, and it queries the pipeline history and attaches the context to
\neach failure:<\/p>\n

dotnet test --report-azdo --report-azdo-flaky-history 14<\/code><\/pre>\n
A test that has been failing on and off for two weeks is labelled with its record:<\/p>\n
[flaky: failed 3\/20 in last 14d]<\/code><\/pre>\n
A test with no such history gets the label that matters:<\/p>\n
[REGRESSION]<\/code><\/pre>\n
That turns \u201csomeone should look at this eventually\u201d into \u201cthis pull request broke something,\u201d and it reaches
\nthe reviewer at the point of decision instead of depending on who remembers what. If you want CI to act on it
\nrather than merely say it, --report-azdo-demote-known-flaky<\/code> turns known-flaky failures into warnings while
\nregressions stay errors.<\/p>\n
In the testfx pipeline we use the history annotations but leave automatic demotion off, so every failure stays
\nblocking and history only guides triage. Other teams will reasonably choose to demote. The decision worth
\nmaking explicitly is whether history should inform a reviewer or change CI severity on its own.<\/p>\n
The same history powers slow-test detection. Instead of one threshold that is wrong for every project,
\n--report-azdo-slow-test-history<\/code> compares each test against its own past. The threshold is a configurable
\nmultiple over a minimum number of runs, so a single cold start won\u2019t trip it.<\/p>\n
Keep the evidence when the run crashes<\/h2>\n
History helps determine whether a failure is new, but triage still stalls if the run loses the evidence needed
\nto investigate it. The run you most need a report from is the one that died. That used to be the run that
\nproduced nothing, because results were serialized at the end.<\/p>\n
TRX results are now streamed to disk as they are produced, so a hard crash no longer takes the whole report
\nwith it. Pair --report-trx<\/code> with the
\ncrash-dump extension<\/a>
\nand the run is supervised by a controller process that finalises the partial report when the host dies:<\/p>\n
dotnet test --report-trx --crashdump<\/code><\/pre>\n
You get a valid TRX containing every test that completed, plus a console list of the ones that didn\u2019t:<\/p>\n
The following tests were still running when the test host crashed:\n[00:00:00] D_crash_the_host<\/code><\/pre>\n
That names the test that took the process down, without rerunning the suite to find it. The extension also
\nwrites a *.crash.sequence.log<\/code> recording every test start and end, so a test that started and never finished
\nis unambiguous even when several were running in parallel.<\/p>\n
Attachments got the same treatment. Crash dumps, hang dumps and extension artifacts are no longer silently
\ndropped on .NET Framework when the path exceeds the Windows MAX_PATH<\/code> limit, and an attachment that can\u2019t be
\ncopied is now surfaced on the console rather than only inside the TRX file. An incomplete run is now visibly<\/em>
\nincomplete, instead of a green-looking report with the evidence quietly missing.<\/p>\n
One run, reports for people and tools<\/h2>\n
Choose the output based on who consumes it. Use TRX for .NET tooling, HTML for direct inspection, and JUnit or
\nCTRF for dashboards and cross-stack automation. A single run can enable any combination, removing the
\nformat-conversion step teams often wire into their pipelines.<\/p>\n\n\n\n\n\n\n\n\n
Format<\/th>\nEnable with<\/th>\nPackage<\/th>\nStatus<\/th>\n<\/tr>\n<\/thead>\n
TRX<\/td>\n--report-trx<\/code><\/td>\nMicrosoft.Testing.Extensions.TrxReport<\/code><\/a><\/td>\nStable<\/td>\n<\/tr>\n
HTML<\/td>\n--report-html<\/code><\/td>\nMicrosoft.Testing.Extensions.HtmlReport<\/code><\/a><\/td>\nStable<\/td>\n<\/tr>\n
JUnit XML<\/td>\n--report-junit<\/code><\/td>\nMicrosoft.Testing.Extensions.JUnitReport<\/code><\/a><\/td>\nPreview<\/td>\n<\/tr>\n
CTRF JSON<\/td>\n--report-ctrf<\/code><\/td>\nMicrosoft.Testing.Extensions.CtrfReport<\/code><\/a><\/td>\nPreview<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
CTRF<\/a> is worth knowing about if you aggregate results across languages: it\u2019s a shared JSON
\nschema, so .NET reports into the same shape as the rest of a polyglot estate.<\/p>\n
Only TRX and JUnit are formats a CI system parses into a results view. HTML and CTRF are for people and
\ndashboards, so they show up as downloadable artifacts instead \u2014 and on Azure DevOps,
\n--report-azdo-upload-artifacts files<\/code> collects them automatically. Pick TRX or JUnit when something downstream
\nreads the results; add HTML or CTRF for whoever has to look at them.<\/p>\n
Report names no longer collide either. Each reporter takes a --report-<format>-filename<\/code> accepting
\nbuild-specific placeholders<\/a>
\nsuch as {asm}<\/code> and {tfm}<\/code>, resolved under the results directory:<\/p>\n
dotnet test --report-trx --report-trx-filename \"reports\/{asm}_{tfm}_{time}.trx\"<\/code><\/pre>\n
Left alone, TRX, HTML and JUnit now default to a deterministic <asm>_<tfm>_<arch><\/code> name. If you multi-target,
\nnet8.0<\/code> and net8.0-windows<\/code> write separate files instead of reporting over each other \u2014 a quiet data-loss bug
\nin matrix builds that is worth checking for in your current setup.<\/p>\n
Stable output for automation<\/h2>\n
People need actionable context in the pull request; automation needs the same context in a stable, structured
\nformat. Scripts, dashboards, IDE integrations and coding agents cannot reliably consume terminal prose.<\/p>\n
--list-tests json<\/code> emits a schema-versioned document describing every discovered test, down to its source
\nlocation:<\/p>\n
{\n  \"schemaVersion\": 1,\n  \"tests\": [\n    {\n      \"uid\": \"11a3aade-7a31-8389-89ce-eab6ff0db7f3\",\n      \"displayName\": \"Total_includes_shipping\",\n      \"location\": { \"file\": \"CheckoutTests.cs\", \"lineStart\": 13, \"lineEnd\": 13 }\n    }\n  ]\n}<\/code><\/pre>\n
That schema is a stable input for test selection, impact analysis or IDE integration, instead of scraping
\nconsole text that changes between releases.<\/p>\n
MTP also adapts its own output for these consumers. In an agent or LLM environment it suppresses the banner,
\nANSI escapes and progress animation, and defaults --show-stdout<\/code> and --show-stderr<\/code> to failed<\/code>, so what
\ncomes back is results rather than redrawn progress bars. You can set the same behaviour by hand \u2014 the platform
\nhonors NO_COLOR<\/code>, and exposes --ansi<\/code> and --progress<\/code> for the rest.<\/p>\n
Try it today<\/h2>\n
    \n
  1. Confirm that one test project runs MTP 2.3 or later.<\/li>\n
  2. Enable the reporter for your CI provider through MSTest.Sdk<\/code> or a direct package reference.<\/li>\n
  3. Add dotnet test --report-gh<\/code> in GitHub Actions or dotnet test --report-azdo<\/code> in Azure DevOps.<\/li>\n
  4. Use the next failure to check whether the report reduced the time you spent searching logs.<\/li>\n<\/ol>\n
    Scale it across a repository<\/h3>\n
    Once you have chosen a reporting policy, keep it in the repository rather than everyone\u2019s shell history. Every
    \noption in the
    \n    MTP CLI reference<\/a> can
    \nlive in a checked-in
    \n    testconfig.json<\/code><\/a>
    \nbeside your test project, under commandLineOptions<\/code>:<\/p>\n
    {\n  \"commandLineOptions\": {\n    \"report-trx\": true,\n    \"report-html\": true,\n    \"report-azdo\": true,\n    \"report-azdo-flaky-history\": 14\n  }\n}<\/code><\/pre>\n
    Local and CI runs then produce the same reports by default, and a developer can still override any of it from
    \nthe command line. Misspelled settings fail at startup with a message naming the file, rather than being
    \nsilently ignored:<\/p>\n
    In testconfig.json under 'commandLineOptions': Unknown option '--report-trxx'<\/code><\/pre>\n
    For a repo-wide rollout:<\/p>\n