The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2857556	dotnet-sdk	Publish new tools list (1ES PT)	Log
2857594	dotnet-sdk	Clone dotnet/dotnet	Log	#55968
2857595	dotnet-sdk	Checkout dotnet-sdk@refs/pull/55968/merge to s	Log	#55968
2857576	dotnet-sdk	Setup Internal Feeds	Log	#55967
2857575	dotnet-aspnetcore	Run build.cmd	Log	#55966
2857561	dotnet-sdk	Build	Log
2857543	dotnet-sdk	🛡 Guardian: 1ES Secret Scanning on Linux (SPMI)	Log	#55961
2857539	dotnet-sdk	Install wasm-tools Workload	Log	#55960
2857532	dotnet-sdk	Run Tests in Helix	Log	#55911
2857530	dotnet-sdk	Run Tests in Helix	Log	#55910
2857502	dotnet-sdk	Run AoT Tests	Log	#55844
2857501	dotnet-sdk	Build	Log	#55844
2857398	dotnet-runtime	Run performance job script	Log
1225187	dotnet/runtime	Publish	Log	dotnet/runtime#121412
1225184	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122234/merge to s	Log	dotnet/runtime#122234
1225034	dotnet/roslyn	Build with analyzers	Log	dotnet/roslyn#81598
2857045	dotnet-runtime	Run performance job script	Log
2857003	dotnet-dotnet	Build	Log
1224670	dotnet/runtime	Send to Helix	Log
1224685	dotnet/runtime	Publish	Log	dotnet/runtime#122266
2856924	dotnet-sdk	Run Tests in Helix	Log	#55911
2856961	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
2856948	dotnet-dotnet-monitor	Build	Log
2856926	dotnet-sdk	Run Tests in Helix	Log	#55910
2856900	dotnet-sdk	Run AoT Tests	Log	#55844
2856909	dotnet-dotnet-monitor	Build	Log
1224594	dotnet/roslyn	Test	Log
1224591	dotnet/roslyn	Test	Log
2856872	dotnet-dotnet-monitor	Build	Log
1224532	dotnet/roslyn	Test	Log	dotnet/roslyn#81550
2856809	dotnet-aspnetcore	🛡 Generate SBoM Manifest	Log	#55906
2856807	dotnet-sdk	Run Tests in Helix	Log	#55911
2856664	dotnet-sdk	Run TestBuild Tests	Log	#55853
2856810	dotnet-efcore	Send job to helix	Log	#55914
2856806	dotnet-sdk	Run Tests in Helix	Log	#55910
2856805	dotnet-efcore	Send job to helix	Log	#55913
1224440	dotnet/roslyn	Build with analyzers	Log
2856688	dotnet-dotnet	Build	Log
2856742	dotnet-dotnet-monitor	Build	Log
1224240	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#121305
2856660	dotnet-sdk	Run Tests in Helix	Log	#55910
1224180	dotnet/runtime	Publish	Log	dotnet/runtime#122111
2856662	dotnet-sdk	Run Tests in Helix	Log	#55911
1224217	dotnet/roslyn	Test	Log	dotnet/roslyn#81362
2845572	dotnet-roslyn	Build	Log
2845564	dotnet-roslyn	Build	Log
2856500	dotnet-wpf	🛡 Guardian: Check if the path to run Secret Scanning is a directory (1ES PT)	Log
1223974	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121305
2856468	dotnet-wpf	OneLocBuild	Log
2853919	dotnet-extensions	Checkout dotnet-extensions@refs/pull/55833/merge to s	Log	#55833
2856436	dotnet-wix	Build WiX Toolset	Log
2856397	dotnet-dotnet	Build (centOSStreamContainer)	Log
2856413	dotnet-maui	Run Integration Tests - Build macOS	Log
2856364	dotnet-maui	Run Integration Tests - Build macOS	Log
1223773	dotnet/runtime	Publish	Log	dotnet/runtime#122285
1223796	dotnet/runtime	Publish native test components	Log	dotnet/runtime#122286
1223608	dotnet/runtime	Send to Helix	Log
1223611	dotnet/runtime	Build product	Log
2855941	dotnet-dotnet	Run Tests	Log
2855943	dotnet-dotnet	Run Tests	Log
2855958	dotnet-dotnet-monitor	Publish Using Darc	Log
2854091	dotnet-dotnet	Build	Log
2856007	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
2855859	dotnet-arcade	Publish packages, blobs and symbols	Log
2855850	dotnet-arcade	Publish packages, blobs and symbols	Log
2854916	dotnet-roslyn	Publish Using Darc	Log
2855807	dotnet-source-indexer	🟣Build source index	Log
2855762	dotnet-runtime	Send tests to Helix (Unix)	Log
1223058	dotnet/runtime	Send tests to Helix (Windows)	Log
1223054	dotnet/runtime	Send to Helix	Log
2855539	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
1222932	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#122224
1222988	dotnet/runtime	AOT compile CoreCLR tests	Log
1222967	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#81387
2855413	dotnet-runtime	Send tests to Helix (Unix)	Log
2855408	dotnet-arcade	Publish packages, blobs and symbols	Log
1222853	dotnet/runtime	Send to Helix	Log
1222788	dotnet/runtime	Zip	Log	dotnet/runtime#122162
1222804	dotnet/runtime	AOT compile CoreCLR tests	Log
2855284	dotnet-dotnet	Build (centOSStreamContainer)	Log
2855183	dotnet-runtime	Send to Helix	Log
1222664	dotnet/runtime	Publish Build Assets	Log
2855074	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
1222616	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121853
1222586	dotnet/runtime	Send tests to Helix (Windows)	Log
1222592	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122257
1222598	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121694
1222595	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121656
1222459	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#121614
2855036	dotnet-dotnet-monitor	Build	Log
1222517	dotnet/runtime	Send to Helix	Log
1222350	dotnet/runtime	Run Trimming Tests	Log	dotnet/runtime#122249
1222414	dotnet/runtime	Send to Helix	Log
1222430	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119432
1222374	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122012
1222401	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#122085
2854861	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2854910	dotnet-dotnet	Build (centOSStreamContainer)	Log
1222300	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122227
1222368	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122247
Displaying 100 of 1848 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
43	357	1848

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez