-
Notifications
You must be signed in to change notification settings - Fork 117
Open
Labels
Description
On the event of crashing shards (e.g, native Android crashes) reported, these are treated as device failures and not test result failures. Within matrix_ids.json, could report something like:
"axes": [
{
"device": "Pixel2.arm-30-en_US-portrait",
"outcome": "failure",
"details": "0 test cases failed, 595 passed, 2 flaky",
"suiteOverview": {
"total": 597,
"errors": 0,
"failures": 0,
"flakes": 2,
"skipped": 0,
"elapsedTime": 264.07,
"overheadTime": 0.0
}
}
]Test result processing of matrix_ids.json and would be confused on this outcome with a failure reported with no failing tests.
FTL returns exit code of 10 with these failing shards (https://firebase.google.com/docs/test-lab/ios/command-line) wondering if an additional device descriptor can be appended to this outcome (if possible through FTL endpoints to have that data added to the axes)