Skip to content

Discrepancy with outcomes and return exit codes with failing shards  #2496

@AaronMT

Description

@AaronMT

On the event of crashing shards (e.g, native Android crashes) reported, these are treated as device failures and not test result failures. Within matrix_ids.json, could report something like:

"axes": [
      {
        "device": "Pixel2.arm-30-en_US-portrait",
        "outcome": "failure",
        "details": "0 test cases failed, 595 passed, 2 flaky",
        "suiteOverview": {
          "total": 597,
          "errors": 0,
          "failures": 0,
          "flakes": 2,
          "skipped": 0,
          "elapsedTime": 264.07,
          "overheadTime": 0.0
        }
      }
    ]

Test result processing of matrix_ids.json and would be confused on this outcome with a failure reported with no failing tests.

FTL returns exit code of 10 with these failing shards (https://firebase.google.com/docs/test-lab/ios/command-line) wondering if an additional device descriptor can be appended to this outcome (if possible through FTL endpoints to have that data added to the axes)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions