Unified data model

The unified data model for test entities (test summary, test suite, test case) implements a super-set of all (so far known) unit test result summary file formats. pyEDAA.Report’s data model is a structural and functional cleanup of the Ant JUnit data model. Naming has been cleaned up and missing features have been added.

As some of the JUnit XML dialects are too divergent from the original Ant + JUnit4 format, these dialects have an independent test entity inheritance hierarchy. Nonetheless, instances of each data format can be converted to and from the unified data model.

Testcase

A test case is the leaf-element in the test entity hierarchy and describes an individual test run. Test cases are grouped by test suites.

Testsuite

A test suite is a group of test cases and/or test suites. Test suites itself can be grouped by test suites. The test suite hierarchy’s root element is a test suite summary.

TestsuiteSummary

The test suite summary is derived from test suite and defines the root of the test suite hierarchy.

Document

The document is derived from a test suite summary and represents a file containing a test suite summary.

        graph TD;
  doc[Document]
  sum[Summary]
  ts1[Testsuite]
  ts2[Testsuite]
  ts21[Testsuite]
  tc11[Testcase]
  tc12[Testcase]
  tc13[Testcase]
  tc21[Testcase]
  tc22[Testcase]
  tc211[Testcase]
  tc212[Testcase]
  tc213[Testcase]

  doc:::root -.-> sum:::summary
  sum --> ts1:::suite
  sum --> ts2:::suite
  ts2 --> ts21:::suite
  ts1 --> tc11:::case
  ts1 --> tc12:::case
  ts1 --> tc13:::case
  ts2 --> tc21:::case
  ts2 --> tc22:::case
  ts21 --> tc211:::case
  ts21 --> tc212:::case
  ts21 --> tc213:::case

  classDef root fill:#4dc3ff
  classDef summary fill:#80d4ff
  classDef suite fill:#b3e6ff
  classDef case fill:#eeccff
    

Testcase Status

TestcaseStatus and TestsuiteStatus are flag enumerations to describe the overall status of a test case or test suite.

Unknown

tbd

Excluded

tbd

Skipped

tbd

Weak

tbd

Passed

tbd

Failed

tbd

Inverted

tbd

Warned

tbd

Errored

tbd

Failed

tbd

SetupError

tbd

TearDownError

tbd

Inconsistent

tbd

@export
class TestcaseStatus(Flag):
  """A flag enumeration describing the status of a test case."""
  Unknown =    0                         #: Testcase status is uninitialized and therefore unknown.
  Excluded =   1                         #: Testcase was permanently excluded / disabled
  Skipped =    2                         #: Testcase was temporarily skipped (e.g. based on a condition)
  Weak =       4                         #: No assertions were recorded.
  Passed =     8                         #: A passed testcase, because all assertions were successful.
  Failed =    16                         #: A failed testcase due to at least one failed assertion.

  Mask = Excluded | Skipped | Weak | Passed | Failed

  Inverted = 128                         #: To mark inverted results
  UnexpectedPassed = Failed | Inverted
  ExpectedFailed =   Passed | Inverted

  Warned =  1024                         #: Runtime warning
  Errored = 2048                         #: Runtime error (mostly caught exceptions)
  Aborted = 4096                         #: Uncaught runtime exception

  SetupError =     8192                  #: Preparation / compilation error
  TearDownError = 16384                  #: Cleanup error / resource release error
  Inconsistent = 32768                   #: Dataset is inconsistent

  Flags = Warned | Errored | Aborted | SetupError | TearDownError | Inconsistent

@export
class TestsuiteStatus(Flag):
  """A flag enumeration describing the status of a test suite."""
  Unknown =    0
  Excluded =   1                         #: Testcase was permanently excluded / disabled
  Skipped =    2                         #: Testcase was temporarily skipped (e.g. based on a condition)
  Empty =      4                         #: No tests in suite
  Passed =     8                         #: Passed testcase, because all assertions succeeded
  Failed =    16                         #: Failed testcase due to failing assertions

  Mask = Excluded | Skipped | Empty | Passed | Failed

  Inverted = 128                         #: To mark inverted results
  UnexpectedPassed = Failed | Inverted
  ExpectedFailed =   Passed | Inverted

  Warned =  1024                         #: Runtime warning
  Errored = 2048                         #: Runtime error (mostly caught exceptions)
  Aborted = 4096                         #: Uncaught runtime exception

  SetupError =     8192                  #: Preparation / compilation error
  TearDownError = 16384                  #: Cleanup error / resource release error

  Flags = Warned | Errored | Aborted | SetupError | TearDownError

Testcase

A Testcase is the leaf-element in the test entity hierarchy and describes an individual test run. Besides a test case status, it also contains statistics like the start time or the test duration. Test cases are grouped by test suites and need to be unique per parent test suite.

A test case (or its base classes) implements the following properties and methods:

Parent

The test case has a reference to it’s parent test suite in the hierarchy. By iterating parent references, the root element (test suite summary) be be found, which has no parent reference (None).

Name

The test case has a name. This name must be unique per hierarchy parent, but can exist multiple times in the overall test hierarchy.

In case the data format uses hierarchical names like pyEDAA.Reports.CLI.Application, the name is split at the separator and multiple hierarchy levels (test suites) are created in the unified data model. To be able to recreate such an hierarchical name, TestsuiteKind is applied accordingly to test suite’s Kind field.

StartTime

The test case stores a time when the individual test run was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test case has fields to capture the setup duration, test run duration and teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up a test run. If the setup duration can’t be distinguished from the test’s runtime, set this value to None.

The test’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test run is the time spend on tearing down a test run. If the teardown duration can’t be distinguished from the test’s runtime, set this value to None.

The test case has a field total duration to sum up setup duration, test duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test case counts for warnings, errors and fatal errors observed in a test run while the test was executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test case implements a dictionary interface, so arbitrary key-value pairs can be annotated per test entity.

Status

The overall status of a test case.

See also: Testcase Status.

AssertionCount, PassedAssertionCount, FailedAssertionCount

The assertion count represents the overall number of assertions (checks) in a test case. It can be distinguished into passed assertions and failed assertions. If it can’t be distinguished, set passed and failed assertions to None.

AssertionCount := PassedAssertionCount + FailedAssertionCount

Copy()

tbd

Aggregate()

Aggregate (recalculate) all durations, warnings, errors, assertions, etc.

__str__()

tbd

@export
class Testcase(Base):
   def __init__(
      self,
      name: str,
      startTime: Nullable[datetime] = None,
      setupDuration: Nullable[timedelta] = None,
      testDuration: Nullable[timedelta] = None,
      teardownDuration: Nullable[timedelta] = None,
      totalDuration:  Nullable[timedelta] = None,
      status: TestcaseStatus = TestcaseStatus.Unknown,
      assertionCount: Nullable[int] = None,
      failedAssertionCount: Nullable[int] = None,
      passedAssertionCount: Nullable[int] = None,
      warningCount: int = 0,
      errorCount: int = 0,
      fatalCount: int = 0,
      parent: Nullable["Testsuite"] = None
   ):
     ...

   @readonly
   def Parent(self) -> Nullable["Testsuite"]:
     ...

   @readonly
   def Name(self) -> str:
     ...

   @readonly
   def StartTime(self) -> Nullable[datetime]:
     ...

   @readonly
   def SetupDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TestDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TeardownDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TotalDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def WarningCount(self) -> int:
     ...

   @readonly
   def ErrorCount(self) -> int:
     ...

   @readonly
   def FatalCount(self) -> int:
     ...

   def __len__(self) -> int:
     ...

   def __getitem__(self, key: str) -> Any:
     ...

   def __setitem__(self, key: str, value: Any) -> None:
     ...

   def __delitem__(self, key: str) -> None:
     ...

   def __contains__(self, key: str) -> bool:
     ...

   def __iter__(self) -> Generator[Tuple[str, Any], None, None]:
     ...

   @readonly
   def Status(self) -> TestcaseStatus:
     ...

   @readonly
   def AssertionCount(self) -> int:
     ...

   @readonly
   def FailedAssertionCount(self) -> int:
     ...

   @readonly
   def PassedAssertionCount(self) -> int:
     ...

   def Copy(self) -> "Testcase":
     ...

   def Aggregate(self, strict: bool = True) -> TestcaseAggregateReturnType:
     ...

   def __str__(self) -> str:
     ...

Testsuite

A Testsuite is a grouping element in the test entity hierarchy and describes a group of test runs. Besides a list of test cases and a test suite status, it also contains statistics like the start time or the test duration for the group of tests. Test suites are grouped by other test suites or a test suite summary and need to be unique per parent test suite.

A test suite (or its base classes) implements the following properties and methods:

Parent

The test suite has a reference to it’s parent test entity in the hierarchy. By iterating parent references, the root element (test suite summary) be be found, which has no parent reference (None).

Name

The test suite has a name. This name must be unique per hierarchy parent, but can exist multiple times in the overall test hierarchy.

In case the data format uses hierarchical names like pyEDAA.Reports.CLI.Application, the name is split at the separator and multiple hierarchy levels (test suites) are created in the unified data model. To be able to recreate such an hierarchical name, TestsuiteKind is applied accordingly to test suite’s Kind field.

StartTime

The test suite stores a time when the first test run was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test suite has fields to capture the suite’s setup duration, test group run duration and suite’s teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up a test suite. If the setup duration can’t be distinguished from the test group’s runtime, set this value to None.

The test group’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test suite is the time spend on tearing down a test suite. If the teardown duration can’t be distinguished from the test group’s runtime, set this value to None.

The test suite has a field total duration to sum up setup duration, test duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test suite counts for warnings, errors and fatal errors observed in a test suite while the tests were executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test suite implements a dictionary interface, so arbitrary key-value pairs can be annotated.

Todo

TestsuiteBase APIs

Testcases

tbd

TestcaseCount

tbd

AssertionCount

The overall number of assertions (checks) in a test case.

Aggregate()

Aggregate (recalculate) all durations, warnings, errors, assertions, etc.

Iterate()

tbd

__str__()

tbd

@export
class Testsuite(TestsuiteBase[TestsuiteType]):
   def __init__(
      self,
      name: str,
      kind: TestsuiteKind = TestsuiteKind.Logical,
      startTime: Nullable[datetime] = None,
      setupDuration: Nullable[timedelta] = None,
      testDuration: Nullable[timedelta] = None,
      teardownDuration: Nullable[timedelta] = None,
      totalDuration:  Nullable[timedelta] = None,
      status: TestsuiteStatus = TestsuiteStatus.Unknown,
      warningCount: int = 0,
      errorCount: int = 0,
      fatalCount: int = 0,
      testsuites: Nullable[Iterable[TestsuiteType]] = None,
      testcases: Nullable[Iterable["Testcase"]] = None,
      parent: Nullable[TestsuiteType] = None
   ):
     ...

   @readonly
   def Parent(self) -> Nullable["Testsuite"]:
     ...

   @readonly
   def Name(self) -> str:
     ...

   @readonly
   def StartTime(self) -> Nullable[datetime]:
     ...

   @readonly
   def SetupDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TestDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TeardownDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TotalDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def WarningCount(self) -> int:
     ...

   @readonly
   def ErrorCount(self) -> int:
     ...

   @readonly
   def FatalCount(self) -> int:
     ...

   def __len__(self) -> int:
     ...

   def __getitem__(self, key: str) -> Any:
     ...

   def __setitem__(self, key: str, value: Any) -> None:
     ...

   def __delitem__(self, key: str) -> None:
     ...

   def __contains__(self, key: str) -> bool:
     ...

   def __iter__(self) -> Generator[Tuple[str, Any], None, None]:
     ...

   # TestsuiteBase API


   @readonly
   def Testcases(self) -> Dict[str, "Testcase"]:
     ...

   @readonly
   def TestcaseCount(self) -> int:
     ...

   @readonly
   def AssertionCount(self) -> int:
     ...

   def Aggregate(self, strict: bool = True) -> TestsuiteAggregateReturnType:
     ...

   def Iterate(self, scheme: IterationScheme = IterationScheme.Default) -> Generator[Union[TestsuiteType, Testcase], None, None]:
     ...

   def __str__(self) -> str:
     ...

TestsuiteSummary

A TestsuiteSummary is the root element in the test entity hierarchy and describes a group of test suites as well as overall statistics for the whole set of test cases. A test suite summary is derived for the same base-class as a test suite, thus they share almost all properties and methods.

A test suite summary (or its base classes) implements the following properties and methods:

Parent

The test suite summary has a parent reference, but as the root element in the test entity hierarchy, its always None.

Name

The test suite summary has a name.

StartTime

The test suite summary stores a time when the first test runs was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test suite summary has fields to capture the suite summary’s setup duration, overall run duration and suite summary’s teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up an overall test run. If the setup duration can’t be distinguished from the test’s runtimes, set this value to None.

The test suite summary’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test suite summary is the time spend on tearing down a test suite summary. If the teardown duration can’t be distinguished from the test’s runtimes, set this value to None.

The test suite summary has a field total duration to sum up setup duration, overall run duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test suite summary counts for warnings, errors and fatal errors observed in a test suite summary while the tests were executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test suite summary implements a dictionary interface, so arbitrary key-value pairs can be annotated.

Todo

TestsuiteBase APIs

Aggregate()

tbd

Iterate()

tbd

__str__()

tbd

@export
class TestsuiteSummary(TestsuiteBase[TestsuiteType]):
   def __init__(
      self,
      name: str,
      startTime: Nullable[datetime] = None,
      setupDuration: Nullable[timedelta] = None,
      testDuration: Nullable[timedelta] = None,
      teardownDuration: Nullable[timedelta] = None,
      totalDuration:  Nullable[timedelta] = None,
      status: TestsuiteStatus = TestsuiteStatus.Unknown,
      warningCount: int = 0,
      errorCount: int = 0,
      fatalCount: int = 0,
      testsuites: Nullable[Iterable[TestsuiteType]] = None,
      parent: Nullable[TestsuiteType] = None
   ):
     ...

   @readonly
   def Parent(self) -> Nullable["Testsuite"]:
     ...

   @readonly
   def Name(self) -> str:
     ...

   @readonly
   def StartTime(self) -> Nullable[datetime]:
     ...

   @readonly
   def SetupDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TestDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TeardownDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TotalDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def WarningCount(self) -> int:
     ...

   @readonly
   def ErrorCount(self) -> int:
     ...

   @readonly
   def FatalCount(self) -> int:
     ...

   def __len__(self) -> int:
     ...

   def __getitem__(self, key: str) -> Any:
     ...

   def __setitem__(self, key: str, value: Any) -> None:
     ...

   def __delitem__(self, key: str) -> None:
     ...

   def __contains__(self, key: str) -> bool:
     ...

   def __iter__(self) -> Generator[Tuple[str, Any], None, None]:
     ...

   # TestsuiteBase API

   def Aggregate(self) -> TestsuiteAggregateReturnType:
     ...

   def Iterate(self, scheme: IterationScheme = IterationScheme.Default) -> Generator[Union[TestsuiteType, Testcase], None, None]:
     ...

   def __str__(self) -> str:
     ...

Document

A Document is a mixin-class …

Path

tbd

AnalysisDuration

tbd

ModelConversionDuration

tbd

Analyze()

tbd

Convert()

tbd

@export
class Document(metaclass=ExtendedType, mixin=True):
   def __init__(self, path: Path):
     ...

   @readonly
   def Path(self) -> Path:
     ...

   @readonly
   def AnalysisDuration(self) -> timedelta:
     ...

   @readonly
   def ModelConversionDuration(self) -> timedelta:
     ...

   @abstractmethod
   def Analyze(self) -> None:
     ...

   @abstractmethod
   def Convert(self):
     ...