Unittesting

pyEDAA.Reports provides a unified and generic unittest summary data model. The data model allows the description of testcases grouped in testsuites. Testsuites can be nested in other testsuites. The data model’s root element is a special testsuite called testsuite summary. It contains only testsuites, but no testcases.

The data model can be filled from various sources like Ant JUnit test reports or OSVVM testsuite summaries (more to be added). Many programming languages and/or unit testing frameworks support exporting results in the Ant JUnit format. See below for supported formats and their variations (dialects).

Attention

The so called JUnit XML format is the weakest file format and standard ever seen. At first was not created by JUnit (version 4). It was added by the built system Ant, but it’s not called Ant XML format nor Ant JUnit XML format. The latest JUnit 5 uses a completely different format called open test reporting. As JUnit is not the formats author, no file format documentation nor XML schema was provided. Also Ant isn’t providing any file format documentation or XML schema. Various Ant JUnit XML adopters have tried to reverse engineer a description and XML schemas, but unfortunately many are not even compatible to each other.

Unified data model

The unified data model for test entities (test summary, test suite, test case) implements a super-set of all (so far known) unit test result summary file formats. pyEDAA.Report’s data model is a structural and functional cleanup of the Ant JUnit data model. Naming has been cleaned up and missing features have been added.

As some of the JUnit XML dialects are too divergent from the original Ant + JUnit4 format, these dialects have an independent test entity inheritance hierarchy. Nonetheless, instances of each data format can be converted to and from the unified data model.

Testcase

A test case is the leaf-element in the test entity hierarchy and describes an individual test run. Test cases are grouped by test suites.

Testsuite

A test suite is a group of test cases and/or test suites. Test suites itself can be grouped by test suites. The test suite hierarchy’s root element is a test suite summary.

TestsuiteSummary

The test suite summary is derived from test suite and defines the root of the test suite hierarchy.

Document

The document is derived from a test suite summary and represents a file containing a test suite summary.

        graph TD;
  doc[Document]
  sum[Summary]
  ts1[Testsuite]
  ts2[Testsuite]
  ts21[Testsuite]
  tc11[Testcase]
  tc12[Testcase]
  tc13[Testcase]
  tc21[Testcase]
  tc22[Testcase]
  tc211[Testcase]
  tc212[Testcase]
  tc213[Testcase]

  doc:::root -.-> sum:::summary
  sum --> ts1:::suite
  sum --> ts2:::suite
  ts2 --> ts21:::suite
  ts1 --> tc11:::case
  ts1 --> tc12:::case
  ts1 --> tc13:::case
  ts2 --> tc21:::case
  ts2 --> tc22:::case
  ts21 --> tc211:::case
  ts21 --> tc212:::case
  ts21 --> tc213:::case

  classDef root fill:#4dc3ff
  classDef summary fill:#80d4ff
  classDef suite fill:#b3e6ff
  classDef case fill:#eeccff
    

Testcase Status

TestcaseStatus and TestsuiteStatus are flag enumerations to describe the overall status of a test case or test suite.

Unknown

tbd

Excluded

tbd

Skipped

tbd

Weak

tbd

Passed

tbd

Failed

tbd

Inverted

tbd

Warned

tbd

Errored

tbd

Failed

tbd

SetupError

tbd

TearDownError

tbd

Inconsistent

tbd

@export
class TestcaseStatus(Flag):
  """A flag enumeration describing the status of a test case."""
  Unknown =    0                         #: Testcase status is uninitialized and therefore unknown.
  Excluded =   1                         #: Testcase was permanently excluded / disabled
  Skipped =    2                         #: Testcase was temporarily skipped (e.g. based on a condition)
  Weak =       4                         #: No assertions were recorded.
  Passed =     8                         #: A passed testcase, because all assertions were successful.
  Failed =    16                         #: A failed testcase due to at least one failed assertion.

  Mask = Excluded | Skipped | Weak | Passed | Failed

  Inverted = 128                         #: To mark inverted results
  UnexpectedPassed = Failed | Inverted
  ExpectedFailed =   Passed | Inverted

  Warned =  1024                         #: Runtime warning
  Errored = 2048                         #: Runtime error (mostly caught exceptions)
  Aborted = 4096                         #: Uncaught runtime exception

  SetupError =     8192                  #: Preparation / compilation error
  TearDownError = 16384                  #: Cleanup error / resource release error
  Inconsistent = 32768                   #: Dataset is inconsistent

  Flags = Warned | Errored | Aborted | SetupError | TearDownError | Inconsistent

@export
class TestsuiteStatus(Flag):
  """A flag enumeration describing the status of a test suite."""
  Unknown =    0
  Excluded =   1                         #: Testcase was permanently excluded / disabled
  Skipped =    2                         #: Testcase was temporarily skipped (e.g. based on a condition)
  Empty =      4                         #: No tests in suite
  Passed =     8                         #: Passed testcase, because all assertions succeeded
  Failed =    16                         #: Failed testcase due to failing assertions

  Mask = Excluded | Skipped | Empty | Passed | Failed

  Inverted = 128                         #: To mark inverted results
  UnexpectedPassed = Failed | Inverted
  ExpectedFailed =   Passed | Inverted

  Warned =  1024                         #: Runtime warning
  Errored = 2048                         #: Runtime error (mostly caught exceptions)
  Aborted = 4096                         #: Uncaught runtime exception

  SetupError =     8192                  #: Preparation / compilation error
  TearDownError = 16384                  #: Cleanup error / resource release error

  Flags = Warned | Errored | Aborted | SetupError | TearDownError

Testcase

A Testcase is the leaf-element in the test entity hierarchy and describes an individual test run. Besides a test case status, it also contains statistics like the start time or the test duration. Test cases are grouped by test suites and need to be unique per parent test suite.

A test case (or its base classes) implements the following properties and methods:

Parent

The test case has a reference to it’s parent test suite in the hierarchy. By iterating parent references, the root element (test suite summary) be be found, which has no parent reference (None).

Name

The test case has a name. This name must be unique per hierarchy parent, but can exist multiple times in the overall test hierarchy.

In case the data format uses hierarchical names like pyEDAA.Reports.CLI.Application, the name is split at the separator and multiple hierarchy levels (test suites) are created in the unified data model. To be able to recreate such an hierarchical name, TestsuiteKind is applied accordingly to test suite’s Kind field.

StartTime

The test case stores a time when the individual test run was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test case has fields to capture the setup duration, test run duration and teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up a test run. If the setup duration can’t be distinguished from the test’s runtime, set this value to None.

The test’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test run is the time spend on tearing down a test run. If the teardown duration can’t be distinguished from the test’s runtime, set this value to None.

The test case has a field total duration to sum up setup duration, test duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test case counts for warnings, errors and fatal errors observed in a test run while the test was executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test case implements a dictionary interface, so arbitrary key-value pairs can be annotated per test entity.

Status

The overall status of a test case.

See also: Testcase Status.

AssertionCount, PassedAssertionCount, FailedAssertionCount

The assertion count represents the overall number of assertions (checks) in a test case. It can be distinguished into passed assertions and failed assertions. If it can’t be distinguished, set passed and failed assertions to None.

AssertionCount := PassedAssertionCount + FailedAssertionCount

Copy()

tbd

Aggregate()

Aggregate (recalculate) all durations, warnings, errors, assertions, etc.

__str__()

tbd

@export
class Testcase(Base):
   def __init__(
      self,
      name: str,
      startTime: Nullable[datetime] = None,
      setupDuration: Nullable[timedelta] = None,
      testDuration: Nullable[timedelta] = None,
      teardownDuration: Nullable[timedelta] = None,
      totalDuration:  Nullable[timedelta] = None,
      status: TestcaseStatus = TestcaseStatus.Unknown,
      assertionCount: Nullable[int] = None,
      failedAssertionCount: Nullable[int] = None,
      passedAssertionCount: Nullable[int] = None,
      warningCount: int = 0,
      errorCount: int = 0,
      fatalCount: int = 0,
      parent: Nullable["Testsuite"] = None
   ):
     ...

   @readonly
   def Parent(self) -> Nullable["Testsuite"]:
     ...

   @readonly
   def Name(self) -> str:
     ...

   @readonly
   def StartTime(self) -> Nullable[datetime]:
     ...

   @readonly
   def SetupDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TestDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TeardownDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TotalDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def WarningCount(self) -> int:
     ...

   @readonly
   def ErrorCount(self) -> int:
     ...

   @readonly
   def FatalCount(self) -> int:
     ...

   def __len__(self) -> int:
     ...

   def __getitem__(self, key: str) -> Any:
     ...

   def __setitem__(self, key: str, value: Any) -> None:
     ...

   def __delitem__(self, key: str) -> None:
     ...

   def __contains__(self, key: str) -> bool:
     ...

   def __iter__(self) -> Generator[Tuple[str, Any], None, None]:
     ...

   @readonly
   def Status(self) -> TestcaseStatus:
     ...

   @readonly
   def AssertionCount(self) -> int:
     ...

   @readonly
   def FailedAssertionCount(self) -> int:
     ...

   @readonly
   def PassedAssertionCount(self) -> int:
     ...

   def Copy(self) -> "Testcase":
     ...

   def Aggregate(self, strict: bool = True) -> TestcaseAggregateReturnType:
     ...

   def __str__(self) -> str:
     ...

Testsuite

A Testsuite is a grouping element in the test entity hierarchy and describes a group of test runs. Besides a list of test cases and a test suite status, it also contains statistics like the start time or the test duration for the group of tests. Test suites are grouped by other test suites or a test suite summary and need to be unique per parent test suite.

A test suite (or its base classes) implements the following properties and methods:

Parent

The test suite has a reference to it’s parent test entity in the hierarchy. By iterating parent references, the root element (test suite summary) be be found, which has no parent reference (None).

Name

The test suite has a name. This name must be unique per hierarchy parent, but can exist multiple times in the overall test hierarchy.

In case the data format uses hierarchical names like pyEDAA.Reports.CLI.Application, the name is split at the separator and multiple hierarchy levels (test suites) are created in the unified data model. To be able to recreate such an hierarchical name, TestsuiteKind is applied accordingly to test suite’s Kind field.

StartTime

The test suite stores a time when the first test run was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test suite has fields to capture the suite’s setup duration, test group run duration and suite’s teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up a test suite. If the setup duration can’t be distinguished from the test group’s runtime, set this value to None.

The test group’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test suite is the time spend on tearing down a test suite. If the teardown duration can’t be distinguished from the test group’s runtime, set this value to None.

The test suite has a field total duration to sum up setup duration, test duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test suite counts for warnings, errors and fatal errors observed in a test suite while the tests were executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test suite implements a dictionary interface, so arbitrary key-value pairs can be annotated.

Todo

TestsuiteBase APIs

Testcases

tbd

TestcaseCount

tbd

AssertionCount

The overall number of assertions (checks) in a test case.

Aggregate()

Aggregate (recalculate) all durations, warnings, errors, assertions, etc.

Iterate()

tbd

__str__()

tbd

@export
class Testsuite(TestsuiteBase[TestsuiteType]):
   def __init__(
      self,
      name: str,
      kind: TestsuiteKind = TestsuiteKind.Logical,
      startTime: Nullable[datetime] = None,
      setupDuration: Nullable[timedelta] = None,
      testDuration: Nullable[timedelta] = None,
      teardownDuration: Nullable[timedelta] = None,
      totalDuration:  Nullable[timedelta] = None,
      status: TestsuiteStatus = TestsuiteStatus.Unknown,
      warningCount: int = 0,
      errorCount: int = 0,
      fatalCount: int = 0,
      testsuites: Nullable[Iterable[TestsuiteType]] = None,
      testcases: Nullable[Iterable["Testcase"]] = None,
      parent: Nullable[TestsuiteType] = None
   ):
     ...

   @readonly
   def Parent(self) -> Nullable["Testsuite"]:
     ...

   @readonly
   def Name(self) -> str:
     ...

   @readonly
   def StartTime(self) -> Nullable[datetime]:
     ...

   @readonly
   def SetupDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TestDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TeardownDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TotalDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def WarningCount(self) -> int:
     ...

   @readonly
   def ErrorCount(self) -> int:
     ...

   @readonly
   def FatalCount(self) -> int:
     ...

   def __len__(self) -> int:
     ...

   def __getitem__(self, key: str) -> Any:
     ...

   def __setitem__(self, key: str, value: Any) -> None:
     ...

   def __delitem__(self, key: str) -> None:
     ...

   def __contains__(self, key: str) -> bool:
     ...

   def __iter__(self) -> Generator[Tuple[str, Any], None, None]:
     ...

   # TestsuiteBase API


   @readonly
   def Testcases(self) -> Dict[str, "Testcase"]:
     ...

   @readonly
   def TestcaseCount(self) -> int:
     ...

   @readonly
   def AssertionCount(self) -> int:
     ...

   def Aggregate(self, strict: bool = True) -> TestsuiteAggregateReturnType:
     ...

   def Iterate(self, scheme: IterationScheme = IterationScheme.Default) -> Generator[Union[TestsuiteType, Testcase], None, None]:
     ...

   def __str__(self) -> str:
     ...

TestsuiteSummary

A TestsuiteSummary is the root element in the test entity hierarchy and describes a group of test suites as well as overall statistics for the whole set of test cases. A test suite summary is derived for the same base-class as a test suite, thus they share almost all properties and methods.

A test suite summary (or its base classes) implements the following properties and methods:

Parent

The test suite summary has a parent reference, but as the root element in the test entity hierarchy, its always None.

Name

The test suite summary has a name.

StartTime

The test suite summary stores a time when the first test runs was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test suite summary has fields to capture the suite summary’s setup duration, overall run duration and suite summary’s teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up an overall test run. If the setup duration can’t be distinguished from the test’s runtimes, set this value to None.

The test suite summary’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test suite summary is the time spend on tearing down a test suite summary. If the teardown duration can’t be distinguished from the test’s runtimes, set this value to None.

The test suite summary has a field total duration to sum up setup duration, overall run duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test suite summary counts for warnings, errors and fatal errors observed in a test suite summary while the tests were executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test suite summary implements a dictionary interface, so arbitrary key-value pairs can be annotated.

Todo

TestsuiteBase APIs

Aggregate()

tbd

Iterate()

tbd

__str__()

tbd

@export
class TestsuiteSummary(TestsuiteBase[TestsuiteType]):
   def __init__(
      self,
      name: str,
      startTime: Nullable[datetime] = None,
      setupDuration: Nullable[timedelta] = None,
      testDuration: Nullable[timedelta] = None,
      teardownDuration: Nullable[timedelta] = None,
      totalDuration:  Nullable[timedelta] = None,
      status: TestsuiteStatus = TestsuiteStatus.Unknown,
      warningCount: int = 0,
      errorCount: int = 0,
      fatalCount: int = 0,
      testsuites: Nullable[Iterable[TestsuiteType]] = None,
      parent: Nullable[TestsuiteType] = None
   ):
     ...

   @readonly
   def Parent(self) -> Nullable["Testsuite"]:
     ...

   @readonly
   def Name(self) -> str:
     ...

   @readonly
   def StartTime(self) -> Nullable[datetime]:
     ...

   @readonly
   def SetupDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TestDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TeardownDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TotalDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def WarningCount(self) -> int:
     ...

   @readonly
   def ErrorCount(self) -> int:
     ...

   @readonly
   def FatalCount(self) -> int:
     ...

   def __len__(self) -> int:
     ...

   def __getitem__(self, key: str) -> Any:
     ...

   def __setitem__(self, key: str, value: Any) -> None:
     ...

   def __delitem__(self, key: str) -> None:
     ...

   def __contains__(self, key: str) -> bool:
     ...

   def __iter__(self) -> Generator[Tuple[str, Any], None, None]:
     ...

   # TestsuiteBase API

   def Aggregate(self) -> TestsuiteAggregateReturnType:
     ...

   def Iterate(self, scheme: IterationScheme = IterationScheme.Default) -> Generator[Union[TestsuiteType, Testcase], None, None]:
     ...

   def __str__(self) -> str:
     ...

Document

A Document is a mixin-class …

Path

tbd

AnalysisDuration

tbd

ModelConversionDuration

tbd

Analyze()

tbd

Convert()

tbd

@export
class Document(metaclass=ExtendedType, mixin=True):
   def __init__(self, path: Path):
     ...

   @readonly
   def Path(self) -> Path:
     ...

   @readonly
   def AnalysisDuration(self) -> timedelta:
     ...

   @readonly
   def ModelConversionDuration(self) -> timedelta:
     ...

   @abstractmethod
   def Analyze(self) -> None:
     ...

   @abstractmethod
   def Convert(self):
     ...

Specific Data Models

JUnit Data Model

Testcase

A test case is the leaf-element in the test entity hierarchy and describes an individual test run. Test cases are grouped by test classes.

Testclass

A test class is the mid-level element in the test entity hierarchy and describes a group of test runs. Test classes are grouped by test suites.

Testsuite

A test suite is a group of test classes. Test suites are grouped by a test suite summary.

TestsuiteSummary

The test suite summary is derived from test suite and defines the root of the test suite hierarchy.

Document

The document is derived from a test suite summary and represents a file containing a test suite summary.

JUnit Dialects

The JUnit format is not well defined, thus multiple dialects developed over time.

        graph TD;
  doc[Document]
  sum[Summary]
  ts1[Testsuite]
  ts11[Testsuite]
  ts2[Testsuite]

  tc111[Testclass]
  tc112[Testclass]
  tc23[Testclass]

  tc1111[Testcase]
  tc1112[Testcase]
  tc1113[Testcase]
  tc1121[Testcase]
  tc1122[Testcase]
  tc231[Testcase]
  tc232[Testcase]
  tc233[Testcase]

  doc:::root -.-> sum:::summary
  sum --> ts1:::suite
  sum ---> ts2:::suite
  ts1 --> ts11:::suite

  ts11 --> tc111:::cls
  ts11 --> tc112:::cls
  ts2  --> tc23:::cls

  tc111 --> tc1111:::case
  tc111 --> tc1112:::case
  tc111 --> tc1113:::case
  tc112 --> tc1121:::case
  tc112 --> tc1122:::case
  tc23 --> tc231:::case
  tc23 --> tc232:::case
  tc23 --> tc233:::case

  classDef root fill:#4dc3ff
  classDef summary fill:#80d4ff
  classDef suite fill:#b3e6ff
  classDef cls fill:#ff9966
  classDef case fill:#eeccff
    

Testcase

Testclass

Testsuite

TestsuiteSummary

Document

JUnit Dialects

As the JUnit XML format was not well specified and no XML Schema Definition (XSD) was provided, many variants and dialects (and simplifications) were created by the various frameworks emitting JUnit XML files.

JUnit Dialect Comparison

Feature	Any JUnit	Ant + JUnit4	CTest JUnit	GoogleTest JUnit	pyTest JUnit
Root element	testsuites	testsuite	testsuite	testsuites	testsuites
Supports properties	☑	☑		⸺
Testcase status	…	…	more status values

Any JUnit

The Any JUnit format uses a relaxed XML schema definition aiming to parse many JUnit XML dialects, which use a <testsuites> root element.

Reading Any JUnit

from pyEDAA.Reports.Unittesting.JUnit import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

Convert to and from Unified Data Model

from pyEDAA.Reports.Unittesting.JUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing Any JUnit

from pyEDAA.Reports.Unittesting.JUnit import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestException as ex:
  ...

Ant + JUnit4

The original JUnit format created by Ant for JUnit4 uses <testsuite> as a root element.

Reading Ant + JUnit4

from pyEDAA.Reports.Unittesting.JUnit.AntJUnit4 import Document

xmlReport = Path("AntJUnit4-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

Convert to and from Unified Data Model

from pyEDAA.Reports.Unittesting.JUnit.AntJUnit4 import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing Ant + JUnit4

from pyEDAA.Reports.Unittesting.JUnit.AntJUnit4 import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestException as ex:
  ...

CTest JUnit

The CTest JUnit format written by CTest uses <testsuite> as a root element.

Reading CTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

xmlReport = Path("CTestJUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

Convert to and from Unified Data Model

from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing CTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestException as ex:
  ...

GoogleTest JUnit

The GoogleTest JUnit format written by GoogleTest (sometimes GTest) uses <testsuites> as a root element.

Reading GoogleTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

xmlReport = Path("GoogleTestJUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

Convert to and from Unified Data Model

from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing GoogleTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestException as ex:
  ...

pyTest JUnit

The pyTest JUnit format written by pyTest uses <testsuites> as a root element.

Reading pyTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

xmlReport = Path("PyTestJUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

Convert to and from Unified Data Model

from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing pyTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestException as ex:
  ...

OSVVM

Open Source VHDL Verification Methodology writes test results as YAML files for its internal data model storage. Some YAML files are written by the VHDL code of the verification framework, others are written by OSVVM-Scripts as a test runner.

Features

Create test entities

The hierarchy of test entities (test cases, test suites and test summaries) can be constructed top-down or bottom-up.

Test Case

from pyEDAA.Reports.Unittesting import Testsuite, Testcase

# Top-down
ts1 = Testsuite("ts1")

tc = Testcase("tc", parent=ts)

# Bottom-up
tc1 = Testcase("tc1")
tc2 = Testcase("tc2")

ts2 = Testsuite("ts2", testcases=(tc1, tc2))

# ts.AddTestcase(...)
tc3 = Testcase("tc3")
tc4 = Testcase("tc4")

ts3 = Testsuite("ts3")
ts3.AddTestcase(tc3)
ts3.AddTestcase(tc4)

# ts.AddTestcases(...)
tc3 = Testcase("tc3")
tc4 = Testcase("tc4")

ts3 = Testsuite("ts3")
ts3.AddTestcases((tc3, tc4))

Test Suite

from pyEDAA.Reports.Unittesting import Testsuite, TestsuiteSummary

# Top-down
ts = Testsuite("ts")

ts1 = Testsuite("ts1", parent=tss)

# Bottom-up
ts2 = Testsuite("ts2")
ts3 = Testsuite("ts3")

ts4 = Testsuite("ts4", testsuites=(ts2, ts3))

# ts.AddTestsuite(...)
ts5 = Testcase("ts5")
ts6 = Testcase("ts6")

ts7 = Testsuite("ts7")
ts7.AddTestsuite(ts5)
ts7.AddTestsuite(ts6)

# ts.AddTestsuites(...)
ts8 = Testcase("ts8")
ts9 = Testcase("ts9")

ts10 = Testsuite("ts10")
ts10.AddTestsuites((ts8, ts9))

Test Suite Summary

from pyEDAA.Reports.Unittesting import Testsuite, TestsuiteSummary

# Top-down

# Bottom-up

Reading unittest reports

A JUnit XML test report summary file can be read by creating an instance of the Document class. Because JUnit has so many dialects, a derived subclass for the dialect might be required. By choosing the right Document class, also the XML schema for XML schema validation gets pre-selected.

Any JUnit

from pyEDAA.Reports.Unittesting.JUnit import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

Ant + JUnit4

from pyEDAA.Reports.Unittesting.JUnit.AntJUnit import Document

xmlReport = Path("AntJUnit4-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

CTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

xmlReport = Path("CTest-JUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

GoogleTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

xmlReport = Path("GoogleTest-JUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

pyTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

xmlReport = Path("pyTest-JUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

Converting unittest reports

Any JUnit dialect specific data model can be converted to the generic hierarchy of test entities.

Note

This conversion is identical for all derived dialects.

Document

from pyEDAA.Reports.Unittesting.JUnit import Document

# Read from XML file
xmlReport = Path("JUnit-Report.xml")
try:
  doc = Document(xmlReport, parse=True)
except UnittestException as ex:
  ...

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert to a tree
rootNode = doc.ToTree()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

# Write to XML file
newDoc.Write()

Annotations

Every test entity can be annotated with arbitrary key-value pairs.

Testcase

# Add annotate a key-value pair
testcase["key"] = value

# Update existing annotation with new value
testcase["key"] = newValue

# Check if key exists
if "key" in testcase:
  pass

# Access annoation by key
value = testcase["key"]

# Get number of annotations
annotationCount = len(testcase)

# Delete annotation
del testcase["key"]

# Iterate annotations
for key, value in testcases:
   pass

Testsuite

# Add annotate a key-value pair
testsuite["key"] = value

# Update existing annotation with new value
testsuite["key"] = newValue

# Check if key exists
if "key" in testsuite:
  pass

# Access annoation by key
value = testsuite["key"]

# Get number of annotations
annotationCount = len(testsuite)

# Delete annotation
del testsuite["key"]

# Iterate annotations
for key, value in testsuite:
   pass

TestsuiteSummary

# Add annotate a key-value pair
testsuiteSummary["key"] = value

# Update existing annotation with new value
testsuiteSummary["key"] = newValue

# Check if key exists
if "key" in testsuiteSummary:
  pass

# Access annoation by key
value = testsuiteSummary["key"]

# Get number of annotations
annotationCount = len(testsuiteSummary)

# Delete annotation
del testsuiteSummary["key"]

# Iterate annotations
for key, value in testsuiteSummary:
   pass

Merging unittest reports

add description here

Testcase

# add code here

Concatenate unittest reports

Todo

Planned feature.

Transforming the reports’ hierarchy

pytest specific transformations

add description here

Testcase

# add code here

Writing unittest reports

A test suite summary can be converted to a document of any JUnit dialect. Internally a deep-copy is created to convert from a hierarchy of the unified test entities to a hierarchy of specific test entities (e.g. JUnit entities).

When the document was created, it can be written to disk.

Any JUnit

from pyEDAA.Reports.Unittesting.JUnit import Document

# Convert a TestsuiteSummary back to a Document
newXmlReport = Path("JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

# Write to XML file
try:
   newDoc.Write()
except UnittestException as ex:
  ...

Ant + JUnit4

from pyEDAA.Reports.Unittesting.JUnit.AntJUnit import Document

# Convert a TestsuiteSummary back to a Document
newXmlReport = Path("JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

# Write to XML file
try:
   newDoc.Write()
except UnittestException as ex:
  ...

CTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

# Convert a TestsuiteSummary back to a Document
newXmlReport = Path("JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

# Write to XML file
try:
   newDoc.Write()
except UnittestException as ex:
  ...

GoogleTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

# Convert a TestsuiteSummary back to a Document
newXmlReport = Path("JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

# Write to XML file
try:
   newDoc.Write()
except UnittestException as ex:
  ...

pyTest JUnit

from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

# Convert a TestsuiteSummary back to a Document
newXmlReport = Path("JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

# Write to XML file
try:
   newDoc.Write()
except UnittestException as ex:
  ...

Command Line Tool

pyedaa-reports unittest --input=Ant-JUnit:data/JUnit.xml

File Formats

Unittest summary reports can be stored in various file formats. Usually these files are XML based. Due to missing (clear) specifications and XML schema definitions, some file formats have developed dialects. Either because the specification was unclear/not existing or because the format was specific for a single programming language, so tools added extensions or misused XML attributes instead of designing their own file format.

Ant and JUnit 4 XML

The so-called JUnit XML format was defined by Ant when running JUnit4 test suites. Because the format was not specified by JUnit4, many dialects spread out. Many tools and test frameworks have minor or major differences compared to the original format. While some modifications seam logical additions or adjustments to the needs of the respective framework, others undermine the ideas and intents of the data format.

Many issues arise because the Ant + JUnit4 format is specific to unit testing with Java. Other languages and frameworks were lazy and didn’t derive their own format, but rather stuffed their language constructs into the concepts and limitations of the Ant + JUnit4 XML format.

JUnit Dialects

• 🚧 Bamboo JUnit (planned)

• ✅ CTest JUnit format

• ✅ GoogleTest JUnit format

• 🚧 Jenkins JUnit (planned)

• ✅ pyTest JUnit format

JUnit 5 XML

JUnit5 uses a new format called Open Test Reporting (see the following section for details). This format isn’t specific to Java (packages, classes, methods, …), but describes a generic data model. Of cause an extension for Java specifics is provided too.

Open Test Reporting

The Open Test Alliance created a new format called Open Test Reporting (OTR) to overcome the shortcommings of a missing file format for JUnit5 as well as the problems of Ant + JUnit4.

OTR defines a structure of test groups and tests, but no specifics of a certain programming languge. The logical structure of tests and test groups is decoupled from language specifics like namespaces, packages or classes hosting the individual tests.

OSVVM YAML

The Open Source VHDL Verification Methodology (OSVVM) defines its own test report format in YAML. While OSVVM is able to convert its own YAML files to JUnit XML files, it’s recommended to use the YAML files as data source, because these contain additional information, which can’t be expressed with JUnit XML.

The YAML files are created when OSVVM-based testbenches are executed with OSVVM’s embedded TCL scripting environment OSVVM-Scripts.

Hint

YAML was chosen instead of JSON or XML, because a YAML document isn’t corrupted in case of a runtime error. The document might be incomplete (content), but not corrupted (structural). Such a scenario is possible if a VHDL simulator stops execution, then the document structure can’t be finalized.

Frameworks / Tools

CTest

• https://github.com/bvdberg/ctest

GoogleTest (gtest)

• https://github.com/google/googletest

JUnit4

• https://github.com/apache/ant

• https://github.com/junit-team/junit4

JUnit5

OSVVM

• https://github.com/OSVVM/OSVVM

• https://github.com/OSVVM/OSVVM-Scripts

pytest

• https://github.com/pytest-dev/pytest

VUnit

• https://github.com/VUnit/vunit

Consumers

GitLab

Jenkins

Dorney (GitHub Action)

• https://github.com/dorny/test-reporter