Unittesting

pyEDAA.Reports provides a unified and generic unittest summary data model. The data model allows the description of testcases grouped in testsuites. Testsuites can be nested in other testsuites. The data model’s root element is a special testsuite called testsuite summary. It contains only testsuites, but no testcases.

The data model can be filled from various sources like Ant JUnit test reports, Open Test Reporting files, pyTooling test reports or OSVVM testsuite summaries (more to be added). Many programming languages and/or unit testing frameworks support exporting results in the Ant JUnit format. See below for supported formats and their variations (dialects).

Attention

The so called JUnit XML format is the weakest file format and standard ever seen. At first was not created by JUnit (version 4). It was added by the built system Ant, but it’s not called Ant XML format nor Ant JUnit XML format. The latest JUnit 5 uses a completely different format called open test reporting. As JUnit is not the formats author, no file format documentation nor XML schema was provided. Also Ant isn’t providing any file format documentation or XML schema. Various Ant JUnit XML adopters have tried to reverse engineer a description and XML schemas, but unfortunately many are not even compatible to each other.

Unified data model

The unified data model for test entities (test summary, test suite, test case) implements a super-set of all (so far known) unit test result summary file formats. pyEDAA.Report’s data model is a structural and functional cleanup of the Ant JUnit data model. Naming has been cleaned up and missing features have been added.

As some of the JUnit XML dialects are too divergent from the original Ant + JUnit4 format, these dialects have an independent test entity inheritance hierarchy. Nonetheless, instances of each data format can be converted to and from the unified data model.

Testcase

A test case is the leaf-element in the test entity hierarchy and describes an individual test run. Test cases are grouped by test suites.

Testsuite

A test suite is a group of test cases and/or test suites. Test suites itself can be grouped by test suites. The test suite hierarchy’s root element is a test suite summary.

TestsuiteSummary

The test suite summary is derived from test suite and defines the root of the test suite hierarchy.

Document

The document is derived from a test suite summary and represents a file containing a test suite summary.

        graph TD;
  doc[Document]
  sum[Summary]
  ts1[Testsuite]
  ts2[Testsuite]
  ts21[Testsuite]
  tc11[Testcase]
  tc12[Testcase]
  tc13[Testcase]
  tc21[Testcase]
  tc22[Testcase]
  tc211[Testcase]
  tc212[Testcase]
  tc213[Testcase]

  doc:::root -.-> sum:::summary
  sum --> ts1:::suite
  sum --> ts2:::suite
  ts2 --> ts21:::suite
  ts1 --> tc11:::case
  ts1 --> tc12:::case
  ts1 --> tc13:::case
  ts2 --> tc21:::case
  ts2 --> tc22:::case
  ts21 --> tc211:::case
  ts21 --> tc212:::case
  ts21 --> tc213:::case

  classDef root fill:#4dc3ff
  classDef summary fill:#80d4ff
  classDef suite fill:#b3e6ff
  classDef case fill:#eeccff

Testcase Status

TestcaseStatus and TestsuiteStatus are flag enumerations to describe the overall status of a test case or test suite.

Unknown: tbd
Excluded: tbd
Skipped: tbd
Weak: tbd
Passed: tbd
Failed: tbd
Inverted: tbd
Warned: tbd
Errored: tbd
Failed: tbd
SetupError: tbd
TearDownError: tbd
Inconsistent: tbd

@export
class TestcaseStatus(Flag):
  """A flag enumeration describing the status of a test case."""
  Unknown =    0                         #: Testcase status is uninitialized and therefore unknown.
  Excluded =   1                         #: Testcase was permanently excluded / disabled
  Skipped =    2                         #: Testcase was temporarily skipped (e.g. based on a condition)
  Weak =       4                         #: No assertions were recorded.
  Passed =     8                         #: A passed testcase, because all assertions were successful.
  Failed =    16                         #: A failed testcase due to at least one failed assertion.

  Mask = Excluded | Skipped | Weak | Passed | Failed

  Inverted = 128                         #: To mark inverted results
  UnexpectedPassed = Failed | Inverted
  ExpectedFailed =   Passed | Inverted

  Warned =  1024                         #: Runtime warning
  Errored = 2048                         #: Runtime error (mostly caught exceptions)
  Aborted = 4096                         #: Uncaught runtime exception

  SetupError =     8192                  #: Preparation / compilation error
  TearDownError = 16384                  #: Cleanup error / resource release error
  Inconsistent = 32768                   #: Dataset is inconsistent

  Flags = Warned | Errored | Aborted | SetupError | TearDownError | Inconsistent

@export
class TestsuiteStatus(Flag):
  """A flag enumeration describing the status of a test suite."""
  Unknown =    0
  Excluded =   1                         #: Testcase was permanently excluded / disabled
  Skipped =    2                         #: Testcase was temporarily skipped (e.g. based on a condition)
  Empty =      4                         #: No tests in suite
  Passed =     8                         #: Passed testcase, because all assertions succeeded
  Failed =    16                         #: Failed testcase due to failing assertions

  Mask = Excluded | Skipped | Empty | Passed | Failed

  Inverted = 128                         #: To mark inverted results
  UnexpectedPassed = Failed | Inverted
  ExpectedFailed =   Passed | Inverted

  Warned =  1024                         #: Runtime warning
  Errored = 2048                         #: Runtime error (mostly caught exceptions)
  Aborted = 4096                         #: Uncaught runtime exception

  SetupError =     8192                  #: Preparation / compilation error
  TearDownError = 16384                  #: Cleanup error / resource release error

  Flags = Warned | Errored | Aborted | SetupError | TearDownError

Testcase

A Testcase is the leaf-element in the test entity hierarchy and describes an individual test run. Besides a test case status, it also contains statistics like the start time or the test duration. Test cases are grouped by test suites and need to be unique per parent test suite.

A test case (or its base classes) implements the following properties and methods:

Parent

The test case has a reference to it’s parent test suite in the hierarchy. By iterating parent references, the root element (test suite summary) be be found, which has no parent reference (None).

Name

The test case has a name. This name must be unique per hierarchy parent, but can exist multiple times in the overall test hierarchy.

In case the data format uses hierarchical names like pyEDAA.Reports.CLI.Application, the name is split at the separator and multiple hierarchy levels (test suites) are created in the unified data model. To be able to recreate such an hierarchical name, TestsuiteKind is applied accordingly to test suite’s Kind field.

Title, Summary, Description

The test case can carry a title, a summary and a description written for a reader, e.g. taken from a doc-string. Formats like JUnit XML can’t express them, so they are None then.

StartTime

The test case stores a time when the individual test run was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test case has fields to capture the setup duration, test run duration and teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up a test run. If the setup duration can’t be distinguished from the test’s runtime, set this value to None.

The test’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test run is the time spend on tearing down a test run. If the teardown duration can’t be distinguished from the test’s runtime, set this value to None.

The test case has a field total duration to sum up setup duration, test duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test case counts for warnings, errors and fatal errors observed in a test run while the test was executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test case implements a dictionary interface, so arbitrary key-value pairs can be annotated per test entity.

Status

The overall status of a test case.

Testsuite

A Testsuite is a grouping element in the test entity hierarchy and describes a group of test runs. Besides a list of test cases and a test suite status, it also contains statistics like the start time or the test duration for the group of tests. Test suites are grouped by other test suites or a test suite summary and need to be unique per parent test suite.

A test suite (or its base classes) implements the following properties and methods:

Parent

The test suite has a reference to it’s parent test entity in the hierarchy. By iterating parent references, the root element (test suite summary) be be found, which has no parent reference (None).

Name

The test suite has a name. This name must be unique per hierarchy parent, but can exist multiple times in the overall test hierarchy.

In case the data format uses hierarchical names like pyEDAA.Reports.CLI.Application, the name is split at the separator and multiple hierarchy levels (test suites) are created in the unified data model. To be able to recreate such an hierarchical name, TestsuiteKind is applied accordingly to test suite’s Kind field.

Title, Summary, Description

The test suite can carry a title, a summary and a description written for a reader, e.g. taken from a doc-string. Formats like JUnit XML can’t express them, so they are None then.

StartTime

The test suite stores a time when the first test run was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test suite has fields to capture the suite’s setup duration, test group run duration and suite’s teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up a test suite. If the setup duration can’t be distinguished from the test group’s runtime, set this value to None.

The test group’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test suite is the time spend on tearing down a test suite. If the teardown duration can’t be distinguished from the test group’s runtime, set this value to None.

The test suite has a field total duration to sum up setup duration, test duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test suite counts for warnings, errors and fatal errors observed in a test suite while the tests were executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test suite implements a dictionary interface, so arbitrary key-value pairs can be annotated.

Todo

TestsuiteBase APIs

Testcases: tbd
TestcaseCount: tbd
AssertionCount: The overall number of assertions (checks) in a test case.
Aggregate(): Aggregate (recalculate) all durations, warnings, errors, assertions, etc.
Iterate(): tbd
__str__(): tbd

@export
class Testsuite(TestsuiteBase[TestsuiteType]):
   def __init__(
      self,
      name: str,
      kind: TestsuiteKind = TestsuiteKind.Logical,
      startTime: Nullable[datetime] = None,
      setupDuration: Nullable[timedelta] = None,
      testDuration: Nullable[timedelta] = None,
      teardownDuration: Nullable[timedelta] = None,
      totalDuration:  Nullable[timedelta] = None,
      status: TestsuiteStatus = TestsuiteStatus.Unknown,
      warningCount: int = 0,
      errorCount: int = 0,
      fatalCount: int = 0,
      testsuites: Nullable[Iterable[TestsuiteType]] = None,
      testcases: Nullable[Iterable["Testcase"]] = None,
      *,
      parent: Nullable[TestsuiteType] = None
   ):
     ...

   @readonly
   def Parent(self) -> Nullable["Testsuite"]:
     ...

   @readonly
   def Name(self) -> str:
     ...

   @readonly
   def StartTime(self) -> Nullable[datetime]:
     ...

   @readonly
   def SetupDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TestDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TeardownDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TotalDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def WarningCount(self) -> int:
     ...

   @readonly
   def ErrorCount(self) -> int:
     ...

   @readonly
   def FatalCount(self) -> int:
     ...

   def __len__(self) -> int:
     ...

   def __getitem__(self, key: str) -> Any:
     ...

   def __setitem__(self, key: str, value: Any) -> None:
     ...

   def __delitem__(self, key: str) -> None:
     ...

   def __contains__(self, key: str) -> bool:
     ...

   def __iter__(self) -> Generator[Tuple[str, Any], None, None]:
     ...

   # TestsuiteBase API


   @readonly
   def Testcases(self) -> Dict[str, "Testcase"]:
     ...

   @readonly
   def TestcaseCount(self) -> int:
     ...

   @readonly
   def AssertionCount(self) -> int:
     ...

   def Aggregate(self, strict: bool = True) -> TestsuiteAggregateReturnType:
     ...

   def Iterate(self, scheme: IterationScheme = IterationScheme.Default) -> Generator[Union[TestsuiteType, Testcase], None, None]:
     ...

   def __str__(self) -> str:
     ...

TestsuiteSummary

A TestsuiteSummary is the root element in the test entity hierarchy and describes a group of test suites as well as overall statistics for the whole set of test cases. A test suite summary is derived for the same base-class as a test suite, thus they share almost all properties and methods.

A test suite summary (or its base classes) implements the following properties and methods:

Parent

The test suite summary has a parent reference, but as the root element in the test entity hierarchy, its always None.

Name

The test suite summary has a name.

Title, Summary, Description

The test suite summary can carry a title, a summary and a description written for a reader, e.g. taken from a doc-string. Formats like JUnit XML can’t express them, so they are None then.

StartTime

The test suite summary stores a time when the first test runs was started. In combination with TotalDuration, the end time can be calculated. If the start time is unknown, set this value to None.

SetupDuration, TestDuration, TeardownDuration, TotalDuration

The test suite summary has fields to capture the suite summary’s setup duration, overall run duration and suite summary’s teardown duration. The sum of all durations is provided by total duration.

TotalDuration := SetupDuration + TestDuration + TeardownDuration

The setup duration is the time spend on setting up an overall test run. If the setup duration can’t be distinguished from the test’s runtimes, set this value to None.

The test suite summary’s runtime without setup and teardown portions is captured by test duration. If the duration is unknown, set this value to None.

The teardown duration of a test suite summary is the time spend on tearing down a test suite summary. If the teardown duration can’t be distinguished from the test’s runtimes, set this value to None.

The test suite summary has a field total duration to sum up setup duration, overall run duration and teardown duration. If the duration is unknown, this value will be None.

WarningCount, ErrorCount, FatalCount

The test suite summary counts for warnings, errors and fatal errors observed in a test suite summary while the tests were executed.

__len__(), __getitem__(), __setitem__(), __delitem__(), __contains__(), __iter__()

The test suite summary implements a dictionary interface, so arbitrary key-value pairs can be annotated.

Todo

TestsuiteBase APIs

Aggregate(): tbd
Iterate(): tbd
__str__(): tbd

@export
class TestsuiteSummary(TestsuiteBase[TestsuiteType]):
   def __init__(
      self,
      name: str,
      startTime: Nullable[datetime] = None,
      setupDuration: Nullable[timedelta] = None,
      testDuration: Nullable[timedelta] = None,
      teardownDuration: Nullable[timedelta] = None,
      totalDuration:  Nullable[timedelta] = None,
      status: TestsuiteStatus = TestsuiteStatus.Unknown,
      warningCount: int = 0,
      errorCount: int = 0,
      fatalCount: int = 0,
      testsuites: Nullable[Iterable[TestsuiteType]] = None,
      *,
      parent: Nullable[TestsuiteType] = None
   ):
     ...

   @readonly
   def Parent(self) -> Nullable["Testsuite"]:
     ...

   @readonly
   def Name(self) -> str:
     ...

   @readonly
   def StartTime(self) -> Nullable[datetime]:
     ...

   @readonly
   def SetupDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TestDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TeardownDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def TotalDuration(self) -> Nullable[timedelta]:
     ...

   @readonly
   def WarningCount(self) -> int:
     ...

   @readonly
   def ErrorCount(self) -> int:
     ...

   @readonly
   def FatalCount(self) -> int:
     ...

   def __len__(self) -> int:
     ...

   def __getitem__(self, key: str) -> Any:
     ...

   def __setitem__(self, key: str, value: Any) -> None:
     ...

   def __delitem__(self, key: str) -> None:
     ...

   def __contains__(self, key: str) -> bool:
     ...

   def __iter__(self) -> Generator[Tuple[str, Any], None, None]:
     ...

   # TestsuiteBase API

   def Aggregate(self) -> TestsuiteAggregateReturnType:
     ...

   def Iterate(self, scheme: IterationScheme = IterationScheme.Default) -> Generator[Union[TestsuiteType, Testcase], None, None]:
     ...

   def __str__(self) -> str:
     ...

Document

A Document is a mixin-class …

Path: tbd
AnalysisDuration: tbd
ModelConversionDuration: tbd
Analyze(): tbd
Convert(): tbd

@export
class Document(metaclass=ExtendedType, mixin=True):
   def __init__(self, path: Path):
     ...

   @readonly
   def Path(self) -> Path:
     ...

   @readonly
   def AnalysisDuration(self) -> timedelta:
     ...

   @readonly
   def ModelConversionDuration(self) -> timedelta:
     ...

   @abstractmethod
   def Analyze(self) -> None:
     ...

   @abstractmethod
   def Convert(self):
     ...

Specific Data Models

JUnit Data Model

Testcase

A test case is the leaf-element in the test entity hierarchy and describes an individual test run. Test cases are grouped by test classes.

Testclass

A test class is the mid-level element in the test entity hierarchy and describes a group of test runs. Test classes are grouped by test suites.

Testsuite

A test suite is a group of test classes. Test suites are grouped by a test suite summary.

TestsuiteSummary

The test suite summary is derived from test suite and defines the root of the test suite hierarchy.

Document

The document is derived from a test suite summary and represents a file containing a test suite summary.

JUnit Dialects

The JUnit format is not well defined, thus multiple dialects developed over time.

        graph TD;
  doc[Document]
  sum[Summary]
  ts1[Testsuite]
  ts11[Testsuite]
  ts2[Testsuite]

  tc111[Testclass]
  tc112[Testclass]
  tc23[Testclass]

  tc1111[Testcase]
  tc1112[Testcase]
  tc1113[Testcase]
  tc1121[Testcase]
  tc1122[Testcase]
  tc231[Testcase]
  tc232[Testcase]
  tc233[Testcase]

  doc:::root -.-> sum:::summary
  sum --> ts1:::suite
  sum ---> ts2:::suite
  ts1 --> ts11:::suite

  ts11 --> tc111:::cls
  ts11 --> tc112:::cls
  ts2  --> tc23:::cls

  tc111 --> tc1111:::case
  tc111 --> tc1112:::case
  tc111 --> tc1113:::case
  tc112 --> tc1121:::case
  tc112 --> tc1122:::case
  tc23 --> tc231:::case
  tc23 --> tc232:::case
  tc23 --> tc233:::case

  classDef root fill:#4dc3ff
  classDef summary fill:#80d4ff
  classDef suite fill:#b3e6ff
  classDef cls fill:#ff9966
  classDef case fill:#eeccff

Testcase

Testclass

Testsuite

TestsuiteSummary

Document

JUnit Dialects

As the JUnit XML format was not well specified and no XML Schema Definition (XSD) was provided, many variants and dialects (and simplifications) were created by the various frameworks emitting JUnit XML files.

JUnit Dialect Comparison

Feature	Any JUnit	Ant + JUnit4	Catch2 JUnit	CTest JUnit	go-junit-report	GoogleTest JUnit	cargo-nextest JUnit	pyTest JUnit	TestLogger JUnit
Root element	testsuites	testsuite	testsuites	testsuite	testsuites	testsuites	testsuites	testsuites	testsuites
Supports properties	☑	☑	☑		☑	⸺			☑
Testcase status	…	…	always run	more status values			reruns		no errors

Any JUnit

The Any JUnit format uses a relaxed XML schema definition aiming to parse many JUnit XML dialects, which use a <testsuites> root element.

A report is validated against Any-JUnit.xsd.

Reading Any JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pathlib import Path
from pyEDAA.Reports.Unittesting.JUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing Any JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit import Document

xmlReport = Path("AnyJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

Ant + JUnit4

The original JUnit format created by Ant for JUnit4 uses <testsuite> as a root element.

A report is validated against Ant-JUnit4.xsd.

Gradle writes this format too, when it runs JUnit4 tests.

Reading Ant + JUnit4

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.AntJUnit4 import Document

xmlReport = Path("AntJUnit4-Report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pathlib import Path
from pyEDAA.Reports.Unittesting.JUnit.AntJUnit4 import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing Ant + JUnit4

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.AntJUnit4 import Document

xmlReport = Path("AntJUnit4-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

Catch2 JUnit

The Catch2 JUnit format written by the JUnit reporter of Catch2 (version 3) uses <testsuites> as a root element. It holds exactly one <testsuite>, named after the test executable.

A report is validated against Catch2-JUnit.xsd.

Each test case and each path of nested sections is a <testcase>, named by its section path (TestCase/Section/Subsection). A test case or section containing sections is a <testcase> of its own.
The classname is <executable>.global, or <executable>.<fixture class> for a test case of a fixture class (:: becomes .).
SKIP() writes a <skipped> element, a failed assertion a <failure> element, an unexpected exception an <error> element. Only the first of them per <testcase> is written.
Each <testcase> carries status="run".

Known issues

The tests, failures, errors and skipped attributes of <testsuite> count assertions, not test cases. Written by pyEDAA.Reports, they count test cases.
The hostname attribute is always tbd.
A test case without assertions and without output is missing.
A failure in a section is reported on the section’s <testcase> only; the <testcase> of the enclosing test case passes.
A failure expected by [!mayfail] or [!shouldfail] writes a <skipped> element followed by a <failure> element. Its test case is read as skipped.
A [!shouldfail] test case passing unexpectedly fails for Catch2, but passes in the report.
The properties random-seed and filters aren’t read.

Reading Catch2 JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.Catch2JUnit import Document

xmlReport = Path("Catch2JUnit-Report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pathlib import Path
from pyEDAA.Reports.Unittesting.JUnit.Catch2JUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing Catch2 JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.Catch2JUnit import Document

xmlReport = Path("Catch2JUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

CTest JUnit

The CTest JUnit format written by CMake’s CTest (ctest --output-junit) uses <testsuite> as a root element.

A report is validated against CTest-JUnit.xsd.

Reading CTest JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

xmlReport = Path("CTestJUnit-Report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pathlib import Path
from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing CTest JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

xmlReport = Path("CTestJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

go-junit-report JUnit

The JUnit format written by go-junit-report (v2) uses <testsuites> as a root element. It converts the output of go test (option -parser gotest, the default) or of go test -json (option -parser gojson).

A <testsuite> is a Go package, named by its import path. It carries an id (its index in the report), and the hostname and timestamp of the conversion, not of the test run.
A <testcase> is a test, a subtest (TestName/Subtest) or an example. Its classname is the package’s import path. A parent test is a test case of its own, besides its subtests.
<failure message="Failed"> holds the failure’s log lines, <skipped message="Skipped"> the reason. An <error> is a build error, a runtime error of the test binary or a test without a result. A passed test’s log is in its <system-out>.

Reading go-junit-report JUnit

from pyEDAA.Reports.Unittesting.JUnit.GoJUnitReport import Document

xmlReport = Path("go-junit-report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pyEDAA.Reports.Unittesting.JUnit.GoJUnitReport import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing go-junit-report JUnit

from pyEDAA.Reports.Unittesting.JUnit.GoJUnitReport import Document

xmlReport = Path("go-junit-report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

Reader

A package without tests, run with code coverage (go test -coverprofile=...), is written as <testsuite name="">: go-junit-report doesn’t recognize go test’s result line of such a package (<tab><import path><tab><tab>coverage: 0.0% of statements) and names the package by its option -package-name, which is empty by default. The line is the test suite’s <system-out>, and the reader names the test suite by the import path in it. A test suite with an empty name and without such a line is rejected.
A panic ends the package’s test binary. Its stack trace is written to the package’s <system-out>, which isn’t read, and the <failure> of the panicking test is empty. A parent test failed by a subtest has an empty <failure> too.
<properties> (coverage.statements.pct, and the properties of the options -p and -go-version) and the id attribute aren’t read.
In the unified data model, a test class becomes a test suite of kind class named by the whole import path; it isn’t split at . into packages.
Option -subtest-mode changes the parent tests: ignore-parent-results writes them as passed, exclude-parents leaves them out.

Writer

id is the test suite’s index in the document.
A test suite without a duration gets the sum of its test cases’ durations as time, as go-junit-report does.
Every <skipped>, <error> and <failure> gets a message attribute; it’s empty for a test case without a message.

GoogleTest JUnit

The GoogleTest JUnit format written by GoogleTest (sometimes GTest) uses <testsuites> as a root element.

A report is validated against GoogleTest-JUnit.xsd.

Reading GoogleTest JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

xmlReport = Path("GoogleTestJUnit-Report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pathlib import Path
from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing GoogleTest JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

xmlReport = Path("GoogleTestJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

cargo-nextest JUnit

The JUnit format written by cargo-nextest, the test runner for Rust, uses <testsuites> as a root element. nextest serializes it with the crate quick-junit.

A report is validated against Nextest-JUnit.xsd.

Mapping to the data model

<testsuites>: its name is the profile’s report name (nextest-run by default), its uuid the run ID (RunID), timestamp and time the run’s start time and duration.
<testsuite>: one per test binary, named by its binary ID, e.g. counter for the unit tests of a library and counter::sequence for its integration test tests/sequence.rs. A setup script gets a test suite @setup-script:<name> with one test case, and its command as <property>. A <testsuite> has neither hostname, nor timestamp, nor time.
<testcase>: its name is the test’s path in the binary, e.g. tests::failing, its classname the binary ID again. In the unified data model, the test cases are therefore in a test suite (class) of the same name as the binary’s test suite. The timestamp is kept as the test case’s start time (StartTime).
Status: <failure> is failed, <error> is errored (e.g. the test process didn’t start, or leaked handles), <skipped> is skipped, a test case without one of these passed.
Retries: a test which passed in a retry (flaky) has a <flakyFailure> (or <flakyError>) per failed attempt, a test failing in every attempt a <rerunFailure> (or <rerunError>) per retry. Their number is RerunCount; their output is not read. A failed test case carries the start time, duration and message of its first attempt, a flaky one those of its last attempt.

Known issues

A test marked #[ignore] is missing from the report, unless the profile sets junit.report-skipped to "ignored" or "all". The console reports 12 tests run: 8 passed, 4 failed, 1 skipped, the report tests="12" skipped="0". A reported skipped test has no timestamp and a time of zero.
Every failure of a Rust test is a <failure type="test failure with exit code 101">: a failed assertion, a panic, an Err returned by the test and a #[should_panic] test which doesn’t panic. Only the message (the first line of the error output) and the text tell them apart. The type is not kept in the data model.
A flaky test configured with flaky-result = "fail" is a <failure type="flaky failure"> with the message test passed on attempt 2/2 but is configured to fail when flaky.
A setup script counts as a test: tests="6" in a report, whose console summary is 4 tests run: 2 passed (1 flaky), 2 failed, 1 skipped.
The test cases are listed in the order they finished; reported skipped tests come first.
Writing: the run ID is written only for a document read from a nextest report, as the unified data model has no field for it. Reruns and the type of a <failure> are not written. The test suite summary needs a start time and a duration; the hostname, start time and duration of a test suite are not written.

Reading cargo-nextest JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.NextestJUnit import Document

xmlReport = Path("target/nextest/ci/junit.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pathlib import Path
from pyEDAA.Reports.Unittesting.JUnit.NextestJUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing cargo-nextest JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.NextestJUnit import Document

xmlReport = Path("Nextest-JUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

pyTest JUnit

The pyTest JUnit format written by pyTest uses <testsuites> as a root element.

A report is validated against PyTest-JUnit.xsd.

Reading pyTest JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

xmlReport = Path("PyTestJUnit-Report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pathlib import Path
from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing pyTest JUnit

from pathlib import Path
from pyEDAA.Reports.Unittesting import UnittestError
from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

xmlReport = Path("PyTestJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

JunitXml.TestLogger JUnit

The JUnit format written by the .NET test logger JunitXml.TestLogger (dotnet test --logger junit, see dotnet test (VSTest)) uses <testsuites> as a root element.

Reading JunitXml.TestLogger JUnit

from pyEDAA.Reports.Unittesting.JUnit.TestLoggerJUnit import Document

xmlReport = Path("TestLoggerJUnit-Report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to and from Unified Data Model

from pyEDAA.Reports.Unittesting.JUnit.TestLoggerJUnit import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

# Convert back to a document
newXmlReport = Path("New JUnit-Report.xml")
newDoc = Document.FromTestsuiteSummary(newXmlReport, summary)

Writing JunitXml.TestLogger JUnit

from pyEDAA.Reports.Unittesting.JUnit.TestLoggerJUnit import Document

xmlReport = Path("TestLoggerJUnit-Report.xml")
try:
  newDoc.Write(xmlReport)
except UnittestError as ex:
  ...

Reading

<testsuites> has no attributes. Each <testsuite> holds the results of one test assembly and names it by its file (name and package). Its timestamp is the start of the test run in UTC, written without a time zone, and is read as UTC. The test suite summary takes its start time from the first test suite. id (always 0) and package are not read.
A test case’s classname is the namespace and class of the test, its name the method with the arguments of a parameterized test. In the unified data model, each part of the namespace becomes a test suite of kind Namespace, the class a test suite of kind Class. A nested class (Outer+Inner) and a parameterized test class (Fixture("a",1)) keep their names.
<failure> is a failed test case, <skipped/> a skipped one, a test case without either passed.
<system-out> and <system-err> of a test case are its captured output; the logger appends the test’s attachments to <system-out> as [[ATTACHMENT|path]]. The test framework’s messages of the whole run in <system-out> and <system-err> of the test suite, and a test case’s traits in <properties>, are not read.

Writing

An errored test case is written as <failure type="failure"> and counted in failures; errors is always 0.
A skipped test case is written as an empty <skipped/>, without its message and details.
The start time is written in UTC. A test suite needs a start time, a duration, a host name and at least one test case; a test case needs a duration, written as at least 0.0000001 seconds.

Known issues of the logger

VSTest knows failed tests only: an exception escaping a test, a time-out or a cancellation is a <failure> as well. The exception’s type is only part of the message, e.g. System.DivideByZeroException : Attempted to divide by zero.
The reason of a skipped test is lost; the TRX file keeps it.
A test whose outcome is neither passed, failed nor skipped (e.g. an inconclusive NUnit test) gets no child element: it is read as passed. The logger counts it in tests only.
A test suite’s time is the sum of its test cases’ durations, not the duration of the run.
With the option MethodFormat=Class or MethodFormat=Full, name repeats the class or the namespace and class. It’s read as written, so the class shows up in the test case’s name too.
A test name the logger can’t split into namespace, class and method is written with classname UnknownNamespace.UnknownType and the whole name as name.

Open Test Reporting Data Model

Open Test Reporting describes tests and containers of tests in a tree. A test engine is a root, its containers (test classes, nested classes, parameterized tests) and tests (test methods, invocations of a parameterized test) are its descendants. The data model of pyEDAA.Reports.Unittesting.OpenTestReporting keeps this tree:

An Execution holds the Infrastructure (host, user, operating system, CPU cores, Java version, …) and the root nodes.
A Node is a test or container with a display name, a start time and a duration. It carries the metadata (tags, JUnit’s unique ID, legacy reporting name and type), the sources (Java class and method), the attachments (data entries, output, files) and a Result.
A result has a status, a reason and the exception ending the test: its type, whether it is an assertion error, and the stack trace.
Each element knows its parent: the infrastructure and a root node their execution, a node its parent node, a result its node.

pyEDAA.Reports.Unittesting.OpenTestReporting.Events.Document reads the event-based format, pyEDAA.Reports.Unittesting.OpenTestReporting.Hierarchy.Document the hierarchical format into this data model. Both convert it into the unified data model.

Reading Open Test Reporting

from pyEDAA.Reports.Unittesting.OpenTestReporting.Events import Document

xmlReport = Path("build/test-results/test/open-test-report.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

for node in doc.Roots:
  print(f"{node.Name}: {node.Result.Status}")

Reading the hierarchical format

from pyEDAA.Reports.Unittesting.OpenTestReporting.Hierarchy import Document

xmlReport = Path("build/test-results/test/hierarchy.xml")
try:
  doc = Document(xmlReport, analyzeAndConvert=True)
except UnittestError as ex:
  ...

Convert to Unified Data Model

from pyEDAA.Reports.Unittesting.OpenTestReporting.Events import Document

# Convert to unified test data model
summary = doc.ToTestsuiteSummary()

print(f"{summary.Tests} tests: {summary.Passed} passed, {summary.Failed} failed")

Conversion to the Unified Data Model

Open Test Reporting	Unified data model
file, e.g. `open-test-report.xml`	test suite summary, named by the file name without extension
container	test suite, kind `Class` for a Java class, else `Logical`
test (JUnit’s type `TEST`, or a node without children)	test case
JUnit’s legacy reporting name, e.g. `my.pack.MyClassTest`, `testAbsolute(int, int, TestReporter)[1]`	name; without JUnit’s metadata, the display name
display name (`name`), e.g. `Tests of MyClass`, `[1] "5", "5"`	title (only together with a legacy reporting name)
`SUCCESSFUL`	`Passed`
`FAILED`	`Failed`
`ERRORED`	`Errored`
`SKIPPED` (e.g. `@Disabled`), `ABORTED` (e.g. a failed assumption)	`Skipped`
no result (started, but not finished) or no status	`Inconsistent`
reason, else the exception’s message	message
stack trace of the exception (`<java:throwable>`)	details
output of source `stdout` / `stderr`	standard output / standard error
data entries, e.g. JUnit’s `TestReporter` entries	key-value pairs of the test case
start time of the first root node; time from start to finish (event-based), `duration` (hierarchical)	start time; total duration
host name of the infrastructure	host name of the root test suites

The names of a container’s children have to be unique: two child containers or two child tests with the same name raise an exception instead of dropping a test case. A test without a container (a root node, which is a test) can’t be converted, because a test case needs a test suite as parent.

Not in the unified data model

The Open Test Reporting data model keeps these, the unified data model doesn’t:

the result of a container: e.g. a failure in @BeforeAll is a FAILED container, whose tests weren’t started, but a test suite’s status is computed from its test cases only;
the difference between SKIPPED and ABORTED;
the exception’s type and whether it is an assertion error: a FAILED test is Failed for an assertion and for another exception alike, as the JUnit Platform reports it;
tags, JUnit’s unique ID and type, the Java class and method (beside the names), file attachments, output of other sources than stdout and stderr, and the timestamps of attachments;
the infrastructure, besides the host name: user name, operating system, CPU cores, extension values like the Java version;
a data entry reported twice under the same key: the test case keeps the last value.

Both data models truncate the times from nanoseconds to microseconds. The event-based format’s durations are computed from truncated times, the hierarchical format’s are truncated, so they can differ by a microsecond.

OSVVM

Open Source VHDL Verification Methodology writes test results as YAML files for its internal data model storage. Some YAML files are written by the VHDL code of the verification framework, others are written by OSVVM-Scripts as a test runner.

Command Line Tool

pyedaa-reports unittest --input=Ant-JUnit:data/JUnit.xml

File Formats

Unittest summary reports can be stored in various file formats. Usually these files are XML based. Due to missing (clear) specifications and XML schema definitions, some file formats have developed dialects. Either because the specification was unclear/not existing or because the format was specific for a single programming language, so tools added extensions or misused XML attributes instead of designing their own file format.

Ant and JUnit 4 XML

The so-called JUnit XML format was defined by Ant when running JUnit4 test suites. Because the format was not specified by JUnit4, many dialects spread out. Many tools and test frameworks have minor or major differences compared to the original format. While some modifications seam logical additions or adjustments to the needs of the respective framework, others undermine the ideas and intents of the data format.

Many issues arise because the Ant + JUnit4 format is specific to unit testing with Java. Other languages and frameworks were lazy and didn’t derive their own format, but rather stuffed their language constructs into the concepts and limitations of the Ant + JUnit4 XML format.

JUnit Dialects

🚧 Bamboo JUnit (planned)
✅ Catch2 JUnit format
✅ CTest JUnit format
✅ go-junit-report JUnit format
✅ GoogleTest JUnit format
🚧 Jenkins JUnit (planned)
✅ JunitXml.TestLogger JUnit format for dotnet test
✅ cargo-nextest JUnit format
✅ pyTest JUnit format

JUnit 5 XML

JUnit5 uses a new format called Open Test Reporting (see the following section for details). This format isn’t specific to Java (packages, classes, methods, …), but describes a generic data model. Of course an extension for Java specifics is provided too.

The JUnit Platform writes this format only on request (junit.platform.reporting.open.xml.enabled=true). Build tools like Gradle write Ant + JUnit4 XML for JUnit 5 and 6 tests by default, see JUnit5.

Open Test Reporting

The Open Test Alliance created a new format called Open Test Reporting (OTR) to overcome the shortcomings of a missing file format for JUnit5 as well as the problems of Ant + JUnit4.

OTR defines a structure of test groups and tests, but no specifics of a certain programming language. The logical structure of tests and test groups is decoupled from language specifics like namespaces, packages or classes hosting the individual tests.

The JUnit Platform writes OTR’s event-based format into open-test-report.xml:

The root element <e:events> holds an <infrastructure> element (host name, user name, operating system, Java version, …) and one event element per state change.
<e:started> opens a container (test engine, class, nested class, parameterized test) or a test with an id, a parentId and a name - the display name. Its <metadata> carries tags and JUnit’s unique ID, its <sources> the Java class or method.
<e:reported> adds attachments while it runs: TestReporter entries and captured output on STDOUT and STDERR.
<e:finished> closes it with a <result> of status SUCCESSFUL, FAILED, ABORTED (a failed assumption) or SKIPPED (a disabled test), with the exception or the reason.
Every event has a time in UTC with nanoseconds, e.g. 2026-10-08T10:56:04.172060276Z.

The event-based format is read by pyEDAA.Reports.Unittesting.OpenTestReporting.Events.Document and converted into the unified data model, see Open Test Reporting Data Model. The file is validated against the schemas, which Open Test Reporting publishes for each namespace.

OTR’s command line tool converts the event-based format into its hierarchical format (convert command): <h:execution> holds the <infrastructure> and a <h:root> per test engine, which nests the containers and tests as <h:child> elements, each with a start time and a duration. It’s read by pyEDAA.Reports.Unittesting.OpenTestReporting.Hierarchy.Document into the same data model.

pyTooling Test Report

pyTooling provides a pytest plugin writing a test report in a format of its own (pytest option --pytooling-xml=PATH). Unlike JUnit XML, test suites nest, and every test suite and test case can carry a title, a summary and a description besides its name. The format is defined by an XML schema; its version is part of the schema’s file name (e.g. TestReport-v0.1.xsd), which every report names in its xsi:noNamespaceSchemaLocation attribute.

The report is read by pyEDAA.Reports.Unittesting.pyTooling.Document into the unified data model. It’s validated against the schema it names, e.g. TestReport-v0.1.xsd.

Visual Studio Test Results (TRX)

Visual Studio and dotnet test (VSTest) write a test run’s results as TRX file (--logger trx): XML in namespace http://microsoft.com/schemas/VisualStudio/TeamTest/2010 with root <TestRun>. Its children are:

<Times>: Creation, start and finish of the test run.
<Results>: A <UnitTestResult> per test: name, duration, start and end time, and its outcome (e.g. Passed, Failed, NotExecuted). <Output> holds the test’s standard output and, in <ErrorInfo>, the message and stack trace of a failed test or the reason of a skipped one.
<TestDefinitions>: A <UnitTest> per test, naming the assembly, the class and the method of a test in <TestMethod>. Each data row of a parameterized test is a test of its own.
<TestEntries>, <TestLists>: Link results and definitions by GUIDs.
<ResultSummary>: The run’s outcome and <Counters> (total, executed, passed, failed, notExecuted, …), the run’s output and the attachments of data collectors, e.g. code coverage reports.

There is no reader yet.

OSVVM YAML

The Open Source VHDL Verification Methodology (OSVVM) defines its own test report format in YAML. While OSVVM is able to convert its own YAML files to JUnit XML files, it’s recommended to use the YAML files as data source, because these contain additional information, which can’t be expressed with JUnit XML.

The YAML files are created when OSVVM-based testbenches are executed with OSVVM’s embedded TCL scripting environment OSVVM-Scripts.

Hint

YAML was chosen instead of JSON or XML, because a YAML document isn’t corrupted in case of a runtime error. The document might be incomplete (content), but not corrupted (structural). Such a scenario is possible if a VHDL simulator stops execution, then the document structure can’t be finalized.

Frameworks / Tools

cargo-nextest

https://github.com/nextest-rs/nextest

Rust’s test harness (libtest, run by cargo test) writes JUnit XML only on a nightly compiler (-Z unstable-options --format junit), one document per test binary on standard output. The test runner cargo-nextest writes one JUnit XML report per run, if the profile in .config/nextest.toml names it, e.g. [profile.ci.junit] with path = "junit.xml"; cargo nextest run --profile ci writes it to target/nextest/ci/junit.xml.

The report has a <testsuites> root named nextest-run with a uuid attribute, and a <testsuite> per test binary, named after the crate - counter for the unit tests in the library, counter::sequence for the integration test tests/sequence.rs. A <testcase>’s name is the test’s path in its crate, e.g. tests::increment, its classname the <testsuite>’s name. Each <testcase> carries its start time in a timestamp attribute, and the test’s output in <system-out> and <system-err>. A failed assertion, a panic, an Err returned by a test and a #[should_panic] test that doesn’t panic are each a <failure> of type test failure with exit code 101. Its message is the first line of the test’s error output, e.g. thread 'tests::failing' (554412) panicked at src/lib.rs:100:9 or Error: Underflow.

Unlike Ant JUnit4, a test marked #[ignore] isn’t in the report by default, the <testsuite> has neither timestamp, time nor hostname, and the test cases are listed in the order they finished. Doc-tests aren’t run by cargo-nextest. Except for uuid on <testsuites> and timestamp on <testcase>, the report is Any JUnit. It is read by the cargo-nextest JUnit dialect.

The example in examples/Rust/Cargo is run by the pipeline job Rust-Cargo. It measures the code coverage by Rust’s source-based coverage (-C instrument-coverage) with cargo-llvm-cov, and writes it as LLVM JSON (llvm-cov export format), lcov tracefile and Cobertura XML by cargo-llvm-cov, and as lcov tracefile, Cobertura XML and covdir JSON by grcov.

Catch2

https://github.com/catchorg/Catch2

Catch2 (version 3) writes a report per reporter given on its command line, e.g. --reporter JUnit::out=catch2-junit.xml --reporter XML::out=catch2.xml:

JUnit reporter: A <testsuites> root without attributes, holding one <testsuite> named after the test executable. Each test case and each path of nested sections (TestCase/Section/Subsection) is a <testcase>; its classname is <executable>.global, or <executable>.<fixture class> for a test case of a fixture class. SKIP() writes a <skipped> element, an exception escaping a test case an <error> element. Unlike Ant JUnit4, the <testsuite>’s tests attribute counts assertions (skips included), not test cases, hostname is tbd, and each <testcase> carries status="run". The report is read by the Catch2 JUnit dialect.
XML reporter: Catch2’s own format (root <Catch2TestRun>, xml-format-version="3"): nested <TestCase> and <Section> elements with tags, source file and line, every failed expression and the result counts. It isn’t JUnit XML, and pyEDAA.Reports doesn’t read it yet.

The example in examples/Cpp/Catch2 is built and run by the pipeline job Cpp-Catch2. It measures the code coverage of the tested library with GCC (--coverage) and writes it as gcov JSON (gcov --json-format) and as lcov tracefile (lcov --capture).

CTest

dotnet test (VSTest)

dotnet test runs the tests of a .NET test project with VSTest, whatever test framework the tests are written in (e.g. xUnit.net, NUnit, MSTest). Each logger given on its command line writes a report:

--logger trx

A TRX file, Visual Studio’s own format.

--logger junit (NuGet package JunitXml.TestLogger)

A <testsuites> root without attributes, holding one <testsuite> per test assembly, named after the assembly’s file (e.g. MyLibrary.Tests.dll). Each test is a <testcase>; its classname is the test class, its name the method, followed by the arguments of a parameterized test, e.g. Absolute(value: -4, expected: 5). The test cases follow in the order they finished, not grouped by class. The report is read by the JunitXml.TestLogger JUnit dialect. Compared to Ant + JUnit4, the logger writes:

the additional attributes id and package on <testsuite>. Only these keep the Any JUnit and pyTest JUnit dialects from reading the report.
an exception escaping a test as <failure>, never as <error>: VSTest knows failed tests only. The exception’s type and message are in the message attribute, the stack trace is the element’s text.
a skipped test as an empty <skipped/>, without its reason. The TRX file keeps the reason.
the test framework’s messages of the whole run into <system-out> and <system-err> of the <testsuite>.

The example in examples/CSharp/xUnit - a class library and its xUnit.net test project - is built and run by the pipeline job CSharp-xUnit. It writes both reports and measures the code coverage with coverlet and with Microsoft’s code coverage collector (see .NET: coverlet, Microsoft’s code coverage collector, ReportGenerator).

Go (go test)

go test writes no JUnit XML: it prints text, or with -json a stream of JSON events (test2json), one object per line. Converters translate it to JUnit XML:

gotestsum: gotestsum --junitfile gotestsum.xml --jsonfile go-test.json -- ./... runs go test -json and writes a <testsuites> root with a <testsuite> per package - also for a package without tests -, each with the property go.version. Each test, subtest (TestName/Subtest) and example is a <testcase>; a parent test is a test case of its own beside its subtests. classname is the package’s import path. A failure and a panic write a <failure> with the test’s output, t.Skip a <skipped> with the test’s output in its message attribute. Unlike Ant JUnit4, a <testsuite> has no hostname and errors, and skipped only if a test was skipped. --junitfile-testsuite-name and --junitfile-testcase-classname shorten the import path to its last element (short) or to the path relative to the module (relative), --junitfile-project-name names the <testsuites>. The report is Any JUnit.
go-junit-report: go-junit-report -parser gojson -in go-test.json -out go-junit-report.xml converts the JSON events - by default go test -v’s text output. It writes the same tree, but a test’s log in the test case’s <system-out>, a panic’s stack trace in the package’s <system-out>, and id, hostname and timestamp - of the conversion - on each <testsuite>; a package without tests is a <testsuite> with an empty name. The report is read by the go-junit-report JUnit dialect.

A panic ends the package’s test binary: the package’s later tests don’t run, and its code coverage is lost.

The example in examples/Go/testing is built and run by the pipeline job Go-Test. It measures the code coverage as Go cover profile and converts it to Cobertura XML and lcov tracefile, see Go Cover Profile.

GoogleTest (gtest)

https://github.com/google/googletest

The pipeline job Cpp-GoogleTest runs the tests in examples/Cpp/GoogleTest directly and by CTest, writing GoogleTest’s and CTest’s JUnit XML reports. The library is compiled with --coverage; gcov writes the gcov JSON report of the direct run.

Gradle

Gradle’s test task writes one Ant + JUnit4 XML file per test class into build/test-results/test/, named TEST-<class>.xml like Ant’s. The Ant + JUnit4 dialect reads them. The example examples/Java/Gradle-JUnit4 runs JUnit 4 tests by Gradle in the pipeline. Compared to Ant’s <junit> task, Gradle writes:

an exception other than a failed assertion as <failure>, never as <error>. errors is always 0.
an empty <properties/> element. Ant lists the JVM’s system properties and the build’s properties.
timestamp in UTC with milliseconds, e.g. 2026-10-08T10:41:52.384Z. Ant writes the local time without fraction or time zone.
an ignored test (@Ignore) as an empty <skipped/>, without the reason, and a failed assumption as <skipped> with message, type and the stack trace. Ant writes both as <skipped message="..."/>.
the exception’s class name and message into message of <failure>, and the unfiltered stack trace as its text. Ant writes the exception’s message only and removes JUnit’s and Ant’s stack frames.

JUnit4

JUnit5

JUnit 5 and its successor JUnit 6 run tests on the JUnit Platform. The examples examples/Java/Gradle-JUnit5 and examples/Java/Gradle-JUnit6 run JUnit 5 and JUnit 6 tests by Gradle in the pipeline and write both: Gradle’s Ant + JUnit4 XML files, read by the Ant + JUnit4 dialect, and the JUnit Platform’s Open Test Reporting file, read by Document. Compared to JUnit 4 tests, Gradle’s files differ:

The test suite is named by the test class’ display name (@DisplayName), the file by the class’ name.
A test case is named by its display name: the method name with its parameter types, e.g. testReturnTrue(), or @DisplayName. An invocation of a parameterized test is named by its index and arguments only, e.g. [1] 5, 5, so the invocations of two parameterized tests of one class can have the same names.
A nested test class (@Nested) is a test suite in a file of its own, e.g. TEST-my.pack.MyClassTest$Divide.xml.
TestReporter entries and, if the JUnit Platform captures the output, STDOUT and STDERR are <property> elements of the <testcase>.
A disabled test (@Disabled) is an empty <skipped/>, without the reason. A failed assumption is a <skipped> with message, type and the stack trace.
The JUnit Platform removes JUnit’s and Gradle’s stack frames calling the test method from stack traces; the JDK’s frames remain.
Compared to JUnit 5, JUnit 6 quotes the arguments in the display name of a parameterized test ([1] "5", "5" instead of [1] 5, 5), and removes the JDK’s frames calling the test method and the assertion’s internal frames from stack traces too.

OSVVM

pytest

https://github.com/pytest-dev/pytest

The pipeline job Python-pytest runs the tests in examples/Python/pytest and writes pytest’s JUnit XML report, measured by coverage.py, which writes its JSON and Cobertura XML reports.

Unittesting

Unified data model

Testcase Status

Testcase

Testsuite

TestsuiteSummary

Document

Specific Data Models

JUnit Data Model

Testcase

Testclass

Testsuite

TestsuiteSummary

Document

JUnit Dialects

Any JUnit

Ant + JUnit4

Catch2 JUnit

CTest JUnit

go-junit-report JUnit

GoogleTest JUnit

cargo-nextest JUnit

pyTest JUnit

JunitXml.TestLogger JUnit

Open Test Reporting Data Model

Conversion to the Unified Data Model

OSVVM

Features

Create test entities

Reading unittest reports

Converting unittest reports

Annotations

Merging unittest reports

Concatenate unittest reports

Transforming the reports’ hierarchy

pytest specific transformations

Writing unittest reports

Command Line Tool

File Formats

Ant and JUnit 4 XML

JUnit 5 XML

Open Test Reporting

pyTooling Test Report

Visual Studio Test Results (TRX)

OSVVM YAML

Frameworks / Tools

cargo-nextest

Catch2

CTest

dotnet test (VSTest)

Go (go test)

GoogleTest (gtest)

Gradle

JUnit4

JUnit5

OSVVM

pytest

VUnit

Consumers

GitLab

Jenkins

Dorney (GitHub Action)