View on GitHub

Cucumber Reports

Known Errors Report

Download this project as a .zip file Download this project as a tar.gz file

What is it?

Known errors report is the report grouping test results errors which match some known patterns.

Where is it used?

In vast majority of cases the same error may appear in many different places in the results report. Also, these is some common functionality. And usually, if some error occurs in the common functionality it would make multiple errors in test results with the same error message. Sometimes it is convenient to group them to see how many errors of known reasons were found.

Major sections

The report contains only one section represented with the results table as it is shown at the below image:

Sample Report

The table itself contains the following columns:

  • Description - contains brief error summary (in bold) with more detailed description. It is used to give detailed information about error found.
  • Priority - indicates the importance of the problem
  • # of Occurrences - indicates frequency of errors matching the same pattern

Configuration options

Top-Level Parameters

At the highest level we should define the following options:

  • Source file - the path to initial JSON report to generate breakdown from
  • Output directory - the directory to send output to
  • Enable breakdown - flag which enables breakdown report generation
  • Known errors configuration file - the path to known errors report configuration file. This is one of the ways to specify configuration from code and the only way to do it via extended Cucumber runner

Configuration File Format

Sample configuration JSON

The configuration file is defined in JSON format. Here is one of the examples:

{  
   "@type":"com.github.mkolisnyk.cucumber.reporting.types.knownerrors.KnownErrorsModel",
   "errorDescriptions":[  
      {  
         "title":"Unexpected message",
         "description":"Message box shows unexpected content",
         "filter":{  
            "alias":"(.*)message is shown",
            "dimensionValue":"FAILED_STEP",
            "expression":"(.*)message is shown",
            "subElements":[  

            ],
            "isFinal":false
         },
         "priority":{  
            "name":"LOW",
            "ordinal":1
         }
      }
   ],
   "orderBy":{  
      "name":"FREQUENCY",
      "ordinal":1
   }
}

What is normally defined

Configuration defines the list of expressions which can indicate known errors. They are defined using filter rules. Additionally, each rule has title, description and the priority.

In addition to the list of rules, at the top-most level we define the parameter to order by.

Core Data Types

Known Errors Report Model

Field Type Description
errorDescriptions List of Breakdown Report Info The list of error description items
orderBy Enumeration The field to order errors by. Possible values: PRIORITY, FREQUENCY, NAME

Known Errors Info

Field Type Description
title String The short text which gives summary to an error
description String The text which contains detailed description of the error
filter Data Dimension Contains the filter which is used to match some specific error
priority Enumeration Defines the priority of the error. Can be one of the following values: LOWEST, LOW, MEDIUM, HIGH, HIGHEST

Generation sample

From code

Generating from the explicitly defined model:

CucumberKnownErrorsReport results = new CucumberKnownErrorsReport();
results.setOutputDirectory("target");
results.setOutputName("cucumber-results");
results.setSourceFile("./src/test/resources/cucumber.json");
KnownErrorsModel model = new KnownErrorsModel(
    new KnownErrorsInfo[] {
       new KnownErrorsInfo(
           "Unable to reach Select Ticket screen",
           "Select ticket isn't shown",
           new DataDimension(DimensionValue.FAILED_STEP, "(.*)see the \"Select Ticket\" screen"),
           KnownErrorPriority.HIGHEST),
       new KnownErrorsInfo(
           "Search Fails",
           "Some search parameters do not bring results",
           new DataDimension(DimensionValue.FAILED_STEP, "(.*)the \"Out Search Results\" screen"),
           KnownErrorPriority.HIGH),
       new KnownErrorsInfo(
           "Unexpected message",
           "Message box shows unexpected content",
           new DataDimension(DimensionValue.FAILED_STEP, "(.*)message is shown"),
           KnownErrorPriority.LOW),
    },
    KnownErrorOrderBy.FREQUENCY);
results.execute(model, false);

Generate from predefined configuration file:

CucumberKnownErrorsReport results = new CucumberKnownErrorsReport();
results.setOutputDirectory("target");
results.setOutputName("cucumber-results-2");
results.setSourceFile("./src/test/resources/detailed-source/cucumber-2.json");
results.execute(new File("./src/test/resources/known-errors-source/sample_model.json"), false);

From Extended Cucumber runner

For Extended Cucumber runner we just need to define the following options:

  • jsonReport - known errors report uses JSON results file as an input
  • outputFolder - output directory
  • knownErrorsReport = true - this flag should be enabled in order to generate the report
  • knownErrorsConfig - path to configuration file.

So, sample known error report generation looks like this:

package com.github.mkolisnyk.cucumber.reporting;

import org.junit.runner.RunWith;

import com.github.mkolisnyk.cucumber.runner.ExtendedCucumber;
import com.github.mkolisnyk.cucumber.runner.ExtendedCucumberOptions;

import cucumber.api.CucumberOptions;

@RunWith(ExtendedCucumber.class)
@ExtendedCucumberOptions(
		jsonReport = "target/cucumber.json",
        toPDF = true,
        knownErrorsReport = true,
        knownErrorsConfig = "src/test/resources/known-errors-source/sample_model.json",
        outputFolder = "target")
@CucumberOptions(plugin = { "html:target/cucumber-html-report",
        "json:target/cucumber.json", "pretty:target/cucumber-pretty.txt",
        "usage:target/cucumber-usage.json", "junit:target/cucumber-results.xml" },
        features = { "./src/test/java/com/github/mkolisnyk/cucumber/features" },
        glue = { "com/github/mkolisnyk/cucumber/steps" },
        tags = {"@passed"})
public class SampleCucumberTest {
}

News