TestNG: Dynamically Naming Tests from data provider parameters

I have recently been involved in several contracts that require me to stretch my knowledge of TestNG. There a lot of cool things you can do with TestNG that are available but not explicitly documented  (you know, with real source code !).

One of the first examples is that of dynamically naming a test based on incoming data.

If you're using a data provider in TestNG it allows you to define a method that will feed the data into a test method. For example this simple data provider gives three sets of data that a test can be run with:

@DataProvider(name="rawDP")
public Object[][] sampleDataProvider()
{
 Object[][] rawData = {
  
   {"SCENARIO_1","First Test Scenario"},
   {"SCENARIO_2","Second Test Scenario"},
   {"SCENARIO_3","Third Test Scenario"}
 };
 
 return rawData;
}


@Test(dataProvider="rawDP")
public void shouldHaveTestNamesBasedOnMethodName(String arg1, String arg2)
{
}

The above method will be passed all three sets of data in sequence with the first argument going into the first string of the method in the second argument going into the second string of the method. When you run the test in eclipse the console output will show

PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_1", "First Test Scenario")
PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_2", "Second Test Scenario")
PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_3", "Third Test Scenario")

But you may want the scenario name to show up in the HTML report or  in some other way.

Through the magic of some TestNG interfaces we can make it so that those arguments are intercepted and used to name the instance of the test before the test method is run.

To do so we define a custom annotation UseAsTestName that is made available at run time.
The annotation has an attribute that indicates which parameter of the parameter set should be used as the test name.

/**
 * Annotation used as an indicator that the Test Method should use the indexed
 * parameter as the test instance name
 *
 * @author jmochel
 */

@Retention(RetentionPolicy.RUNTIME)
public @interface UseAsTestName {

 /**
  * Index of the parameter to use as the Test Case ID.
  */

 int idx() default 0;

}

Then we have code in the test's parent class that runs before each method is run (as indicated by the @BeforeMethod annotation).

public class UseAsTestName_TestBase implements ITest {
 
 /**
  * Name of the current test. Used to implement {@link ITest#getTestName()}
  */

 private String testInstanceName = "";

 /**
  * Allows us to set the current test name internally to this class so that
  * the TestNG framework can use the {@link ITest} implementation for naming
  * tests.
  *
  * @param testName
  */

 private void setTestName(String anInstanceName) {
  testInstanceName = anInstanceName;
 }

 /**
  * See {@link ITest#getTestName()}
  */

 public String getTestName() {
  return testInstanceName;
 }

 /**
  * Method to transform the name of tests when they are called with the
  * testname as one of the parameters. Only takes effect if method has
  * {@link UseAsTestName} annotation on it..
  *
  * @param method
  *            The method being called.
  *
  * @param parameterBlob
  *            The set of test data being passed to that method.
  */

 @BeforeMethod(alwaysRun = true)
 public void extractTestNameFromParameters(Method method, Object[] parameters) {

  /*
   * Verify Parameters
   */

  checkNotNull(method);
  checkNotNull(parameters);

  /*
   * Empty out the name from the previous test
   */

  setTestName(method.getName());

  /*
   * If there is a UseAsTestCaseID annotation on the method, use it to get
   * a new test name
   */

  UseAsTestName useAsTestName = method
    .getAnnotation(UseAsTestName.class);

  if (useAsTestName != null) {
   
   /*
    * Check that the index it uses is viable.
    */

   if (useAsTestName.idx() > parameters.length - 1) {
    throw new IllegalArgumentException(
      format("We have been asked to use an incorrect parameter as a Test Case ID. The {0} annotation on method {1} is asking us to use the parameter at index {2} in the array and there are only {3} parameters in the array.",
        UseAsTestName.class.getSimpleName(),
        method.getName(), useAsTestName.idx(),
        parameters.length));
   }

   /*
    * Is the parameter it points to assignable as a string.
    */

   Object parmAsObj = parameters[useAsTestName.idx()];

   if (!String.class.isAssignableFrom(parmAsObj.getClass())) {
    throw new IllegalArgumentException(
      format("We have been asked to use a parameter of an incorrect type as a Test Case Name. The {0} annotation on method {1} is asking us to use the parameter at index {2} in the array that parameter is not usable as a string. It is of type {3}",
        UseAsTestName.class.getSimpleName(),
        method.getName(), useAsTestName.idx(),
        parmAsObj.getClass().getSimpleName()));
   }

   /*
    * Get the parameter at the specified index and use it.
    */

   String testCaseId = (String) parameters[useAsTestName.idx()];

   setTestName(testCaseId);
  }
 }

}

Because we have a Method type in the methods parameter list, TestNG automatically inserts the method object pertaining to the method being called. The same for the Object[] which TestNG automatically inserts the row of data associated with this invocation of the test method.

@BeforeMethod(alwaysRun = true)
public void extractTestNameFromParameters(Method method, Object[] parameters) {

In the before method we clear out the old test name and use the information provided by the method parameter and the parameters parameter to create a new test name and set that.  When the XML and eclipse Console reports are being generated by the tests they use the ITest getTestName() method to get the name of the test.

The tests that use this look like:

public class UseAsTestNameTest extends TestBase {
 
 @DataProvider(name="rawDP")
 public Object[][] sampleDataProvider()
 {
  Object[][] rawData = {
    {"SCENARIO_1","First Test Scenario"}, 
    {"SCENARIO_2","Second Test Scenario"},
    {"SCENARIO_3","Third Test Scenario"}
  };
  
  return rawData;
 }
 
 
 @Test(dataProvider="rawDP")
 public void shouldHaveTestNamesBasedOnMethodName(String arg1, String arg2)
 {
 }
 
 @UseAsTestName()
 @Test(dataProvider="rawDP")
 public void shouldHaveTestNamesStartingWithANA(String arg1, String arg2)
 {
  getTestName().equals(arg1);
 }
 
 @UseAsTestName(idx=1)
 @Test(dataProvider="rawDP")
 public void shouldHaveTestNamesStartingWithThe(String arg1, String arg2)
 {
  getTestName().equals(arg2);
 } 
}

The output looks something like:

PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_1", "First Test Scenario")
PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_2", "Second Test Scenario")
PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_3", "Third Test Scenario")
PASSED: SCENARIO_1("SCENARIO_1", "First Test Scenario")
PASSED: SCENARIO_2("SCENARIO_2", "Second Test Scenario")
PASSED: SCENARIO_3("SCENARIO_3", "Third Test Scenario")
PASSED: First Test Scenario("SCENARIO_1", "First Test Scenario")
PASSED: Second Test Scenario("SCENARIO_2", "Second Test Scenario")
PASSED: Third Test Scenario("SCENARIO_3", "Third Test Scenario")

The one gotcha in all of this is that the reports that are generated for the results are generated from the XML that TestNG generates.

The XML uses the name we generate and puts it in what's called the "instance name" attribute. There are many different HTML versions of the reports and some of them correctly use the instance name and some of them don't. The HTML reports run on Jenkins correctly use the instance names and show the tests with that as their name. The default HTML reports that get generated will only run it from eclipse don't use the instance name correctly. The eclipse console does correctly use the instance name thus we can see it there.
Some HTML reports will show it in some will not.