Blogs to watch for useful TestNG info

As I come across the I will update this post with blogs that have some useful info on working with TestNG

http://rationaleemotions.wordpress.com

TestNG : Using Guice for dependency injection

In the previous post I discussed how to use annotations to make it possible for TestNG to rename tests.

The next few posts are going to deal with some useful ways to use dependency injection and TestNG to radically simplify integration testing.

Recently I have been working on a contract that requires a lot of integration testing. And the great challenge of integration testing is that tests can fail because the you are testing against can change (as well as the code). My current client has a framework of VMs that can be deployed in many different configurations with different elements being simulated and some being real.

In addition, a deployed node may have one purpose but, depending on the customer being served the node may use a different protocol. Plus we have some environments where a node may be a single node in one deployment but a cluster in another.  Most of this information is available in the database associated with the deployment.

It is my belief that you never need more than two pieces of information to run an integration test. The first is the name of the test set to be run, the second is the name or address of some machine to act as a point source of information (database server or some configuration server such as Zookeeper).

My goals in this current project are to:

1. Minimize the amount of information the person writing the test needs to know in order to write the tests. If a session needs to be created, they shouldn't need to know what the underlying protocols are.
2. Replace the many diverse configurations files (YAML, Properties and XML) with an IP address and the name of a test set to run. 
3. Allow the test writers to write tests without regard for whether or not they are pointing at a cluster of nodes or a single node in the deployed environment.

For the first the obvious pattern is to hide the implementations behind interfaces.

For the second the configuration information should be made available to the implementations without passing through the test writers hands. 

This all argues for Dependency Injection. 

Since we are using TestNG as our test framework, it is natural to use Google Guice as our DI framework. TestNG has an annotation that allows us to specify the DI binding factory to use when the test starts up.

For this example I'm going to use a simple interface for persistent storage called Repo. Repos are designed to store CoolObjs. Note the use of the Google Guava Optional class. This simply gives you a typesafe way of handling when an object is not returned.

Repo.java

package org.saltations.testng.usingguice;

import com.google.common.base.Optional;

/**
 * Repository Interface. Represents a repository that objects can be saved or
 * recovered from...
 */

public interface Repo {

 /** 
  * Store an object 
  */
 void store(CoolObj obj);

 /**
  * Retrieves an existing object
  *  
  * @param id The id of the {@link CoolObj}
  * 
  * @return An instantiated cool object.
  */
 Optional<CoolObj> retrieve(String id);
}

CoolObj.java 

Note the use of Lombok annotations to simplify POJO development. I mentioned Lombok in a previous post. You can generate getters/setters and constructors by hand.

package org.saltations.testng.usingguice;

import lombok.AllArgsConstructor;
import lombok.Data;

@Data
@AllArgsConstructor
public class CoolObj {

 private String id;
 private String stuff;

}

When testing we are going to use three different implementations of the Repo interface. The first one is the RepoMockImpl use for unit testing. It uses an internal map in memory to store and retrieve the objects. The second one is the LocalRepoImpl that is used for integration testing. It stores all of the objects using a key-value store on disk. The third implementation is the ServerRepoImpl that points to a remote repository server.

The details of each of the implementations of the repository are not important. The key thing is they require different types of configuration information. The ServerRepoImpl requires the IP address/host name  of the machine being pointed to.

The test that uses this looks like:

TestUsingInjectedServices.java

package org.saltations.testng.usingguice;

import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;

import org.testng.annotations.Guice;
import org.testng.annotations.Test;

import com.google.common.base.Optional;
import com.google.inject.Inject;

@Test
@Guice(moduleFactory=TestDIFactory.class)
public class TestUsingInjectedServices {

 @Inject
 private Repo repo;
 
 public void shouldSaveAndRetrieveACoolObj()
 {
  CoolObj coolObj= new CoolObj("goodObject", "Cool Tidbits");
  
  repo.store(coolObj);
  
  Optional potentialObj = repo.retrieve(coolObj.getId());
  
  assertTrue(potentialObj.isPresent());
  assertEquals(potentialObj.get(), coolObj);
 }
}

The elements that hook everything together are the

1. Guice annotation which tells the TestNG framework to use the TestDIFactory class as the factory used to supply Guice with the Modules that in turn provide the bindings between the Repo service and its implementation.
2. Inject annotation that Guice to inject the appropriately configured service implementation for the Repo interface in the field repo.
3. The groups attribute in the TestNG.xml file that is used by the TestDIFactory to determine which Module implementation is used to supply the bindings.

If this test were run with the group configured as "unit" we would get the unit test bindings. If we run this test with the group configured as "integration" we would get the integration test bindings.

Configuration information comes in through the TestNG.xml file as TestNG parameters and as TestNG groups .

The Guice factory is in a form specified by TestNG.

TestDIFactory.java

package org.saltations.testng.usingguice;

import static java.text.MessageFormat.format;

import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.List;

import lombok.NoArgsConstructor;

import org.testng.IModuleFactory;
import org.testng.ITestContext;

import com.google.common.collect.Lists;
import com.google.inject.Binder;
import com.google.inject.Module;

/**
 * Guice Factory for TestNG Tests. This factory gives back a Guice module that
 * supplies the bindings that are appropriate for the kind of tests being done.
 * i.e. the factory will give back a module that supplies unit test service
 * implementations for unit tests, integration test service implementations for
 * integration tests, etc...
 * 
 * @author jmochel
 */

@NoArgsConstructor
public class TestDIFactory implements IModuleFactory {

 /**
  * Key for the test parameter in TestNG.xml that contains the name of our repo server.
  */

 private static final String REPO_SERVER_NAME = "repo-server-name";

 /**
  * Module that provides unit test service implementations for unit tests
  */

   public class UnitTestServicesProvider extends GeneralModule implements
   Module {

  public UnitTestServicesProvider(ITestContext ctx, Class clazz) {
   super(ctx, clazz);
  }

  public void configure(Binder binder) {
   binder.bind(Repo.class).to(RepoMockImpl.class);
  }
 }

 /**
  * Module that supplies integration test service implementations for
  * integration tests
  */

 private class IntegrationServicesProvider extends GeneralModule implements
   Module {

  public IntegrationServicesProvider(ITestContext ctx, Class clazz) {
   super(ctx, clazz);
  }

  public void configure(Binder binder) {
   binder.bind(Repo.class).to(LocalRepoImpl.class);
  }
 }

 /**
  * Module that supplies real word service implementations for
  * testing the application against a real world system.
  */

 private class WorkingServicesProvider extends GeneralModule implements
   Module {

  /**
   * Repo Server address.
   */

  private InetAddress address;

  public WorkingServicesProvider(ITestContext ctx, Class clazz) {
   super(ctx, clazz);

   /*
    * Confirm that the repository server address exists and that it points somewhere real. 
    */

   String repoServerName = ctx.getCurrentXmlTest().getAllParameters().get(REPO_SERVER_NAME);

   if (repoServerName == null || repoServerName.isEmpty() )
   {
    throw new IllegalArgumentException(format("Unable to find {0} in the test parameters. We expected to find it configured in the TestNG.xml parameters.", REPO_SERVER_NAME));
   }

   try {
    address = InetAddress.getByName(repoServerName);
   } catch (UnknownHostException e) {
    throw new IllegalArgumentException(format("Unable to find host {1} specified by parameter{0} in the test parameters.", REPO_SERVER_NAME, repoServerName));
   } 
  }

  public void configure(Binder binder) {
   binder.bind(Repo.class).toInstance(new ServerRepoImpl(address));
  }
 }

 /*
  * @see org.testng.IModuleFactory#createModule(org.testng.ITestContext,
  * java.lang.Class)
  */

 public Module createModule(ITestContext ctx, Class clazz) {

  /*
   * Get a list of included groups (comes from the TestNG.xml) and choose
   * which Guice module to return based on the types of tests being done.
   */

  List groups = Lists.newArrayList(ctx.getIncludedGroups());

  Module module = null;

  if (groups.contains("unit")) {
   module = new UnitTestServicesProvider(ctx, clazz);
  } else if (groups.contains("integration")) {
   module = new IntegrationServicesProvider(ctx, clazz);
  } else {
   module = new WorkingServicesProvider(ctx, clazz);
  }

  return module;
 }
}

For completeness: GeneralModule.java

package org.saltations.testng.usingguice;

import static com.google.common.base.Preconditions.checkNotNull;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NonNull;

import org.testng.ITestContext;

import com.google.inject.Binder;
import com.google.inject.Module;

/**
 * Parent class for Guice Modules used in TestNG. Contains the ITestContext and
 * Test Class because I so often need the tests to make decisions based on the
 * contents of the TesNG.xml (and made available in the ITestContext current XML
 * parameters).
 */

@Data
@AllArgsConstructor
public abstract class GeneralModule implements Module {

 /**
  * Context for the test
  */

 @NonNull
 private ITestContext ctx;

 /**
  * Class to be tested
  */
 
 @NonNull
 private Class clazz;

 public abstract void configure(Binder binder);
}

What Happens

The tests are started up using the TestNG.XML file. When the TestNG framework instantiates the test it does so after using the TestDIFactory createModule call to determine which Guice Module it will use to configure Guice. In the createModule call we look at the ITestContext and determine what group this test class should be run has. That module is then instantiated and handed back to Guice. Guice then walks through the instantiated classes and when it encounters a @Inject annotation it asks the module for what class or object should be injected into that field. The person writing the tests does not need to know what the Repo service is pointing to.

One of the key gotchas here is that the TestDIFactory and Modules are instantiated on a per test class basis. If we are running five test classes in a unit test group all five classes will have a separate instantiation process. If you wish for a service to be shared as a singleton then you will want to use a static element in the TestDIFactory or the individual module and then bind the interface to the implementation using the "toInstance" method rather than the "to" method.

What if Repo involved a Generic?

If Repo was actually Repo<ObjectType> the the code would look like:

binder.bind(new TypeLiteral<Repo<ObjectClass>>(){}).to(new TypeLiteral<RepoImpl<ObjectClass>>(){});

Pros and Cons

The pros are pretty clear to me. By using this I am able to hide all of the configuration details from the test writers. It is a little bit more work up front but it pays off handsomely in the simplicity of the testing code. From my standpoint one of the big pluses is that it allows me to verify all the configurations upfront and instantiate the classes with the configuration embedded in them situated that configuration can be used for the test services and (as we see in a later post) for configuring the tests themselves.

TestNG: Dynamically Naming Tests from data provider parameters

I have recently been involved in several contracts that require me to stretch my knowledge of TestNG. There a lot of cool things you can do with TestNG that are available but not explicitly documented  (you know, with real source code !).

One of the first examples is that of dynamically naming a test based on incoming data.

If you're using a data provider in TestNG it allows you to define a method that will feed the data into a test method. For example this simple data provider gives three sets of data that a test can be run with:

@DataProvider(name="rawDP")
public Object[][] sampleDataProvider()
{
 Object[][] rawData = {
  
   {"SCENARIO_1","First Test Scenario"},
   {"SCENARIO_2","Second Test Scenario"},
   {"SCENARIO_3","Third Test Scenario"}
 };
 
 return rawData;
}


@Test(dataProvider="rawDP")
public void shouldHaveTestNamesBasedOnMethodName(String arg1, String arg2)
{
}

The above method will be passed all three sets of data in sequence with the first argument going into the first string of the method in the second argument going into the second string of the method. When you run the test in eclipse the console output will show

PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_1", "First Test Scenario")
PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_2", "Second Test Scenario")
PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_3", "Third Test Scenario")

But you may want the scenario name to show up in the HTML report or  in some other way.

Through the magic of some TestNG interfaces we can make it so that those arguments are intercepted and used to name the instance of the test before the test method is run.

To do so we define a custom annotation UseAsTestName that is made available at run time.
The annotation has an attribute that indicates which parameter of the parameter set should be used as the test name.

/**
 * Annotation used as an indicator that the Test Method should use the indexed
 * parameter as the test instance name
 *
 * @author jmochel
 */

@Retention(RetentionPolicy.RUNTIME)
public @interface UseAsTestName {

 /**
  * Index of the parameter to use as the Test Case ID.
  */

 int idx() default 0;

}

Then we have code in the test's parent class that runs before each method is run (as indicated by the @BeforeMethod annotation).

public class UseAsTestName_TestBase implements ITest {
 
 /**
  * Name of the current test. Used to implement {@link ITest#getTestName()}
  */

 private String testInstanceName = "";

 /**
  * Allows us to set the current test name internally to this class so that
  * the TestNG framework can use the {@link ITest} implementation for naming
  * tests.
  *
  * @param testName
  */

 private void setTestName(String anInstanceName) {
  testInstanceName = anInstanceName;
 }

 /**
  * See {@link ITest#getTestName()}
  */

 public String getTestName() {
  return testInstanceName;
 }

 /**
  * Method to transform the name of tests when they are called with the
  * testname as one of the parameters. Only takes effect if method has
  * {@link UseAsTestName} annotation on it..
  *
  * @param method
  *            The method being called.
  *
  * @param parameterBlob
  *            The set of test data being passed to that method.
  */

 @BeforeMethod(alwaysRun = true)
 public void extractTestNameFromParameters(Method method, Object[] parameters) {

  /*
   * Verify Parameters
   */

  checkNotNull(method);
  checkNotNull(parameters);

  /*
   * Empty out the name from the previous test
   */

  setTestName(method.getName());

  /*
   * If there is a UseAsTestCaseID annotation on the method, use it to get
   * a new test name
   */

  UseAsTestName useAsTestName = method
    .getAnnotation(UseAsTestName.class);

  if (useAsTestName != null) {
   
   /*
    * Check that the index it uses is viable.
    */

   if (useAsTestName.idx() > parameters.length - 1) {
    throw new IllegalArgumentException(
      format("We have been asked to use an incorrect parameter as a Test Case ID. The {0} annotation on method {1} is asking us to use the parameter at index {2} in the array and there are only {3} parameters in the array.",
        UseAsTestName.class.getSimpleName(),
        method.getName(), useAsTestName.idx(),
        parameters.length));
   }

   /*
    * Is the parameter it points to assignable as a string.
    */

   Object parmAsObj = parameters[useAsTestName.idx()];

   if (!String.class.isAssignableFrom(parmAsObj.getClass())) {
    throw new IllegalArgumentException(
      format("We have been asked to use a parameter of an incorrect type as a Test Case Name. The {0} annotation on method {1} is asking us to use the parameter at index {2} in the array that parameter is not usable as a string. It is of type {3}",
        UseAsTestName.class.getSimpleName(),
        method.getName(), useAsTestName.idx(),
        parmAsObj.getClass().getSimpleName()));
   }

   /*
    * Get the parameter at the specified index and use it.
    */

   String testCaseId = (String) parameters[useAsTestName.idx()];

   setTestName(testCaseId);
  }
 }

}

Because we have a Method type in the methods parameter list, TestNG automatically inserts the method object pertaining to the method being called. The same for the Object[] which TestNG automatically inserts the row of data associated with this invocation of the test method.

@BeforeMethod(alwaysRun = true)
public void extractTestNameFromParameters(Method method, Object[] parameters) {

In the before method we clear out the old test name and use the information provided by the method parameter and the parameters parameter to create a new test name and set that.  When the XML and eclipse Console reports are being generated by the tests they use the ITest getTestName() method to get the name of the test.

The tests that use this look like:

public class UseAsTestNameTest extends TestBase {
 
 @DataProvider(name="rawDP")
 public Object[][] sampleDataProvider()
 {
  Object[][] rawData = {
    {"SCENARIO_1","First Test Scenario"}, 
    {"SCENARIO_2","Second Test Scenario"},
    {"SCENARIO_3","Third Test Scenario"}
  };
  
  return rawData;
 }
 
 
 @Test(dataProvider="rawDP")
 public void shouldHaveTestNamesBasedOnMethodName(String arg1, String arg2)
 {
 }
 
 @UseAsTestName()
 @Test(dataProvider="rawDP")
 public void shouldHaveTestNamesStartingWithANA(String arg1, String arg2)
 {
  getTestName().equals(arg1);
 }
 
 @UseAsTestName(idx=1)
 @Test(dataProvider="rawDP")
 public void shouldHaveTestNamesStartingWithThe(String arg1, String arg2)
 {
  getTestName().equals(arg2);
 } 
}

The output looks something like:

PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_1", "First Test Scenario")
PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_2", "Second Test Scenario")
PASSED: shouldHaveTestNamesBasedOnMethodName("SCENARIO_3", "Third Test Scenario")
PASSED: SCENARIO_1("SCENARIO_1", "First Test Scenario")
PASSED: SCENARIO_2("SCENARIO_2", "Second Test Scenario")
PASSED: SCENARIO_3("SCENARIO_3", "Third Test Scenario")
PASSED: First Test Scenario("SCENARIO_1", "First Test Scenario")
PASSED: Second Test Scenario("SCENARIO_2", "Second Test Scenario")
PASSED: Third Test Scenario("SCENARIO_3", "Third Test Scenario")

The one gotcha in all of this is that the reports that are generated for the results are generated from the XML that TestNG generates.

The XML uses the name we generate and puts it in what's called the "instance name" attribute. There are many different HTML versions of the reports and some of them correctly use the instance name and some of them don't. The HTML reports run on Jenkins correctly use the instance names and show the tests with that as their name. The default HTML reports that get generated will only run it from eclipse don't use the instance name correctly. The eclipse console does correctly use the instance name thus we can see it there.
Some HTML reports will show it in some will not.