Awesum Stuff

Parallel TestNG Soft Assertions – The RIGHT way !

In a recent tryst with TestNG, I was running some tests in parallel(way to go TestNG!) and had some Soft Assertions(read Assertions that don’t halt test method execution on failure) in place. The weird thing was the unpredictability of my test results. Sometimes the results were fine but most of the time, I was getting shared assertions between the test methods. Example: If there are 10 methods and 2 were supposed to fail, then 4 were showing up as failed.

First let me briefly explain how Soft Assertions work:-

  1. TestNG offers you a SoftAssert class for implementing Soft Assertions.
  2. We need to instantiate the SoftAssert class in order to use it in our Test Methods for assertion.
  3. Any assertion that we put like softAssertInstance.assertNotNull(someObj) will not fail the test immediately.
  4. The Pass/Failure of the test will be governed by the softAssertInstance.assertAll() method.
  5. TestNG achieves this feature because the assertAll method is called at the end and this finally decides the fate of the test.

Now the million dollar question is where will the SoftAssert instance go ? Some possible solutions:-

  1. Put the SoftAssert instance at the top of the Test class as an instance variable (as most blogs suggest)
  2. Create a separate SoftAssert instance inside each test method.

Tempted by the first approach, I decided to put the Soft Assert instance inside my BaseTest but this was giving me unexpected results in parallel executions(but working like a charm for single threaded executions!).

I realized that the problem was that my Soft Assert instance was being shared across threads and I needed to come up with a way to have a unique instance for each test method. I tried several approaches:-

a) Created a thread local object and set the Soft Assert thread specific instance to that(did not work!)

public class BaseTest {
    private SoftAssert softAssert;
    private ThreadLocal threadLocal = new ThreadLocal();    

    @BeforeMethod(alwaysRun = true)
    public void createSoftAssert(Method method)
    {
        softAssert = new SoftAssert();
        threadLocal.set(softAssert);
    }

    public SoftAssert getSoftAssert() {
        return (SoftAssert)threadLocal.get();
    }
}

//This is how my test class would look like

public class TestClass1 extends BaseTest {

    @Test
    public void TempTest1()
    {
        MyAssertion.AreEqual(1,1,getSoftAssert());
        MyAssertion.assertAll(getSoftAssert());
    }
}

b) Maintain a HashMap of SoftAssert objects for each thread in my BaseTest (overkill!)

public class BaseTest {
    private static Map<String,SoftAssert> softAsserts = new ConcurrentHashMap<String, SoftAssert>();

    @BeforeMethod(alwaysRun = true)
    public void createSoftAssert(Method method)
    {
        SoftAssert softAssert = new SoftAssert();
        softAsserts.put(method.getName(),softAssert);
    }

    public SoftAssert getSoftAssert(ITestContext context) {
        return softAsserts.get(context.getName());
    }
}

//This is how my test class would look like in this case</pre>
public class TestClass1 extends BaseTest {

    @Test
    public void TempTest1(ITestContext context)
    {
        MyAssertion.AreEqual(1,1,getSoftAssert(context));
        MyAssertion.assertAll(getSoftAssert(context));
    }
}

Finally I decided to go with the approach of creating a SoftAssert instance in all my test methods individually and my tests ran fine in parallel threads.

Sample code for this is available on GitHub:-

https://github.com/kohli-harshit/testng-softassert

PS: The only recommendation that I would want to give from the code above is to create SoftAssert inside the Test Methods, rest everything is a way of implementation.

Happy Automating!

Harshit Kohli

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s