Category Archives: Kiwi

One Unit Test Framework to Rule Them All

I kid myself about changing my mind daily regarding the “best” unit testing frameworks. I’ve been advocating for Kiwi for several months now. Well, here goes again.

Kiwi

Pros: I absolutely love Kiwi. It allows test to be well organized (RSpec style), has excellent mock support (including Class mocks), is fast, and is easy to use (runs on top of OCUnit logic tests), and has asynchronous support.

Cons: Debugging tests is difficult. Whenever I try to debug tests, I typically get dumped into assembler code. Yech!

Maybe I just don’t understand how to debug block code.

I’ve also been considering whether having disorganized tests is actually a symptom (smell) of having classes that are too big. I’ve been reviewing an associate’s code that uses OCHamcrest and Mockito, and I must say that I find it quite readable.

So, the current HowsMyFuel TDD project (April 22, commit 068c19d75e) fails unit tests, and it isn’t clear why. I could probably guess, and use trial-and-error to fix it, but I’d prefer to have a clear diagnostic approach to debugging and fixing tests.

So I’m going to rewrite the tests using OCUnit, OCHamcrest, and OCMockito and see if the error occurs, and if so, how tough it is to debug and fix.

Moving OCUnit tests between Logic and Application tests

Sometimes logic tests will crash because the code being exercised is trying to access things that don’t exist unless the application is loaded. For example, simply trying to instantiate an instance of UIFont will work in an OCUnit Application test, but crash in an OCUnit Logic test.

Another example I recently ran into was during testing of the AppDelegate. The application:didFinishLaunchingWithOptions created by the Empty Application project template can be run and tested with logic tests, but the Single View project template’s version will crash, requiring it to be run in an application test.

The difference between a logic test and an application test is that the application test target is configured to execute in the context of the loaded application. This is done by setting the bundle loader and test host build settings. The Apple developer documentation Setting Up Application Unit Tests describes in detail how to do this.

It is interesting to note, that it was the call to makeKeyAndVisible that crashed, and that call is present in both versions. It wasn’t until after I created a view and made it the rootViewController prior to calling makeKeyAndVisible that it started crashing.

This creates somewhat of a problem for TDD. The basic philosophy behind TDD is to create new tests, watch the tests fail, then implement code to make the tests pass. There is an expectation that new code added will not break previously passing tests unless a mistake is made. But this is exactly what happened.

So I’m currently working on a workflow where I create all tests as Logic tests, then move them to the Application test target if/when the application/bundle is needed. Both targets can be added to the Cmd+U Test action, keeping things fairly simple.

For an example of this code, refer to my HowsMyFuel project on Github. Take a look at the commit comments to see how the code progressed from Logic tests only, to crashing Logic tests, to finally adding Application tests and moving the failing test over.

Update 6/5/13: I now use Application tests almost exclusively. Once the situation described above is understood, then the change in behavior can be acknowledged, and tests updated appropriately. This may even result in an increased understanding of the run time behavior, which is a good thing.

Kiwi + Application Tests = ATDD?

I’ve been playing with various ways that Kiwi can be used for testing code. It certainly has powerful capability for performing low level unit testing in a very organized way. I’ve been doing this using strictly OCUnit logic tests.

I don’t tend to use OCUnit application tests much, since they tend to be slower. And frankly, it didn’t even occur to me to try running Kiwi under application tests until last week. I ran into a situation where the code that I’ve been writing is 100% unit tested, and passing all tests, but the app doesn’t do what I want it to do.

Upon thinking about how this happened, I realized that my TDD focus has caused me to think at too low a level. I had skipped too quickly through the much needed high level design in order to quickly get to writing tests.

So I began thinking about how I could write higher level tests first, akin to acceptance tests, and use those to drive TDD at a lower level. This prompted me to think about whether Kiwi would run on Application tests, which I now know works.

So I’m heading down a direction where I’ll be using Kiwi in OCUnit application tests to define expected behavior (story acceptance tests), and then Kiwi in OCUnit logic tests to implement those behaviors via TDD. I’m exciting about this approach, and I’ll be updating this post going forward regarding how it is working.

Floating Point Rounding Errors in Kiwi

I ran into a problem this morning caused by unexpected floating point rounding differences. Comparing floating point numbers can be tricky. For example, a floating point number may represent the value one as 0.99999… So testing for the value of a floating point number may require specifying a precision (eg. 1.0 +/- 0.001). This is because 0.999 does not equal 0.999999.

Kiwi provides some support for this. I can test a floating point value using this type construct:

   [[theValue(myFloat) should] equal:1.0 withDelta:0.001];

However, I ran into a problem when specifying expected floating point arguments for a stub. There doesn’t appear to be a way to specify precision for expected arguments.

For example, the following code tests to ensure that the code is persisting a specified floating point value to NSUserDefaults:

// Code under test
- (void)setMyValue:(float)myValue {    
   [[NSUserDefaults standardUserDefaults]setFloat:myValue 
                                           forKey:@"my_value"];
}
// Kiwi test
   ...
   it(@"persists myValue to NSUserDefaults", ^ {
      id mockNSDefaults = [NSUserDefaults mock];
      [[NSUserDefaults stubAndReturn:mockNSDefaults] standardUserDefaults];
      [[mockNSDefaults should] receive:@selector(setFloat:forKey:)
                         withArguments:theValue(123.4),@"my_value"];
      sut.myValue = 123.4;
   });

This code looks pretty straight forward. It creates a mock object for an NSUserDefaults instance, and returns it when the class method [NSUserDefaults standardUserDefaults] is called (hurray class method stubs!). It then verifies that the value 123.4 is passed to the setFloat:forKey method. Evidently 123.4 isn’t exactly 123.4. It appears that that one of the values is being handled with higher precision (double) than the other (float). Then when compared, they are not exactly equal.

This problem can be fixed a couple ways:

  1. Specify the precision of the values (123.4f instead of 123.4).
    This should work in most cases, but might not depending on what’s going on under the covers.
  2. Use constants that convert from decimal to binary exactly.
    For example, the value 128.0 passes in the above example, but 123.4 doesn’t.

In the above example, I simply changed 123.4 to 123.4f in the first occurrence of 123.4, and the tests then passed as expected.

Although this problem occurred in Kiwi, you may run into similar situations in other frameworks when dealing with floating point numbers. Caveat Emptor.

Class Mocks in Kiwi

Having been working with OCUnit and OCMock for awhile now, one of the common problems that I’ve run into is the pervasive use of code that breaks the “Tell, don’t ask” rule. For example, the following code is typical for retrieving a persisted user default:

NSString *myValue = [[NSUserDefaults standardUserDefaults] 
                     stringForKey:@"my_key"];

This code is calling a class method on NSUserDefaults. So how does one mock the returned value, and or verify that this code is being made correctly, using the correct key, etc?

To date, my approach has been to extract this one-line code to a separate method that returns the value. For example:

   ...
[self getMyDefault];
...
- (NSString *)getMyDefault {
  return [[NSUserDefaults standardUserDefaults] 
          stringForKey:@"my_key"];
}

This is ok, and allows testing the calling method using either a partial mock or subclass and override. But how do we then verify that the new getMyDefault called method is working? Up until recently, I’ve been ignoring testing the new method, because there didn’t appear to be a reasonable way to do so. But in Kiwi, this is really easy:

it(@"gets the persisted user default", ^{
    id mockNSDefaults = [NSUserDefaults mock];
    [[NSUserDefaults stubAndReturn:mockNSDefaults] 
                     standardUserDefaults];
    [[mockNSDefaults should] receive:@selector(stringForKey:)
                           andReturn:@"my expected value"
                       withArguments:@"my_key"];
    [[[myObject getMyDefault] should] equal:@"my expected value"];
});

And voila! The above method creates a mock NSUserDefaults object, then uses Class method stubbing to return it when [NSUserDefaults standardUserDefaults] is called.

 

Kiwi has great support for mocking Class methods, as shown above. Refer to the Kiwi website for more details.

Looking at Other Unit Test Tools: OCHamcrest, OCMockito, and Kiwi

I’ve recently started looking at other unit testing frameworks. I’ve been using OCUnit, GHUnit, and OCMock. But I’m finding that as my tests become more extensive and numerous, the tests themselves are becoming more difficult to read. I have been able to test everything that I’ve tried using these tools, but the resulting test code is sometimes hard to read. So my objective in looking at other tools is not for additional functionality, but for better readability.

The first tools that I looked at were OCHamcrest and OCMockito by Jon Reid. I like the syntax provided by these. But I ran into a problem when converting some of my tests because they currently use partial mocks. It appears that the Java version Mockito provides something like partial mocks, using what is called “spy”, but that this capability hasn’t been ported to OCMockito yet.

So while contemplating whether to try pushing forward using categories, swizzling, or subclassing to replace partial mocks, another iOS engineer recommended that I give Kiwi a look. So I did, and it looks very promising. I guess I haven’t given Kiwi a good look before because I heard that it was only for BDD, not unit testing. This turns out not to be the case.

I am going to give Kiwi a workout by converting the tests in one of the existing unit test files in What’s My Speed: the WeatherService. This file contains an assortment of OCUnit tests using both mock and partial mock OCMock objects.

Adding Kiwi to the project and file

The first step is to add Kiwi to the project. I’m going to just add the static library and headers, but there are instructions for adding it as a subproject. I built the code from source following the instructions on the Kiwi website. I then added the static lib and headers to the project, added the Kiwi headers directory to the test target headers search path, and then the kiwi.h to the WeatherServiceTests.m file:

#import "Kiwi.h"

I’m going to try leaving OCMock in the file until all of the tests have been completed. Then rebuild to verify everything ok.

Note: I had to clean and build twice before it would build without errors. This is a long time Xcode bug. Sometimes Xcode appears to cache things. Upon making changes to header paths and such, it sometimes takes a couple builds before strange errors go away. In this case, it was reporting that it couldn’t find WeatherService.h. Build cleaning and rebuilding twice and the reported error went away.

I also encountered an error with missing header files, including NSObject+KiwiSpyAdditions.h. It appears that building Kiwi inside the Xcode IDE results in only part of the header files being copied to the build directory. I fixed this by manually copying the headers from the Kiwi/Kiwi source directory to my project’s Kiwi headers directory.

Converting the first few simple tests

Next I’ll convert the first few tests. These are simple tests that verify the basic singleton operation of the object. So here is the existing tests before converting. I’ve removed some lines that aren’t related to these tests, and I’ll add them back as we go.

#import <SenTestingKit/SenTestingKit.h>
#import <OCMock/OCMock.h>
#import "WeatherService.h"
#import "WeatherService-Private.h"

@interface WeatherServiceTests : SenTestCase

@property (nonatomic, strong) WeatherServiceForTesting *weatherService;

@end

@implementation WeatherServiceTests

- (void)setUp {    
    self.weatherService = [[WeatherServiceForTesting alloc]init];
}

- (void)tearDown {
    self.weatherService = nil;
}

- (void)testInstantiation {
    STAssertNotNil(self.weatherService, @"Test instance is nil");
}

- (void)testSharedInstanceNotNil {
    WeatherService *ws = [WeatherService sharedInstance];
    STAssertNotNil(ws, @"sharedInstance is nil");
}

- (void)testSharedInstanceReturnsSameSingletonObject {
    WeatherService *ws1 = [WeatherService sharedInstance];
    WeatherService *ws2 = [WeatherService sharedInstance];
    STAssertEquals(ws1, ws2, @"sharedInstance didn't return same object twice");
}

Ok, pretty straight forward tests, no mocks needed. Let’s convert these to Kiwi:

#import <SenTestingKit/SenTestingKit.h>
#import <OCMock/OCMock.h>
#import "WeatherService.h"
#import "WeatherService-Private.h"
#import "Kiwi.h"

SPEC_BEGIN(WeatherServiceKiwiTests)

describe(@"Singleton (by choice)", ^{

    it(@"should instantiate using init", ^ {

        [[[WeatherService alloc]init] shouldNotBeNil];
    });

    it(@"should instantiate using sharedInstance", ^{
        [[WeatherService sharedInstance] shouldNotBeNil];
    });

    it(@"should return the same instance twice using sharedInstance", ^{
        WeatherService *a = [WeatherService sharedInstance];
        WeatherService *b = [WeatherService sharedInstance];
        [[a should] beIdenticalTo:b];
    });

    it(@"should not return the same instance twice using init", ^{
        WeatherService *a = [[WeatherService alloc] init];
        WeatherService *b = [[WeatherService alloc] init];
        [[a shouldNot] beIdenticalTo:b];
    });

});
SPEC_END

Now let’s test to make sure the tests are actually working. Cmd+U to execute tests, and everything appears ok. Are the tests actually working? To verify this, I reverse the test logic by replacing “should” with “shouldNot”, and “ShouldNotBeNil” with “shouldBeNil”, rerunning the tests I see the failures. So I have some confidence that the tests are doing what I expect them to be doing.

Our next test methods further verify that init is doing what we expect. It calls 2 other methods, that each do just one thing.

- (WeatherService *)init {
    self = [super init];
    if (self) {
        [self startTimer];
        [self updateWeatherInfoForZipcode:kDEFAULT_ZIPCODE];
    }
    return self;
}

Ok, so with OCMock we used a partial mock in two tests:

- (void)testInitCallsStartTimer {
    id mock = [OCMockObject partialMockForObject:self.weatherService];
    [[mock expect]startTimer];
    id __unused initedMock = [mock init];
    [mock verify];
}
- (void)testInitCallsUpdateWeatherInfoForZipcode {
    id mock = [OCMockObject partialMockForObject:self.weatherService];
    [[mock expect]updateWeatherInfoForZipcode:kDEFAULT_ZIPCODE];
    id __unused initedMock = [mock init];
    [mock verify];
}

Since Kiwi appears to have great support mocks, this should be pretty straight forward. Note that Kiwi’s mock support allows defining stubs and expectations on both mocks and objects. This eliminates the need for partial mocks altogether!.

describe(@"init", ^{

    it(@"starts the timer", ^ {
        id weatherService = [[WeatherService alloc]init];
        [[weatherService should] receive:@selector(startTimer)];
        id __unused initedMock = [weatherService init];
    });

    it(@"updates the weather info", ^{
        id weatherService = [[WeatherService alloc]init];
        [[weatherService should] receive:@selector(updateWeatherInfoForZipcode:) withArguments:kDEFAULT_ZIPCODE];
        id __unused initedMock = [weatherService init];
    });

});

Ok, this new code looks pretty similar. It’s shorter by one line because [mock verify] isn’t needed. And for this small set of fairly simple tests, the difference in readability isn’t much, but I’m seeing the potential for greatly improved readability. The structure of the tests feels much more organized. I need to learn how to really take advantage of that. I’m going to stop this blog here, and continue converting the rest of the tests to Kiwi. I’ll probably have more to say about this in future posts as I learn more about using Kiwi.