Category Archives: OCUnit

SenTestingKit, the unit testing framework integrated into Xcode.

Converting OCUnit to XCTest

My first attempt at converting a real project to XCTest in Xcode 5 went fairly well. The project was fairly simple, and had 582 OCUnit Application tests using OCHamcrest assertions. I performed this using Xcode 5 DP3, so the issues I encountered should have been fixed by the time you read this.
Xcode 5 supports both OCUnit and XCTest, so the app loaded and the tests ran ok. Interestingly, the Test Navigator displayed all of the OCUnit tests nicely. I had wondered whether or not the Test Navigator only worked with XCTests. It works with both though.
Once all the tests were passing, I then used the “Edit -> Refactor -> Convert to XCTest” menu option to convert the OCUnit test target to XCTest. I wasn’t sure what was going to happen, but had Git ready to undo any changes if/when things got messed up. It turned out that I didn’t need to do this.
The conversion worked almost perfectly. This isn’t a tough operation, basically just changing the #import from SenTestingKit to XCTest, and changing the test case class parent from SenTestCase to XCTestCase.
This conversion reused the same target, without renaming it. So things look pretty much the same in the Test Navigator before and after the conversion.
Then I tried to run the tests. I expected that the tests would fail, or maybe even fail to compile due to OCHamcrest. What happened though was that Xcode 5 crashed. However, upon restarting Xcode 5, I could then select to run the tests without crashing, but got a linker error.
The conversion operation did not add the XCTest framework to the converted target. It did remove the SenTestingKit framework though. So this is probably just a bug in DP3. I submitted radar 14470998 which is fixed in the latest code.
After adding XCTest to the test target’s link build step, all tests ran and passed!
I have wondered, and talked with other developers that I work with, about why Apple would simply rename SenTestingKit to XCTest. It doesn’t appear that much if anything is functionally different. I’ve come to the conclusion (and the hope) that this is just the beginning. Having a totally separate, new framework allows Apple to make changes and additions going forward, without worrying about breaking legacy SenTestingKit users. Let’s hope that this is a new start for unit testing support from Apple.

 

“Duplicate Classes with Same Name” Bug

I ran into the most puzzling issue this past week. While working with the new MMDrawerController package on Github (written by some friends here at Mutual Mobile), I ran into a situation where the test runtime believes that there are 2 separate classes, both named “MMDrawerBarButtonItem”.

The failing source code for the project is on Github in WhosWorkingOnWhat at commit c19a6a7de5cda609c75b70d93aed91687fdfcf00. You can uncomment the code in that commit to see that the log shows that there are 2 subclasses of UIBarButtonItem with the same name “MMDrawerBarButtonItem”.

What’s happening in testNavBarLeftButton is that the code under test assigning an instance of one class named “MMDrawerBarButtonItem” to the leftBarButtonItem. The test is then trying to confirm that an instance of the correct class has been assigned to the left button,  but when the test compares it with [MMDrawerBarButtonItem class], it is getting the second class with that name, and therefore failing the assertion!

This is a most peculiar bug, and so far has me completely stumped. Any ideas on how my tests are managing to create 2 classes with the same name?

Update 5/28/13:

Last week one of the senior iOS engineers ran into a similar problem on his project, and determined the cause and a workaround. The problem is caused by the combination of CocoaPods and OCUnit Application tests.

CocoaPods was used to add a 3rd party library. In my case, it is MMDrawerController, in his it was something else. Then in order to test code that interacts with that 3rd party library, the Podfile was updated to add the same library to the unit test target. And voila, when tests are run, 2 instances of the classes in the 3rd party library will exist in the combined product code + unit test code test target.

Normally the compiler and linker would detect this duplication, but no errors are generated  when the duplicates exists separately in the product target and the unit test target, which then get combined when running the unit tests.

My workaround was to simply stop using CocoaPods. My friends solution involved putting dependencies into a separate static library. There’s probably a better solution, and I’ll update this when it is identified.

OCUnit Logic Tests Are Dead

… long live OCUnit Application tests.

I’ve been espousing the use of logic tests for quite awhile now. But recently, I’ve had a chance to work with some projects that have a fairly large number of OCUnit application unit tests. To my surprise, these tests run very fast, roughly sub-second per hundreds o tests.

Of course, test execution times will vary depending on the individual project and tests, but it appears that in general, there’s no need to jump through hoops using OCUnit Logic tests, since simply using OCUnit Application tests should work just fine.

This also applies to GHUnit. I’ve seen a project with over 600 tests execute in under 2 seconds.

Anyone have observations that disagree with this?

 

Moving OCUnit tests between Logic and Application tests

Sometimes logic tests will crash because the code being exercised is trying to access things that don’t exist unless the application is loaded. For example, simply trying to instantiate an instance of UIFont will work in an OCUnit Application test, but crash in an OCUnit Logic test.

Another example I recently ran into was during testing of the AppDelegate. The application:didFinishLaunchingWithOptions created by the Empty Application project template can be run and tested with logic tests, but the Single View project template’s version will crash, requiring it to be run in an application test.

The difference between a logic test and an application test is that the application test target is configured to execute in the context of the loaded application. This is done by setting the bundle loader and test host build settings. The Apple developer documentation Setting Up Application Unit Tests describes in detail how to do this.

It is interesting to note, that it was the call to makeKeyAndVisible that crashed, and that call is present in both versions. It wasn’t until after I created a view and made it the rootViewController prior to calling makeKeyAndVisible that it started crashing.

This creates somewhat of a problem for TDD. The basic philosophy behind TDD is to create new tests, watch the tests fail, then implement code to make the tests pass. There is an expectation that new code added will not break previously passing tests unless a mistake is made. But this is exactly what happened.

So I’m currently working on a workflow where I create all tests as Logic tests, then move them to the Application test target if/when the application/bundle is needed. Both targets can be added to the Cmd+U Test action, keeping things fairly simple.

For an example of this code, refer to my HowsMyFuel project on Github. Take a look at the commit comments to see how the code progressed from Logic tests only, to crashing Logic tests, to finally adding Application tests and moving the failing test over.

Update 6/5/13: I now use Application tests almost exclusively. Once the situation described above is understood, then the change in behavior can be acknowledged, and tests updated appropriately. This may even result in an increased understanding of the run time behavior, which is a good thing.

Class Mocks in Kiwi

Having been working with OCUnit and OCMock for awhile now, one of the common problems that I’ve run into is the pervasive use of code that breaks the “Tell, don’t ask” rule. For example, the following code is typical for retrieving a persisted user default:

NSString *myValue = [[NSUserDefaults standardUserDefaults] 
                     stringForKey:@"my_key"];

This code is calling a class method on NSUserDefaults. So how does one mock the returned value, and or verify that this code is being made correctly, using the correct key, etc?

To date, my approach has been to extract this one-line code to a separate method that returns the value. For example:

   ...
[self getMyDefault];
...
- (NSString *)getMyDefault {
  return [[NSUserDefaults standardUserDefaults] 
          stringForKey:@"my_key"];
}

This is ok, and allows testing the calling method using either a partial mock or subclass and override. But how do we then verify that the new getMyDefault called method is working? Up until recently, I’ve been ignoring testing the new method, because there didn’t appear to be a reasonable way to do so. But in Kiwi, this is really easy:

it(@"gets the persisted user default", ^{
    id mockNSDefaults = [NSUserDefaults mock];
    [[NSUserDefaults stubAndReturn:mockNSDefaults] 
                     standardUserDefaults];
    [[mockNSDefaults should] receive:@selector(stringForKey:)
                           andReturn:@"my expected value"
                       withArguments:@"my_key"];
    [[[myObject getMyDefault] should] equal:@"my expected value"];
});

And voila! The above method creates a mock NSUserDefaults object, then uses Class method stubbing to return it when [NSUserDefaults standardUserDefaults] is called.

 

Kiwi has great support for mocking Class methods, as shown above. Refer to the Kiwi website for more details.

Looking at Other Unit Test Tools: OCHamcrest, OCMockito, and Kiwi

I’ve recently started looking at other unit testing frameworks. I’ve been using OCUnit, GHUnit, and OCMock. But I’m finding that as my tests become more extensive and numerous, the tests themselves are becoming more difficult to read. I have been able to test everything that I’ve tried using these tools, but the resulting test code is sometimes hard to read. So my objective in looking at other tools is not for additional functionality, but for better readability.

The first tools that I looked at were OCHamcrest and OCMockito by Jon Reid. I like the syntax provided by these. But I ran into a problem when converting some of my tests because they currently use partial mocks. It appears that the Java version Mockito provides something like partial mocks, using what is called “spy”, but that this capability hasn’t been ported to OCMockito yet.

So while contemplating whether to try pushing forward using categories, swizzling, or subclassing to replace partial mocks, another iOS engineer recommended that I give Kiwi a look. So I did, and it looks very promising. I guess I haven’t given Kiwi a good look before because I heard that it was only for BDD, not unit testing. This turns out not to be the case.

I am going to give Kiwi a workout by converting the tests in one of the existing unit test files in What’s My Speed: the WeatherService. This file contains an assortment of OCUnit tests using both mock and partial mock OCMock objects.

Adding Kiwi to the project and file

The first step is to add Kiwi to the project. I’m going to just add the static library and headers, but there are instructions for adding it as a subproject. I built the code from source following the instructions on the Kiwi website. I then added the static lib and headers to the project, added the Kiwi headers directory to the test target headers search path, and then the kiwi.h to the WeatherServiceTests.m file:

#import "Kiwi.h"

I’m going to try leaving OCMock in the file until all of the tests have been completed. Then rebuild to verify everything ok.

Note: I had to clean and build twice before it would build without errors. This is a long time Xcode bug. Sometimes Xcode appears to cache things. Upon making changes to header paths and such, it sometimes takes a couple builds before strange errors go away. In this case, it was reporting that it couldn’t find WeatherService.h. Build cleaning and rebuilding twice and the reported error went away.

I also encountered an error with missing header files, including NSObject+KiwiSpyAdditions.h. It appears that building Kiwi inside the Xcode IDE results in only part of the header files being copied to the build directory. I fixed this by manually copying the headers from the Kiwi/Kiwi source directory to my project’s Kiwi headers directory.

Converting the first few simple tests

Next I’ll convert the first few tests. These are simple tests that verify the basic singleton operation of the object. So here is the existing tests before converting. I’ve removed some lines that aren’t related to these tests, and I’ll add them back as we go.

#import <SenTestingKit/SenTestingKit.h>
#import <OCMock/OCMock.h>
#import "WeatherService.h"
#import "WeatherService-Private.h"

@interface WeatherServiceTests : SenTestCase

@property (nonatomic, strong) WeatherServiceForTesting *weatherService;

@end

@implementation WeatherServiceTests

- (void)setUp {    
    self.weatherService = [[WeatherServiceForTesting alloc]init];
}

- (void)tearDown {
    self.weatherService = nil;
}

- (void)testInstantiation {
    STAssertNotNil(self.weatherService, @"Test instance is nil");
}

- (void)testSharedInstanceNotNil {
    WeatherService *ws = [WeatherService sharedInstance];
    STAssertNotNil(ws, @"sharedInstance is nil");
}

- (void)testSharedInstanceReturnsSameSingletonObject {
    WeatherService *ws1 = [WeatherService sharedInstance];
    WeatherService *ws2 = [WeatherService sharedInstance];
    STAssertEquals(ws1, ws2, @"sharedInstance didn't return same object twice");
}

Ok, pretty straight forward tests, no mocks needed. Let’s convert these to Kiwi:

#import <SenTestingKit/SenTestingKit.h>
#import <OCMock/OCMock.h>
#import "WeatherService.h"
#import "WeatherService-Private.h"
#import "Kiwi.h"

SPEC_BEGIN(WeatherServiceKiwiTests)

describe(@"Singleton (by choice)", ^{

    it(@"should instantiate using init", ^ {

        [[[WeatherService alloc]init] shouldNotBeNil];
    });

    it(@"should instantiate using sharedInstance", ^{
        [[WeatherService sharedInstance] shouldNotBeNil];
    });

    it(@"should return the same instance twice using sharedInstance", ^{
        WeatherService *a = [WeatherService sharedInstance];
        WeatherService *b = [WeatherService sharedInstance];
        [[a should] beIdenticalTo:b];
    });

    it(@"should not return the same instance twice using init", ^{
        WeatherService *a = [[WeatherService alloc] init];
        WeatherService *b = [[WeatherService alloc] init];
        [[a shouldNot] beIdenticalTo:b];
    });

});
SPEC_END

Now let’s test to make sure the tests are actually working. Cmd+U to execute tests, and everything appears ok. Are the tests actually working? To verify this, I reverse the test logic by replacing “should” with “shouldNot”, and “ShouldNotBeNil” with “shouldBeNil”, rerunning the tests I see the failures. So I have some confidence that the tests are doing what I expect them to be doing.

Our next test methods further verify that init is doing what we expect. It calls 2 other methods, that each do just one thing.

- (WeatherService *)init {
    self = [super init];
    if (self) {
        [self startTimer];
        [self updateWeatherInfoForZipcode:kDEFAULT_ZIPCODE];
    }
    return self;
}

Ok, so with OCMock we used a partial mock in two tests:

- (void)testInitCallsStartTimer {
    id mock = [OCMockObject partialMockForObject:self.weatherService];
    [[mock expect]startTimer];
    id __unused initedMock = [mock init];
    [mock verify];
}
- (void)testInitCallsUpdateWeatherInfoForZipcode {
    id mock = [OCMockObject partialMockForObject:self.weatherService];
    [[mock expect]updateWeatherInfoForZipcode:kDEFAULT_ZIPCODE];
    id __unused initedMock = [mock init];
    [mock verify];
}

Since Kiwi appears to have great support mocks, this should be pretty straight forward. Note that Kiwi’s mock support allows defining stubs and expectations on both mocks and objects. This eliminates the need for partial mocks altogether!.

describe(@"init", ^{

    it(@"starts the timer", ^ {
        id weatherService = [[WeatherService alloc]init];
        [[weatherService should] receive:@selector(startTimer)];
        id __unused initedMock = [weatherService init];
    });

    it(@"updates the weather info", ^{
        id weatherService = [[WeatherService alloc]init];
        [[weatherService should] receive:@selector(updateWeatherInfoForZipcode:) withArguments:kDEFAULT_ZIPCODE];
        id __unused initedMock = [weatherService init];
    });

});

Ok, this new code looks pretty similar. It’s shorter by one line because [mock verify] isn’t needed. And for this small set of fairly simple tests, the difference in readability isn’t much, but I’m seeing the potential for greatly improved readability. The structure of the tests feels much more organized. I need to learn how to really take advantage of that. I’m going to stop this blog here, and continue converting the rest of the tests to Kiwi. I’ll probably have more to say about this in future posts as I learn more about using Kiwi.

Refactoring to Improve Testability

One of the observations that I made earlier is that refactoring for clean code as advocated by Robert C. Martin would greatly simplify unit testing. So let me give you an example, using some code I’m actually working with, not just something I threw together to support my argument.

This is from my personal motorcycle app that I use to display navigation, time, temp, and weather forecast while riding. I currently have a Weather model file that provides basic weather properties such as current temp. During initialization of this object, it needs to setup a timer to periodically check for any changes in the weather, as well as register to receive location notifications. This particular object was written long ago, before I became a 100% code coverage fanatic. I didn’t know how to test the init method back then. But now that I do, I’m going to refactor this code both to make Uncle Bob happy, as well as to add full unit test coverage for it.

So here’s the original init method:

Original code image

This code looks pretty typical for an init method, and there isn’t anything functionally wrong with it. Ignoring the NSLog statement, and standard call to super and return of self, it does 3 things:

  1. Set the default current location. Location is needed for getting local weather, and since I’m on a motorcycle, it will change periodically. I initialize it with a constant that is the zipcode where my home is located.
  2. Setup a timer to fire off every 5 minutes to update weather.
  3. Register to receive location change notifications from a Location object I also created.

I’ve used comments to identify what is happening on 2 of these things, the 3rd being so obvious I didn’t bother. However, I’ve learned over the past few years that comments should be considered smells. I believe that a better approach is to extract the commented code to a separate method, and name the method using the text from the comment. Let’s see how that also helps make unit testing easier.

Let’s extract the first thing to its own helper method: setLocationToDefault. We can use the refactor command as shown here:

Using refactor to extract a method

Following that, we’ll do the same thing for the other 2 things that init is doing. This results in the following refactored code:

Refactored init code

Now our code reads like prose, and the extracted helper methods do exactly what you would expect. These are 2 attributes of good code.

Create a class extension file to contain the declarations of the helper methods. This file can then be #imported in the unit test case file, but kept private to normal class users:

Prior to making these changes, our unit tests for init would require verifying that:

  1. The current location (lastLocation property) is set to our default value.
  2. The timer was set to callback every 5 minutes
  3. A location notification is set.

The first item is very easy to test. But the inclusion of the other 2 things in init could complicate and make the first test unreliable and/or outright fail. What if a side effect of the other 2 causes the current location to change? There is a potential race condition here. Even if it didn’t break at first, it might in the future as changes are made to the code. This code is fragile.

In this particular case, it probably isn’t a problem. But it shows the type of problem that unit testing can encounter whenever a method does multiple things. In Clean Code, Uncle Bob admonishes us to have each method do exactly 1 thing (the Curly principle?). That may sound a bit extreme, but look how it has simplified things here, and made things safer to extend or change later.

After refactoring, the unit tests for init can focus on making sure that it calls the methods that do these things, and not worry about exactly how they are done, how long it takes, or whether they have conflicting effects. We’ll use a partial mock object to do this, as shown here:

Init test using partial mock

In the code above, self.weather refers to the test weather object created in the setUp method. The partial mock created will intercept the expected calls, but pass anything else on to the test object. This code therefore tests only the code in the init method. The other unit tests that verify that the 3 things happen correctly would call the helper methods directly, thereby eliminating the possibility of any conflicting side effects.

OCUnit Application vs. Logic Tests

One of the things that I learned at WWDC 2012 is that Apple has changed the definition of OCUnit Logic and OCUnit Application tests. Apple does not split OCUnit test into “Application” and “Logic” tests anymore. Instead, OCUnit test targets can optionally specify a “test host” in the target’s build settings. What we think of currently as “Application” unit tests are tests that have designated a test host. What we think of as “Logic” unit tests do not.

One of the useful side effects of this is that both types of tests can be run in the same  scheme and action. Previously I had been creating two separate schemes in my projects; one for the Application tests and one for the Logic tests. Both can actually be combined into a single scheme and run together if desired.

 

Back to GHUnit

I was quite embarrassed by all of the Xcode unit testing bugs I encountered during the recording of Unit Testing iOS Applications on Xcode 4 back in February at lynda.com. The default test template crashed, the tests kept passing when they should have failed, sometimes failing when they should have passed, and so forth. The system had been upgraded to Xcode 4.2 days before I got there, and there were numerous severe bugs in the unit test support. Sheesh! I eventually got over my embarrassment and determined that I needed to perform frequent deep-cleans and Xcode restarts as needed, and disable LLDB and used GDB instead. Come on, Apple! Does anybody there test the unit test support in Xcode before releasing it?

At WWDC 2012 I searched for “the unit test guy” to try to find out what they were doing to try to fix this situation. I had hoped to be convinced that Apple was making changes to fix this situation, maybe even unit testing their unit testing framework. The response though was defensive. I was told that the documentation was all wrong, and to go create radar bug reports if I had problems.

After that disappointment though, I had a more encouraging meeting with a developer in the Core Data group. She advocates the use of unit tests, and explained that her group received special treatment by upper management because of the fact that their code includes unit tests. It is therefore very robust and can undergo major changes very quickly and safely.

That’s the good news.

The bad news is that they do not use the unit testing framework in Xcode. They use an old version of SenTestingKit. In fact, I couldn’t find anyone at Apple that uses the unit testing framework in Xcode.

This helps explain how Xcode 4.2 was released with crashes in the default unit tests. Apple clearly does not test their unit test frameworks.

This brings me to conclude that the OCUnit support in Xcode is not important to Apple. It clearly isn’t tested, and I have no evidence that anyone at Apple is actually using it.

So after having spent the past year advocating the use of the new OCUnit support in Xcode 4, I’m reversing my position now and advocating using GHUnit instead.

I like the tight integration of OCUnit into Xcode 4, but I hate having to guess whether a test failure is really a test failure or an OCUnit or Xcode bug.