I was quite embarrassed by all of the Xcode unit testing bugs I encountered during the recording of Unit Testing iOS Applications on Xcode 4 back in February at lynda.com. The default test template crashed, the tests kept passing when they should have failed, sometimes failing when they should have passed, and so forth. The system had been upgraded to Xcode 4.2 days before I got there, and there were numerous severe bugs in the unit test support. Sheesh! I eventually got over my embarrassment and determined that I needed to perform frequent deep-cleans and Xcode restarts as needed, and disable LLDB and used GDB instead. Come on, Apple! Does anybody there test the unit test support in Xcode before releasing it?
At WWDC 2012 I searched for “the unit test guy” to try to find out what they were doing to try to fix this situation. I had hoped to be convinced that Apple was making changes to fix this situation, maybe even unit testing their unit testing framework. The response though was defensive. I was told that the documentation was all wrong, and to go create radar bug reports if I had problems.
After that disappointment though, I had a more encouraging meeting with a developer in the Core Data group. She advocates the use of unit tests, and explained that her group received special treatment by upper management because of the fact that their code includes unit tests. It is therefore very robust and can undergo major changes very quickly and safely.
That’s the good news.
The bad news is that they do not use the unit testing framework in Xcode. They use an old version of SenTestingKit. In fact, I couldn’t find anyone at Apple that uses the unit testing framework in Xcode.
This helps explain how Xcode 4.2 was released with crashes in the default unit tests. Apple clearly does not test their unit test frameworks.
This brings me to conclude that the OCUnit support in Xcode is not important to Apple. It clearly isn’t tested, and I have no evidence that anyone at Apple is actually using it.
So after having spent the past year advocating the use of the new OCUnit support in Xcode 4, I’m reversing my position now and advocating using GHUnit instead.
I like the tight integration of OCUnit into Xcode 4, but I hate having to guess whether a test failure is really a test failure or an OCUnit or Xcode bug.
Hey there! Love your idea to start a unit testing blog. I’ve been doing iOS unit testing for over a year now, and TDD in general for about 3 years.
I was curious, did you configure your GHUnit test classes to be ARC-enabled?
I seem to have a compilation error using GHAssertNotNULL, complaining about:
“Implicit conversion of Objective-C pointer type ‘NSString *’ to C pointer type ‘const void *’ requires a bridged cast”
What did you do to solve this, or did you just disable ARC on these specific classes? I was curious what you decided on, if you could let me know that would be great. Thanks!
Hi Kyle,
Thanks for the note. Sorry for the slow response. I’ve been on 2 weeks vacation.
I don’t recall having this sort of trouble. I’ll try setting up an ARC project with GHUnit again and see if I encounter similar problems.
It is very quick fix, change to GHAssertNotNil instead of GHAssertNotNULL. I got the same issue when run the first example from GHUnit website. I guess, it is only a typo, or a new version of method.
> I couldn’t find anyone at Apple that uses the unit testing framework in Xcode.
*Raises hand* I used it.
This was in the Xcode 3.* days. I worked at Apple and I used the Xcode unit testing framework, to test some functionality in the Xcode application itself. Never had the severity of problems you describe.
And in the Xcode 4.* days, from what I’ve seen, they’ve improved certain aspects of it quite a bit.
So I’d be curious about the details of the kinds of problems you’re seeing, though I’m not sure blog comments are the best place to suss it all out. Are you going to be in the San Francisco Bay Area anytime soon? If nothing else, I could buy you a beer at WWDC 2013.
I’ve subscribed to iOS Unit Testing and will be following it with interest.
I heard about this post from here: https://twitter.com/rentzsch/status/272842700942897153
I’m delighted to hear about others using OCUnit, especially within Apple. I’d like to hear from someone currently doing so within Apple.
I still like using OCUnit myself, although many on my team have switched to GHUnit exclusively. I can’t argue with their position that they don’t want to have to spend extra time figuring out how to get the test framework to work whenever Xcode releases a new version.
As mentioned in the post, the biggest problem that I ran into was in going from 4.1 to 4.2. I had not started doing much unit testing in Xcode prior to 4, so I cannot attest to the stability of OCUnit in Xcode 3.
At Mutual Mobile, we have unit tests in every project, with coverage ranging from just model code to nearly everything. And we’re working on driving the percentage higher, and augmenting unit tests with automated integration, UI, and other tests.
So let me be clear: I love using OCUnit, and I do use it a lot. I love the tight integration into Xcode. And I’ll continue to use it. But I’ve been bitten by it, and remain convinced that Apple doesn’t test OCUnit before shipping new Xcode versions, and documentation for it is strictly an afterthought.
Caveat emptor.