I was quite embarrassed by all of the Xcode unit testing bugs I encountered during the recording of Unit Testing iOS Applications on Xcode 4 back in February at lynda.com. The default test template crashed, the tests kept passing when they should have failed, sometimes failing when they should have passed, and so forth. The system had been upgraded to Xcode 4.2 days before I got there, and there were numerous severe bugs in the unit test support. Sheesh! I eventually got over my embarrassment and determined that I needed to perform frequent deep-cleans and Xcode restarts as needed, and disable LLDB and used GDB instead. Come on, Apple! Does anybody there test the unit test support in Xcode before releasing it?
At WWDC 2012 I searched for “the unit test guy” to try to find out what they were doing to try to fix this situation. I had hoped to be convinced that Apple was making changes to fix this situation, maybe even unit testing their unit testing framework. The response though was defensive. I was told that the documentation was all wrong, and to go create radar bug reports if I had problems.
After that disappointment though, I had a more encouraging meeting with a developer in the Core Data group. She advocates the use of unit tests, and explained that her group received special treatment by upper management because of the fact that their code includes unit tests. It is therefore very robust and can undergo major changes very quickly and safely.
That’s the good news.
The bad news is that they do not use the unit testing framework in Xcode. They use an old version of SenTestingKit. In fact, I couldn’t find anyone at Apple that uses the unit testing framework in Xcode.
This helps explain how Xcode 4.2 was released with crashes in the default unit tests. Apple clearly does not test their unit test frameworks.
This brings me to conclude that the OCUnit support in Xcode is not important to Apple. It clearly isn’t tested, and I have no evidence that anyone at Apple is actually using it.
So after having spent the past year advocating the use of the new OCUnit support in Xcode 4, I’m reversing my position now and advocating using GHUnit instead.
I like the tight integration of OCUnit into Xcode 4, but I hate having to guess whether a test failure is really a test failure or an OCUnit or Xcode bug.