Slather freezing/hanging/stalling when generating coverage report

[brockboland]

Foreword: This might be a bug for fastlane, and it got long because I'm trying to include all the details I have available.

My team has been using Slather on a couple projects in the past month or two, and just in the past week or two, I've noticed it hanging on Travis after running tests. It causes the build to fail, after ten minutes without any output (this is the output from fastlane on Travis, which includes the actual command it runs):

[19:33:28]: $ slather coverage --input-format profdata --scheme MyScheme --cobertura-xml --output-directory test_output --ignore R.generated.swift --ignore MyClient/CoreDataAndModels/private/\* --ignore MyClient/\*TestingAppDelegate.\* --ignore MyClient/Utilities/DLog.swift --ignore Pods/\* MyClient.xcodeproj
[19:33:30]: ▸ Slathering...
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

This seems to happen most of the time, around 4/5 builds. If I just keep re-starting the build, eventually it will pass, so it's not failing completely. I thought this was just a Travis issue, but I just found the same thing happened locally.

Has anyone ever seen behavior like this? Or have any idea what might be hanging in slather? Unforunately, fastlane doesn't pass through a --verbose flag to slather, though it doesn't look like that flag results in too much more output, so I'm not sure it would help.

Some notes on setup:

• When running slather from fastlane, it fails about 4/5 times on Travis, and less often locally.
• When I just copy the slather command that fastlane runs and do that locally (after running the tests from fastlane), it works every time.
• Using Xcode 7.3
• When slathering does work, it only takes about 3-5 seconds
• The project is using CocoaPods, and thus, a workspace, which is specified in .slather.yml since fastlane doesn't yet support the --workspace flag (it'll be in the next version: Add workspace command line option to slather action fastlane/fastlane#4109)
• Project has two schemes: unit tests, and UI tests. Our test lane runs one and slathers the results, then runs the other and slathers that, then uses a custom script to combine the resulting cobertura.xml output files (that script isn't the problem: it's not even getting that far before it hangs).
• Usually fails after the first set of tests, before running the second, but not always

[neonichu]

Haven't yet seen or heard about such an issue so far. I'm guessing you're not able to share the project this happens on?

[chillpop]

TLDR: pass in the build directory explicitly to avoid this (--build-directory flag).

This appears to be an issue with xcodebuild becoming deadlocked.

I am able to recreate the issue by repeated running slather in two terminals:
for n in {1..5}; do slather coverage --input-format profdata --scheme MyApp --verbose --workspace MyApp.xcworkspace MyApp.xcodeproj; done

When one of these calls freezes, looking at ActivityMonitor, there is an xcodebuild process that is also frozen. The stack trace for the frozen xcodebuild indicates that the process is waiting for a lock:

Call graph:
    2614 Thread_27382025   DispatchQueue_1: com.apple.main-thread  (serial)
    + 2614 start  (in libdyld.dylib) + 1  [0x7fff8e6955ad]
    +   2614 ???  (in xcodebuild)  load address 0x10d0a0000 + 0x1861  [0x10d0a1861]
    +     2614 -[Xcode3CommandLineBuildTool run]  (in Xcode3Core) + 5479  [0x10e092faa]
    +       2614 -[Xcode3CommandLineBuildTool _showBuildSettings]  (in Xcode3Core) + 1873  [0x10e08a542]
    +         2614 -[PBXTarget createDependenciesInTargetBuildContextIfNeeded]  (in DevToolsCore) + 226  [0x111e297a9]
    +           2614 -[PBXTargetBuildContext(DependencyGraphEvents) processDependencyGraphEvents]  (in DevToolsCore) + 126  [0x111e96557]
    +             2614 -[PBXTargetBuildContext lockDependencyGraph]  (in DevToolsCore) + 52  [0x111e77d31]
    +               2614 -[NSRecursiveLock lock]  (in Foundation) + 22  [0x7fff91d944fb]
    +                 2614 _pthread_mutex_lock_slow  (in libsystem_pthread.dylib) + 300  [0x7fff9502c5f5]
    +                   2614 _pthread_mutex_lock_wait  (in libsystem_pthread.dylib) + 89  [0x7fff9502ee4a]
    +                     2614 __psynch_mutexwait  (in libsystem_kernel.dylib) + 10  [0x7fff891b9de6]

(This deadlock can also be recreated by repeatedly calling xcodebuild in two terminals:
for n in {1..5}; do xcodebuild -workspace MyApp.xcworkspace -scheme MyApp -showBuildSettings test; done)

When passing in the build directory using the --build-directory or -b flags, the freeze does not occur when looping multiple runs of slather.

I suspect the issue was that fastlane was using xcodebuild for some prior action (probably scan to run tests) that was causing slather to freeze when getting the Derived Data directory automatically using xcodebuild.

[brockboland]

Thanks @chillpop!

[neonichu]

Scary 😱

[ksuther]

Maybe slather is running into this bug? http://www.openradar.me/radar?id=5549442786656256

[yunnnyunnn]

Faced the same issue on CircleCI. Specifying build_directory in .slather.yml solved the problem.
Thanks @chillpop !