Flaky Test: Kestrel.FunctionalTests sometimes fails with "unable to communicate with test host process" or assert

[mikeharder]

Kestrel.FunctionalTests sometimes fails with "unable to communicate with test host process" (on netcoreapp2.0) or an assert (net46). A workaround is to use release instead of debug (build.cmd /p:Configuration=Release).

Assert

Block being leased from disposed pool!
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.MemoryPool.Lease()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.MemoryPool.Rent(Int32 size)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.AllocateWriteHeadUnsynchronized(Int32 count)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IPipeWriter.Alloc(Int32 minimumSize)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline.<ReadInputAsync>d__10.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline.ReadInputAsync()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline.<RunAsync>d__9.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline.RunAsync()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.<RunAdaptedPipeline>d__26.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.RunAdaptedPipeline()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.<ApplyConnectionAdaptersAsync>d__25.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.ApplyConnectionAdaptersAsync()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.<>c.<StartRequestProcessing>b__20_0(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

CI Build Failure

http://aspnetci/viewLog.html?buildId=197746&tab=buildLog&buildTypeId=XPlat_Linux_UbuntuUniverseCoreCLR&logTab=tree&filter=important&consoleStyle=true#_focus=93201

[davidfowl]

This is fixed with 58284bd

[davidfowl]

Nope, I can still reproduce it.

[davidfowl]

As described today by @halter73 this is happening because of a race between when the connection closes and when the libuv connections are walked during shutdown. Normally, the shutting down the server waits a for a certain amount of time before ending all connections, but since we made the change to immediately abort the connection on on a FIN, we lose track of that connection even though we may still be in the process of setting up the FrameConnection. This seems to be causing a bunch of random test failures.

• Since the server doesn't wait on connections we can run the connection adapters after the uv_loop shuts down causing the debug assert here.

[natemcmaster]

Hit same error just now, but stack trace came from Frame.ProduceEnd.

[davidfowl]

@natemcmaster Can you paste it?

[natemcmaster]

sadly I don't have it anymore. Was about to copy/paste it when the dialogue box window was closed due to testhost.exe dying.

[natemcmaster]

The RequestTests.LargeUpload test just triggered this failure. Stack trace matched the one Mike shared in original post. It didn't include Produce.End this time.

[Tratcher]

http://aspnetci/viewLog.html?buildId=201188&buildTypeId=CoreCLR_Darwin&tab=buildLog#_

[Tratcher]

http://aspnetci/viewLog.html?buildId=204841&buildTypeId=CoreCLR_Linux&tab=buildLog#_focus=38604

Unhandled Exception: System.ObjectDisposedException: Safe handle has been closed   at
System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)   at 
System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)   at 
Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.NativeMethods.uv_async_send(UvAsyncHandle handle)   at 
Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.async_send(UvAsyncHandle handle)   at 
Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvHandle.ReleaseHandle()   at System.Runtime.InteropServices.SafeHandle.InternalFinalize()   at 
System.Runtime.InteropServices.SafeHandle.Finalize()

[halter73]

Should be fixed by #1707. I'll reopen if we see this kind of failure again.