Kestrel faills periodically:Error -107 ENOTCONN

[agoretsky]

I've upgraded my app to rc2 and now my app periodically fails after some hours of running.
I am using ubuntu 15.04, mono 4.2.3.4 + asp.net core rc2 (https://www.microsoft.com/net/core#ubuntu)

Unhandled Exception: Microsoft.AspNetCore.Server.Kestrel.Networking.UvException: Error -107 ENOTCONN socket is not connected at Microsoft.AspNetCore.Server.Kestrel.Networking.Libuv.Check (Int32 statusCode) <0x40abee50 + 0x0004f> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.Networking.Libuv.tcp_getpeername (Microsoft.AspNetCore.Server.Kestrel.Networking.UvTcpHandle handle, Microsoft.AspNetCore.Server.Kestrel.Networking.SockAddr& addr, System.Int32& namelen at Microsoft.AspNetCore.Server.Kestrel.Networking.UvTcpHandle.GetPeerIPEndPoint () <0x40ad1940 + 0x0004b> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.Http.Connection.Start () <0x40ad0e00 + 0x0010b> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.Http.ListenerSecondary.ReadStartCallback (Microsoft.AspNetCore.Server.Kestrel.Networking.UvStreamHandle handle, Int32 status) <0x40ace1d0 + 0x003a3> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.Http.ListenerSecondary+<>c.<ConnectedCallback>b__13_1 (Microsoft.AspNetCore.Server.Kestrel.Networking.UvStreamHandle handle, Int32 status2, System.Object state) <0x40acdde0 + 0x0005b> i at Microsoft.AspNetCore.Server.Kestrel.Networking.UvStreamHandle.UvReadCb (IntPtr handle, Int32 status, Microsoft.AspNetCore.Server.Kestrel.Networking.uv_buf_t& buf) <0x40acdbf0 + 0x0006a> in <filename unknown>:0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x40f33640 + 0x0002b> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.KestrelThread.Stop (TimeSpan timeout) <0x40f80dc0 + 0x0052b> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.KestrelEngine.Dispose () <0x40f80cc0 + 0x00073> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Dispose () <0x40f7a380 + 0x0003e> in <filename unknown>:0 at Microsoft.AspNetCore.Hosting.Internal.WebHost.Dispose () <0x40f7a110 + 0x0006f> in <filename unknown>:0 at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run (IWebHost host, CancellationToken token, System.String shutdownMessage) <0x40aab510 + 0x003a5> in <filename unknown>:0 at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run (IWebHost host) <0x40aaac80 + 0x001db> in <filename unknown>:0 at Must.Backend.Program.Main (System.String[] args) <0x408dee00 + 0x000d7> in <filename unknown>:0 [ERROR] FATAL UNHANDLED EXCEPTION: Microsoft.AspNetCore.Server.Kestrel.Networking.UvException: Error -107 ENOTCONN socket is not connected at Microsoft.AspNetCore.Server.Kestrel.Networking.Libuv.Check (Int32 statusCode) <0x40abee50 + 0x0004f> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.Networking.Libuv.tcp_getpeername (Microsoft.AspNetCore.Server.Kestrel.Networking.UvTcpHandle handle, Microsoft.AspNetCore.Server.Kestrel.Networking.SockAddr& addr, System.Int32& namelen at Microsoft.AspNetCore.Server.Kestrel.Networking.UvTcpHandle.GetPeerIPEndPoint () <0x40ad1940 + 0x0004b> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.Http.Connection.Start () <0x40ad0e00 + 0x0010b> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.Http.ListenerSecondary.ReadStartCallback (Microsoft.AspNetCore.Server.Kestrel.Networking.UvStreamHandle handle, Int32 status) <0x40ace1d0 + 0x003a3> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.Http.ListenerSecondary+<>c.<ConnectedCallback>b__13_1 (Microsoft.AspNetCore.Server.Kestrel.Networking.UvStreamHandle handle, Int32 status2, System.Object state) <0x40acdde0 + 0x0005b> i at Microsoft.AspNetCore.Server.Kestrel.Networking.UvStreamHandle.UvReadCb (IntPtr handle, Int32 status, Microsoft.AspNetCore.Server.Kestrel.Networking.uv_buf_t& buf) <0x40acdbf0 + 0x0006a> in <filename unknown>:0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x40f33640 + 0x0002b> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.KestrelThread.Stop (TimeSpan timeout) <0x40f80dc0 + 0x0052b> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.KestrelEngine.Dispose () <0x40f80cc0 + 0x00073> in <filename unknown>:0 at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Dispose () <0x40f7a380 + 0x0003e> in <filename unknown>:0 at Microsoft.AspNetCore.Hosting.Internal.WebHost.Dispose () <0x40f7a110 + 0x0006f> in <filename unknown>:0 at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run (IWebHost host, CancellationToken token, System.String shutdownMessage) <0x40aab510 + 0x003a5> in <filename unknown>:0 at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run (IWebHost host) <0x40aaac80 + 0x001db> in <filename unknown>:0 at Must.Backend.Program.Main (System.String[] args) <0x408dee00 + 0x000d7> in <filename unknown>:0 Jun 27 03:24:15 systemd[1must.service: Main process exited, code=exited, status=1/FAILURE Jun 27 03:24:15 docker[25413Error response from daemon: no such id: must. Jun 27 03:24:15 docker[25413Error: failed to stop containers: [must.] Jun 27 03:24:15 systemd[1must.service: Control process exited, code=exited status=1 Jun 27 03:24:15 systemd[1must.service: Unit entered failed state. Jun 27 03:24:15 systemd[1must.service: Failed with result 'exit-code'.

[davidfowl]

• Mono isn't officially supported (it's a best effort sorta thing)
• That looks like it happens when the server is shutting down, are you running kestrel directly or with nginx in front of it?

[agoretsky]

haproxy in front of it, will move our dependencies to .NET Core.

[halter73]

@agoretsky Also try upgrading to 1.0.0 which was released this morning. While this error may still be logged (even on .NET Core) when a client quickly disconnects after establishing a connection, the entire app shouldn't fail anymore.

Feel free to comment if this isn't the case, and I can reopen.

[gzpbx]

@halter73, In our production (asp.net core 1.1.0 on CentOS 7.2), this issue happened all the time ——

2017-01-24T15:23:45.0219570+08:00 [ERROR] (Piao//) ListenerSecondary.OnConnection
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -107 ENOTCONN socket is not connected
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.ThrowError(Int32 statusCode)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_getpeername(UvTcpHandle handle, SockAddr& addr, Int32& namelen)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle.GetPeerIPEndPoint()
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Connection..ctor(ListenerContext context, UvStreamHandle socket)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.ListenerSecondary.ReadStartCallback(UvStreamHandle handle, Int32 status)

[halter73]

@gzpbx The error should be pretty benign now. Previously it was fatal. Would you prefer this error wasn't logged at all, or would logging it as a warning or information be sufficient?

[mayamoon]

@gzpbx
@halter73
I have the same issue in my production(asp.net core 1.1.0 on CentOS 7.2),it will cause a lot of error log in my log system. if this is a normal situation,please ignore it.

[sparksterz]

@halter73 I'm actually having a similar problem as @gzpbx . We're also hosting a Kestrel server on asp .net core 1.1 in our case using the official microsoft/aspnetcore1.1 container. Similar to @agoretsky we have an HA Proxy in front of our services.

Within the last couple of weeks we've seen the containers fail however and give a similar stack trace:
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -107 ENOTCONN socket is not connected at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_getpeername(UvTcpHandle handle, SockAddr& addr, Int32& namelen) at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle.GetPeerIPEndPoint() at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Connection..ctor(ListenerContext context, UvStreamHandle socket) at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Listener.DispatchConnection(UvStreamHandle socket) at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.TcpListener.OnConnection(UvStreamHandle listenSocket, Int32 status)

We run our integration tests and eventually our services will get into this state and it prevents the service from properly responding. Any information about where to dig to find out more would be useful. I'm a bit at a loss considering the stack trace is buried in Kestrel and we haven't been able to reproduce it consistently with exact circumstances.

[halter73]

@sparksterz Are you still having this issue? If so, is your project.json targeting netcoreapp1.1 or net451?

It's known that this exception can occur for very short-lived connections though it should be rare. Also, as I mentioned above, this error should no longer be fatal.

Since you have an integration test that reproduces this issue, can you provide a repro? Feel free to open a new issue. Thanks.

[sparksterz]

It's targeting netcoreapp1.1 I also updated it to the 1.1.1 container as well as putting it behind nginx to help see if request throttling may have been the issue and still no dice. We're still unable to figure out exactly what triggers the issue. I think our real problem is related to something on the Amazon end.

Kestrel operates just fine for our tests when running in Windows and a local dynamo db. Put it in a container and run the tests locally and we get some transient Amazon errors, but no Kestrel errors. Once we put it up in Amazon load balanced behind HA Proxy, something triggers it to lose its marbles. I'll have to check if I can make an open source sample in the mean time, but if it's something that's only triggered when deployed in a real AWS environment, that could prove to be a challenge.