Worker process exiting with code 0 leaves worker channel dangling

[jviau]

Found when investigating a CRI.

If a worker process exits with code 0, our process exit handler will just close the process and not notify anything that the process is gone. The worker channel will stay active, and functions invocations will continue to try and dispatch to the worker. All invocations to this worker will then fail as the process is gone.

We need to identify when a worker exits with code 0 and we are not shutting down so we can broadcast process exit and restart the worker.

Example exception:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.<SOME_FUNCTION>---> System.InvalidOperationException : No process is associated with this object.
   at System.Diagnostics.Process.EnsureState(State state)
   at System.Diagnostics.Process.EnsureState(State state)
   at System.Diagnostics.Process.get_Id()
   at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.AddAdditionalTraceContext(InvocationRequest invocationRequest,ScriptInvocationContext context) at /_/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 1701
   at async Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.SendInvocationRequest(ScriptInvocationContext context) at /_/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 893
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs : 103
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 82
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) at /_/src/WebJobs.Script/Description/FunctionGenerator.cs : 225
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at /_/src/Microsoft.Azure.WebJobs.Host/Executors/FunctionInvoker.cs : 53
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at /_/src/Microsoft.Azure.WebJobs.Host/Executors/FunctionExecutor.cs : 586
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at /_/src/Microsoft.Azure.WebJobs.Host/Executors/FunctionExecutor.cs : 528
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at /_/src/Microsoft.Azure.WebJobs.Host/Executors/FunctionExecutor.cs : 311
   End of inner exception
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at /_/src/Microsoft.Azure.WebJobs.Host/Executors/FunctionExecutor.cs : 357
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at /_/src/Microsoft.Azure.WebJobs.Host/Executors/FunctionExecutor.cs : 113