c# – HttpClient异步请求未完成循环中发送的大批量

我想我已经设法做了一个可重复显示这个问题的测试,至少在我的系统上是这样.此问题涉及用于坏端点的HttpClient(非远程端点,目标已关闭).

问题是完成任务的数量不足总数,通常是几个.我不介意请求不起作用,但这只会导致应用程序在等待结果时挂在那里.

我从下面的测试代码中得到以下结果：

经历：237.2009884秒.
批处理数组中的任务：8000个已完成的任务：7993

如果我将batchsize设置为8而不是8000,则完成.对于8000,它会在WhenAll上进行阻塞.

我想知道其他人是否会得到相同的结果,如果我做错了什么,如果这似乎是一个错误.

using System; using System.Diagnostics; using System.Linq; using System.Net.Http; using System.Threading; using System.Threading.Tasks; namespace CustomArrayTesting { /// <summary> /// Problem: a large batch of async http requests is done in a loop using HttpClient, and a few of them never complete /// </summary> class ProgramTestHttpClient { static readonly int batchSize = 8000; //large batch size brings about the problem static readonly Uri Target = new Uri("http://localhost:8080/BadAddress"); static TimeSpan httpClientTimeout = TimeSpan.FromSeconds(3); // short Timeout seems to bring about the problem. /// <summary> /// Sends off a bunch of async httpRequests using a loop, and then waits for the batch of requests to finish. /// I installed asp.net web api client libraries Nuget package. /// </summary> static void Main(String[] args) { httpClient.Timeout = httpClientTimeout; stopWatch = new Stopwatch(); stopWatch.Start(); // this timer updates the screen with the number of completed tasks in the batch (See timerAction method bellow Main) TimerCallback _timerAction = timerAction; TimerCallback _resetTimer = ResetTimer; TimerCallback _timerCallback = _timerAction + _resetTimer; timer = new Timer(_timerCallback, null, TimeSpan.FromSeconds(1), Timeout.InfiniteTimeSpan); // for (int i = 0; i < batchSize; i++) { Task<HttpResponseMessage> _response = httpClient.PostAsJsonAsync<Object>(Target, new Object());//WatchRequestBody() Batch[i] = _response; } try { Task.WhenAll(Batch).Wait(); } catch (Exception ex) { } timer.Dispose(); timerAction(null); stopWatch.Stop(); Console.WriteLine("Done"); Console.ReadLine(); } static readonly TimeSpan timerRepeat = TimeSpan.FromSeconds(1); static readonly HttpClient httpClient = new HttpClient(); static Stopwatch stopWatch; static System.Threading.Timer timer; static readonly Task[] Batch = new Task[batchSize]; static void timerAction(Object state) { Console.Clear(); Console.WriteLine("Elapsed: {0} seconds.", stopWatch.Elapsed.TotalSeconds); var _tasks = from _task in Batch where _task != null select _task; int _tasksCount = _tasks.Count(); var _completedTasks = from __task in _tasks where __task.IsCompleted select __task; int _completedTasksCount = _completedTasks.Count(); Console.WriteLine("Tasks in batch array: {0} Completed Tasks : {1} ", _tasksCount, _completedTasksCount); } static void ResetTimer(Object state) { timer.Change(timerRepeat, Timeout.InfiniteTimeSpan); } } }

有时它会在完成Access Violation未处理的异常之前崩溃.调用堆栈只是说：

> mscorlib.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode = 1225, uint numBytes = 0, System.Threading.NativeOverlapped* pOVERLAP = 0x08b38b98) [Native to Managed Transition] kernel32.dll!@BaseThreadInitThunk@12() ntdll.dll!___RtlUserThreadStart@8() ntdll.dll!__RtlUserThreadStart@8()

大部分时间它不会崩溃,但只是永远不会等待时间.在任何情况下,每个请求都会抛出以下第一次机会异常：

A first chance exception of type 'System.Net.Sockets.SocketException' occurred in System.dll A first chance exception of type 'System.Net.WebException' occurred in System.dll A first chance exception of type 'System.AggregateException' occurred in mscorlib.dll A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll

我使调试器停止在Object dispos异常上,并得到了这个调用堆栈：

> System.dll!System.Net.Sockets.NetworkStream.UnsafeBeginWrite(byte[] buffer, int offset, int size, System.AsyncCallback callback, object state) + 0x136 bytes System.dll!System.Net.PooledStream.UnsafeBeginWrite(byte[] buffer, int offset, int size, System.AsyncCallback callback, object state) + 0x19 bytes System.dll!System.Net.ConnectStream.WriteHeaders(bool async = true) + 0x105 bytes System.dll!System.Net.HttpWebRequest.EndSubmitRequest() + 0x8a bytes System.dll!System.Net.HttpWebRequest.SetRequestSubmitDone(System.Net.ConnectStream submitStream) + 0x11d bytes System.dll!System.Net.Connection.CompleteConnection(bool async, System.Net.HttpWebRequest request = {System.Net.HttpWebRequest}) + 0x16c bytes System.dll!System.Net.Connection.CompleteConnectionWrapper(object request, object state) + 0x4e bytes System.dll!System.Net.PooledStream.ConnectionCallback(object owningObject, System.Exception e, System.Net.Sockets.Socket socket, System.Net.IPAddress address) + 0xf0 bytes System.dll!System.Net.ServicePoint.ConnectSocketCallback(System.IAsyncResult asyncResult) + 0xe6 bytes System.dll!System.Net.LazyAsyncResult.Complete(System.IntPtr userToken) + 0x65 bytes System.dll!System.Net.ContextAwareResult.Complete(System.IntPtr userToken) + 0x92 bytes System.dll!System.Net.LazyAsyncResult.ProtectedInvokeCallback(object result, System.IntPtr userToken) + 0xa6 bytes System.dll!System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* nativeOverlapped) + 0x98 bytes mscorlib.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* pOVERLAP) + 0x6e bytes [Native to Managed Transition]

异常消息是：

{"Cannot access a disposed object.\r\nObject name: 'System.Net.Sockets.NetworkStream'."} System.Exception {System.ObjectDisposedException}

请注意我很少看到的与未处理的访问冲突异常的关系.

因此,当目标停止时,似乎HttpClient不健壮.顺便说一下,我在Windows 7 32上这样做.

最佳答案

我使用反射器查看了HttpClient的来源.对于同步执行的操作部分(当它被启动时),就我所见,似乎没有应用于返回任务的超时.有一些超时实现在HttpWebRequest对象上调用Abort(),但他们似乎又错过了异步函数这一侧返回任务的任何超时取消或故障.回调方面可能有一些东西,但有时回调可能“失踪”,导致返回的任务永远不会完成.

我发布了一个问题,询问如何为任何任务添加超时,并且回答者提供了这个非常好的解决方案(这里作为扩展方法)：

public static Task<T> WithTimeout<T>(this Task<T> task, TimeSpan timeout) { var delay = task.ContinueWith(t => t.Result , new CancellationTokenSource(timeout).Token); return Task.WhenAny(task, delay).Unwrap(); }

因此,像这样调用HttpClient应该可以防止任何“任务变坏”永无止境：

Task<HttpResponseMessage> _response = httpClient.PostAsJsonAsync<Object>(Target, new Object()).WithTimeout<HttpResponseMessage>(httpClient.Timeout);

我认为还有一些事情要求不太可能丢失：
1.将超时从3秒增加到30秒,使得我在这个问题上发布的程序中完成了所有任务.
2.增加允许使用的并发连接数,例如System.Net.ServicePointManager.DefaultConnectionLimit = 100;

点击查看更多相关文章