1.　http: request canceled (Client.Timeout exceeded while awaiting headers)

1.1 代码层

看到此错误，多是因为在写DB时没有使用Task

可参考如下写法： Logic package

public async Task<CsvInfoResult> SaveCsv(string list){
	var tasks = new List<Task>();
    ......
    await _csvService.SaveItemList(csvResult, _mapper, items);
    ......
}

Service package

public Task SaveItemList(CsvInfoResult csvResult, IMapper mapper, List<OperItem> items)
{
    var tasks = new List<Task>();

    var taskdb = Task.Run(() =>
    {
        for (int i = 0; i < itemList.Count; i++)
        {
            opItem = itemList[i];
            ItemResult result = csvResult.Resultlist[i];
            if (result.Status != HttpStatus.SUCCESSFULCODE) continue;
            if (_abcDbContext.Database.CanConnectAsync().Result)
            {
                try
                {
                    _abcDbContext.Database.BeginTransaction();
                    SaveItem(mapper, opItem, result);
                    _abcDbContext.Database.CommitTransaction();
                }
                catch (Exception ex)
                {
                    csvResult.Status = HttpStatus.DB_DATA_UPDATE_FAILURE;
                    result.Status = HttpStatus.DB_DATA_UPDATE_FAILURE;
                    result.Msg = ex.Message;
                    _abcDbContext.Database.RollbackTransaction();
                    _logger.LogError(ex.StackTrace);
                }
            }
        }
     });
     tasks.Add(taskdb);
     return Task.WhenAll(tasks);
}

1.2 系统层

Windows

On Windows OS, Running Linux Containers with WSL2, i followed the following steps,

1. Ran the command docker logout
1. Ran the command, docker network prune, so as to remove all the preconfigured settings of the network.
1. From Docker Settings, Enabled the DNS server configuration with 8.8.8.8
1. Restarted the Docker Now executed login command with registry to login, docker login {registry}

Ubuntu

You can easily test this out by using the following steps (in Ubuntu)

1. Select the IPv4 Settings tab.
1. Disable the “Automatic” toggle switch and enter the DNS resolvers' IP addresses, separated by a comma. We’ll use the Google DNS nameservers:

8.8.8.8,8.8.4.4

If it works, then you may be able to reset the DNS to "Automatic"

1.3 架构层

受制于开发语言或框架的限制，单次请求处理的数据量肯定能达到其上限值。如何找到一种办法来彻底解决此问题。

受学友的启发，有两种方案可供参考：

1. 重I/O型的MQ
1. 重内存的Redis

1.4 测试单次请求处理的最大数据量

以下测试基准：HTTP timeout=10s

502条--OK
1001条--timeover
901条--timeover
801条--timeover
701条--timeover
651条--timeover
641条--timeover
631条--timeover
621条--timeover
611条--timeover
609条--timeover
608条--OK(timeover/OK)
606条--OK(timeover/OK)
601条--OK(timeover/OK)

可以看出：600~608--不稳定 609+--timeover ~600 OK 600+的情况要追加 HTTP timeout的时间了

2. This MySqlConnection is already in use

参考：www.cnblogs.com/ricolee/p/T… 记得所有使用DBContext检索的地方加上 ToList()

3. A second operation started on this context before a previous asynchronous operation completed

参考：www.cnblogs.com/xishuai/p/e…

因为EF DBContext并不是线程安全的组件。

Startup.cs

        public void ConfigureServices(IServiceCollection services)
        {
        ...
                    services.AddDbContextPool<AbcDbContext>(options => options.UseMySql(DBConnectionString.ConnectionString, ServerVersion.AutoDetect(DBConnectionString.ConnectionString))
                                                                .UseLoggerFactory(LoggerFactory.Create(builder =>
                                                                {
                                                                    builder.AddLog4Net("Config/log4net.config").AddConsole().AddDebug();
                                                                })));
        ...
        }

这里虽然能够保证一个请求生成一个AbcDbContext,但在写mysql时并不能使用多线程。

出错的代码如下：

            var tasks = new List<Task>();
            for (int i = 0; i < itemList.Count; i++)
            {
            	opItem = itemList[i];
                ItemResult result = csvResult.Resultlist[i];
                if (result.Status != HttpStatus.SUCCESSFULCODE) continue;
            	var taskdb = Task.Run(() =>
            	{
                    if (_abcDbContext.Database.CanConnectAsync().Result)
                    {
                        try
                        {
                            _abcDbContext.Database.BeginTransaction();

                            SaveItem(mapper, opItem, result);

                            _abcDbContext.Database.CommitTransaction();
                        }
                        catch (Exception ex)
                        {
                            csvResult.Status = HttpStatus.DB_DATA_UPDATE_FAILURE;
                            result.Status = HttpStatus.DB_DATA_UPDATE_FAILURE;
                            result.Msg = ex.Message;

                            _abcDbContext.Database.RollbackTransaction();

                            _logger.LogError(ex.StackTrace);
                        }
                    }
            	});
                tasks.Add(taskdb);
             }

            return Task.WhenAll(tasks);

修改后的代码如下：

            var tasks = new List<Task>();

            var taskdb = Task.Run(() =>
            {
                for (int i = 0; i < itemList.Count; i++)
                {
                    opItem = itemList[i];
                    ItemResult result = csvResult.Resultlist[i];
                    if (result.Status != HttpStatus.SUCCESSFULCODE) continue;
                    if (_abcDbContext.Database.CanConnectAsync().Result)
                    {
                        try
                        {
                            _abcDbContext.Database.BeginTransaction();
                            SaveItem(mapper, opItem, result);
                            _abcDbContext.Database.CommitTransaction();
                        }
                        catch (Exception ex)
                        {
                            csvResult.Status = HttpStatus.DB_DATA_UPDATE_FAILURE;
                            result.Status = HttpStatus.DB_DATA_UPDATE_FAILURE;
                            result.Msg = ex.Message;
                            _abcDbContext.Database.RollbackTransaction();
                            _logger.LogError(ex.StackTrace);
                        }
                    }
                }
            });
            tasks.Add(taskdb);
            return Task.WhenAll(tasks);

4 xUnitf测试时，每次执行结果都不一样

原因：因为有一些共享资源，例如DB、redis等，导致互相影响。解决方案：让其执行有序

4.1 测试类有序

注解程序如下：

using System.Collections.Generic;
using System.Linq;
using Xunit;
using Xunit.Abstractions;

namespace XUnit.Project.Orderers
{
    public class DisplayNameOrderer : ITestCollectionOrderer
    {
        public IEnumerable<ITestCollection> OrderTestCollections(
            IEnumerable<ITestCollection> testCollections) =>
            testCollections.OrderBy(collection => collection.DisplayName);
    }
}

样例程序：注意点--如果是以类文件呈现的，assembly部分只在第一个类文件标注就可。

using Xunit;

// Need to turn off test parallelization so we can validate the run order
[assembly: CollectionBehavior(DisableTestParallelization = true)]
[assembly: TestCollectionOrderer("XUnit.Project.Orderers.DisplayNameOrderer", "XUnit.Project")]

namespace XUnit.Project
{
    [Collection("A Test Collection")]
    public class TestsInCollection1
    {
        public static bool Collection1Run;

        [Fact]
        public static void Test()
        {
            Assert.True(TestsInCollection2.Collection2Run);     // Abc
            Assert.True(TestsInCollection3.Collection3Run);     // Mno
            Assert.False(TestsInCollection1.Collection1Run);    // Xyz

            Collection1Run = true;
        }
    }

    [Collection("B Test Collection")]
    public class TestsInCollection2
    {
        public static bool Collection2Run;

        [Fact]
        public static void Test()
        {
            Assert.False(TestsInCollection2.Collection2Run);    // Abc
            Assert.False(TestsInCollection3.Collection3Run);    // Mno
            Assert.False(TestsInCollection1.Collection1Run);    // Xyz

            Collection2Run = true;
        }
    }

    [Collection("C Test Collection")]
    public class TestsInCollection3
    {
        public static bool Collection3Run;

        [Fact]
        public static void Test()
        {
            Assert.True(TestsInCollection2.Collection2Run);     // Abc
            Assert.False(TestsInCollection3.Collection3Run);    // Mno
            Assert.False(TestsInCollection1.Collection1Run);    // Xyz

            Collection3Run = true;
        }
    }
}

4.2 类的测试用例有序

注解程序如下：

接口TestPriorityAttribute.cs

using System;

namespace XUnit.Project.Attributes
{
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
    public class TestPriorityAttribute : Attribute
    {
        public int Priority { get; private set; }

        public TestPriorityAttribute(int priority) => Priority = priority;
    }
}

实装PriorityOrderer（优先级）

using System.Collections.Generic;
using System.Linq;
using Xunit.Abstractions;
using Xunit.Sdk;
using XUnit.Project.Attributes;

namespace XUnit.Project.Orderers
{
    public class PriorityOrderer : ITestCaseOrderer
    {
        public IEnumerable<TTestCase> OrderTestCases<TTestCase>(
            IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
        {
            string assemblyName = typeof(TestPriorityAttribute).AssemblyQualifiedName!;
            var sortedMethods = new SortedDictionary<int, List<TTestCase>>();
            foreach (TTestCase testCase in testCases)
            {
                int priority = testCase.TestMethod.Method
                    .GetCustomAttributes(assemblyName)
                    .FirstOrDefault()
                    ?.GetNamedArgument<int>(nameof(TestPriorityAttribute.Priority)) ?? 0;

                GetOrCreate(sortedMethods, priority).Add(testCase);
            }

            foreach (TTestCase testCase in
                sortedMethods.Keys.SelectMany(
                    priority => sortedMethods[priority].OrderBy(
                        testCase => testCase.TestMethod.Method.Name)))
            {
                yield return testCase;
            }
        }

        private static TValue GetOrCreate<TKey, TValue>(
            IDictionary<TKey, TValue> dictionary, TKey key)
            where TKey : struct
            where TValue : new() =>
            dictionary.TryGetValue(key, out TValue result)
                ? result
                : (dictionary[key] = new TValue());
    }
}

样例程序：

using Xunit;
using XUnit.Project.Attributes;

namespace XUnit.Project
{
    [TestCaseOrderer("XUnit.Project.Orderers.PriorityOrderer", "XUnit.Project")]
    public class ByPriorityOrder
    {
        public static bool Test1Called;
        public static bool Test2ACalled;
        public static bool Test2BCalled;
        public static bool Test3Called;

        [Fact, TestPriority(5)]
        public void Test3()
        {
            Test3Called = true;

            Assert.True(Test1Called);
            Assert.True(Test2ACalled);
            Assert.True(Test2BCalled);
        }

        [Fact, TestPriority(0)]
        public void Test2B()
        {
            Test2BCalled = true;

            Assert.True(Test1Called);
            Assert.True(Test2ACalled);
            Assert.False(Test3Called);
        }

        [Fact]
        public void Test2A()
        {
            Test2ACalled = true;

            Assert.True(Test1Called);
            Assert.False(Test2BCalled);
            Assert.False(Test3Called);
        }

        [Fact, TestPriority(-5)]
        public void Test1()
        {
            Test1Called = true;

            Assert.False(Test2ACalled);
            Assert.False(Test2BCalled);
            Assert.False(Test3Called);
        }
    }
}

5. Mysql 提升写库性能

5.1 Mysql8.0版本，如果出现下面错误。

{"Loading local data is disabled; this must be enabled on both the client and server sides"}

解决方案：查看：show variables like 'local_infile';结果：Off 设置：set global local_infile=on;注意：使用root权限

5.2 减少输入参数的分拣

解决方案：MySqlBulkLoader使用、然后调用存储过程

            Encoding utf8WithoutBom = new UTF8Encoding(false);

            File.WriteAllText(tmpPath, csv, utf8WithoutBom);

            using (MySqlConnection conn = new MySqlConnection(DBConnectionString.ConnectionString))
            {

                MySqlTransaction tran = null;
                try
                {
                    Stopwatch stopwatch = new Stopwatch();
                    stopwatch.Start();

                    conn.Open();

                    string query = "TRUNCATE TABLE " + tableName;
                    MySqlCommand cmd = new MySqlCommand(query, conn);
                    cmd.ExecuteNonQuery();

                    tran = conn.BeginTransaction();

                    MySqlBulkLoader bulk = new MySqlBulkLoader(conn)
                    {
                        FieldTerminator = "\t",
                        FieldQuotationCharacter = '"',
                        EscapeCharacter = '"',
                        FileName = tmpPath,
                        NumberOfLinesToSkip = 1,
                        TableName = tableName,
                        CharacterSet = "UTF8",
                    };
                    
                    bulk.Load();
                    tran.Commit();
                    stopwatch.Stop();

                    File.Delete(tmpPath);

                    _logger.LogInformation("MySqlBulkLoaderで使用時間:{0}ミリ秒", stopwatch.ElapsedMilliseconds);
                }
                catch (MySqlException)
                {
                    if (tran != null) tran.Rollback();

                    _logger.LogError(Resource.ErrorLocalinfile);

                    throw;
                }
            }

C# EntityFramework DbContext(MySQL) 踩坑记录

1. http: request canceled (Client.Timeout exceeded while awaiting headers)