Question

把很多TXT文件内容写入MYSQL数据库，有一亿条数据，如何提高写入效率？求高手赐教修改如下代码

Private Function F_WritetoDatabase(ByRef myArr As ArrayList) As Boolean
Dim connection As MySqlConnection
Dim connectionString As String
Dim strInsert As String
Dim hasstr As String
Dim myhash As Hashtable = New Hashtable()
' Dim myIndex As Integer
Dim remyArr As String
myhash.Clear()
For i = 0 To myArr(0).ToString.Length - 1
remyArr = myArr(0)(i).ToString
' If (Asc(remyArr) < 0) Or ((65 <= Asc(remyArr)) And (Asc(remyArr) <= 90)) Or ((97 <= Asc(remyArr)) And (Asc(remyArr) <= 122)) Or (Asc(remyArr) = 42) Then
'remyArr = remyArr.Replace(remyArr, """" & remyArr & """")
myArr(0)(i) = """" & remyArr & """"
' myIndex = myArr(i).ToString().Trim().IndexOf(",")

' End If
myhash.Add(i, myArr(0)(i).ToString())
Next

connectionString = "Database=schema;Data Source=localhost;User Id=root;Password=1234;"
connection = New MySqlConnection(connectionString)

connection.Open()

If (connection.State = ConnectionState.Open) Then
' MsgBox("稻香")

End If

hasstr = myhash(0).ToString().Trim() & "," & myhash(1).ToString().Trim() & "," & myhash(2).ToString().Trim() & "," & myhash(3).ToString().Trim() & "," & myhash(4).ToString().Trim() &
"," & myhash(5).ToString().Trim() & "," & myhash(6).ToString().Trim() & "," & myhash(7).ToString().Trim() & "," & myhash(8).ToString().Trim() & "," & myhash(9).ToString().Trim() &
"," & myhash(10).ToString().Trim() & "," & myhash(11).ToString().Trim() & "," & myhash(12).ToString().Trim() & "," & myhash(13).ToString().Trim() & "," & myhash(14).ToString().Trim()

strInsert = "insert into share(time,stockcode,name,closingprice,highestprice,lowestprice,openingprice," &
"beforeclosingprice,riseandfalllines,riseandfallamplitude,turnoverrate,volume,turnover,totalmarketvalue,currentmarket) " &
"values(" & hasstr & ")"
Dim cmdInsert As MySqlCommand = New MySqlCommand(strInsert, connection)

Try
cmdInsert.ExecuteNonQuery() '执行具体的插入查询动作
Catch ex As ApplicationException
ex.Message.ToString()
connection.Close()
End Try

connection.Close()

' MsgBox("全部完成，共写入数据库" & myhash.Count - 1 & "条！")

Return True

End Function

Answer 1

非root用户运行MySQL，当MySQL配置比较高时，MySQL运行中生效的参数值与配置的值不一样,所以具体分析一下MySQL是怎么调整这些参数值的。 这篇文章的目的是为了说明在系统资源不够的情况下，MySQL 是怎么调整者三个参数的。说明此文涉及到三个参数open_files_limit、 max_connections、 table_open_cache。与这三个参数相关的系统资源是打开文件数限制，即文件描述符（fd）限制。系统参数与文件描述符的关系 - max_connection & fd : 每一个MySQL connection      都需要一个文件描述符；- table_open_cache & fd 打开一张表至少需要一个      文件描述符，如打开MyISAM需要两个fd ；- 系统最大打开文件数可以通过 ulimit -n查看。MySQL调整参数的方式

• 根据配置（三个参数的配置值或默认值）计算 request_open_files（需要的文件描述符）；

• 2.获取有效的系统的限制值effective_open_files；  3.根据effective_open_files调整request_open_files；  4.根据调整后的request_open_files,计算实际生效的参数值（show variables 可查看参数值）。计算request_open_filesrequest_open_files有三个计算公式：1.      // 最大连接数+同时打开的表的最大数量+其他（各种日志等等）2.     limit_1= max_connections+table_cache_size * 2 + 10;3.   4.      //假设平均每个连接打开的表的数量（2-4）5.      //源码中是这么写的：6.      //We are trying to allocate no less than 7.      // max_connections*5 file handles8.      limit_2= max_connections * 5;9.   10.    //mysql 默认的默认是500011.    limit_3= open_files_limit ? open_files_limit : 5000;12.  13.     所以open_files_limit期待的最低14.     request_open_files= max(limit_1,limit_2,limit_3);计算effective_open_files:MySQL 的思路： 




• 在有限值的的范围内MySQL 尽量将effective_open_files的值设大。

• 修正request_open_files




requested_open_files= min(effective_open_files, request_open_files)

重新计算参数值

• 修正open_files_limit




• open_files_limit = effective_open_files




• 修正max_connections




• max_connections 根据 request_open_files 来做修正。1.  limit = requested_open_files - 10 - TABLE_OPEN_CACHE_MIN * 2;

• 如果配置的max_connections值大于limit，则将max_connections 的值修正为limit

• 其他情况下 max_connections 保留配置值 




• 修正table_cache_size




• table_cache_size 会根据 request_open_files 来做修正1.   // mysql table_cache_size 最小值，4002.   limit1 = TABLE_OPEN_CACHE_MIN3.   // 根据 requested_open_files 计算4.   limit2 = (requested_open_files - 10 - max_connections) / 25.   limit = max(limit1,limt2);

• 如果配置的table_cache_size 值大于limit，则将 table_cache_size 的值修正为limit

• 其他情况下table_cache_size 保留配置值




• 举例




• 以下用例在非 root 用户下运行

参数设置：




• //mysql

max_connections = 500

• table_open_cache = 999

//ulimit -n 

1500

生效的值：




• open_files_limit = 1500   max_connections = min[(1500 - 10 - 800),500] = 500

table_open_cache = ( 1500 - 10 - 500) / 2 =495

Answer 2

这是vb.net么？应该使用参数话查询，替代拼接sql，for循环外部创建MySqlCommand，for循环内部赋值执行。另外可以再加上事务一般更快