我是这个论坛的新手,但不是这个网站的新手。几个星期以来,我一直在寻找如何使用C++11快速处理大型数据文件。我正在尝试使用一个成员函数来捕获跟踪文件名,打开并处理数据。跟踪文件包含200万行数据,每行都由读/写操作和十六进制地址构成:
r abcdef123456
然而,对于一个包含这么多数据的文件,我需要快速读入并解析这两个值。我第一次尝试读取该文件是:
void getTraceData(string filename)
{
ifstream inputfile;
string file_str;
vector<string> op, addr;
// Open input file
inputfile.open(filename.c_str());
cout << "Opening file for reading: " << filename << endl;
// Determine if file opened successfully
if(inputfile.fail())
{
cout << "Text file failed to open." << endl;
cout << "Please check file name and path." << endl;
exit(1);
}
// Retrieve and store address values and operations
if(inputfile.is_open())
{
cout << "Text file opened successfully." << endl;
while(inputfile >> file_str)
{
if((file_str == "r") || (file_str == "w"))
{
op.push_back(file_str);
}
else
{
addr.push_back(file_str);
}
}
}
inputfile.close();
cout << "File closed." << endl;
}
它工作了,运行了,并读取了文件。不幸的是,程序运行和读取文件需要8分钟。我将第一个程序修改为第二个程序,以尝试更快地读取文件。它做到了,只需几秒钟就将文件读入缓冲区,而不是8分钟。使用ifstream:
void getTraceData()
{
// Setup variables
char* fbuffer;
ifstream ifs("text.txt");
long int length;
clock_t start, end;
// Start timer + get file length
start = clock();
ifs.seekg(0, ifs.end);
length = ifs.tellg();
ifs.seekg(0, ifs.beg);
// Setup buffer to read & store file data
fbuffer = new char[length];
ifs.read(fbuffer, length);
ifs.close();
end = clock();
float diff((float)end - (float)start);
float seconds = diff / CLOCKS_PER_SEC;
cout << "Run time: " << seconds << " seconds" << endl;
delete[] fbuffer;
}
但是,当我添加代码的解析部分以获取每一行,并逐行解析缓冲区内容以将两个值存储在两个单独的变量中时,程序在while循环中无声地退出,while循环包含缓冲区的getline:
void getTraceData(string filename)
{
// Setup variables
char* fbuffer;
ifstream ifs("text.txt");
long int length;
string op, addr, line;
clock_t start, end;
// Start timer + get file length
start = clock();
ifs.seekg(0, ifs.end);
length = ifs.tellg();
ifs.seekg(0, ifs.beg);
// Setup buffer to read & store file data
fbuffer = new char[length];
ifs.read(fbuffer, length);
ifs.close();
// Setup stream buffer
const int maxline = 20;
char* lbuffer;
stringstream ss;
// Parse buffer data line-by-line
while(ss.getline(lbuffer, length))
{
while(getline(ss, line))
{
ss >> op >> addr;
}
ss.ignore( strlen(lbuffer));
}
end = clock();
float diff((float)end - (float)start);
float seconds = diff / CLOCKS_PER_SEC;
cout << "Run time: " << seconds << " seconds" << endl;
delete[] fbuffer;
delete[] lbuffer;
}
我想知道,一旦我的文件被读入缓冲区,我该如何检索它并将其存储到变量中?为了增加价值,我的基准时间不到2分钟。以读取和处理数据文件。但现在,我只专注于输入文件,而不是程序的其他部分或它运行的机器(代码可移植到其他机器)。语言是C++11,操作系统是Linux计算机。很抱歉发了这么长的帖子。
