使用python删除txt文件中的重复行(以及一些包含特定字符串的行)的最快方法是什么?

我几乎总是用生成器来处理文件。这使得代码快速、易于修改和测试。

首先,构建一个删除重复项的生成器:

def remove_duplicates(seq):
    found = set()
    for item in seq:
        if item in found:
            continue
        found.add(item)
        yield item

>>> print "\n".join(remove_duplicates(["aa", "bb", "cc", "aa"]))
aa
bb
cc

显然是这样。接下来,创建一个函数,告诉您行是否正常:

def is_line_ok(line):
    if "bad text1" in line:
        return False
    if "bad text2" in line:
        return False
    return True

这有用吗?

>>> is_line_ok("this line contains bad text2.")
False
>>> is_line_ok("this line's ok.")
True
>>> 

所以现在我们可以用 remove_duplicates 和 itertools.ifilter 我们的职能是:

>>> seq = ["OK", "bad text2", "OK", "Also OK"]
>>> print "\n".join(remove_duplicates(ifilter(is_line_ok, seq)))
OK
Also OK

此方法适用于任何返回字符串的iterable,包括文件:

with open(input_file, 'r') as f_in:
    with open(output_file, 'w') as f_out:
       f_out.writelines(remove_duplicates(ifilter(is_line_ok, f_in)))

list(set(line for line in file.readlines()
         if 'badstring' not in line
         and 'garbage' not in line))

not in 测验。

goodLines = set()
badString = 'bad string'

with open(inFilename, 'r') as f:
    for line in f:
        if badString not in line:
            goodLines.add(line)

# and let's output these lines (sorted, unique) in another file...

with open(outFilename, 'w') as f:
    f.writelines(sorted(goodLines))