代码之家 › 专栏 › 技术社区 › Fravadona

在两个文件中查找不区分大小写和重音的相似行

grep awk bash

Fravadona · 技术社区 · 3 年前

我有两个 未引用和单列 TSV文件(从数据库导出)有几千个人的名字,我需要找到两个文件中出现的名字。两个文件都是 UTF-8 , CRLF 终止,并从BOM表开始 0xEF 0xBB 0xBF .

一个简单的 join 或 comm 命令本可以做到这一点,但名称上有一些不同:

# cat file1.tsv
A.  Einstein
Louis Pasteur 
Diego Armando Maradona
Isaac Newton
 Frava DâonÃ¤
D RÃ¼ge
FranÃ§oise BarrÃ©-Sinoussi

# cat file2.tsv
Diego Maradona
Albert Einstein
Francoise, BARRE  SINOUSSI
Louis Pasteur
frava d'ona
Marie-Louise Von FRANZ
Dimitri RuÌge

中的预期匹配 file2.tsv 将是:

Diego Maradona
Albert Einstein
Francoise, BARRE  SINOUSSI
Louis Pasteur
frava d'ona
Dimitri RuÌge

这是我写的 bash sed awk grep 动态生成用于匹配姓氏的正则表达式的脚本:

#!/bin/bash

# U+0300 = 0xCC80 = 52352
# U+033F = 0xCCBF = 52415
# U+0340 = 0xCD80 = 52608
# U+036E = 0xCDAE = 52654

_COMBINING_CHARS_=()

for i in {52352..52415} {52608..52654}
do
    hex=$(printf %04X "$i")
    _COMBINING_CHARS_+=( "$(printf '\x'"${hex:0:2}"'\x'"${hex:2:2}")" )
done

_COMBINING_CHARS_ERE_=$(IFS='|'; printf %s "${_COMBINING_CHARS_[*]}")

# Function that removes the BOM, CRLF, and COMBINING characters:
sanitize() {
    LANG=C sed -E \
        -e $'1s/^\xEF\xBB\xBF//' \
        -e $'s/\r$//' \
        -e "s/$_COMBINING_CHARS_ERE_//g" \
    -- "$@"
}

# Function that generates a regex for the _lastname_:
toERE() {
    awk '
        {
            if ( $0 ~ /,/) {
                n = split($0, a, ",");
                $0 = a[n];
            } else {
                $0 = $NF
            }
            sub("^[[:space]]+","");
            sub("[[:space]]+$","");
            gsub("[[:space:]-]+"," ");
        }

        {
            ere = ""
            sep = "";
            for ( nf = 1; nf <= NF; nf++ ) {
                n = split($nf, c, "");
                for ( i = 1; i <= n; i++ ) {
                    ere = ere "[[=" c[i] "=]]"
                }
                ere = sep ere 
                sep = "[[:space:]-]+"
            }
            print ere "[[:space:]]*$"
        }
    ' < <(sanitize "$@")
}

grep -E -f <(toERE "$1") <(sanitize "$2")

不幸的是,给定输入的结果是:

grep: illegal byte sequence

UTF-8多字节字符似乎是个问题,但我想不出一种方法来处理它 awk

0 回复 | 直到 3 年前

James Brown 3 年前

怎么样 agrep ? man agrep : agrep-在文件中搜索字符串或正则表达式,具有近似匹配功能 。这并不像我们将看到的那样完美:

$ while IFS= read -r line
do 
    echo -n "$line: "
    agrep -B -y  "$line" file1
done < file2

输出

Diego A. Maradona: agrep: 1 word matches within 6 errors
Maradona, Diego Armando
Albert Einstein: agrep: 1 word matches within 5 errors
A. Einstein
Louis Pasteur: Louis Pasteur
frava dona: agrep: 2 words match within 4 errors
Maradona, Diego Armando
FrÃ¤va Dona

很好的例子,因为我们已经在最后三行看到了一个问题。

Dudi Boy 3 年前

建议使用以下技巧:

 cat file1.csv file1.csv | sort | uniq -d

解释

cat file1.csv file1.csv 一个接一个地组合bot文件

sort 把相似的线条放在一起

uniq -d 仅打印有重复项的行

Fravadona 3 年前

^{最后我完成了这项工作

ruby

但我发布了一个

awk

然而,解决方案。}

有两个问题:

POSIX [= =] 不适用于变音符号字符
awk 不知道多字节字符

通过将输入文件转换为ASCII,可以解决这一问题。 iconv 可以在一定程度上准确地处理罗马尼亚字符,这正是我所需要的:

#!/bin/bash

to_ascii() {
    case $(uname) in
    Darwin)
        iconv -f UTF-8 -t UTF-8-MAC "$@" |
        iconv -f UTF-8 -t ASCII//TRANSLIT//IGNORE
        ;;
    Linux)
        iconv -f UTF-8 -t ASCII//TRANSLIT//IGNORE "$@"
        ;;
    esac |
    sed $'s/\r$//'
}

现在我们只需要做一点规范化处理,在姓氏中找到完美的匹配:

awk '
    {
        gsub("-+","-");
        gsub("\27+","\27");
        gsub("[.[:space:]]+"," ");
        sub("^[[:space:]]+","");
        sub("[[:space:]]+$","");
    }
    {
        if ($0 ~ /,/) {
            n = split($0,a,"[[:space:]]*,[[:space:]]*");
            lastname = a[n];
        } else {
            lastname = $NF;
        }
        gsub("[-[:space:]]+"," ",lastname);
        lastname = tolower(lastname);
    }
    FNR == NR {
        keys[lastname] = $0;
        next;
    }
    {
        count = 0;
        for (n in keys) {
            if (n == lastname) {
                matches[++count] = keys[n]
            }
        }
        if (count > 0) {
            print $0
            for (i = 1; i <= count; i++) {
                print "\t" matches[i]
            }
        }
    }
' <(to_ascii "$2") <(to_ascii "$1")

输出

A Einstein
    Albert Einstein
Louis Pasteur
    Louis Pasteur
Diego Armando Maradona
    Diego Maradona
Frava D'ona
    frava d'ona
D Ruge
    Dimitri Ruge
Francoise Barre-Sinoussi
    Francoise, BARRE SINOUSSI