比较常用词并用占位符替换

你可以用这个 awk :

awk -v RS='[[:space:]]+' 'FNR == NR {seen[$1]; next} $1 in seen {$1 = "xxx"} {ORS=RT} 1' remove.txt corpus.txt

this is a xxx message to check if xxx words are removed correctly. The second line may or may not have xxx words. The third line also need not be as clean as first and second line.
There can be paragraphs in the text corpus and the entire file should be checked for trash.

$1 in seen {$1 = "xxx"}  # if word is from remove list then set it to xxx
{ORS=RT}                 # set output record separator as RT
1                        # print each record

有了你们展示的样品,你们能试一下吗。用GNU编写和测试 awk

awk -v RS='[[:space:]]+' '
FNR == NR{
  seen[$1]
  next
}
{
  $1=($1 in seen?"XXX":$1)
  ORS=RT
  print
}
' remove.txt corpus.txt

说明:

awk -v RS='[[:space:]]+' '     ##Starting awk program from here and setting RS as spaces here.
FNR == NR{                     ##Checking condition which will be TRUE when remove.txt is being read.
  seen[$1]                     ##Creating seen with index of 1st field.
  next                         ##next will skip all further statements from here.
}
{
  $1=($1 in seen?"XXX":$1)     ##Checking condition if $1 is in seen then set it to XXX else keep $1.
  ORS=RT                       ##Setting ORS value as RT.
  print                        ##Printing current line here.
}
' remove.txt corpus.txt        ##Mentioning Input_file names here.