代码之家 › 专栏 › 技术社区 › Yahia

在perl中填充哈希的最快方法

populate hash perl performance

Yahia · 技术社区 · 7 年前

我试图在perl中从大约564k行的文件中填充一个哈希,代码需要1.6到2.1秒才能执行,而C#中的等效代码需要大约0.8秒才能完成。在Perl中有更好的方法吗?

到目前为止,我已经尝试过:

# 1 - this version take ~ +1.6 seconds to fill the hash from file with ~ 564000
my %voc;
open(F,"<$voc_file");
while(defined(my $line=<F>)) {
    chomp($line);
    $voc{$line} = 1;
}
close(F);

还有这个

# 2 - this version take ~ +2.1 seconds to fill the hash from file with ~ 564000
my %voc;
my @voc_keys;
my @array_of_ones;
open(F,"<$voc_file");
my $voc_keys_index = 0;
while(defined(my $line=<F>)) {
    chomp($line);
    $voc_keys[$voc_keys_index] = $line;
    $array_of_ones[$voc_keys_index] = 1;
    $voc_keys_index ++;
}
@voc{@voc_keys} = @array_of_ones;
close(F);

在c中,我使用:

var voc = new Dictionary<String, int>();
foreach (string line in File.ReadLines(pathToVoc_file))
{
    var trimmedline = line.TrimEnd(new char[] { '\n' });
    voc[trimmedline] = 1;
}

只需要700-800毫秒

2 回复 | 直到 7 年前

ysth 7 年前

绝对避免将1存储为数据并使用exists可以节省时间和内存。通过从循环中删除块,可以弥补更多的不足:

my %voc;
open(F,"<$voc_file");
chomp, undef $voc{$_} while <F>;
close(F);

基准测试结果(使用20个字符行):

Benchmark: running ikegami, original, statementmodifier, statementmodifier_undef for at least 10 CPU seconds...
   ikegami: 10 wallclock secs ( 9.54 usr +  0.46 sys = 10.00 CPU) @  2.10/s (n=21)
  original: 10 wallclock secs ( 9.62 usr +  0.45 sys = 10.07 CPU) @  2.09/s (n=21)
statementmodifier: 10 wallclock secs ( 9.61 usr +  0.48 sys = 10.09 CPU) @  2.18/s (n=22)
statementmodifier_undef: 11 wallclock secs ( 9.85 usr +  0.48 sys = 10.33 CPU) @  2.23/s (n=23)

基准:

use strict;
use warnings;
use Benchmark 'timethese';

my $voc_file = 'rand.txt';

sub check {
    my ($voc) = @_;
    unless (keys %$voc == 564000) {
        warn "bad number of keys ", scalar keys %$voc;
    }
    chomp(my $expected_line = `head -1 $voc_file`);
    unless (exists $voc->{$expected_line}) {
        warn "bad data";
    }
    return;
}

timethese(-10, {
    'statementmodifier' => sub {
        my %voc;
        open(F,"<$voc_file");
        chomp, $voc{$_} = 1 while <F>;
        close(F);
        #check(\%voc);
        return;
    },
    'statementmodifier_undef' => sub {
        my %voc;
        open(F,"<$voc_file");
        chomp, undef $voc{$_} while <F>;
        close(F);
        #check(\%voc);
        return;
    },
    'original' => sub {
        my %voc;
        open(F,"<$voc_file");
        while(defined(my $line=<F>)) {
            chomp($line);
            $voc{$line} = 1;
        }
        close(F);
        #check(\%voc);
        return;
    },
    'ikegami' => sub {
        my %voc;
        open(F,"<$voc_file");
        while(defined(my $line=<F>)) {
            chomp($line);
            undef $voc{$line};
        }
        close(F);
        #check(\%voc);
        return;
    },
});

(原始错误答案替换为此。)

ikegami Gilles Quénot 7 年前

当然C会更快。

您可以通过替换

$voc{$line} = 1;  ...  if ($voc{$key}) { ... }  ...

具有

undef $voc{$line};  ...  if (exists($voc{$key})) { ... }  ...