我目前正在使用Cloudera上的计数器开发MapReduce map only程序。Mapper类将递增一个特定计数器,我想在MapReduce作业完成后显示每个计数器的最终值。下面是我的映射器类代码:
public class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
public static enum MY_COUNTER {
C1,
C2
}
//mapper logic that produces String variable 'final'
if (final.equals("Foo")) context.getCounter(MY_COUNTER.C1).increment(1);
else context.getCounter(MY_COUNTER.C2).increment(1);
//context.write() method
}
下面是我的驾驶员等级代码:
public class MyDriver extends Configured implements Tool {
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new MyDriver(), args);
System.exit(exitCode);
}
public int run(String[] args) throws Exception {
Job job = Job.getInstance(getConf(), "My MapReduce");
//Job configuration:
//Sets mapper to MyMapper class
//Sets num of Reduce tasks to 0
//Other necessary job config
boolean success = job.waitForCompletion(true);
if (success) {
Counter counter1 = job.getCounters().findCounter("MY_COUNTER", "C1");
System.out.println(counter1.getDisplayName() + ": " + counter1.getValue());
Counter counter2 = job.getCounters().findCounter("MY_COUNTER", "C2");
System.out.println(counter2.getDisplayName() + ": " + counter2.getValue());
return 0;
}
else return 1;
}
}
运行jar文件时,作业成功执行。因为我设置了
job.waitForCompletion()
参数为true时,它会将所有MapReduce进度打印到终端。我可以从那里看到我的计数器的值。
18/03/27 09:59:58 INFO mapreduceJob: Counters: 35
//all built-in counters
MyMapper$MY_COUNTER
C1=837
C2=119
但是,当我在作业完成后打印计数器的值时(从
if(success)
作为MyDriver类的一部分),打印的值都是零。
C1: 0
C2: 0
有没有关于我可能错在哪里的建议?
注意:我使用的是Hadoop
2.6.0-cdh5.12.0