代码之家  ›  专栏  ›  技术社区  ›  JD Long

Hadoop流式处理最大行长度

  •  3
  • JD Long  · 技术社区  · 14 年前

    我正在为AmazonElasticMapReduce开发一个Hadoop流式工作流,它涉及到序列化一些二进制对象并将它们流到Hadoop中。Hadoop是否具有流式输入的最大行长度?

    我开始只是用越来越大的线测试,但我想我会先问这里。

    1 回复  |  直到 14 年前
        1
  •  5
  •   JD Long    14 年前

    There appears to be no imposed limits on line length. Since asking the question I have been writing code that serializes binary objects, encodes them in base64, then puts them in a stream for processing. As a result, some of the lines are quite long. Hadoop chews right along with no complaints.