代码之家 › 专栏 › 技术社区 › goks

从hdfs到greenplum的sqoop导出不起作用

greenplum sqoop hdfs hadoop

goks · 技术社区 · 6 年前

我正在尝试将数据从HDFS位置导出到Greenplum用户定义的模式(而不是默认模式)。

已尝试sqoop eval检查连接。

sqoop eval --connect "jdbc:postgresql://sample.com:5432/sampledb" --username sample_user --password xxxx --query "SELECT * FROM sample_db.sample_table LIMIT 3"

结果: 工作正常

尝试 --schema 选项

/usr/bin/sqoop export --connect "jdbc:postgresql://sample.com:5432/sampledb" --username sampleuser --password samplepassword --table sample_table --schema sample_schema --export-dir=/sample/gp_export --input-fields-terminated-by ',' --update-mode allowinsert

结果:

Warning: /usr/hdp/2.3.6.0-3796/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/06/25 11:09:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.3.6.0-3796
18/06/25 11:09:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/06/25 11:09:41 ERROR tool.BaseSqoopTool: Error parsing arguments for export:
18/06/25 11:09:41 ERROR tool.BaseSqoopTool: Unrecognized argument: --schema
18/06/25 11:09:41 ERROR tool.BaseSqoopTool: Unrecognized argument: sample_schema
18/06/25 11:09:41 ERROR tool.BaseSqoopTool: Unrecognized argument: --export-dir=/sample/gp_export
18/06/25 11:09:41 ERROR tool.BaseSqoopTool: Unrecognized argument: --input-fields-terminated-by
18/06/25 11:09:41 ERROR tool.BaseSqoopTool: Unrecognized argument: ,
18/06/25 11:09:41 ERROR tool.BaseSqoopTool: Unrecognized argument: --update-mode
18/06/25 11:09:41 ERROR tool.BaseSqoopTool: Unrecognized argument: allowinsert

额外添加的 '--' 之前 '--schema' 基于sqoop文档 https://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html

 /usr/bin/sqoop export --connect "jdbc:postgresql://sample.com:5432/sampledb" --username sampleuser --password samplepassword --table sample_table -- --schema sample_schema --export-dir=/sample/gp_export --input-fields-terminated-by ',' --update-mode allowinsert

结果:

Warning: /usr/hdp/2.3.6.0-3796/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/06/25 11:06:26 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.3.6.0-3796
18/06/25 11:06:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
Export requires an --export-dir argument or --hcatalog-table argument.
Try --help for usage instructions.

有人能指导我吗?谢谢

2 回复 | 直到 6 年前

goks 6 年前

感谢@cricket_007的澄清。

--schema 参数应该是sqoop命令中的最后一个。所以下面的代码是有效的。

    /usr/bin/sqoop export --connect "jdbc:postgresql://sample.com:5432/sampledb"  \ 
--username sampleuser --password samplepassword  \ 
--export-dir=/sample/gp_export --input-fields-terminated-by ','  \ 
--table sample_table -- --schema sample_schema

但是 UPSERT Postgressql不支持操作。这里有一张未结的圣战联盟门票。 https://issues.apache.org/jira/browse/SQOOP-1270

roh 6 年前

之后 --export-dir 你不需要 = 查看下面的示例。另一个建议是使用 --verbose 当你遇到这些问题时。

sqoop export --libjars /path/some.jar \
--connect 'jdbc:sqlserver://IP:1433;database=db' \
--username someName -password somePassword -m 10 \
--verbose --mysql-delimiters \
--export-dir /HDFS/Path/someFile.csv \
--table "RDBMSTABLENAME"