我在排除前缀为“load*”的文件从AmazonS3同步到Google云存储时遇到了一个问题,同时我也希望将它们从数据完整性检查中排除。
我尝试过gsutils rsync帮助中提到的-x方法,但是文件没有被排除,它仍在检查我想要排除的文件。
用例:
-
有几个大文件名为load*.csv(gb),它不会改变和
我希望他们被排除在外。
-
有一些小文件(KB)将添加到存储桶中
分钟,我想从S3同步到谷歌云存储。
-
我只希望作业同步不在
目标存储桶。
我有什么办法可以做到这一点吗?
命令:
gsutil -m rsync -Cnr -x "dirX/dirY/.*/LOAD*$" s3://bucket-A gs://bucket-B
输出:
Building synchronization state...
At destination listing 10000...
At destination listing 20000...
At destination listing 30000...
At destination listing 40000...
At destination listing 50000...
At destination listing 60000...
At destination listing 70000...
At destination listing 80000...
Non-MD5 etag ("1f46bed70ce51ad32c271f423463c1f8-2") present for key <Key: bucket-A,dirX/dirY/dir1/LOAD00000001.csv>, data integrity checks are not possible.
At destination listing 90000...
At destination listing 100000...
At destination listing 110000...
At destination listing 120000...
At destination listing 130000...
At destination listing 140000...
At destination listing 150000...
At destination listing 160000...
At destination listing 170000...
Non-MD5 etag ("46a7c3d9e894d6e6a6437a97b49815a5-31") present for key <Key: faasos-etl,bucket-A,dirX/dirY/dir2/LOAD00000001.csv>, data integrity checks are not possible.
目录结构如下:
bucketA/dirX/dirY/dir1/LOAD000001.csv
bucketA/dirX/dirY/dir1/LOAD000002.csv
bucketA/dirX/dirY/dir2/LOAD000001.csv
bucketA/dirX/dirY/dir2/LOAD000002.csv
bucketA/dirX/dirY/dirN/LOAD000001.csv
