代码之家  ›  专栏  ›  技术社区  ›  Nitishkumar Singh user3744342

Google云存储同时上传多个大文件时引发错误

  •  5
  • Nitishkumar Singh user3744342  · 技术社区  · 6 年前

    我们正在尝试将多个文件(即100个或更多)上传到谷歌云存储,但我们得到 随机低于错误。
    平均文件大小:30MB-40 MB

     08:57:15.488 [ForkJoinPool-1-worker-26] INFO  
     demo.storage.GCPStorageComponent - Error writing request body to
     server com.google.cloud.storage.StorageException: Error writing
     request body to server
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
         at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:581)
         at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:51)
         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
         at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
    at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:47)
         at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:160)
         at demo.storage.GCPStorageComponent.uploadFile(GCPStorageComponent.java:56)
         at demo.storage.StorageRepository.uploadFile(StorageRepository.java:56)
         at demo.restful.UploadRepository.uploadFileToBucket(UploadRepository.java:132)
         at demo.restful.UploadRepository.processClipResults(UploadRepository.java:56)
         at demo.restful.UploadRepository.lambda$getClipModelCallableList$0(UploadRepository.java:85)
         at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
         at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
     Caused by: java.io.IOException: Error writing request body to server
         at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3536)
         at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3519)
         at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:55)
         at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
         at com.google.api.client.http.AbstractInputStreamContent.writeTo(AbstractInputStreamContent.java:72)
         at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:80)
         at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
         at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:563)
         ... 17 more
    

    下面是我们用来上传文件的代码。 我们正在使用多部分方法上载1 MB的文件。

    public void uploadFile(String bucketName, String bucketFileName, Path localFilePath)
            throws IOException, StorageException {
        BlobInfo blobInfo = BlobInfo.newBuilder(bucketName, bucketFileName).build();
        ByteBuffer buffer = ByteBuffer.allocate(10_000_000); //i.e. 1 MB
        try (WriteChannel writer = storage.writer(blobInfo);
                ByteChannel inChannel = Files.newByteChannel(localFilePath);) { // write to bucket
            while (inChannel.read(buffer) > 0) {
                buffer.flip();
                writer.write(buffer);
                buffer.clear();
            }
        } catch (com.google.cloud.storage.StorageException ex) {
            LOGGER.info(ex.getMessage());
            ex.printStackTrace();
        }
    }
    

    编辑: 正在从谷歌云内部进行访问
    Github Issue 已为相同的打开

    1 回复  |  直到 6 年前