嗨,我在排队应用程序中使用卡夫卡。我每秒向我的应用程序发送75k条记录,卡夫卡将对其进行排队。我的应用程序部署在openstack虚拟机上。由于卡夫卡在将这些记录存储到磁盘时存在一些基础设施问题,我面临与记录损坏有关的CRC问题。以下是例外情况:
org.apache.kafka.common.KafkaException: Error deserializing key/value for partition TcpMessage-3 at offset 1331363158
at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:628) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher.handleFetchResponse(Fetcher.java:566) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher.access$000(Fetcher.java:69) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher$1.onSuccess(Fetcher.java:139) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher$1.onSuccess(Fetcher.java:136) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:274) [kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) [kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) [kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) [kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:908) [kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853) [kafka-clients-0.9.0.1.jar:?]
at com.affirmed.mediation.edr.kafka.tcpMessage.TcpMessageConsumer.doWork(TcpMessageConsumer.java:196) [EdrServer.jar:?]
at com.affirmed.mediation.edr.kafka.tcpMessage.TcpMessageConsumer.run(TcpMessageConsumer.java:255) [EdrServer.jar:?]
Caused by: org.apache.kafka.common.record.InvalidRecordException: **Record is corrupt (stored crc = 2053731240, computed crc = 2767221639)**
at org.apache.kafka.common.record.Record.ensureValid(Record.java:226) ~[kafka-clients-0.9.0.1.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:617) ~[kafka-clients-0.9.0.1.jar:?]
... 15 more
那么,有没有办法使用卡夫卡只用于排队,而不将记录存储到磁盘上?如果是,我们如何实现?
