代码之家  ›  专栏  ›  技术社区  ›  pkaramol

kubernetes上的Jenkins奴隶吊舱随机失效

  •  0
  • pkaramol  · 技术社区  · 4 年前

    我已经设置了一个Jenkins主机(在VM上),这就是配置 jnlp 奴隶 kubernetes

    在极少数情况下,管道发生故障,并显示以下消息:

    java.io.IOException: Pipe closed
        at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:260)
        at java.io.PipedInputStream.receive(PipedInputStream.java:226)
        at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
        at java.io.OutputStream.write(OutputStream.java:75)
        at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.setupEnvironmentVariable(ContainerExecDecorator.java:510)
        at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:474)
        at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:333)
        at hudson.Launcher$ProcStarter.start(Launcher.java:455)
    

    库伯内特斯 Stackdriver 在Stackdriver中,可以看到pod 设法连接到主机,例如。

    Handshaking
    Agent discovery successful
    Trying protocol: JNLP4-Connect
    Remote Identity confirmed: <some_hash_here>
    Connecting to <jenkins-master-url>:49187
    started container
    loading plugin ...
    

    但是过了一段时间它失败了,下面是相关的日志:

    org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
    INFO: Disabled slave engine reconnects.
    hudson.remoting.jnlp.Main$CuiListener status
    Terminated
    hudson.remoting.Request$2 run
     Failed to send back a reply to the request hudson.remoting.Request$2@336ec321: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@29d0e8b2:JNLP4-connect connection to <jenkins-master-url>/<public-ip-of-jenkins-master>:49187": channel is already closed
    "Processing signal 'terminated'"
    .
    .
    .
    

    如何进一步排除此随机错误?

    0 回复  |  直到 4 年前
        1
  •  0
  •   alexzimmer96    4 年前

    你能看看Stackdriver的Kubernetes吊舱事件吗?我们在不同的CI系统(GitlabCI)中也有类似的行为。我们的构建也会随机失败。结果发现,容器中的JVM超出了其内存限制,被Kubernetes(OOMKilled)杀死,CI系统将此识别为构建错误。