long running pod not using reissued certificate #636
Description
What steps will reproduce the bug?
- running in nifi 2.6 in kubernetes mode with operator 1.5.1
- running a pod for a long time
- certificate are reissued
- pod is running with old certificate and no restart or anything is triggered
- pods are running but failed in communication ...healthchecks aren'nt triggering a pod restart
What is the expected behavior?
- better health checks regarding to certificate
- automatic restarts after cert reissue
What do you see instead?
- running zombie pods
Possible solution
- better healthcheck -> possible restart
NiFiKop version
v1.5.1
Golang version
Kubernetes version
1.3.1
NiFi version
2.6
Additional context
Logerror:
2026-01-08 15:25:42,641 WARN [Clustering Tasks Thread-3] o.apache.nifi.controller.FlowController Failed to send heartbeat org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message at org.apache.nifi.cluster.protocol.AbstractNodeProtocolSender.sendProtocolMessage(AbstractNodeProtocolSender.java:209) at org.apache.nifi.cluster.protocol.AbstractNodeProtocolSender.heartbeat(AbstractNodeProtocolSender.java:132) at org.apache.nifi.controller.cluster.ClusterProtocolHeartbeater.send(ClusterProtocolHeartbeater.java:75) at org.apache.nifi.controller.FlowController$HeartbeatSendTask.run(FlowController.java:3293) at org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Suppressed: java.net.SocketException: Connection or outbound has closed at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1297) at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:125) at java.base/java.io.BufferedOutputStream.implFlush(BufferedOutputStream.java:252) at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:240) at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:184) at org.apache.nifi.cluster.protocol.AbstractNodeProtocolSender.sendProtocolMessage(AbstractNodeProtocolSender.java:200) ... 10 common frames omitted Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1327) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1147) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:447) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:206) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:922) at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1291) at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:125) at java.base/java.io.BufferedOutputStream.implFlush(BufferedOutputStream.java:252) at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:240) at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:131) at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext.lambda$createMarshaller$0(JaxbProtocolContext.java:86) at org.apache.nifi.cluster.protocol.AbstractNodeProtocolSender.sendProtocolMessage(AbstractNodeProtocolSender.java:207) ... 10 common frames omitted Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:318) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) at java.base/sun.security.validator.Validator.validate(Validator.java:256) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:230) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) at org.apache.nifi.security.ssl.StandardX509ExtendedTrustManager.checkServerTrusted(StandardX509ExtendedTrustManager.java:61) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1311) ... 28 common frames omitted Caused by: java.security.cert.CertPathValidatorException: validity check failed at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224) at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144) at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:313) ... 34 common frames omitted Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sat Jan 03 18:47:38 CET 2026 at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:182) at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:534) at java.base/sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) at java.base/sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) at java.base/sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ... 39 common frames omitted