Pod Connection Failed - No route to host

Owned by Jeyanthi (Unlicensed)
Last updated: Feb 22, 2019

Problem

Failed to connect to pod ip (Example:10.44.0.2) port 8080 from Kubernetes master. Error: No route to host. In BuildOn, container is running on port 8080. Following URL will be used to fetch the running jenkins job status inside container curl http://10.44.0.2:8080/job/hcabca1/1/wfapi/describe

Cause

Firewall block in Slave machines or weave network issue.

Fix

  1. To know the firewall block, kernel panic messages can be read using  "dmesg"  command. Try telnet between master/slaves for the pod ip and port 8080. In order to connect pod ip in port 8080 from kubernetes master, ensure that the slave machines firewall has port 8080 ALLOW. "ufw status" will tell the status of firewall. If the firewall is active then rules will be listed.
  2. Check weave net pods logs and ensure weave cni network is working properly. As it is occurring initial setup time itself, try to remove the file /var/lib/weave/weave-netdata.db  and restart weave (weave reset or terminate the weave pods and it will be spawned back). Reference:  https://www.weave.works/docs/net/latest/tasks/ipam/troubleshooting-ipam/  Section: "Seeded by Different Peers"