DNS Resolution Failure for Internal Virtual Machines

Overview

You are trying to resolve DNS for an internal Virtual Machine (VM) and observe the following error. At the same time, the DNS resolution for external domains should be working.

unable to resolve host np-e9dc030e-1.us-east1-a.compute.internal: Temporary failure in name resolution

Prerequisites

• SSH access to VM experiencing DNS failures

Cause

The SDN (Software Defined Network) fabric layer on which the Virtual Machines (VMs) run is responsible for intercepting DNS queries exiting the VMs. The service responsible for this only intercepts UDP packets. systemd-resolved always tries UDP for DNS lookup. If there are multiple DNS failures, systemd-resolved it switches to using TCP and that results in DNS failures.

Steps

1. Step 1: Identify that only internal DNS resolution is failing

   • To validate this, run the following commands
     

     # ubuntu@np-e9dc030e-1:~$ nslookup google.com
     Server: 127.0.0.53
     Address: 127.0.0.53#53
     
     Non-authoritative answer:
     Name: google.com
     Address: 172.253.115.138
     Name: google.com
     Address: 172.253.115.139
     Name: google.com
     Address: 172.253.115.113
     Name: google.com
     Address: 172.253.115.102
     Name: google.com
     Address: 172.253.115.100
     Name: google.com
     Address: 172.253.115.101
     Name: google.com
     Address: 2607:f8b0:4004:c06::8b
     Name: google.com
     Address: 2607:f8b0:4004:c06::71
     Name: google.com
     Address: 2607:f8b0:4004:c06::8a
     Name: google.com
     Address: 2607:f8b0:4004:c06::65

   • The internal DNS queries should be failing with the following error:
     

     # nslookup np-e9dc030e-1.us-east1-a.compute.internal
     unable to resolve host np-e9dc030e-1.us-east1-a.compute.internal: Temporary failure in name resolution

2. Step 2: Review the journalctl logs

   • Check if the DNS queries are being sent out as TCP
     

     # journalctl -u systemd-resolved
     .....
     Apr 16 15:46:13 np-e9dc030e-1.us-east1-a.compute.internal systemd-resolved[2569]: Using degraded feature set TCP instead of UDP+EDNS0 for DNS server 8.8.4.4.

3. Step 3: Restart systemd-resolved

   • From the journalctl logs, if you do observe systemd-resolved switching to TCP, restart the service to recover internal DNS resolution
     

     # systemctl restart systemd-resolved