Information Technologies: Problem joining ESXi 5.0 host to Active Directory Domain

Wednesday, March 07, 2012

Problem joining ESXi 5.0 host to Active Directory Domain

I was trying to join an ESXi 5.0 host to our AD domain and encountered the following error.

I have verified that configuration such as NTP, DNS and domain name are all correct. Anyway, I don’t have such problem with ESXi 4.x hosts. Digging the system log provides some clues to the issue.

In the hostd.log file, I have the following error message.

2012-03-07T17:11:19.078Z [FFC40B90 error 'ActiveDirectoryAuthentication' opID=84971F43-0000004C] vmwauth NoSuchDomainException: Exception 0x0000054b: The specified domain either does not exist or could not be contacted.

In the syslog.log file, I have the following error message.

2012-03-07T17:14:04Z netlogond[1837994]: [LWNetDnsQueryWithBuffer() /build/mts/release/bora-396388/likewise/esxi-esxi/src/linux/netlogon/utils/lwnet-dns.c:1185] DNS lookup for '_ldap._tcp.dc._msdcs.deinfotech.org' failed with errno 0, h_errno = 1

With the error messages, I found the kb article Unable to add ESXi host to the Active Directory domain. Basically, the problem is due to the DNS return packet size greater than 512 bytes. This will cause the ESXi host to send the query over TCP rather than UDP to ensure reliable response. However, the DNS TCP port 53 is not open by default in the ESXi firewall.

If you click on the Configuration tab and click on the Security Profile link under the Software group, only UDP port 53 has been defined and open.

According to the kb article, we can create custom rule set to open TCP port 53. However, there is a note in the kb article saying the the custom firewall rule will not be persistent across reboots. The kb article User defined xml firewall configurations are not persistent across ESXi host reboots the persistent problem and the workaround.

Following the instruction, here is what I did.

Create a custom firewall rule set for the TCP port 53. Below is the content and I saved it as dnsservice.xml.

<ConfigRoot>
<service id='0032'>
    <id>dns tcp</id>
    <rule id='0000'>
      <direction>inbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>53</port>
    </rule>
    <rule id='0001'>
      <direction>outbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>53</port>
    </rule>
    <enabled>true</enabled>
    <required>false</required>
</service>
</ConfigRoot>

Using WinSCP, I uploaded my custom firewall file (dnsservice.xml) to a vmfs volume (e.g. /vmfs/volumes/datastore1).

Next, I edit the /etc/rc.local file to add in the following two lines and save it.

cp /vmfs/volumes/datastore1/dnsservice.xml /etc/vmware/firewall
esxcli network firewall refresh

Using PuTTy, I SSH to the ESXi host and run /sbin/auto-backup.sh to make sure the state.tgz file is updated.

Next I rebooted the ESXi host. After the reboot, I am able to see the TCP port 53 which I have defined being open up. If you take a look at /etc/vmware/firewall, the dnsservice.xml file will be there.