UCS Blade unable to ping or connect to vCenter

Hi All,

I came across a strange issue where 2 blades was unable to ping I could get 2 ping and then Request time outs

Problem : 2 Servers( in the same chassis server 3,4 ) unable to ping their gateway. Ping drops are after 1-2 packets. We cannot ping/ssh to them from outside and they cannot be added to vCenter. Other blades in the same chassis are working ( 1,2)

Steps Taken :

  1. Confirmed that we do not have any IP conflicts
  2. Made sure that we do not use the same subnet anywhere else in the network
  3. Also checked and confirmed that the Mgmt(CIMC) subnet is different

ucs1

  1. Started a ping from inside a Host ( Server-3 ) to check if you are able to get to the gateway
  2. Checked vobd logs  to find if you can find any  entrie for duplicate ip/mac
  3. Checked the Mgmt Interface : vmk0
  4. It is connected to Port Group : Management Network  on VLAN ID, Check if we are  using the right vlan
  5. Found that the MAC Address of the vmk0 is same as vnic (vmnic0) : This is a known issue VMware KB article here. This is the vnic mac address of the service profile which is attached to the esx blade.
  6. Deleted and re-created the vmk0 interface
  7. Confirmed that the MAC Address has changed
  8. Started a continuous ping which work fine now
  9. Host can now be added to the vCenter

 

 

So let me go through of the process of how we found the duplicate mac address.

The vnic mac address was taken by vmknic  as you might know vmknic mad starts with 00:50:56 but in my case it was same as the mac address of vnic from the service profile.

login to the host and give the command

#esxcfg-nics -l

this gives the list of vnics assigned to the service profile, note the management nics in my case vnic 0 and 1

then give the below command to get the vmkernal nics

#esxcfg-vmknic -l

as you can see above the mac address is not starting from 00:50:56 and it is been duplicated from the vnic template from the service profile vnic0 which is the root cause for the connectivity issue.

Now to fix this we have to delete the virtual switch and re-create it.

#esxcfg-vswitch -l

This lists the virtual switches

#esxcfg-vmknic -d -p ‘Management Network’

you have to specify exact name of the port

Now we will need to add the vmknic with the same Management network

#esxcfg-vmknic -a -i <IPaddress> -n <Netmask> ‘Management Network’ 

and also give the command #esxcfg-vswitch -l  to check the Management network created as shown below.

note you can not give a gateway at this point you can specify frm DCA

p3

 you should now be able to ping to the host as it has got a new mac address to check it you can give the command

#esxcfg-vmknic -l

p34

 

for a continus ping you can give

#ping <gateway ip> -c 100

this will give 100 continuous pings

 

Reference:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1031111

2 thoughts on “UCS Blade unable to ping or connect to vCenter”

  1. In my opinion it should not be a problem if the vmnic has the same MAC address as the vNIC from the UCS Service Profile. That is how it should be.
    At least we have some setups where this is the case and there is no issue.
    Can you elaborate why the vNIC MAC from the UCS SP and the MAC from the corresponding vmnic in ESXi should be different?

    Thanks.

  2. VMware Certified Advanced Professional 6 (Desktop and Mobility Deployment) – The industry-recognized VCAP6-DTM Deploy certification validates that you know how to deploy and optimize VMware Horizon 6 (with View) environments. It proves that you have the understanding and skills required to leverage best practices to provide a scalable and dependable Business Mobility platform for your company. Some of the topics include: Configuring and managing Horizon View components, configuring cloud pod archituecture, configuring Group Policy settings related to Horizon View, Configuring and optimizing desktop images for Horizon View & Mirage, Configuring and managing App Volumes AppStacks, Configuring desktop pools, Configuring and deploying ThinApp packaged applications, Configuring VMWare Identity Manager, etc.Szumigalski.com was designed by Sebastian to spread his passion for PowerShell & VMWare. Sebastian is an IT skilled professional working in Singapore for more that fifteen years who is frequently searching for fresh expertise to refine his technical skills & understanding. Since then, Sebastian has became a member of PowerShell User Group & VMWare VMug group, and has been engaged in all of the get-togethers held in Singapore. This blog will show you just how Sebastian manage to automate his daily work by making use of PowerShell. You’ll find study courses meant for the VCAP6-DTM test, which were personally published by Sebastian. Sebastian is certified with VCAP6-DTM, and is expert with virtualization & server maintenance from four years experience of automation. The call for VMWare prepared admins and engineers are ever-increasing in today’s competitive technology market. Become familiar more details on PowerShell & VMWare with Szumigalski.com!

Leave a Reply

Your email address will not be published. Required fields are marked *