UCS Blade unable to ping or connect to vCenter

Hi All,

I came across a strange issue where 2 blades was unable to ping I could get 2 ping and then Request time outs

Problem : 2 Servers( in the same chassis server 3,4 ) unable to ping their gateway. Ping drops are after 1-2 packets. We cannot ping/ssh to them from outside and they cannot be added to vCenter. Other blades in the same chassis are working ( 1,2)

Steps Taken :

  1. Confirmed that we do not have any IP conflicts
  2. Made sure that we do not use the same subnet anywhere else in the network
  3. Also checked and confirmed that the Mgmt(CIMC) subnet is different

ucs1

  1. Started a ping from inside a Host ( Server-3 ) to check if you are able to get to the gateway
  2. Checked vobd logs  to find if you can find any  entrie for duplicate ip/mac
  3. Checked the Mgmt Interface : vmk0
  4. It is connected to Port Group : Management Network  on VLAN ID, Check if we are  using the right vlan
  5. Found that the MAC Address of the vmk0 is same as vnic (vmnic0) : This is a known issue VMware KB article here. This is the vnic mac address of the service profile which is attached to the esx blade.
  6. Deleted and re-created the vmk0 interface
  7. Confirmed that the MAC Address has changed
  8. Started a continuous ping which work fine now
  9. Host can now be added to the vCenter

 

 

So let me go through of the process of how we found the duplicate mac address.

The vnic mac address was taken by vmknic  as you might know vmknic mad starts with 00:50:56 but in my case it was same as the mac address of vnic from the service profile. Continue reading “UCS Blade unable to ping or connect to vCenter”

The system detected an address conflict for IP address 0.0.0.0 with the system after a reboot

Hi All,

This  week I have seen an interesting issue on Windows Servers 2008 /2012 . Just to let you know these are VM’s and I was unable to connect to  these server. So I login on vCenter to check the windows host and found the server was having network error.

After checking the event viewer i found this error

The system detected an address conflict for IP address 0.0.0.0 with the system having network hardware address xx-xx-xx-CE-44-3F. Network operations on this system may be disrupted as a result. Time stamp 28/10/2014 hh:mm:ss.

Quick Fix

Reboot the server and it got the network back. I have to mention giving ipconfig was showing the correct IP address even when it was having the network error.
Further investigation reviled few interesting and worrying facts, apparently this issue is only affecting Windows Vista and above, we also found that it is a known issue, However it will arise only when a windows server is rebooted.

Root Cause

The root cause is part of the detection flows defined by the RFC 5227 (IPv4 Address Conflict Detection).

The error is caused by the method used by Windows to detect an address conflict (http://tools.ietf.org/html/rfc5227#section-2.1.1) and one of the packets used by the cisco security feature called “ip device tracking”, used for the NAC Layer 2 validation.

Unfortunately the IOS version (15.2) used by the most of the Cisco core switches, enables this feature by default and there is no way to disable it. The only options are: downgrade the IOS or tuning some parameters.

Solution

a. downgrade the IOS

b. modify a parameter of Cisco ‘IP device tracking’  feature in order to potentially solve this issue.

On each interfaces

ip device tracking maximum 0

References:

https://social.technet.microsoft.com/Forums/en-US/82e7b9a6-c58e-4b73-9fac-8e73f3347b91/0000-ip-address-conflicts-on-the-network-vms-lose-network-after-reboot?forum=winserverhyperv
http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/8021x/116529-problemsolution-product-00.html

Other blog:

http://networkengineering.stackexchange.com/questions/6617/cisco-3560x-switch-mac-reported-with-same-ip-as-other-equipment

Cisco UCS Invicta Storage – How to Provision a LUN to ESX Host

Hi All,

I am writing this post to show the process on how to provision LUN’s from Cisco UCS Invicta  to ESX host in 3 steps hope you enjoy the post.

Note:

  1.  Create a default LUN and add MAP ID 0 ‘zero’ when ever you are provisioning a new ESX  rule of thumb for Invicta is add a default lun  lun0_ default with MAP ID 0
  2. Keep all the LUN’s with the same MAP id on each host when provisioning them. ( Caution : If you do not give the same MAP ID for a LUN which is already provisioned and the lun has VM running. The LUN comes up as a new lun to be added to VMware so the Administrator if he is not aware of this issue may end up formating it assuming its a new LUN to provision using VMFS 5.
  3. You need to remember that you can right click and drag and drop luns to map them.

In our environment I have created a default lun with 10 GB and never provision this on VMware, however always add this default lun first  with MAP ID 0 to any ESX host you want to provision.

Login to the invicta with admin or superuser account.

Step 1: Creating a LUN

Click on LUN Configuration and LUNs and Click on Create LUN

 1.LUN Page

As I have mentioned in the Notes at the top that we need to first create a default lun which is what I am creating below. just fill in the details I call it lun0_default

Create a Default lun0

The size of this lun can even be 1 GB as you are not going to use this lun . I have put 10 GB a I have 65 TB of thin provisioned disk and I am not going to provision this lun its just what is recommended by Cisco documentation to have a default  lun 0 provisioned on each host with MAP ID 0

Create LUN Page

I have created another LUN of 1 TB calling it lun1 as shown below

4.creating_default_lun

Step 2.  Adding a Host Initiator Group

Click on LUN Configuration and Initiator Groups and Click on Create New Group

Continue reading “Cisco UCS Invicta Storage – How to Provision a LUN to ESX Host”