Cisco Live 2016

Hi All,

It has been a long time since I have published any post. I have been quite busy these days.

I am going to attend Cisco Live in Berlin, Germany next week. So I just thought this is going to be a good opportunity to share my views of Cisco Live and the latest and greatest technologies Cisco has to offer.

For those of you who are interested to keep an eye on Cisco Live Cisco Live webcasts and broadcasts are free to attend!
To attend a webcast, you will need a Cisco Live account. Register now for your free account or log in if you already have one

The list of events are given below:

http://www.ciscolive.com/emea/ ( Online Events – Cisco Live )

 Stay tuned for more updates…

 

Troubleshooting The vMotion migration failed because the ESX host were not able to connect over the vMotion network.

Hi All,

Today I will take you thought the troubleshooting for vMotion using CLI, If you have not done network troubleshooting on esx host you will find some really good esxi commands on this post. And if you are using a converged infrastructure its important to understand end to end connectivity especially if you have different vlans for each service like vMotion FT, Management traffic and VM-Traffic.

I build a brand new Data Center and after configuring the dv switch I decided to test the vMotion first and this is the error I saw on the first attempt the vm was stuck at 14 % and after a while it gave the below error message.

as shown below:

vmotion error_task

 

Migrate virtual machine error message

vmotion error

If you read the above error message you will find the IP’s of the host which I am migrating from and to.

for example lets consider the below two host this troubleshooting is done on esx-02. vmnic2 is FT vmnic3 is vMotion so I am comparing the mac address of booth the host on the ucs service profile to compare tht the vnic template has the right mac address and if they are connected to the right vlan. Continue reading “Troubleshooting The vMotion migration failed because the ESX host were not able to connect over the vMotion network.”

UCS Blade unable to ping or connect to vCenter

Hi All,

I came across a strange issue where 2 blades was unable to ping I could get 2 ping and then Request time outs

Problem : 2 Servers( in the same chassis server 3,4 ) unable to ping their gateway. Ping drops are after 1-2 packets. We cannot ping/ssh to them from outside and they cannot be added to vCenter. Other blades in the same chassis are working ( 1,2)

Steps Taken :

  1. Confirmed that we do not have any IP conflicts
  2. Made sure that we do not use the same subnet anywhere else in the network
  3. Also checked and confirmed that the Mgmt(CIMC) subnet is different

ucs1

  1. Started a ping from inside a Host ( Server-3 ) to check if you are able to get to the gateway
  2. Checked vobd logs  to find if you can find any  entrie for duplicate ip/mac
  3. Checked the Mgmt Interface : vmk0
  4. It is connected to Port Group : Management Network  on VLAN ID, Check if we are  using the right vlan
  5. Found that the MAC Address of the vmk0 is same as vnic (vmnic0) : This is a known issue VMware KB article here. This is the vnic mac address of the service profile which is attached to the esx blade.
  6. Deleted and re-created the vmk0 interface
  7. Confirmed that the MAC Address has changed
  8. Started a continuous ping which work fine now
  9. Host can now be added to the vCenter

 

 

So let me go through of the process of how we found the duplicate mac address.

The vnic mac address was taken by vmknic  as you might know vmknic mad starts with 00:50:56 but in my case it was same as the mac address of vnic from the service profile. Continue reading “UCS Blade unable to ping or connect to vCenter”

The system detected an address conflict for IP address 0.0.0.0 with the system after a reboot

Hi All,

This  week I have seen an interesting issue on Windows Servers 2008 /2012 . Just to let you know these are VM’s and I was unable to connect to  these server. So I login on vCenter to check the windows host and found the server was having network error.

After checking the event viewer i found this error

The system detected an address conflict for IP address 0.0.0.0 with the system having network hardware address xx-xx-xx-CE-44-3F. Network operations on this system may be disrupted as a result. Time stamp 28/10/2014 hh:mm:ss.

Quick Fix

Reboot the server and it got the network back. I have to mention giving ipconfig was showing the correct IP address even when it was having the network error.
Further investigation reviled few interesting and worrying facts, apparently this issue is only affecting Windows Vista and above, we also found that it is a known issue, However it will arise only when a windows server is rebooted.

Root Cause

The root cause is part of the detection flows defined by the RFC 5227 (IPv4 Address Conflict Detection).

The error is caused by the method used by Windows to detect an address conflict (http://tools.ietf.org/html/rfc5227#section-2.1.1) and one of the packets used by the cisco security feature called “ip device tracking”, used for the NAC Layer 2 validation.

Unfortunately the IOS version (15.2) used by the most of the Cisco core switches, enables this feature by default and there is no way to disable it. The only options are: downgrade the IOS or tuning some parameters.

Solution

a. downgrade the IOS

b. modify a parameter of Cisco ‘IP device tracking’  feature in order to potentially solve this issue.

On each interfaces

ip device tracking maximum 0

References:

https://social.technet.microsoft.com/Forums/en-US/82e7b9a6-c58e-4b73-9fac-8e73f3347b91/0000-ip-address-conflicts-on-the-network-vms-lose-network-after-reboot?forum=winserverhyperv
http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/8021x/116529-problemsolution-product-00.html

Other blog:

http://networkengineering.stackexchange.com/questions/6617/cisco-3560x-switch-mac-reported-with-same-ip-as-other-equipment

Cisco UCS Invicta Storage – How to Provision a LUN to ESX Host

Hi All,

I am writing this post to show the process on how to provision LUN’s from Cisco UCS Invicta  to ESX host in 3 steps hope you enjoy the post.

Note:

  1.  Create a default LUN and add MAP ID 0 ‘zero’ when ever you are provisioning a new ESX  rule of thumb for Invicta is add a default lun  lun0_ default with MAP ID 0
  2. Keep all the LUN’s with the same MAP id on each host when provisioning them. ( Caution : If you do not give the same MAP ID for a LUN which is already provisioned and the lun has VM running. The LUN comes up as a new lun to be added to VMware so the Administrator if he is not aware of this issue may end up formating it assuming its a new LUN to provision using VMFS 5.
  3. You need to remember that you can right click and drag and drop luns to map them.

In our environment I have created a default lun with 10 GB and never provision this on VMware, however always add this default lun first  with MAP ID 0 to any ESX host you want to provision.

Login to the invicta with admin or superuser account.

Step 1: Creating a LUN

Click on LUN Configuration and LUNs and Click on Create LUN

 1.LUN Page

As I have mentioned in the Notes at the top that we need to first create a default lun which is what I am creating below. just fill in the details I call it lun0_default

Create a Default lun0

The size of this lun can even be 1 GB as you are not going to use this lun . I have put 10 GB a I have 65 TB of thin provisioned disk and I am not going to provision this lun its just what is recommended by Cisco documentation to have a default  lun 0 provisioned on each host with MAP ID 0

Create LUN Page

I have created another LUN of 1 TB calling it lun1 as shown below

4.creating_default_lun

Step 2.  Adding a Host Initiator Group

Click on LUN Configuration and Initiator Groups and Click on Create New Group

Continue reading “Cisco UCS Invicta Storage – How to Provision a LUN to ESX Host”

advanced boot or secure boot configuration cannot be applied to the specified server There are not enough resources overall

If you are looking at this Failure Reason: advanced boot or secure boot configuration cannot be applied to the specified server There are not enough resources overall, You might have come across this error.

advance boot or secure boot configuration cannot be applied

Although the above error looks like a very generic error when you apply a service profile to a Server. Most of the on-line search takes you in the direction of making sure you are using the legacy CD ROM drive.

With the process of elimination I found that while using the Flex Flash ‘SD Card’ in Boot Option this error was trigged. once I removed the SD Card, I was able to apply the profile with out any problems.  But I wanted to use the Flex Flash with RAID-1,( SD Cards) to boot.  more info on  Cisco UCS FlexFlash Configuration of Secure Digital (SD) Card

As you might be aware that boot from SD card is a new feature introduced in version 2.2 , we then started looking at he version of UCS Manager and Firmware running on the environment. I had recently upgraded this UCS to the 2.2(1d) so that was not an issue.

We thought to double check the firmware configuration, when going through this options we create a Host Firmware Policy specifically only for M3 and just selecting the M 200 M3 CIMC and Bios with the latest version of firmware.  more info on firmware can be found CISCO UCS Firmware Update Process

Once this option was applied to the Service profile The blade started to boot up.

The only thing different to this was I had created a custom firmware on Host Firmware Policy called 2.2 firmware but ended up selected quite a lot of servers with latest version of software .. However in my UCS Environment I have only got M 200 M3.

So to fix this issue, make  sure in ‘Host Firmware Package’ make sure you have customized firmware profile for every different server you have, so that you can individually apply it to the service profile you are assign to that specific server.

Example M2 and M3 Blades we can have two Host Firmware Packages which you can add to specific Service profiles individually.

Additional resources:

If you want to learn more about the process of doing a Firmware update please checkout my link CISCO UCS Firmware Update Process

If you want to learn more about the process of doing a Firmware update please checkout my link Cisco UCS FlexFlash Configuration of Secure Digital (SD) Card

 

 

 

Update Firmware of Cisco UCS B series Server gives Error “The firmware of the following components cannot be updated because they are using host/management firmware policy:”

If you are reading this post you might have come accross the Error “The firmware of the following components cannot be updated because they are using host/management firmware policy:”

This week I was workig on upgrading the firmware of  CISCO UCS  from version 2.1.(1 f) to 2.2.(1d). When doing the firmware update after uploading the firmware imagers to the FI, I went to the ‘Update Firmware’ and Selected ‘All’ and select ‘Bundle’ to select the specific bundile I wanted to install in my case 2.2(1d)

updatefirmwareselection

After making the selection as shown above Click Apply and OK. Which is when I saw the error message below, This bacally means all the chassis CIMC Controler / Adapter, BISO cannot be updated because the service profile attached is using hosted management of firmware.

firmware Update error

 So, their is no ways way out form this as if you say Yes then these server will miss out form updates.

One option we have tested is de-associate the sersvice profile , as this is Server Firmware so, its related to Service profile association.  I was able to Update Firmware successfully.

Howerver I am still invistigating how to avoid this step if I have to do a Firmware upgrade on Production Live enviernment, this is a Show Stoper. Luckely for us we have migrated all our production to a diffrent UCS enviernment, so we did not have any thing running on this UCS.

I belive the way the service profile firmware police should be done in a diffrent method,  and then associated to each Physical Blades.

I will keep you posted, once I get another process of doing the Update Firmware with out de-associating the Service Profiles. For more information on how the firmware is done check  out my post CISCO UCS Firmware Update Process

Please feel free to comment on your opinion.

Cisco UCS Firmware Overview

Hello Everyone,

Cisco has recently launched a lot of  Firmware for CISCO UCS, it is becoming rather hard to keep up with the new release. We had 5 New Release in Version 2.2 starting from 2.2.(1b),  2.2.(1c), 2.2.(1d),  2.2.(1e), 2.2.(2c).

The good thing is, in every release CISCO  is adding some good features. In one of  the release 2.2.(1d) they have introduced boot from SD, FlexFlash, this option was not available on the older versions of  UCS Manager.

If you are intreasted in this, and want to find how you can boot the ESXi or any other OS off the Secure Digital (SD)  FlexFlash  and how this can be implemented do check  my post  here: Cisco UCS FlexFlash Configuration of Secure Digital (SD) Card

So, for the newbee  to Cisco UCS,  its confusing about the firmware. In this post I am trying to clarify few aspects of this, feel free to post comment below if you have any questions and I will get back to you.

The CISCO UCS firmware is bundled into two packages as given below:

ucs_firmware

 

1. Infrastructure Firmware

In Infrastructure Software firmware we update the firmware of the following

  • Cisco UCS Manager software
  •  Kernel and system firmware for the fabric interconnects
  •  I/O module firmware

2. Server Firmware

Cisco UCS B-Series Blade Server Software Bundle

  • CIMC firmware
  • BIOS firmware
  • Adapter firmware
  • Board controller firmware
  • Third-party firmware images required by the new server

This also includes the firmware of Adapter, BIOSFC Adapters,  HBA Option ROM, Storage, Controller, Local Disk

Cisco UCS C-Series Rack-Mount UCS-Managed Server Software

  • CIMC firmware
  • BIOS firmware
  • Adapter firmware
  • Board controller firmware
  • Third-party firmware images required by the new server

As most of you might be aware the UCS manager is the GUI Java based which allows you to fully manage the CISCO Unified Computing System.

Note I have noticed that the Java version  7 update 40 is only compatible with the UCS Manager. 

So, where does the UCS Manager run ?

This was one of my first questions, Well the Software of the UCS Manager runs on the FI ( Fabric Interconnect 62xx) as we have a pair of Fabric Interconnect it has got resilience.

The UCS System is designed to be highly scalable and available, so upgrading the firmware should not bring any outage to the environment. However Cisco still suggests to arrange a Change windows and get this done out of hours or the quite period of your business.

This UCS Manager is bundled in the Infrastructure Firmware Software along with the Kernal update of the FI which is the NXOS, Finally we have the I/O Module firmware which is a part of the Infrastructure firmware software too.

The Other Bundle is the Server Firmware which has the firmware for BIOS, the Blades, Adapter, Board controller etc.

If you want to learn more about the process of doing a Firmware update please checkout my link CISCO UCS Firmware Update Process

Hope this information is useful, feel free to comment.

CISCO UCS Firmware Update Process

Hi All,

Recently Cisco has released yet another release of the Cisco UCS firmware version 3.0.

This version of firmware is for 6324 Fabric Interconnect and  B200 M3, C220 M3, and C240 M3 servers.updatestarted

This is released on 24-JUL-2014, you can find more details on the release notes for this firmware version:

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/ucs_3_0_rn.html#pgfId-225587

Just to give everyone an idea there are two from of Firmware that needs to be updated on UCS system

1. Infrastructure Firmware

In Infrastructure Software firmware we update the firmware of the following

  • Cisco UCS Manager software
  •  Kernel and system firmware for the fabric interconnects
  •  I/O module firmware

2. Server Firmware

Cisco UCS B-Series Blade Server Software Bundle

  • CIMC firmware
  • BIOS firmware
  • Adapter firmware
  • Board controller firmware
  • Third-party firmware images required by the new server

This also includes the firmware of Adapter, BIOSFC Adapters,  HBA Option ROM, Storage, Controller, Local Disk

Cisco UCS C-Series Rack-Mount UCS-Managed Server Software BundleCIMC firmware

  • BIOS firmware
  • Adapter firmware
  • Board controller firmware
  • Third-party firmware images required by the new server

So,  without wasting any time these are the steps to follow:

Step One

  1. Create an all Configuration Backup
  2. Verifying HA status of Fabric Interconnects ( as we do not have any compute it should not affect us in anyway)
  3. Verifying the status of I/O Modules
  4. Verifying the Status of the Servers and Adaptors
  5. Download Firmware Images to the Fabric Interconnect.

 

Step Two

  1. Disable the Call Home ( Email alerts for reboot)
  2. Updating the Firmware on the Adapters, CIMCs and IOMs

 

If the Service profiles are configured using host/management firmware policy this will create an error and we might need to think of Plan B (If we do not get any error message we can progress with upgrade)

 

  1. Activating the Firmware on the Adapters
  2. Activating the Firmware on the CIMCs
  3. Activating the Board Controller Firmware
  4. Activating the I/O Modules
  5. Activating the Subordinate Fabric Interconnect Firmware
  6. Making the primary FI the Subordinate by login to FI ssh to the primary and to make it subordinate first check

# connect local-mgmt

 

FI-01-A(local-mgmt)# show cluster extended-state

Cluster Id: xxxxxxxxxxxxxxxxxxxxxxxx

 

Start time: Mon Jun 23 14:41:56 2014

Last election time: Tue Oct  7 11:54:01 2014

 

A: UP, SUBORDINATE

B: UP, PRIMARY

 

A: memb state UP, lead state SUBORDINATE, mgmt services state: UP

B: memb state UP, lead state PRIMARY, mgmt services state: UP

   heartbeat state PRIMARY_OK

 

INTERNAL NETWORK INTERFACES:

eth1, UP

eth2, UP

 

HA READY

Detailed state of the device selected for HA storage:

Chassis 1, serial: xxxxx, state: active

Chassis 2, serial: xxxxx, state: active

Chassis 3, serial: xxxxx, state: active

 

In my case as shown above FI B is the Primary so I can connect t0 FI B with hte below command .

 

FI-01-A(local-mgmt)# connect local-mgmt B

# cluster lead A

  1. Activating the Primary Fabric Interconnect Firmware

 

Step Three

  1. Listing the BIOS Upgrade Versions
  2. Create a Host Firmware Package
  3. Applying the Host Firmware Package
  4. Enabling Call Home.

 

Additional Notes

 

Prerequisite for Firmware update

 http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/upgrading/from2-0/to2-0MR/b_UpgradingCiscoUCSFrom2-0To2-0MR/b_UpgradingCiscoUCSFrom1-4To2-0_chapter_010.html#concept_DD68485034FB4E258D5827880D37225C

Process of Upgrading and Activating the Firmware

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/upgrading/from2-0/to2-0MR/b_UpgradingCiscoUCSFrom2-0To2-0MR/b_UpgradingCiscoUCSFrom1-4To2-0_chapter_0100.html#task_F1119EB4EDBA42578ADE662462B5C4CE

UCS Firmware deep dive

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/firmware-mgmt/gui/2-1/b_GUI_Firmware_Management_21/b_GUI_Firmware_Management_21_chapter_0100.html#concept_2474DE9D2048450CA1A37A8A3D3EE597

 

You can also find the process of upgrading on this youtube video  https://www.youtube.com/watch?v=b6LQt2DYVWA

its broken down into 3 steps.