Category Archives: cisco

Moved Data Centre

The Data Centre move happened last weekend – all went well and we did it (a team of 11) in 24 hours. I had to look after the firewalls, ACEs, proxy, WAAS as a priority and chipped in whereever else when required. For a business that does several hundred million dollars of business through its web site (as one example) – It was a great effort to have the business back online 2 days prior to expectation. Some of the guys I work with really showed their expertise and planning skills.

One thing I learned was that if unsure, always bring up the standby FWSM or ACE when :

A. The primary unit is online and functional and is active.

B. The underlying network (vlans, switch configs etc) are up and stable.

I had a few dramas where the failover states of the FWSM or ACE seemed ok, but because a few things were changed during the move things started to go awry. In some cases both FWSMs or ACEs were up, but because of some underlying changes to the network they would both think they are active and the network can get turned into a banana. (eg. failover vlan removed!)

In hindsight, I should have left the secondary modules out until all other network work was completed. Just because you think the modules are working at one point in time doesn’t mean they will be behaving as expected at a later point if there are changes being made to the network.

Cisco ACE log bug

That old chestnut – the weird ace log issue – now has a bug ID ! (CSCso15332). It only appears to happen if the rservers are offline in the original context. Bizarre.

Here is what Cisco has said on the matter:

Conditions:

The customer was using the same syslog host in two contexts, had configured http and https probes in both contexts, and was logging probe failures in both contexts. The rservers in the second context were unavailable when the issue began; later, those rservers became available and the inappropriate logging ceased.

Workaround:

None.

Further Problem Description:

The customer experienced this problem for approximately ten days sproadically on one rserver. Even during that period there were at most a moderate (20 to 30) number of messages a day.

NTP on the 6509

Had some minor drama with the core switch clock being out by well over an hour. Not causing any issue except frustration when viewing logs and trying to compare timestamps across devices.

All the network devices get their time sync from the wan router (which in turn contacts our ntp server for its time).

To fix it up – remove all the ntp related statements from the cisco 6509, then manually set the clock to within 5 mins of the actual time as reported by the ntp server, then re-apply the ‘ntp server’ statement(s). After a short while the clock will re-sync correctly with the ntp server specified.

Never apply the ‘ntp clock-period’ statement – this is done by the device itself. I think someone manually applied this and that was why the time started to get way out.
ie. Make sure you never copy the ‘ntp clock-period’ statement from one device to another. The clock-period is set by NTP; to regulate the internal clock on the device. This is the number of ticks in one second. Therefore, NTP doesn’t change the time of the switch, it adjusts the clock-period so that the system clock is synchronized with the ntp server.