Tag Archives: cisco

WAAS and Isilon

While I’ve been in the USA there’s been some interesting developments in an EMC Isilon/Cisco WAAS interoperability issue the guys back in Sydney have been dealing with.

A few weeks ago we upgraded Isilon from 6.0 to 6.5 level. Since then, users who were saving files to Isilon shares and exiting their application, but if they went straight back into the application and tried to open the file they were presented with an error. Secondly, the directory where the file was located had multiple .tmp files being generated and not being deleted per save and free space was now being used quickly. These .tmp files are expected behaviour in an MS Office saving operation (see below).

We bypassed CIFS acceleration and the problem went away (native TFO, LZ and DRE was still active). Therefore, as a workaround the CIFS accelerator was disabled to the Isilon nodes only to give Cisco/Isilon enough time to work out what was happening. That in itself took longer as it should have to get some action, but eventually some analysis of the wireshark captues by the support teams from the vendors came to some interesting findings. Obviously, we would prefer if the CIFS accelerator could work as we expect.

The analysis came back and noted there was a change in the file path passed during the save operation from a backslash ‘\’ to a forward slash ‘/’.

I’ll quote the case notes here:

 ===Quoted from an Isilon support representative===

— Client deletes temp file

— Client renames xls to temp file

— Cisco core kicks in and does an enumeration for /temp file

— Cluster returns file not found (because Isilon doesn’t support the /)

— Client opens the file \temp

— Cisco returns file not found because they got back /temp did not exist on the cluster

— At this point, since the client failed to open \temp file, it stops working on the file and ends up not deleting the file.

So that brings us to what changed for the upgrade?

Well, according to Cisco, the reason the WAAS sends a / instead of a \ to Isilon is because during the Session Setup Response, Isilon returns the field “Native OS: Unix”.  Cisco have never tested against Isilon but they have tested against Samba which works when you pass /.

Since we upgraded from 6.0 (samba implementation) to 6.5 (likewise implementation) that is the change.  Both our 6.0 and 6.5 releases set Native OS: Unix; however, while the 6.0 version supported / the 6.5. version does not.

As for why Windows works; it is because when Windows sets the Session Setup Response of Native OS: Windows …., Cisco WAAS leaves the \ as \.

At this point, both Cisco and Isilon are investigating what can be changed from a code perspective.

===End of case note===

Isilon upgrades are currently one-way. There is no rollback option. So, we must await the vendors to fix it. We are now in the forth week since the upgrade; and it has taken a fair bit of ‘prodding’ by the storage engineer on our staff to things this far along.

Not only has this highlighted the need for us to improve our UAT testing prior to deployment to Production (e.g. saving instead of opening/copying only – in this case only if you have a 2nd isilon cluster available to test!), but of course you cannot take what vendors say as gospel. That is, look at the following link: http://www.cisco.com/en/US/netsol/ns962/index.html and note the statement:  “Isilon’s entire product line is certified to work with Cisco Wide Area Application Services (WAAS)”. You could argue that I am taking that line too literally, but that’s probably giving these vendors more credit than they deserve. You can’t say that a product is certified and then never revisit testing between the products after any future software upgrade of either product. That’s ordinary.

ACS 5.x Database Purge

I received an alert:

Purge is successful. The size of records present in view data base is 22.58 GB. The physical size of the view data base on the disk 96.1 GB. If you want to reduce the physical size of the view data base, run acsview-db-compress command from acs-config mode through command line.

Use the acsview-db-compress command to compress the view database file size. This command compresses the ACS View database by rebuilding each table in the database and release the unused space. As a result, the physical size of the database is reduced.

Ok so time to fix the database during a maintenance window.

acsadmin(config-acs)# acsview-db-compress
You chose to compress ACS View database. This operation will take more time if the size 
of the database is big. During this operation, ACS services will be stopped. Services will 
be started automatically when the compression is over. Do you want to continue (y/n)?  y

Please wait till ACS services come back after the view db is compressed. Refer ADE.log 
for more details about the view db compress.

How long to wait? Who knows – so I decided to jump in and run it to see how long….I pressed ‘y’ (as above) and waited. I could not find any command to show any status/progress indication – so I had to rely on nagios and the following command only.


admin# show application status acs

Application initializing...
Status is not yet available.
Please check again in a minute.


Yes ok – thanks Cisco!….. Anyway 3 hours 20 mins later:


admin# show application status acs


Process 'database'                  running
Process 'management'                running
Process 'runtime'                   running
Process 'adclient'                  running
Process 'view-database'             running
Process 'view-jobmanager'           running
Process 'view-alertmanager'         running
Process 'view-collector'            running
Process 'view-logprocessor'         running


…so cleaning up about 75GB of whitespace in the database took about 3 hours… so you can expect about 25GB an hour perhaps. Plan your outage window accordingly.

Call Manager 7 – admin password recovery

I had a vm for cucm7.1(3) that I’d had off for a while – and I needed to fire it up again for some lab testing. Of course, my usual set of passwords didnt work to get into the thing. So, at the console of the vm, I did the following:

1. Login at the console using username: pwrecovery  password: pwreset.

2. Edit the vm’s settings to ensure no cdrom images or physical cdroms are connected.

3. Follow the prompts, and re-enable the call manager iso image in the cdrom drive when prompted to ‘insert the cdrom’. Its basically asking for the call manager install media.

This resets the user admin account to ‘admin’ with a new password you specify.