While I’ve been in the USA there’s been some interesting developments in an EMC Isilon/Cisco WAAS interoperability issue the guys back in Sydney have been dealing with.

A few weeks ago we upgraded Isilon from 6.0 to 6.5 level. Since then, users who were saving files to Isilon shares and exiting their application, but if they went straight back into the application and tried to open the file they were presented with an error. Secondly, the directory where the file was located had multiple .tmp files being generated and not being deleted per save and free space was now being used quickly. These .tmp files are expected behaviour in an MS Office saving operation (see below).

We bypassed CIFS acceleration and the problem went away (native TFO, LZ and DRE was still active). Therefore, as a workaround the CIFS accelerator was disabled to the Isilon nodes only to give Cisco/Isilon enough time to work out what was happening. That in itself took longer as it should have to get some action, but eventually some analysis of the wireshark captues by the support teams from the vendors came to some interesting findings. Obviously, we would prefer if the CIFS accelerator could work as we expect.

The analysis came back and noted there was a change in the file path passed during the save operation from a backslash ‘\’ to a forward slash ‘/’.

I’ll quote the case notes here:

 ===Quoted from an Isilon support representative===

— Client deletes temp file

— Client renames xls to temp file

— Cisco core kicks in and does an enumeration for /temp file

— Cluster returns file not found (because Isilon doesn’t support the /)

— Client opens the file \temp

— Cisco returns file not found because they got back /temp did not exist on the cluster

— At this point, since the client failed to open \temp file, it stops working on the file and ends up not deleting the file.

So that brings us to what changed for the upgrade?

Well, according to Cisco, the reason the WAAS sends a / instead of a \ to Isilon is because during the Session Setup Response, Isilon returns the field “Native OS: Unix”.  Cisco have never tested against Isilon but they have tested against Samba which works when you pass /.

Since we upgraded from 6.0 (samba implementation) to 6.5 (likewise implementation) that is the change.  Both our 6.0 and 6.5 releases set Native OS: Unix; however, while the 6.0 version supported / the 6.5. version does not.

As for why Windows works; it is because when Windows sets the Session Setup Response of Native OS: Windows …., Cisco WAAS leaves the \ as \.

At this point, both Cisco and Isilon are investigating what can be changed from a code perspective.

===End of case note===

Isilon upgrades are currently one-way. There is no rollback option. So, we must await the vendors to fix it. We are now in the forth week since the upgrade; and it has taken a fair bit of ‘prodding’ by the storage engineer on our staff to things this far along.

Not only has this highlighted the need for us to improve our UAT testing prior to deployment to Production (e.g. saving instead of opening/copying only – in this case only if you have a 2nd isilon cluster available to test!), but of course you cannot take what vendors say as gospel. That is, look at the following link: http://www.cisco.com/en/US/netsol/ns962/index.html and note the statement:  “Isilon’s entire product line is certified to work with Cisco Wide Area Application Services (WAAS)”. You could argue that I am taking that line too literally, but that’s probably giving these vendors more credit than they deserve. You can’t say that a product is certified and then never revisit testing between the products after any future software upgrade of either product. That’s ordinary.