In my search to improve my Virtualization, that is my Virtual Server Infrastructure, I can to a point where local storage (local disks/drives) just would not cut it anymore.  They were too slow, the RAID required for ESX(i) to work with was too expensive for my budget, and I did not want to deal with the headaches that came with maintaining them on local servers, so I started to look around, google, and do my research and it turned out that iSCSI seems to be the current market winner.  That is that iSCSI has captivated the market with its promise of a cheaper storage that supposedly can compare to FiberChannel in some respects.  With this in mind I bought the Best Dell Gigabit iSCSI Switch I could find in my price range, the Dell PowerConnect 3448.

My Trials

I first tried to use an Unmanaged Switch, the D-Link DGS-1024D to try to handle my iSCSI connections but I eventually found some settings on both ends of the connection that ended up pushing too much through that switch and it kept dying.  It was not the Best Gigabit iSCSI Switch like I had hoped.

Then I thought that the Best Gigabit iSCSI Switch was going to be too expensive for my budget, so I tried running Crossover Gigabit Ethernet cables directly from my Virtual Hosts to my iSCSI Storage and it worked better.  I started to be able to push through more data without having a switch to overload but this did not account for bursting and I soon ran into limitations of the straight NICs were they dropped too many packets for a stable connection.  Clearly I needed an Ethernet Switch in the middle, it had to be Gigabit, I wanted the Best in my price range, and my research revealed that Dell’s iSCSI implementation, meaning the code their switches use to handle that kind of traffic, was their own, where as HP had used Cisco’s and a good number of reviewed said that Dell’s iSCSI worked better then Cisco’s version, so the hunt for the Best Dell Gigabit iSCSI Switch I could afford was back on.

After more and more research, I finally found the Best Dell Gigabit iSCSI Switch and it turned out to be the Dell PowerConnect 3448.  Having checked the retail and wholesales prices I noticed that you cannot get one new anymore so I resorted to EBay and ended up coming in under $200 with the shipping costs.  There was the usual extra $65 charge from FedEx to pay their broker and act like it is an ‘import fee’ but it was really well priced for what I got.

My iSCSI Recommendations

The details of configuring it are pretty dry but I will tell you the points that I found to work the best for iSCSI and so far this also holds true for Cisco switches I’ve implemented this on as well:

• Use a Layer 2 switch for iSCSI
• Make sure your switch specifically can handle iSCSI traffic.  Some specifically cannot
• Use a dedicated switch for iSCSI
• Use a dedicated VLAN and subnet for iSCSI
• Turn off QoS on the iSCSI switch ports
• Turn STP to ‘Port Fast’ on the iSCSI switch ports and set it to rSTP on the entire switch if you can
• Turn on FlowControl on the iSCSI switch ports
• Turn off Storm Control on the iSCSI switch ports
• Use MPIO if you can, but if you cannot complete MPIO all the way, then don’t use it.  I’ll make another post about this in the future

My Defense

I’d like to take a chance to explain some of my points above in case anyone else is doing research and comes across this.

Layer 2: I’d say this is best because a Layer 3 switch can route and you do not want that.  If iSCSI packets leave your subnet then iSCSI will fail.  My going Layer 2 only, you are putting less load on the switch by not having it worry about the Layer 3 part and that leaves you more horsepower to handle the packets.

QoS Off:  QoS is short of Quality of Service and in network switches, this means that the switch has to inspect all packets coming into any port, tag them based on their content, then add extra decisions about routing them based on this.  It usually will cause lots of slowness and disconnections for iSCSI.

STP:  This stands for Spanning Tree Protocol and when done right it can be a networking person’s best friend, but it detect when you unplug and plug in any port and once it detects that it can take up to 50 seconds to re-detect everything and your ports will be down that entire time.  Since iSCSI has ways of tripping this port detection it means disconnections for you.  Setting STP to ‘Port Fast’ on your iSCSI ports will tell it not to do that scan that takes up to 50 seconds.

StormControl: This is a feature of the switch to detect when something is sending out a storm of packets and block it for a short time or until the storm goes away.  By blocking it of course I mean drop all packets from that source and of course it normally mistakes iSCSI’s bursts for packet storms so this will cause random disconnects during production data spikes.

FlowControl: This is one of the few Ethernet Switch features that can actually help iSCSI.  Basically if this is turn on on the switch, the iSCSI Initiator(s) [The iSCSI NIC(s) on the VM hosts] and on the NAS/SAN (Storage) then what happens is the switch will eventually fill up its queue for packets on an iSCSI port and will send back a ‘pause frame’ which is a special packet of data that asks the sender to pause sending nicely while it tries to deal with the load it has, and then is sends a ‘resume frame’.  This method replaces the usual one of just dropping packets when the port’s queue is full and will present as slowness instead of disconnections when your iSCSI overloads the switch.

Following all of this, you still need to monitor the resources on your switch to make sure you are not overloading it and if you are then you may want to consider MPIO to help with that.  MPIO is short for MultiPathing and really it means setting up multiple paths for the iSCSI data to travel between the Storage and VM Host.  Since this can take the form of multiple NICs on both the VM Host and Storage and in fact going through two different switches with two different subnets, then you would be splitting up the switch work between two or more switches and then better handle it.