Category Archives: Virtualization

iSCSI, Virtualization

Best Dell Gigabit iSCSI Switch

In my search to improve my Virtualization, that is my Virtual Server Infrastructure, I can to a point where local storage (local disks/drives) just would not cut it anymore.  They were too slow, the RAID required for ESX(i) to work with was too expensive for my budget, and I did not want to deal with the headaches that came with maintaining them on local servers, so I started to look around, google, and do my research and it turned out that iSCSI seems to be the current market winner.  That is that iSCSI has captivated the market with its promise of a cheaper storage that supposedly can compare to FiberChannel in some respects.  With this in mind I bought the Best Dell Gigabit iSCSI Switch I could find in my price range, the Dell PowerConnect 3448.

My Trials

D-Link DGS-1024D
D-Link DGS-1024D

I first tried to use an Unmanaged Switch, the D-Link DGS-1024D to try to handle my iSCSI connections but I eventually found some settings on both ends of the connection that ended up pushing too much through that switch and it kept dying.  It was not the Best Gigabit iSCSI Switch like I had hoped.

Then I thought that the Best Gigabit iSCSI Switch was going to be too expensive for my budget, so I tried running Crossover Gigabit Ethernet cables directly from my Virtual Hosts to my iSCSI Storage and it worked better.  I started to be able to push through more data without having a switch to overload but this did not account for bursting and I soon ran into limitations of the straight NICs were they dropped too many packets for a stable connection.  Clearly I needed an Ethernet Switch in the middle, it had to be Gigabit, I wanted the Best in my price range, and my research revealed that Dell’s iSCSI implementation, meaning the code their switches use to handle that kind of traffic, was their own, where as HP had used Cisco’s and a good number of reviewed said that Dell’s iSCSI worked better then Cisco’s version, so the hunt for the Best Dell Gigabit iSCSI Switch I could afford was back on.

Dell PowerConnect 5448
Dell PowerConnect 5448

After more and more research, I finally found the Best Dell Gigabit iSCSI Switch and it turned out to be the Dell PowerConnect 3448.  Having checked the retail and wholesales prices I noticed that you cannot get one new anymore so I resorted to EBay and ended up coming in under $200 with the shipping costs.  There was the usual extra $65 charge from FedEx to pay their broker and act like it is an ‘import fee’ but it was really well priced for what I got.

My iSCSI Recommendations

The details of configuring it are pretty dry but I will tell you the points that I found to work the best for iSCSI and so far this also holds true for Cisco switches I’ve implemented this on as well:

  • Use a Layer 2 switch for iSCSI
  • Make sure your switch specifically can handle iSCSI traffic.  Some specifically cannot
  • Use a dedicated switch for iSCSI
  • Use a dedicated VLAN and subnet for iSCSI
  • Turn off QoS on the iSCSI switch ports
  • Turn STP to ‘Port Fast’ on the iSCSI switch ports and set it to rSTP on the entire switch if you can
  • Turn on FlowControl on the iSCSI switch ports
  • Turn off Storm Control on the iSCSI switch ports
  • Use MPIO if you can, but if you cannot complete MPIO all the way, then don’t use it.  I’ll make another post about this in the future

My Defense

I’d like to take a chance to explain some of my points above in case anyone else is doing research and comes across this.

Layer 2: I’d say this is best because a Layer 3 switch can route and you do not want that.  If iSCSI packets leave your subnet then iSCSI will fail.  My going Layer 2 only, you are putting less load on the switch by not having it worry about the Layer 3 part and that leaves you more horsepower to handle the packets.

QoS Off:  QoS is short of Quality of Service and in network switches, this means that the switch has to inspect all packets coming into any port, tag them based on their content, then add extra decisions about routing them based on this.  It usually will cause lots of slowness and disconnections for iSCSI.

STP:  This stands for Spanning Tree Protocol and when done right it can be a networking person’s best friend, but it detect when you unplug and plug in any port and once it detects that it can take up to 50 seconds to re-detect everything and your ports will be down that entire time.  Since iSCSI has ways of tripping this port detection it means disconnections for you.  Setting STP to ‘Port Fast’ on your iSCSI ports will tell it not to do that scan that takes up to 50 seconds.

StormControl: This is a feature of the switch to detect when something is sending out a storm of packets and block it for a short time or until the storm goes away.  By blocking it of course I mean drop all packets from that source and of course it normally mistakes iSCSI’s bursts for packet storms so this will cause random disconnects during production data spikes.

FlowControl: This is one of the few Ethernet Switch features that can actually help iSCSI.  Basically if this is turn on on the switch, the iSCSI Initiator(s) [The iSCSI NIC(s) on the VM hosts] and on the NAS/SAN (Storage) then what happens is the switch will eventually fill up its queue for packets on an iSCSI port and will send back a ‘pause frame’ which is a special packet of data that asks the sender to pause sending nicely while it tries to deal with the load it has, and then is sends a ‘resume frame’.  This method replaces the usual one of just dropping packets when the port’s queue is full and will present as slowness instead of disconnections when your iSCSI overloads the switch.

Following all of this, you still need to monitor the resources on your switch to make sure you are not overloading it and if you are then you may want to consider MPIO to help with that.  MPIO is short for MultiPathing and really it means setting up multiple paths for the iSCSI data to travel between the Storage and VM Host.  Since this can take the form of multiple NICs on both the VM Host and Storage and in fact going through two different switches with two different subnets, then you would be splitting up the switch work between two or more switches and then better handle it.

Virtualization

Vyatta Core 6.5 Software Router VM on ESXi 5.1

In keeping with my previous posts, I should have posted about running Vyatta Core 6.5 Software Router VM on ESXi 5.1 about two months ago however, I was not able to at the time so I am publishing it now.vyatta

 

 My Router History

Having gone through a progression of routers for my home I think I should first list them out so that the increasing need can be understood.

  1. At first I started with a basic home router years ago before WiFi was even an option.  My first home router was actually a Linksys model so old it did not come with a switch at all.
  2. My second home router was one with Wifi and a switch built in and for the two computers I had it worked well.  Thing is I began to grow my army of computers I kept at home and eventually I ran into difficulties with my home router simply because of built in firmware was not capable of handling as many connections as I wanted to keep open.  I was slowly doing more with my home computers; running Web Sites, MUDs (google it), an FTP site, and some other things.
  3. Eventually I was talking with a colleague at work who had taken his Linksys router and replaced the firmware with one called OpenWRT.  Having checked it out, this firmware replacement for home routers offers to completely remove the Web GUI and allows you to only access it through SSH, and while this is secure it was not what I was looking for, but it did lead me to find DD-WRT which similarly offers an improved firmware with far less issues then stock firmware and far more features, but it does so through a Web Interface but still offers telnet and/or SSH access if you want.  It allows you to take a decent router like the Linksys WRT54GL and turn it into a powerful processing device plus a good WiFi AP and a great router.  So I went forward with this and for a few years I was quite happy using this.  With DD-WRT I was able to keep up with what few constant servers I was running but still had all the functionality I wanted out of my home computers.
  4. Then I began to get serious about things like wanting to be able to QoS my connection and provide the best possible experience for my VoIP packets but still allow streaming data to flow well and this was just too much for a small Broadcom CPU running at 200 MHz with 8 MB of RAM to handle on top of VLAN managing, the bi-directional NAT, the Firewall, the WiFi AP, and everything else I was asking of it, so I resolved to upgrade to a more professional router, like a corporation would use.  I already had tried pfSence while setting up VoIP for another company and I disliked it, so I went with a VM running SmoothWall Express 3.0 Polar.  In order to achieve this I needed my ESXi tower to have a second Network Card (NIC) and a second virtual switch, as illustrated by this image.VM Router Networking
  5. With SmoothWall Express installed and running I was able to get a Linksys E1550 router and with DD-WRT running on it I am able to use it to run my VPN server and a three SSID Wireless Access Point.  This means that I have a main WiFi with access to everything, a guess WiFi which is firewalled to only be able to access the internet and not anything else on my network, and a third WiFi which uses WEP for a few older devices which still need it.  The SmoothWall Express 3.0 was running fine until I can across something it could not do and that was route.  It makes a great Firewall and Gateway but it does not have the ability to write different routes.  Being that I used my Linksys E1550 with DD-WRT on it as a VPN server, I wanted the entire network to have a few extra routes in the routing table of the Gateway so that it could route packets for the subnets my VPN clients had to the VPN itself effectively creating a VPN tunnel between sites with my site as the main point.

Vyatta Core 6.5 Software Router VM on ESXi 5.1

At this point I came across Vyatta Core which is a free version of a paid Software Firewall so in one respect it is like SmoothWall Express and pfSence, but it goes much further.  In SmoothWall Express and pfSence for example you do about 99% of your configuration through the Web Interface and the console or SSH is saved for installation and hacking, where as in Vyatta the Web Interface is saved for the paid version only and the free version is configured entirely by SSH which works in a similar way to the Cisco interface over Telnet.

As it turned out it was actually a little challenging to wrap my head around how Vyatta handles everything but now I understand how much more powerful things are when you have to go through such effort to customize them in the first place.

I scoured the internet for articles that seemed somewhat relevant and it was not easy to find as a few articles seemed to start off well through the process of configuring a Vyatta system and then were never finished.  I did manage to assemble the list here for anyone else who wants to try it for themselves.

Virtualization, VoIP

FreePBX 2.11 on ESXi 5.1

The Power Out of 2013

Have been in Toronto for so many years it did not come as much of a surprise when the power was out for a week at the very end of 2013. Having most of my servers on an ESXi tower made things easy to restart.

VMware ESXi

As soon as the power came back online my modem and switches kicked in and so did my UPS and ESXi tower and within 5 minutes the first virtual machine (VM) was booting up. Being my router I considered my home online a few minutes later. Within a half hour, both of my domain controllers, my WSUS, my Ubuntu web server, and a few other back end VMs where online and for the most part I was good again but still my FreePBX VoIP server was offline, since I was not home to power it on. The downside to having dedicated physical servers keeps piling up and with access to virtualization technologies like ESXi 5.1 I saw no reason why my FreePBX server could not work as a VM as well.

Researchfreepbx

Having done some googling I found lots of post talking about the downside of virtualizing a phone server and how future virtual servers will be able to handle ‘real-time’ applications like FreePBX, but I did notice these had all been written over a year before.

So I went on the FreePBX IRC channel, #freepbx on freenode, and spoke to the chat room full of people who work on FreePBX all the time and even from them I got mixed results. Some of them said ‘soon maybe’ while others gave a very firm ‘I wouldn’t’.

Throwing all of this aside it decided to give it a shot. After all I actually had nothing to loose but my time. I figured that I could make the virtual server with a different internal IP address and hostname, and it would not cause any issues. The reason for my assumption was that it was all still based on IP address still. The phones registered to a specific IP/port and the firewall only forwarded the SIP and RTP packets to the internal IP I tell it to. So as long as I make it with something different and manually change it later, then ‘no harm, no foul.’

Installing FreePBX 2.11 on ESXi 5.1

So I made an empty VM and downloaded the latest 64-bit distro ISO of FreePBX which was v2.11 on CentOS 6.4.

I installed it as usual on Asterisk 11 (a choice when you boot to the ISO) and it installed without issue.  After some checking I even found that some of the minor issues, like the CDR not working out of the box, that I had seen in previous releases were not an issue here and everything seemed to work out of the box.  Of source I knew full well that the real problems that would come from virtualizing a Real-Time application like a VoIP server would come from performance related factors and so would not show up until I tried to use it.

I continued the installation by adding Webmin to the mix on this installation just as it was on my previous FreePBX 2.10 on an Acer mini-PC and so I was able to have the FreePBX Web GUI, the CLI from having SSH access, and the Webmin Web GUI.

With everything up at the same time it was only a matter of copying over settings like the extensions, trunks, inbound routes, and outbound routes, plus my announcements, and so on.

* I did try twice actually to take a backup of my FreePBX 2.10 and restore it on my new 2.11 but this resulted in restoring 2.10 entirely as the backup is really just the DB and the files so it grabs everything and is good only for a lost system and not for bringing settings over.  In both cases I had to format and re-install in order to undo the damage done by restoring.

Testing

Once my FreePBX settings, backup schedule and so on where setup I licensed the server with Schmoozecom and began to test it.  I waiting for a weekend.  One Sunday I powered down the physical FreePBX 2.10 server and told FreePBX to static change it’s IP to the same as the old server, then rebooted and waited.  To my surprise it seemed to work without issue.

I tested the inbound and outbound calling from all the lines and everything seemed to work fine, except on of my Linksys phones, an SPA942, refused to work right and just kept showing ‘~PLEASE WAIT~” on its display screen.  Knowing what I had setup I went to the phone itself and told it to restore to factory settings.  After a minutes the phone booted up and found the DHCP address waiting for it.  The phone then checked and found the DHCP contained a TFTP server’s IP address as well and went there looking for specific files.  Since I had setup my FreePBX 2.11 with a TFTP server ( a manually installed module) and told the DHCP server to use the FreePBX server’s IP for the TFTP server it made programming the phone easy, since I just needed to have the configuration files already there.  I also found that by using something called the OSS Endpoint Manager, FreePBX 2.10 or 2.11 configured the phone’s configuration files for me.   So the phone booted up, got an IP from DHCP and found the TFTP server with the files that the Endpoint Manager made for me.  The phone rebooted after a minute and came back up showing what I wanted it to say on its display screen and I knew that it had taken the first configuration file.  After another few minutes the phone rebooted again and came back up this time with the lines I wanted it to have already configured and in order.  I tested the phone and it seemed to be working perfectly.

To my surprise, this worked for my Cisco 7960 and my Linksys ATA device as well.

* This also means that if in the future I need to change an extension’s password (“secret”) then once I do I just tell the Endpoint Manager to rebuild it’s configuration file and reboot the phone and it will pick up the new settings all on its own.

Results

So far it has been a business week and I have not seen any issues with my FreePBX 2.11 running on my ESXi 5.1 tower.  All these advances in phone technology are actually starting to make things easier.