iSCSI, Virtualization

Best Dell Gigabit iSCSI Switch

In my search to improve my Virtualization, that is my Virtual Server Infrastructure, I can to a point where local storage (local disks/drives) just would not cut it anymore.  They were too slow, the RAID required for ESX(i) to work with was too expensive for my budget, and I did not want to deal with the headaches that came with maintaining them on local servers, so I started to look around, google, and do my research and it turned out that iSCSI seems to be the current market winner.  That is that iSCSI has captivated the market with its promise of a cheaper storage that supposedly can compare to FiberChannel in some respects.  With this in mind I bought the Best Dell Gigabit iSCSI Switch I could find in my price range, the Dell PowerConnect 3448.

My Trials

D-Link DGS-1024D
D-Link DGS-1024D

I first tried to use an Unmanaged Switch, the D-Link DGS-1024D to try to handle my iSCSI connections but I eventually found some settings on both ends of the connection that ended up pushing too much through that switch and it kept dying.  It was not the Best Gigabit iSCSI Switch like I had hoped.

Then I thought that the Best Gigabit iSCSI Switch was going to be too expensive for my budget, so I tried running Crossover Gigabit Ethernet cables directly from my Virtual Hosts to my iSCSI Storage and it worked better.  I started to be able to push through more data without having a switch to overload but this did not account for bursting and I soon ran into limitations of the straight NICs were they dropped too many packets for a stable connection.  Clearly I needed an Ethernet Switch in the middle, it had to be Gigabit, I wanted the Best in my price range, and my research revealed that Dell’s iSCSI implementation, meaning the code their switches use to handle that kind of traffic, was their own, where as HP had used Cisco’s and a good number of reviewed said that Dell’s iSCSI worked better then Cisco’s version, so the hunt for the Best Dell Gigabit iSCSI Switch I could afford was back on.

Dell PowerConnect 5448
Dell PowerConnect 5448

After more and more research, I finally found the Best Dell Gigabit iSCSI Switch and it turned out to be the Dell PowerConnect 3448.  Having checked the retail and wholesales prices I noticed that you cannot get one new anymore so I resorted to EBay and ended up coming in under $200 with the shipping costs.  There was the usual extra $65 charge from FedEx to pay their broker and act like it is an ‘import fee’ but it was really well priced for what I got.

My iSCSI Recommendations

The details of configuring it are pretty dry but I will tell you the points that I found to work the best for iSCSI and so far this also holds true for Cisco switches I’ve implemented this on as well:

  • Use a Layer 2 switch for iSCSI
  • Make sure your switch specifically can handle iSCSI traffic.  Some specifically cannot
  • Use a dedicated switch for iSCSI
  • Use a dedicated VLAN and subnet for iSCSI
  • Turn off QoS on the iSCSI switch ports
  • Turn STP to ‘Port Fast’ on the iSCSI switch ports and set it to rSTP on the entire switch if you can
  • Turn on FlowControl on the iSCSI switch ports
  • Turn off Storm Control on the iSCSI switch ports
  • Use MPIO if you can, but if you cannot complete MPIO all the way, then don’t use it.  I’ll make another post about this in the future

My Defense

I’d like to take a chance to explain some of my points above in case anyone else is doing research and comes across this.

Layer 2: I’d say this is best because a Layer 3 switch can route and you do not want that.  If iSCSI packets leave your subnet then iSCSI will fail.  My going Layer 2 only, you are putting less load on the switch by not having it worry about the Layer 3 part and that leaves you more horsepower to handle the packets.

QoS Off:  QoS is short of Quality of Service and in network switches, this means that the switch has to inspect all packets coming into any port, tag them based on their content, then add extra decisions about routing them based on this.  It usually will cause lots of slowness and disconnections for iSCSI.

STP:  This stands for Spanning Tree Protocol and when done right it can be a networking person’s best friend, but it detect when you unplug and plug in any port and once it detects that it can take up to 50 seconds to re-detect everything and your ports will be down that entire time.  Since iSCSI has ways of tripping this port detection it means disconnections for you.  Setting STP to ‘Port Fast’ on your iSCSI ports will tell it not to do that scan that takes up to 50 seconds.

StormControl: This is a feature of the switch to detect when something is sending out a storm of packets and block it for a short time or until the storm goes away.  By blocking it of course I mean drop all packets from that source and of course it normally mistakes iSCSI’s bursts for packet storms so this will cause random disconnects during production data spikes.

FlowControl: This is one of the few Ethernet Switch features that can actually help iSCSI.  Basically if this is turn on on the switch, the iSCSI Initiator(s) [The iSCSI NIC(s) on the VM hosts] and on the NAS/SAN (Storage) then what happens is the switch will eventually fill up its queue for packets on an iSCSI port and will send back a ‘pause frame’ which is a special packet of data that asks the sender to pause sending nicely while it tries to deal with the load it has, and then is sends a ‘resume frame’.  This method replaces the usual one of just dropping packets when the port’s queue is full and will present as slowness instead of disconnections when your iSCSI overloads the switch.

Following all of this, you still need to monitor the resources on your switch to make sure you are not overloading it and if you are then you may want to consider MPIO to help with that.  MPIO is short for MultiPathing and really it means setting up multiple paths for the iSCSI data to travel between the Storage and VM Host.  Since this can take the form of multiple NICs on both the VM Host and Storage and in fact going through two different switches with two different subnets, then you would be splitting up the switch work between two or more switches and then better handle it.

Security

Spam Email Filtering

I know it has been a while since I got down to blogging here, and it really has been quite busy for me, but as it turns out there is a lot a blog can do for you, so I am back.  This time I am here to talk about Spam Email Filtering.

As I’m sure many of you already know, spam Emails are the bane of anyone’s life who checks their Emails regularly.  I may not mention it here but I have almost a dozen different Email accounts by now and though most of them do route to my primary Gmail account, which makes things so much easier, I still have some which I handle myself and that means dealing with spam emails and having to filter them.

In case you don’t know, spam Emails are a way to make money these days.  Perfectly normal companies are never happy with how much money they make so they are always looking for ways to get advertising out to ever more people and some even resort to paying for spam Email services.  This means that they pay someone to figure out what your Email address(es) is/are and send you their advertising Emails in a way that will not be blocked.  Technically this is considered hacking but because it pays the bills, no one wants to call it that.

As annoying as the advertisements are, they are still not as bad as those random spam Emails that come from real hackers trying to steal your information, push you to a phishing site, or infect your computer system with something malicious.   I’ll get into Malware in a later blog, promise.

This time I just wanted to give you a basic understand of just how much we IT pros struggle with spam Emails and how much they really do annoy us too.

I plan to help fight them on this domain and a few others by adding to what I’ve already done.  So far I am using spamassassin along with a host of other tools which help to filter over 70% of what my domains really receive but it doesn’t seem to be enough anymore, so I am going to try to add another layer by using a free service called MX Guard Dog.  This is that they don’t ask for money, they trade in credits and for simply posting the following link, I get some credits.  How many really depends on how often people visit my site (but I’m told not how often people click the link).  So I am not saying click the link, I just want to inform you and by reading this blog, you are helping to support my efforts to filter spam.

spam email

Click on the link if you want, up to you.

Cloud

Cloud Email is the Future

Cloud Email is The Future

With every day the world turns and the world evolves and moves forward.  With every day, we advance technology, we move from having a phone in every home to everyone having cell phones and switching land lines to VoIP, with mail switching to Email and communication getting faster, more electronic and more important.

Email Changed the Worldcloud email is the future

When the Internet was new and fresh it was used mainly for chat rooms and later evolved the web browser and web pages and eventually electronic versions of Post/Mail came about.  We called it Email because we wanted to use it just like mail and these days that is what we see everywhere.  People chat by Email now at work more then phone calls, chat programs and meetings.  And every office meeting, phone call or chat is followed up by a summary Email that spawns an entirely new Email thread that continues the discussion and summarizes everything that was discussed.  At home we use Email to receive invitations to events (Facebook and Evite), we use it to find out who get new jobs (LinkedIn), we use it to receive our bills (Epost).  We also use it to send our friends and family funny cat pictures and updates about the kids.

Fast Mobile Email

We don’t even wait anymore to get to work or home to get our Emails.  We receive them right on our smartphones.  It is common now to see your average office job worker with a Blackberry from work and a smartphone of their own.  So it is common to carry at least two Emails with you at all times so you can always be within 15 minutes of your Emails, because they are that important these days.

Cloud Email is the Future

With Emails being as important as they are, not many people are noticing that their current Email providers are not really keeping up with the world.  Having lived in Canada for so many years it was common to see two kinds of Email addresses by friends and family would have.  The @rogers.com or @sympatico.ca Email accounts are common because they are free.  When you buy internet at home, your ISP (Internet Service Provider) normally gives you a few Email accounts for free on their own servers because it doesn’t really cost them anything but if they did not offer this you would not take them.  What people don’t notice is these kinds of Email accounts are very old.  What I mean is the world and the Internet are changing, and Emails tied to your ISP and using their old Email technology are just not enough anymore.  Cloud Email is the Future.

 What is Cloud Email and why is it the Future ?

What is Cloud Email and why Cloud Email is the Future?  Well Cloud Email is when someone else provides you the service of hosting your Emails for you and allows you to access them from anywhere at anytime without restrictions.

You may ask if and how this is different from the traditional Email services you are used to getting from your ISP.  Well Cloud Email also means that an Email Client (i.e. Microsoft Outlook, Mozilla ThunderBird, Apple Mail) is not required in order to access these.

Apple

 

Even though iPhones, iPad and Mac computers all come with an Email Client the free Cloud Email service provided by Apple (@me.com) does not require you to use an Email Client, though most still do.  When you get any Apple device and you start to install apps on it you will find you need an Apple account and Apple is happy to provide you a free @me.com Email address to go along with that but you may not be aware that you can go to the iCloud website (www.icloud.com) and sign in with your credentials and do everything from there.  This allows you to have access to your Contact, your Emails and a few other things Apple provides there right on your desktop and without you having to pull out your smartphone or running it inside of a Client application, just a secure web site will do.

Hotmail

Hotmail is one of the oldest Cloud Email providers.  What they offer is a free Email address which you can get access to either by correctly setting up an Email Client or by just visiting their web site www.hotmail.com.   There are of course limitations like a maximum amount of storage space for your Emails and a maximum size for each sent Email but these are generally higher limits then the Email accounts your ISP provides and you will not loose the account if you change Internet providers or move countries.

Gmail

If you need more space and larger Emails, you can always go with the Google solution to Cloud Email, Gmail.  A Gmail account is free and easy to get (once you provide you are a real person).  It can be accessed from an Email Client but they actually prefer that you use their web site.  It gives you a much larger storage space for your Emails (15GB currently) and allow for Emails and attachments larger then the standard 10 MB limit.  In fact the limit is 1 GB per Email (including all Email text and code).  When Gmail first came out the majority of people who got one were in fact using it to store and/or transfer larger files because of these increased limits but I find it means I do not have to be so strict on deleting my Emails.

One of my favorite advantages to using Gmail (my personal preference) is that the Spam filter in Gmail is basically the best Email Spam Filter I have ever seen.  It gives you control enough to say an Email is or is not Spam and it will learn from what you select but the automatic selections are better then I could have imagined.

One of my other favorite features of Gmail is the fact that I have configured my Gmail account to check my 9 (that’s right 9) other Email accounts every hour for new Emails, pulls them down, filters them for Spam, then marks them as having come from that other Email address but still puts them in my Inbox.  I like to call it a Unified Inbox because it means that my 10 Email addresses show up in a single Inbox and that helps me beyond belief.  Thing is, on top of checking it also allows me to setup each Email address so I can send out as that Email address too.  It basically behaves like an intelligent Email Client so I don’t need any other.  Another advantage of this is that on my Android phone, I only need to connect my Gmail account to my Gmail app and all my Emails appear there thanks to all the backend work by Gmail, saving me having to keep 10 Email accounts running and finishing my battery.

Virtualization

Vyatta Core 6.5 Software Router VM on ESXi 5.1

In keeping with my previous posts, I should have posted about running Vyatta Core 6.5 Software Router VM on ESXi 5.1 about two months ago however, I was not able to at the time so I am publishing it now.vyatta

 

 My Router History

Having gone through a progression of routers for my home I think I should first list them out so that the increasing need can be understood.

  1. At first I started with a basic home router years ago before WiFi was even an option.  My first home router was actually a Linksys model so old it did not come with a switch at all.
  2. My second home router was one with Wifi and a switch built in and for the two computers I had it worked well.  Thing is I began to grow my army of computers I kept at home and eventually I ran into difficulties with my home router simply because of built in firmware was not capable of handling as many connections as I wanted to keep open.  I was slowly doing more with my home computers; running Web Sites, MUDs (google it), an FTP site, and some other things.
  3. Eventually I was talking with a colleague at work who had taken his Linksys router and replaced the firmware with one called OpenWRT.  Having checked it out, this firmware replacement for home routers offers to completely remove the Web GUI and allows you to only access it through SSH, and while this is secure it was not what I was looking for, but it did lead me to find DD-WRT which similarly offers an improved firmware with far less issues then stock firmware and far more features, but it does so through a Web Interface but still offers telnet and/or SSH access if you want.  It allows you to take a decent router like the Linksys WRT54GL and turn it into a powerful processing device plus a good WiFi AP and a great router.  So I went forward with this and for a few years I was quite happy using this.  With DD-WRT I was able to keep up with what few constant servers I was running but still had all the functionality I wanted out of my home computers.
  4. Then I began to get serious about things like wanting to be able to QoS my connection and provide the best possible experience for my VoIP packets but still allow streaming data to flow well and this was just too much for a small Broadcom CPU running at 200 MHz with 8 MB of RAM to handle on top of VLAN managing, the bi-directional NAT, the Firewall, the WiFi AP, and everything else I was asking of it, so I resolved to upgrade to a more professional router, like a corporation would use.  I already had tried pfSence while setting up VoIP for another company and I disliked it, so I went with a VM running SmoothWall Express 3.0 Polar.  In order to achieve this I needed my ESXi tower to have a second Network Card (NIC) and a second virtual switch, as illustrated by this image.VM Router Networking
  5. With SmoothWall Express installed and running I was able to get a Linksys E1550 router and with DD-WRT running on it I am able to use it to run my VPN server and a three SSID Wireless Access Point.  This means that I have a main WiFi with access to everything, a guess WiFi which is firewalled to only be able to access the internet and not anything else on my network, and a third WiFi which uses WEP for a few older devices which still need it.  The SmoothWall Express 3.0 was running fine until I can across something it could not do and that was route.  It makes a great Firewall and Gateway but it does not have the ability to write different routes.  Being that I used my Linksys E1550 with DD-WRT on it as a VPN server, I wanted the entire network to have a few extra routes in the routing table of the Gateway so that it could route packets for the subnets my VPN clients had to the VPN itself effectively creating a VPN tunnel between sites with my site as the main point.

Vyatta Core 6.5 Software Router VM on ESXi 5.1

At this point I came across Vyatta Core which is a free version of a paid Software Firewall so in one respect it is like SmoothWall Express and pfSence, but it goes much further.  In SmoothWall Express and pfSence for example you do about 99% of your configuration through the Web Interface and the console or SSH is saved for installation and hacking, where as in Vyatta the Web Interface is saved for the paid version only and the free version is configured entirely by SSH which works in a similar way to the Cisco interface over Telnet.

As it turned out it was actually a little challenging to wrap my head around how Vyatta handles everything but now I understand how much more powerful things are when you have to go through such effort to customize them in the first place.

I scoured the internet for articles that seemed somewhat relevant and it was not easy to find as a few articles seemed to start off well through the process of configuring a Vyatta system and then were never finished.  I did manage to assemble the list here for anyone else who wants to try it for themselves.

Virtualization, VoIP

FreePBX 2.11 on ESXi 5.1

The Power Out of 2013

Have been in Toronto for so many years it did not come as much of a surprise when the power was out for a week at the very end of 2013. Having most of my servers on an ESXi tower made things easy to restart.

VMware ESXi

As soon as the power came back online my modem and switches kicked in and so did my UPS and ESXi tower and within 5 minutes the first virtual machine (VM) was booting up. Being my router I considered my home online a few minutes later. Within a half hour, both of my domain controllers, my WSUS, my Ubuntu web server, and a few other back end VMs where online and for the most part I was good again but still my FreePBX VoIP server was offline, since I was not home to power it on. The downside to having dedicated physical servers keeps piling up and with access to virtualization technologies like ESXi 5.1 I saw no reason why my FreePBX server could not work as a VM as well.

Researchfreepbx

Having done some googling I found lots of post talking about the downside of virtualizing a phone server and how future virtual servers will be able to handle ‘real-time’ applications like FreePBX, but I did notice these had all been written over a year before.

So I went on the FreePBX IRC channel, #freepbx on freenode, and spoke to the chat room full of people who work on FreePBX all the time and even from them I got mixed results. Some of them said ‘soon maybe’ while others gave a very firm ‘I wouldn’t’.

Throwing all of this aside it decided to give it a shot. After all I actually had nothing to loose but my time. I figured that I could make the virtual server with a different internal IP address and hostname, and it would not cause any issues. The reason for my assumption was that it was all still based on IP address still. The phones registered to a specific IP/port and the firewall only forwarded the SIP and RTP packets to the internal IP I tell it to. So as long as I make it with something different and manually change it later, then ‘no harm, no foul.’

Installing FreePBX 2.11 on ESXi 5.1

So I made an empty VM and downloaded the latest 64-bit distro ISO of FreePBX which was v2.11 on CentOS 6.4.

I installed it as usual on Asterisk 11 (a choice when you boot to the ISO) and it installed without issue.  After some checking I even found that some of the minor issues, like the CDR not working out of the box, that I had seen in previous releases were not an issue here and everything seemed to work out of the box.  Of source I knew full well that the real problems that would come from virtualizing a Real-Time application like a VoIP server would come from performance related factors and so would not show up until I tried to use it.

I continued the installation by adding Webmin to the mix on this installation just as it was on my previous FreePBX 2.10 on an Acer mini-PC and so I was able to have the FreePBX Web GUI, the CLI from having SSH access, and the Webmin Web GUI.

With everything up at the same time it was only a matter of copying over settings like the extensions, trunks, inbound routes, and outbound routes, plus my announcements, and so on.

* I did try twice actually to take a backup of my FreePBX 2.10 and restore it on my new 2.11 but this resulted in restoring 2.10 entirely as the backup is really just the DB and the files so it grabs everything and is good only for a lost system and not for bringing settings over.  In both cases I had to format and re-install in order to undo the damage done by restoring.

Testing

Once my FreePBX settings, backup schedule and so on where setup I licensed the server with Schmoozecom and began to test it.  I waiting for a weekend.  One Sunday I powered down the physical FreePBX 2.10 server and told FreePBX to static change it’s IP to the same as the old server, then rebooted and waited.  To my surprise it seemed to work without issue.

I tested the inbound and outbound calling from all the lines and everything seemed to work fine, except on of my Linksys phones, an SPA942, refused to work right and just kept showing ‘~PLEASE WAIT~” on its display screen.  Knowing what I had setup I went to the phone itself and told it to restore to factory settings.  After a minutes the phone booted up and found the DHCP address waiting for it.  The phone then checked and found the DHCP contained a TFTP server’s IP address as well and went there looking for specific files.  Since I had setup my FreePBX 2.11 with a TFTP server ( a manually installed module) and told the DHCP server to use the FreePBX server’s IP for the TFTP server it made programming the phone easy, since I just needed to have the configuration files already there.  I also found that by using something called the OSS Endpoint Manager, FreePBX 2.10 or 2.11 configured the phone’s configuration files for me.   So the phone booted up, got an IP from DHCP and found the TFTP server with the files that the Endpoint Manager made for me.  The phone rebooted after a minute and came back up showing what I wanted it to say on its display screen and I knew that it had taken the first configuration file.  After another few minutes the phone rebooted again and came back up this time with the lines I wanted it to have already configured and in order.  I tested the phone and it seemed to be working perfectly.

To my surprise, this worked for my Cisco 7960 and my Linksys ATA device as well.

* This also means that if in the future I need to change an extension’s password (“secret”) then once I do I just tell the Endpoint Manager to rebuild it’s configuration file and reboot the phone and it will pick up the new settings all on its own.

Results

So far it has been a business week and I have not seen any issues with my FreePBX 2.11 running on my ESXi 5.1 tower.  All these advances in phone technology are actually starting to make things easier.

Making IT affordable

spam email