In previous versions of Team Foundation Server there was not a way to install and configure TFS to be run in a completely high available environment. TFS 2008 supports the data tier running in a SQL Server Cluster. If the TFS application tier server crashed, there could be a “warm standby” configured to take over but required a manual process to do this. TFS 2010 supports running multiple Application tier servers using Network Load Balancing (NLB). Last year I first heard of topology improvements in TFS 2010 in Brian Harry’s post about the Administrator, Operations,and Setup Improvements in Team Foundation Server 2010. I think this is an extremely important feature as companies are utilizing more features in Team Foundation Server and expecting these services to always be available.
In this post I am going to explain:
- Configuring the second TFS application tier
- Enabling and configuring Network Load Balancing in Windows Server 2008 R2
- Testing TFS using the NLB Application tier severs
- Lessons learned
Team Foundation Server 2010 Farm Network Topology Diagram
This is a diagram shows the topology of the TFS configuration I created to demonstrate the NLB option.
I set up this environment on a laptop. I installed Windows Server 2008 R2 x64 on the laptop as the host OS to utilize Hyper-V. I added Active Directory Domain Services (ADDS) and DNS to this machine and configured it as the domain controller. I installed SQL Server 2008 R2 x64 on it also to be the data tier for the TFS installation. Both TFS Application Tier Servers were created as Hyper-V virtual machines with Windows 2008 R2 x64. To make this configuration 100% redundant I would have needed to install the reporting services on both of the TFS Servers, used a SharePoint 2007 Farm for the portal, and a SQL Server cluster for the data tier. In fact, now that I have all of this working, I am going to uninstall everything and try to get WSS and SSRS installed on both application tiers also utilizing the NLB. This would allow for 100% redundant and high availability for all of the TFS 2010 Components with only 4 servers (creating a 2 server SQL Server Cluster). I will post a follow up on how this goes..
Team Foundation Server Installation
The TFS installation and configuration for the first server (TFS2010A) was done just as if it was going to be the only server. Here are the settings after I installed the first server.
For the second server (TFS2010B), the only pre-requisites that were required were IIS 7 and SQL Client connectivity tools. Below are the steps for configuring the second TFS Server.
In the TFS configuration, choose the “Application-Tier Only” installation option.
The first step was the Welcome step. There were no options on this step. The next step is to specify the configuration database that was created when the first server was configured.
Next, specify the service account for the the new application tier. I chose to use the same domain account that I used for the first server.
This screen shows the summary of the settings that were chosen before the verification is run.
After the first time I ran the verification I received two errors. One was that .Net 3.5sp1 wasn’t installed. I’m not sure why I reported this error. When I checked it, it was installed. It could be related to other error. The other error, TF255040, was that I didn’t have Reporting Services or SQL Server Connectivity tools installed.
I installed the connectivity tools and reran the verification process and it passed this time.
The configuration completed successfully.
The TFS administrative console now shows both application tier servers.
Setting up Network Load Balancing in Windows 2008 R2 and Hyper-V
Now that both servers are installed and configured correctly, the next step is to set up Network Load Balancing (NLB). This will allows users to connect to TFS through a single endpoint and allow NLB to balance the traffic and route all of the traffic to one server if the other is unavailable. This provides high availability in the event of an outage or when the servers need to be updated.
The first thing to do before setting up NLB is to pick a static IP address and create a DNS (A) Record for the shared name. In this example, TFS2010 is the endpoint that clients such as Visual Studio 2010. Here is a snapshot of the A records.
Here is an article I used to help enable and configure NLB in Windows Server 2008 R2. Below are the steps I performed to configure it.
http://technet.microsoft.com/en-us/library/cc731695.aspx
For each TFS Application Tier server, install Network Load Balancing by going to Server Manager > Features > Add Features > Network Load Balancing
Once NLB is installed on all of the App Tiers, run the Network Load Balancing Manager by typing nlbmgr at the command prompt. Then connect to one of the hosts. I chose TFS2010a first. Right click on the Network Load Balance node and choose “Add New Cluster”. Add the current server to the node by walking through the wizard. You should be able to leave the defaults unless you want to limit the NLB to just port 8080. The last step of the wizard is to assign the Cluster a shared IP Address. This IP Address is how all of the clients will access and see it. Make sure this is different than the IP addresses of the any of the nodes in the cluster.
Next, add the other host to the cluster by right clicking on the cluster and choosing “Add Host to Cluster”. Enter the name of the Host to be added to the cluster.
Here is what the Cluster looks like when it is configured.
Connect to TFS from Visual Studio 2010
Testing the high availability servers
To test the high availability configuration, I am going to take down one of the TFS application tier servers and then both. First I created the team project while both servers were available.
Then I created a C# windows project, checked it in, and then checked out one of the files.
Next, I disabled the NIC on TFS2010a.
I did a Undo checkout on the file
NLB rerouted the traffic to TFS2010B and it worked perfect.
Next, I Disabled the NIC on TFS2010b so now both are disabled and it should error.
Tried to check out a file and got a TFS not available error as expected.
Last, I re-enabled the NIC on TFS2010A
I performed the check out again and worked perfect.
The NLB worked as expected.
Lessons Learned
I’m a developer. I started my IT career as a server and desktop administrator but that was a long time ago. So setting up a domain to test this scenario was fun but I ran into a few unexpected problems. Here are a couple things that slowed me down.
- Installed DHCP and DNS with dynamic IP. Make sure you assign a static IP address to the virtual internal network card on the DC. Actually now that I have assigned static IP addresses to both of the TFS Servers, DHCP doesn’t even need to be installed.
- I couldn’t reach SQL Server 2008 R2 from the TFS Servers. I got an TF255049 error in the TFS configuration. I installed the SQL Server Management Studio and couldn’t connect using that. I set up aliases and toggled named pipes on and off. Still couldn’t connect. Finally I stumbled upon the network protocol configuration for SQL. TCP and Named Pipes were disabled. Seems very strange but probably a security precaution. I enabled TCP/IP and it worked perfect.
NLB Issue
I ran into an issue trying to configure NLB on Windows Server 2008 R2 using Hyper-V
“The interface is misconfigured”
Cluster IP address (IP) not added to TCPIP properties
Dedicated IP address (IP) not added to TCPIP properties
Fix is to enable MAC spoofing in the settings of each VM in Hyper-V.
Enjoy!
Mike