In previous versions of Team Foundation Server there was not a way to install and configure TFS to be run in a completely high available environment.  TFS 2008 supports the data tier running in a SQL Server Cluster.  If the TFS application tier server crashed, there could be a “warm standby” configured to take over but required a manual process to do this.   TFS 2010 supports running multiple Application tier servers using Network Load Balancing (NLB).  Last year I first heard of topology improvements in TFS 2010 in Brian Harry’s post about the Administrator, Operations,and Setup Improvements in Team Foundation Server 2010.  I think this is an extremely important feature as companies are utilizing more features in Team Foundation Server and expecting these services to always be available.

In this post I am going to explain:

• Configuring the second TFS application tier
• Enabling and configuring Network Load Balancing in Windows Server 2008 R2
• Testing TFS using the NLB Application tier severs
• Lessons learned

Team Foundation Server 2010 Farm Network Topology Diagram

This is a diagram shows the topology of the TFS configuration I created to demonstrate the NLB option.

I set up this environment on a laptop.  I installed Windows Server 2008 R2 x64 on the laptop as the host OS to utilize Hyper-V.   I added Active Directory Domain Services (ADDS) and DNS to this machine and configured it as the domain controller.  I installed SQL Server 2008 R2 x64 on it also to be the data tier for the TFS installation.  Both TFS Application Tier Servers were created as Hyper-V virtual machines with Windows 2008 R2 x64.  To make this configuration 100% redundant I would have needed to install the reporting services on both of the TFS Servers, used a SharePoint 2007 Farm for the portal, and a SQL Server cluster for the data tier.  In fact, now that I have all of this working, I am going to uninstall everything and try to get WSS and SSRS installed on both application tiers also utilizing the NLB.  This would allow for 100% redundant and high availability for all of the TFS 2010 Components with only 4 servers (creating a 2 server SQL Server Cluster). I will post a follow up on how this goes..

Team Foundation Server Installation

The TFS installation and configuration for the first server (TFS2010A) was done just as if it was going to be the only server.  Here are the settings after I installed the first server.

For the second server (TFS2010B), the only pre-requisites that were required were IIS 7 and SQL Client connectivity tools.    Below are the steps for configuring the second TFS Server.

In the TFS configuration, choose the “Application-Tier Only” installation option.

The first step was the Welcome step.  There were no options on this step.  The next step is to specify the configuration database that was created when the first server was configured. 

Next, specify the service account for the the new application tier.  I chose to use the same domain account that I used for the first server.

This screen shows the summary of the settings that were chosen before the verification is run.

After the first time I ran the verification I received two errors.  One was that .Net 3.5sp1 wasn’t installed.  I’m not sure why I reported this error.  When I checked it, it was installed.  It could be related to other error.  The other error, TF255040, was that I didn’t have Reporting Services or SQL Server Connectivity tools installed.

I installed the connectivity tools and reran the verification process and it passed this time.

The configuration completed successfully.

The TFS administrative console now shows both application tier servers.

Setting up Network Load Balancing in Windows 2008 R2 and Hyper-V

Now that both servers are installed and configured correctly, the next step is to set up Network Load Balancing (NLB).  This will allows users to connect to TFS through a single endpoint and allow NLB to balance the traffic and route all of the traffic to one server if the other is unavailable.  This provides high availability in the event of an outage or when the servers need to be updated.

The first thing to do before setting up NLB is to pick a static IP address and create a DNS (A) Record for the shared name.  In this example, TFS2010 is the endpoint that clients such as Visual Studio 2010.  Here is a snapshot of the A records.

Here is an article I used to help enable and configure NLB in Windows Server 2008 R2.  Below are the steps I performed to configure it.
http://technet.microsoft.com/en-us/library/cc731695.aspx

For each TFS Application Tier server, install Network Load Balancing by going to Server Manager > Features > Add Features > Network Load Balancing

Once NLB is installed on all of the App Tiers, run the Network Load Balancing Manager by typing nlbmgr at the command prompt.  Then connect to one of the hosts.  I chose TFS2010a first.  Right click on the Network Load Balance node and choose “Add New Cluster”.  Add the current server to the node by walking through the wizard.  You should be able to leave the defaults unless you want to limit the NLB to just port 8080. The last step of the wizard is to assign the Cluster a shared IP Address.  This IP Address is how all of the clients will access and see it.  Make sure this is different than the IP addresses of the any of the nodes in the cluster.

Next, add the other host to the cluster by right clicking on the cluster and choosing “Add Host to Cluster”.  Enter the name of the Host to be added to the cluster.

Here is what the Cluster looks like when it is configured.

Connect to TFS from Visual Studio 2010

Testing the high availability servers

To test the high availability configuration, I am going to take down one of the TFS application tier servers and then both.  First I created the team project while both servers were available.

Then I created a C# windows project, checked it in, and then checked out one of the files.

Next, I disabled the NIC on TFS2010a.

I did a Undo checkout on the file

NLB rerouted the traffic to TFS2010B and it worked perfect.

Next, I Disabled the NIC on TFS2010b so now both are disabled and it should error.

Tried to check out a file and got a TFS not available error as expected.

Last, I re-enabled the NIC on TFS2010A

I performed the check out again and worked perfect.

The NLB worked as expected.

Lessons Learned

I’m a developer.  I started my IT career as a server and desktop administrator but that was a long time ago.  So setting up a domain to test this scenario was fun but I ran into a few unexpected problems.   Here are a couple things that slowed me down.

• Installed DHCP and DNS with dynamic IP.  Make sure you assign a static IP address to the virtual internal network card on the DC.  Actually now that I have assigned static IP addresses to both of the TFS Servers, DHCP doesn’t even need to be installed.
• I couldn’t reach SQL Server 2008 R2 from the TFS Servers.  I got an TF255049 error in the TFS configuration.  I installed the SQL Server Management Studio and couldn’t connect using that.  I set up aliases and toggled named pipes on and off.  Still couldn’t connect.  Finally I stumbled upon the network protocol configuration for SQL.  TCP and Named Pipes were disabled.   Seems very strange but probably a security precaution.  I enabled TCP/IP and it worked perfect.

NLB Issue

I ran into an issue trying to configure NLB on Windows Server 2008 R2 using Hyper-V

“The interface is misconfigured”

Cluster IP address (IP) not added to TCPIP properties

Dedicated IP address (IP) not added to TCPIP properties

Fix is to enable MAC spoofing in the settings of each VM in Hyper-V.

Enjoy!

Mike

We have a lot of exciting things going on at Deliveron Consulting Services. I wanted to post a few of the updates.

Get Social with Deliveron

  Become a Facebook fan of Deliveron: http://www.facebook.com/pages/Deliveron-Consulting-Services/92712059791

  Follow Deliveron on Twitter: http://www.twitter.com/deliveron

   Join the LinkedIn Deliveron Consulting Services Network group.

Deliveron Website Updates

We have implemented a few website updates.  These include:

• Deliveron Jumpstart Solutions including the TFS Jumpstart information and flyer. (I am very excited about this!!)
• “The Deliveron Story”
• Areas of Expertise
• Our Partners and Memberships

View the first edition of the Deliveron Monthly Newsletter.  Sign up on the website to receive future newsletters and announcements.

Upcoming Events

2/3 – Omaha SQL/BI User Group - Solution Consultant Moe Elatta from Deliveron will be presenting on SQL Server Reporting Services 2008 scale-out configuration and new charting enhancements. Register here...

2/18 - MSDN Webinar - Enhancing the Business Process Automation Capabilities of SharePointRegister here...

2/25 - Lunch and Learn - SharePoint Features You NeedRegister here...

I wanted to take a look at writing SQL Reports for TFS.  I was going to install the Business Intelligence Studio for SQL Server 2005 but I thought it would be better to use the latest edition of SQL Reporting Services in SQL Server 2008.   So I decided to upgrade my TFS Test Server to SQL Server 2008.  This is a single server install of TFS.   I ran the SQL Server 2008 Upgrade Wizard.     Most of the steps were straight-forward.  I only ran into one roadblock that I was able to fix and I documented below.  Here’s the steps.

First I chose the “Upgrade from SQL Server 2000 or SQL Server 2005” option.

I kept the default settings for all of the steps except when it asked what instance I would like to upgrade.

All of the checks passed but one.  I got the following error.

Rule "Security Group SID (Security Identifier)" failed.

http://go.microsoft.com/fwlink/?LinkId=94001

I went to the link but it didn’t help.  I searched on the web for for the error and I found the fix here.  Basically I had to manually replace the invalid SIDs with the new one.

To find the correct SIDs, I ran the command prompt utility:  whoami /groups

If you have multiple editions of SQL Server installed make sure you are in the correct one in the registry.  Look at the Edition key

I pasted the correct SIDs into the registry keys

I fixed the ones in MSSQL.2 but I was still getting the error.  I checked the log file %ProgramFiles%\Microsoft SQL Server\100\Setup Bootstrap\LOG\Detail.txt and here is the error.

2009-04-14 23:24:25 Slp: Sco: Attempting to open registry subkey SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.3\Setup
2009-04-14 23:24:25 Slp: SqlSidRule.GetSidValue - reading value name ASGroup
2009-04-14 23:24:25 Slp: Sco: Attempting to get registry value ASGroup
2009-04-14 23:24:25 Slp: Sco: Attempting to get account from sid S-1-5-21-3621347759-1475631895-1437307089-1014
2009-04-14 23:24:25 Slp: Sco: LookupAccountSid failed for SID S-1-5-21-3621347759-1475631895-1437307089-1014
2009-04-14 23:24:25 Slp: SqlSidRule.EnsureGroupExists - Exception caught and ignored. Returning false. Exception message is No mapping between account names and security IDs was done.

I grabbed the SID for the ASUser and it worked!

The wizard continued and finished.  I launched Visual Studio and TFS is working properly.

Now I can look at the SQL Reporting :)