70-240 in 15 minutes a week: Windows 2000 IPSec

Sunday Dec 2nd 2001 by ServerWatch Staff

Article 27 in Dan DiNicolo's 70-240 in 15 minutes a week series covers IPSec and Certificate Services in Windows 2000. This includes a look at understanding, using and configuring IPSec policies and related settings, including those found in Group Policy.

by Dan DiNicolo

Welcome to article number 27 in my 70-240 in 15 minutes a week series. This article covers IPSec and Certificate Services in Windows 2000. This includes a look at understanding, using and configuring IPSec policies and related settings, including those found in Group Policy. This article is the second-last in the 70-240 in 15 minutes a week series, and again falls under the networking services portion of the exam.

The material covered in this article includes:

- IPSec Overview
- Configuring IPSec Policies

IPSec Overview

IPSec is a security protocol included in the Windows 2000 TCP/IP stack. Its function is simple - to secure TCP/IP communications within or between networks according to the parameters and rules that you lay out. While the concept itself is simple enough, the actual implementation of policies that will do what you want actually requires a deeper understanding of TCP/IP. Not only must you understand how communication takes place on the network, but you must also be able to translate your security needs into specific explicit policies. That requires knowledge of protocols, port numbers, and more, as well as a plan for how and where you need to encrypt or confirm the integrity of traffic. While it might immediately seem like a good idea to encrypt all traffic on your network, that isn't necessarily practical or a good idea. Understand that you pay a price for encryption in the form of higher CPU utilization. This might not be a big deal for an individual workstation with a high-speed processor, but consider also the load on the server that it handling (and encrypting) multiple simultaneous encrypted connections. Instead, focus on using encryption where required for security purposes, creating a setup that meets the needs of your organization. It will seldom be a one-size-fits-all scenario.

If you are considering using IPSec, you first need to understand a bit about how it works. First of all, IPSec encryption / authentication differs from what you may already be familiar with in terms of security products. Most people have worked or are familiar with application-level encryption, where a program (such as PGP or similar) actually handles the encryption/decryption/signing processes. In contrast, IPSec actually resides in the protocol stack, independent of applications. Because of this, you are able to encrypt the data from any application prior to transmitting it over the network. Imagine that a client application on computer 1 wants to send encrypted data to a server application on computer 2. While there certainly needs to be some coordination between the two systems (we'll get to that in a bit), the applications on either side actually have nothing to do with the encryption process. The client application would simply begin passing data down the protocol stack like normal, where it would be 'intercepted' by IPSec, encrypted, and sent to the server system. On the server, the data moves up the stack, and is decrypted by IPSec prior to being passed to the application. For all intents and purposes, the encryption is transparent to the person on the client side - something very favorable indeed.

While many books and articles drown on about the various reasons for using encryption on a network, I'll spare you the details. Everyone who has worked with networks knows that if a data stream hasn't been secured, then it is theoretically possible for some intermediary person to possibly read or change the data. The various applications and attacks that make that possible are well documented elsewhere, and certainly some are more common than others. All things being equal, it is up to you to decide what is considered sensitive information on your network, and how you wish to proceed with policies that meet your needs. IPSec actually provides both encryption and authentication services, and understanding the difference is key to your planning process. 

IPSec is most commonly looked at as a protocol for securing network traffic via encryption. The way in which this is handled is via something called ESP, the Encapsulating Security Payload. The purpose of ESP in to provide data encryption - while packets are still visible on the network if you used a network sniffer or similar tool, the actual contents (often referred to as the payload, or less technically, the 'important stuff') of the packet beyond the original IP header are encrypted. How the encryption process happens I'll cover in a moment. The other function that IPSec is capable of is authentication and integrity settings via AH, the Authentication Header. That is, instead of actually encrypting a packet, it is rather signed, such that the receiver can be sure that the data came from the original sender and vice versa. The concern in this case is that the data hasn't been modified in transit, and that somebody (or some machine) isn't trying to impersonate someone else. While AH provides data integrity, it does nothing to actually secure the data stream, unless used in conjunction with ESP. Both AH and ESP can be used alone, or in conjunction with one another.

A small note here - be aware that ESP traffic uses IP protocol 50 to identify itself, while AH uses IP protocol 51. This is important in cases where you want these types of traffic to pass through a firewall or filtering device. Note also that IPSec will require that traffic can be passed over UDP port 500, over which ISAKMP/Oakley transfers (to be discussed shortly) take place.

Another important IPSec concept to consider is the mode in which you would have it participate on the network. The two IPSec modes are referred to as Transport mode and Tunnel mode. Transport mode allows for IPSec connections between 2 or more systems, used on demand, as the policy that you have created dictates. For example, you might specify that when any computer on the HR subnet attempts to communicate with the HR server, all traffic must be encrypted and signed. Or, you might specify that when any computer in the organization accesses the employee benefits database, all communication should be signed only. The level of granularity that you wish to use is up to you, just remember that the more complex your policies, the more difficult they may be to create and administer. Also remember that in order for a system to use IPSec in Transport mode, that system must use the IPSec protocol. Of the Microsoft operating systems, only Windows 2000 (and now XP) support these capabilities natively. Tunnel mode functions differently, and with a different purpose in mind. It creates a secured (and/or authenticated) tunnel between to endpoints. The idea is that traffic that passes through the tunnel in encrypted on one end and decrypted on the other, a scenario that would most commonly apply to moving between two private networks via an unsecured network such as the Internet. The benefits of using IPSec in Tunnel more are many. Firstly, it is relatively simple to set up (especially compared to Transport mode). Secondary, clients or servers on either side on the tunnel need not have any concept of IPSec or how it works. They simply send traffic destined for a system on the other side of the tunnel, and when it hits the first server the packets are encrypted, and subsequently decrypted on the other end. As you have probably considered, this does mean that the packets do travel over parts of both local networks in an unencrypted state. Whether that works for you will depend upon your specific security needs. Use a combination or tunnel mode and transport mode if it better meets your needs.

Many other documents also spend an incredible amount of time and energy explaining the internal processes that IPSec goes through in attempting to negotiate security. While this is certainly important, it is also easy to get bogged down in details and forget exactly what is going on. Essentially, if two computers want to use IPSec for the purpose of securing direct communication between them, they will each have to be configured with an IPSec policy. These policies are configured via local or group policy in Windows 2000, and will be discussed shortly. The policies need to work out the necessary details such that computers understand not only which protocols need to be secured, but also what key settings need to be used, authentication mechanisms, and so forth. The downside of IPSec is that there are many places where your configuration can be wrong. However, if you have everything mapped out properly in advance, the actual configuration should be fairly straightforward. Imagine if I have configured policies on two systems, and done it correctly. When the first system (say the client) attempts to communicate using IPSec with the second (say the server), they will first need to agree on a variety of factors including the key length to be used, security protocols, and something called the Security Parameter Index (SPI - this is what uniquely distinguishes one secured session from another). All of these things constitute something called a Security Association, the basis for the IPSec communication. You may have heard of something called the Internet Security Association and Key Management Protocol (ISAKMP) - this is the protocol used to actually establish security associations. There are actually two phases of security association negotiation, Phase I and Phase II. Phase I is responsible for the initial setup of a secure and authenticated channel between the two systems. Phase II is responsible for the actual security and negotiation of the data transfer, and the process of refreshing session keys used during the process. At the end of the Phase I SA lifetime (which is configurable), any Phase II SAs that exist may still exist, as they have their own lifetimes. However, if a Phase II data transfer were received without a Phase I SA in place, data transfer would be rejected. 

Before getting into the actual configuration of IPSec policies, you should also have an awareness of a key determination protocol called OAKLEY. Basically, OAKLEY is the one on each computer that works in conjunction with ISAKMP and is responsible for generating and managing the authenticated keys that are ultimately used to secure IPSec communications. 

Configuring IPSec Policies

Now that that's all settled, lets take a look at actually setting up IPSec policies. Actually, before we get there, you should have an awareness of the three IPSec policies that exist by default, and what they means. Found in the Computer Configuration section of group policy under \Windows Settings\Security Settings\IPSec Policies, the there policies that exist are Client (Respond Only), Secure Server (Require Security), and Server (Request Security). Note that none of the policies are assigned by default, and that only one IPSec policy can be assigned on a system at any given point in time. The there policies are discussed after the screen shot below:

Client (Respond Only) - As the name suggests, this policy is meant for client systems primary. This policy outlines that IPSec security should only be used in response to a request by a server. As such, a client with this policy assigned will make all requests unsecured, but if a server requests security, is capable of responding using IPSec security.

Server (Request Security) - This policy, meant mainly for servers, will request that a client use a secured connection if assigned. The caveat here is that the server will only ask. If a client is not capable of using security (or has no policy configured), it will still allow unsecured communication. Think of this policy as provided best-effort IPSec security.

Server (Require Security) - Again a policy meant for servers, this one is a little harsh, and probably won't meet your needs. If configured, this server requires IPSec security from all connections, right from the get-go. Make an unsecured request and you won't be asked, informed, or otherwise - you'll simply be denied.

The three policies provided are simply a starting point, since you will likely need something a little more granular that meets your specific needs. To create a new IPSec policy, right-click and choose 'Create IP Security Policy'.

Doing so will open the IPSec Security Policy wizard, which will walk you through the policy creation process step-by-step. The first step is giving the policy a name, and an associated description. I would strongly recommend a name that will be descriptive enough to explain what the policy is used for.

The Request for Secure communication screen comes next. This allows you to control how this policy will respond when other servers on the network request secured communication.

The next step is configuring the default authentication mechanism to be used. In Active Directory environments, Kerberos is probably the easiest choice, although the use of certificates or a pre-shared key are possible given that you may want to configure connections with systems that only support these methods. For the purpose of simplicity, I'll choose the pre-shared key.

The last screen that appears is actually a little confusing. While you have now created a new policy, you still need to edit its properties. If you choose to uncheck the box below, you can always still configure policy settings later by right clicking on a policy in Group Policy and choosing Properties.

If you choose to edit properties of the policy, you will be presented with the property screen shown below.

Note that by default, the only security rule that exists is the Default Response rule that we chose to allow in the previous wizard. This rule can be edited, or we can add and remove rules from this screen. We'll add a new rule in a moment, but first take a look at the Advanced button from the General tab below.

Note the settings on this screen above. The first setting, Master key Perfect Forward Secrecy, is used if you want to ensure that the same keying material or previous keys are ever used again in selecting a master key. The other settings configure how often new keys should be generated in minutes and seconds, while the Methods button allows you to configure the integrity and encryption mechanisms used during exchanges (such as SHA or MD5 for integrity, and DES or 3DES for encryption).

Now that you're familiar with some of the property settings, lets configure a new rule for accessing the HR server, as the name I gave the policy suggests. This particular rule will require transport mode, since I don't want to define any type of tunnel. I also should have some concept of which port or ports I need to secure for a particular application if applicable. For my example, I am going to assume that we want all IP traffic between this client and my HR server encrypted. You should experiment with more granular policies on your own.

Clicking the Add button on the rules tab opens the Create IP Security Rule Wizard, as shown below.

The first step is deciding whether or not the rule specifies a tunnel. If not, the system will be configured to use transport mode.

The Network type screen comes next. From here we can decide whether you want the rule to be used for all network connections, only LAN, or only Remote access. I've chosen that it should apply to all.

It may seem strange that you are again presented with the authentication mechanism screen again after this, but remember that each rule can have its own. The one that we specified earlier was the authentication mechanism for the default response rule. I'll skip the screen shot since it's exactly the same as the one presented previously.

The next screen is where the going gets tough, since you need to specify the exact types of traffic that you wish to secure. You might be able to escape easily by choosing to secure all IP and ICMP traffic, but I suggest taking a look at adding a rule that meets your needs a little better. 

After clicking the Add button to add a new filter list, you are presented with the screen below:

This gets even more complex since you can add multiple filters using this tool. If you click the Add button, you are moved into yet another wizard that will walk you through the process of creating a filter. I am going to create only one (note the descriptive name), that specifies that all of the computers on a given subnet must use IPSec to communicate with the HR Server

After choosing the traffic source (my subnet), I must next choose the traffic destination. This can be any of the options listed below:

I have chose the IP address of my HR server, Next step is choosing the IP protocol type to be included in my filter. I am going to choose ANY, although I could be more specific and choose a given TCP port or protocol ID if required.

The next screen allows you to edit your filter without the wizard. Remember that the policy is not yet complete; we must choose to enable our new rule in the policy by selecting it.

The next step is to actually specify the filter actions. What will happen when a user does something that meets rules of the filter? Remember that we haven't yet said whether AH or ESP should be used, whether these packets should be allowed or denied, and so forth. I know this is a long process, but we're almost there.

The Filter Action screen allows us to control how this will work. I suggest leaving the defaults alone (and unselected) and using the Add button (though you may be sick of Wizards at this point) to select what you want to happen. Without using the wizard, you will be presented with this screen:

Note that this screen allows you to either Block, Permits, or Negotiate security when a filter rule if encountered. I will choose Negotiate Security, and click the Add button, and choose that High security (ESP and AH) should be used, as shown below.

You can also configure custom settings, if you require them. Be sure to give your filter actions a descriptive name (and select it), as I have below.

The next screen simply finishes the process. Remember that the policy is still not assigned - right click the policy and choose Assign to do this, with the results shown below.

Also remember that IPSec configuration is not a one-way street. You will also need to configure the appropriate settings on your servers to ensure that the policies can negotiate security and function correctly.

A couple of additional notes on IPSec:

- in order to test whether IPSec is working properly, using the IPSECMON utility. It will show you any security negotiations and sessions currently in place.
- Note that you can use Services in Computer Management to stop and restart the IPSec Policy Agent if necessary. This is also the equivalent of stopping and reloading the IPSec driver.

Another week completed, and certainly a long one. Next week's article will in fact be the last in the 70-240 series, and will cover Certificate Services in Windows 2000. I'm not sure if another series will directly follow this one as of yet, as many people have emailed me to ask. I am currently considering a number of options, and will let you know as soon as I do. In the meantime, please post all technical questions to my message board, and feel free to contact me with any comments by email. Best of luck with your studies this week.


Mobile Site | Full Site
Copyright 2017 © QuinStreet Inc. All Rights Reserved