Exploring Windows 2003 Security: IP Security

by Marcin Policht

We detail the improvements made to IPSec on Windows Server 2003 as well as offer a general overview of the technology.

The most recent article of our Windows Server 2003 series covered some of the security-related improvements in the networking functionality of Windows Server 2003 platform. This installment continues that discussion, with a focus on IPSec technology. IPSec was first implemented by Microsoft (based on its joint development effort with Cisco) in Windows 2000. Its primary purpose was to provide the ability to transmit IP-based traffic in a secure manner between two IPSec enabled hosts, regardless of the level of protection offered on intermediate networks.

Before delving into our discussion of the improvements in the area of IPSec on Windows Server 2003 , we will overview the technology's main principles and the way it operates. In essence, IPSec combines a number of different parameters that both identify IP traffic to be secured and specify security settings to be assigned to it:

  • Security Protocols -- Authentication Header (AH) and Encapsulating Security Payload (ESP): Authentication Header offers authentication (checking credentials of hosts participating in data exchange), integrity (verifying that content has not been modified in transit), and replay protection (preventing from capturing IP packets and re-transmitting them to impersonate original sender), but not encryption (ensuring data confidentiality), while Encapsulating Security Payload includes all of them. Encryption algorithms (assigned to ESP) and hash algorithms used for verifying data integrity (assigned to both AH and ESP) are configurable.

  • Mode of Operation -- Transport and Tunnel: In transport mode, only payload (the data portion) of the IP packet is secured (this is done by appending relevant security protocol information to the original packet). In tunnel mode, the entire IP packet (including the header of original IP packet) is protected (this is done by encapsulating the original packet into a new IP packet). Transport mode is used much more commonly, since it constitutes the easiest way to provide VPN-based end-to-end secure communication between two computers (obviously both systems must support L2TP/IPSec). Tunnel mode is used primarily for communication between systems where L2TP/IPSec is not fully supported, so a tunnel is created to secure a portion of communication channel (typically to a gateway or end point that does not support L2TP/IPSec). Tunneling also requires static IP addresses at both endpoints and does not support filtering of IP traffic based on protocol or port number.

  • Security Association: A combination of parameters, established using Internet Key Exchange (IKE) protocol, determine connection security. These parameters consist of common security protocols, authentication methods, and mode of operation agreed on by both hosts participating in the IPSec connection. IKE negotiates these parameters in two phases. The primary goal of the first one, called main mode, is to generate master key, which is used for encrypting the second phase (and for authenticating two connection partners until the second phase is completed). During the second phase, the master key secures negotiation between two hosts, resulting in the selection of a mutual security protocol, encryption method, and hash algorithm for the fully secured connection. This phase also results in the creation of a session key with configurable expiration parameters. Based on their value, a new session key is periodically re-created.
    Note that the IKE traffic is exempt from IPSec filtering (understandably so, since it is required to establish secure communication before the IPSec session is started). However, by default, this is the only exception of this kind in Windows Server 2003. In contrast, in Windows 2000 and XP, IPSec filtering exemption was less strict and also allowed all broadcast, multicast, Kerberos, and Resource Reservation Protocol (RSVP) traffic. Detailed information about modifying this default is provided in Microsoft Knowledge Base article 810207.

  • Authentication Methods Used to Verify the Identity of Hosts During IKE Negotiation to Establish Security Association: Kerberos (limited to Active Directory, relies on forwarding computer credentials to Windows 200x domain controllers), certificates (requires a trusted Certificate Authority), or preshared keys (matching string of characters assigned to each computer). Note that preshared key authentication is not recommended due to its inherent weaknesses (e.g., such keys are stored in plaintext), and it should be used only if implementation of the remaining two methods is not possible.

  • IP Filters: These filters specify protocol, port number, and source and destination of IP addresses to which all security parameters listed above will apply. This enables the system administrator to be selective when specifying what type of IP traffic should be secured.

  • IP Filter Actions: The filter actions determine what type of action is triggered when traffic matching the IP Filter criteria is detected. Actions can belong to one of three general categories -- permit, block, and negotiate (where the last one can be further configured by selecting security methods allowed to be negotiated).

IPSec policies are configured and stored as part of local and Active Directory group policies (although Windows Server 2003 also provides an option to use a persistent store for the location of locally assigned IPSec policy, independent of group policies. (This is accomplished with the NETSH command line utility, as described later in this article.) In either case, there are three pre-configured IPSec policies -- Client (respond only), Server (request security), and Secure Server (require security), listed in the order of increasing security level. Creation of new ones is simplified by IP Security Policy Wizard, with associated sub-wizards (e.g., IP Security Rule Wizard, IP Filter Wizard, and IP Security Filter Action Wizard); however, even with help from the wizards, a number of available configuration settings can be initially overwhelming. After a policy is defined, it must be assigned (by selecting the Assign option from the context-sensitive menu of a selected IPSec policy) to become effective. Obviously, in case of a local policy, assignment applies to the local system, while in case of an Active Directory group policy, its impact depends on a container to which the policy is linked (as well as Security Group and WMI filtering settings).

With the conclusion of this brief overview of IPSec technology, let's investigate the most relevant improvements in its Windows 2003 implementation:

  • New and improved management tools
  • Increased security of the master key (during the IKE negotiation stage)
  • More-flexible authentication methods
  • The ability to operate in NAT-ed environments
  • Interaction with Network Load Balancing

New and Improved Management Tools

Windows 2003 provides four primary tools for managing IPSec:

  1. IP Security Monitor: Unlike its fairly limited Windows 2000 counterpart IPSECMON.EXE, this new version is implemented as an MMC snap-in and displays much more extensive information about IPSec-related settings. Its primary purpose is monitoring IPSec policies and security associations for local and remote computers. For each computer, the tree pane lists Active Policy, Main Mode, and Quick Mode top-level folders, displaying currently assigned IPSec policy and parameters of main and quick mode IKE negotiations, respectively.
  2. IPSec Context: NETSH is a rather cumbersome (due to lack of a friendly GUI interface) command line utility, introduced in Windows 2000, that can be used to modify and display a number of different network configuration settings. Its main benefits are speed, its capability to be executed in non-GUI scenarios (e.g., via Telnet), and scriptability. In Windows Server 2003, IPSec context enables administrators to take advantage of these features when managing IPSec settings. IPSec context also offers a number of additional configuration options not available from the IP Security Monitor, such as:

    • Configure bootmode property, which determines the type of traffic allowed during boot time prior to the initialization of IPSec Services. Choices include permit, block, or stateful (which limits inbound traffic to communication initiated locally) values. This can be set directly from the command prompt, by running

      netsh ipsec dynamic set config property=bootmode value=permit
    • Configure bootexemptions property, should you opt to block traffic. This is done by specifying exemptions in the form Protocol:SourcePort:DestinationPort:Direction, e.g. TCP:80:80:outbound, which requires executing the command of:

      netsh ipsec dynamic set config property=bootexemptions value="TCP:80:80:outbound"
    • Set IPSec and IKE diagnostics levels and logging intervals, by using ipsecdiagnostics (possible values range from 0 to 7), ikelogging (0 or 1), and ipsecloginterval (between 60 and 86400 seconds) properties.

    • Set persistent policy for enhanced security using the set store command with the persistent option. This way, selected IPSec policy gets assigned to the local system at startup, before the application of those assigned through local or Active Directory group policy setttings (and it remains assigned afterward).

  3. IP Security Policies: Although the interface remained relatively unchanged, there is no longer a limitation of two choices (My IP Address and Any IP address) when specifying the source and destination IP addresses of IP Filters for IPSec policies. In particular, it is possible to set base filters on a specific DNS name, an IP subnet, or even apply them to communication with source or destination being evaluated dynamically, by selecting such options as default gateway, DNS, WINS, or DHCP server.
  4. Group Policy Modeling and Group Policy Results: These include IPSec extensions, which simplifies both designing and troubleshooting of IPSec policies. For more information about Group Policy related Windows 2003 improvements, refer to the previous article in this series.

More Secure Master Key (During the IKE Negotiation Stage)

As described previously, one of the tasks performed during the first phase of IKE negotiation is the generation of the IKE master key, which, in turn, is used to secure communication during the second phase (and frequently is re-used to generate a new session key, when its expiration interval passes). Starting with Windows Server 2003, you can increase security by specifying a longer key (2048 bit) to be used by the Diffie-Hellman algorithm, based on which a master key is created.

Diffie-Hellman parameters are configurable from the Key Exchange Security Methods dialog box. (They are specified using a separate setting for each of the security methods listed there.) This dialog box is reached by first clicking on the Settings ... button on the General tab of the IPSec policy Properties dialog box and then clicking on the Methods... button on the Key Exchange Settings dialog box.

More-Flexible Authentication Methods

As already explained, three authentication methods are available when establishing an IPSec session. Starting with Windows 2003, for certificate-based authentication, you can enable the certificate-to-account mapping option, as long as you operate in Active Directory environment. To turn this option on, launch the IP Security Policies MMC snap-in, double-click on the policy to be configured, choose a rule from the IPSec Policy properties dialog box to be modified (listed on the Rules tab), select the Authentication Methods tab of Edit Rule Properties dialog box, and, finally, edit the certificate authentication method (by checking on the checkbox labeled "Enable certificate to account mapping"). Once this setting is enabled, you can take advantage of user rights (such as Access this computer from the network or Deny access to this computer from the network) assigned via Group Policies to target computers to control which ones will be able to establish IPSec session.

Network Address Translation Awareness

One of the most significant problems with IPSec implementation in Windows 2000 (typically used for VPN L2TP/IPSec connections) was its incompatibility with Network Address Translation (NAT). This meant that two computers could not communicate using IPSec if their traffic was passing a NAT router. Although this could be circumvented by applying PPTP tunneling (with its inherent encryption methods) rather than IPSec/L2TP, or by setting a tunnel between an IPSec client and a firewall (before a NAT device was reached), these workarounds frequently compromised overall security.

Microsoft resolved this problem in Windows 2003. Its implementation of L2TP/IPSec is no longer based on proprietary IPSec encapsulation mechanism but rather on IETF RFP standards, incorporating NAT Transparency (NAT-T). Changes have also been made to RRAS NAT implementation. In addition, recently published updates to Windows XP and Windows 2000 L2TP/IPSec NAT-T VPN client software, as well as VPN client software for older versions of Windows (downloadable from the Microsoft Web site) allow connecting to Windows 2003 VPN server using L2TP/IPSec residing behind a NAT device from practically any version of Windows. However, the NAT device will likely need to be configured to allow the passing of L2TP (UDP ports 500 and 1701), NAT-T (UDP port 4500), and ESP (IP protocol 50) traffic (specifics would depend on whether the NAT device resides on the client side, the server side, or both). Obviously, there must also be some way of authenticating the IPSec session (typically via certificate or preshared key).

For more information about the update, refer to Microsoft Knowledge Base article 818043.

Interaction With Network Load Balancing

Windows 2003 Network Load Balancing provides support for IPSec connections. The steps necessary for such configuration are outlined in Microsoft Knowledge Base article 820752, so we will provide only a short summary here. First, you must ensure that appropriate IPSec policy apply to the shared IP address of the cluster that will be used for secure communication. Next, using NLB Manager, in the Cluster Properties dialog box, on the Port Rules tab, allow UDP traffic on ports 500 and 4500 and set Affinity to Single or Class C.

This article was originally published on Monday Nov 17th 2003
Mobile Site | Full Site