What is the IPsec protocol and how does it work (2024)

What is IPsec?

IPSec stands for internet protocol security. This term refers to a set of communication rules used to establish secure connections over a network. IPsec protocols connect devices and add encryption to keep data safe as it travels between them.

The IPsec protocol suite can be used by individuals or larger organizations and can even act as the main protocol for a variety of VPNs. But how does IPsec work in practice?

How does IPsec work?

IPsec involves five main steps.

1. Host recognition. The host system recognizes that a data packet should be secured and sent via IPsec protocols. At this point, the data packet is encrypted and authenticated, ready for transfer.
2. Negotiation. The two host systems that will communicate through IPsec agree on the protocols that will be used and authenticate themselves to each other. A secure connection is established between them, along which negotiations can take place to determine what algorithms and rules are in place. These negotiations take two forms, main and aggressive.
   • Main mode: The host system that starts the process suggests encryption and authentication algorithms and negotiations continue until both systems settle on the accepted protocols.
   • Aggressive mode: The host system that starts the process proposes its preferred encryption and authentication methods but does not negotiate or change its preferences. If the other host system agrees, the process continues to the next step. If it doesn’t, the process does not continue.
3. Circuit. Using the secure connection created in the previous step, an IPsec circuit is established. The host systems agree on and exchange the encryption and decryption keys they will use, along with cryptographic nonces (randomized numbers used for authentication).
4. Transmission. Encrypted IP packets are transferred between the host systems. On arrival, a data packet is encrypted using the previously exchanged encryption keys.
5. Termination. Once the data has been transferred or the session times out, the IPsec connection is closed. The private keys used for the transfer are deleted, and the process comes to an end.

As demonstrated above, IPsec is a collection of many different functions and steps, similar to the OSI model and other networking frameworks. At the heart of that collection are the protocols and encryption algorithms.

IPSec protocols and encryption algorithms

IPsec uses two primary protocols to provide security services, the Authentication Header (AH) protocol and the Encapsulating Security Payload (ESP) protocol, along with several others. Not all of these protocols and algorithms have to be used — the specific selection is determined during the Negotiations stage.

• Authentication Header (AH). The Authentication Header protocol authenticates data origin and integrity and provides replay protection.
• Encapsulating Security Payload (ESP). Encryption is added by the Encapsulating Security Payload protocol.
• Internet Key Exchange (IKE). The Internet Key Exchange protocol ensures that both host systems have the keys needed to encrypt and decrypt the data packets.
• Triple Data Encryption Standard (3DES). Triple Data Encryption Standard is an encryption algorithm that applies a cipher to data three times for additional security.
• Advanced Encryption Standard (AES). Advanced Encryption Standard encrypts data in blocks of 128 bits.

IPSec security policies and authentication

IPSec provides several authentication methods, including:

• Pre-shared key (PSK) authentication. A shared secret key is known to both the sender host system and the receiver, and is used to authenticate the transferred data.
• Digital certificates. A trusted certificate authority (CA) provides digital certificates to authenticate the communication. This allows the host system receiving the data to verify that the sender is who they claim to be.
• Kerberos authentication. The Kerberos protocol provides a centralized authentication service, allowing devices that use it to authenticate each other.

Different IPsec implementations may use different authentication methods, but the result is the same: the secure transference of data. The protocol suite can also be implemented in two modes: transport mode and tunnel mode.

Differences between transport and tunnel IPsec modes

The transport and tunnel IPsec modes have several key differences.

Transport mode

• Encryption is only applied to the payload of the IP packet, with the original IP header left in plain text.
• Transport mode is mainly used to provide end-to-end communication between two devices.
• Transport mode is primarily used in situations where the two host systems communicating are trusted and have their own security procedures in place.
• Crucially, transport mode offers less security than tunnel mode.

Tunnel mode

• Encryption is applied to both the payload and the IP header, and a new IP header is added to the encrypted packet.
• Tunnel mode provides a secure connection between points, with the original IP packet wrapped inside a new IP packet for additional protection.
• Tunnel mode can be used in cases where endpoints are not trusted or are lacking security mechanisms.
• Tunnel mode provides more security for data in transit.

In short, both modes have their uses, but tunnel mode is more secure. Security is a key benefit for IPsec, which is why the protocol suite is often used to create VPNs.

What are IPsec VPNs?

An IPsec VPN, or virtual private network, is a VPN that uses the IPsec protocol to create an encrypted tunnel on the internet.

A VPN routes traffic along an encrypted tunnel, protecting data from unwanted intrusions. An IPsec VPN does this using the IPsec protocol to establish a connection and encrypt data packets in transit and is particularly useful for businesses and large organizations with out-of-office workers who need remote access to resources.

A company could set up an IPsec VPN between a remote worker’s device and an internal server, giving an employee secure access to the same systems and data that someone working in their office would have.

An IPsec VPN can be configured in several ways:

• Site-to-site. A site-to-site VPN connects two or more networks with an encrypted tunnel. This means that users on both networks can interact as if they were in the same space.
• Client-to-site. Client-to-site VPNs allow individual devices to connect to a network remotely. With this option, a remote worker can operate on the same network as the rest of their team, even if they aren’t in the same location.
• Client-to-client. The client-to-client VPN model allows multiple devices to connect with encrypted tunnels, allowing for secure file sharing and communications. It should be noted that this method is rarely applied since it is difficult to manage and scale.

Whether you’re using a site-to-site VPN or a remote access VPN (client-to-site or client-to-client, for example) most IPsec topologies come with both advantages and disadvantages.

Advantages and disadvantages of IPSec VPNs

Let’s take a closer look at the advantages and disadvantages of an IPsec VPN.

Advantages of an IPSec VPN

An IPsec VPN offers several key advantages, especially for large organizations and businesses.

1. Security: An IPSec VPN provides robust network security by encrypting and authenticating data as it travels between points on the network.
2. Flexibility: An IPSec VPN is versatile and can be configured for different use cases, like site-to-site, client-to-site, and client-to-client. This makes it a good option for organizations of all shapes and sizes.
3. Dispersed teams: If an organization has a team spread across multiple locations, with remote workers or several offices, an IPsec VPN can seamlessly connect all parties.

Disadvantages of an IPSec VPN

Of course, the IPsec VPN is not without its disadvantages:

1. Minor speed reduction: An IPsec VPN adds additional encryption and authentication processes to a network, making data throughput fractionally slower, but this won’t be noticeable for most users.
2. Complexity: An IPsec VPN can be complex to configure and troubleshoot, requiring knowledgeable IT staff or external support.
3. CPU overheads: IPsec uses a large amount of computing power to encrypt and decrypt data moving through the network. This can degrade network performance.

How to set up IPsec VPN protocol

Follow the steps below to set up an IPsec VPN.

1. Decide on a VPN topology. This means determining the structure of the VPN (site-to-site, client-to-site, or client-to-client) and setting the IP addresses and subnet masks for each VPN endpoint.
2. Choose an IPsec implementation. An IPsec implementation is the specific software suite that you will be running on operating systems. Examples of IPsec implementations include StrongSwan, Openswan, and LibreSwan.
3. Configure IPsec settings. Establish the specific settings of your implementation, including authentication method, encryption algorithm, and key management protocol.
4. Configure network settings. In addition to your IPsec settings, you will need to configure the network as a whole to work with a VPN, establishing IP addresses, subnet masks, and routing rules.
5. Configure firewalls. Make sure that firewalls at both ends of the VPN are set up to allow IPsec traffic to pass through their defenses.
6. Test the connection. Once all steps have been taken, make sure that data is traveling seamlessly through the IPsec VPN, and troubleshoot any connection issues.

What is the difference between an IPSec VPN and SSL VPN?

IPsec and SSL VPNs have one main difference: the endpoint of each protocol. In most cases, an IPsec VPN lets a user connect remotely to a network and all its applications.

On the other hand, an SSL VPN creates tunnels to specific apps and systems on a network. This limits the ways in which the SSL VPN can be used but lowers the likelihood of a compromised endpoint leading to a wider network breach. Of course, both an Ipsec and SSL VPN can be useful, but which one you choose depends on the needs and structure of your organization.

Does NordVPN use IPsec?

NordVPN supports the IKEv2/IPsec protocol for manual configurations. IKEv2/IPsec is a combination of the IPsec and Internet Key Exchange version 2 (IKEv2) protocols. IKEv2/IPsec allows for a secure VPN connection, without compromising on internet speeds.

In its applications, NordVPN offers the OpenVPN protocol and NordLynx, a protocol based on WireGuard. NordLynx provides unrivaled speeds, making NordVPN the fastest VPN in the world.

Can I manually connect to the NordVPN IPsec protocol?

Yes, you can manually connect to NordVPN on all major operating systems. For OS-specific guides, see the list below.

• Manually connect on Windows
• Manually connect on Linux
• Manually connect on macOS
• Manually connect on Android
• Manually connect on iOS

Of course, you can use NordVPN without manually connecting to a protocol. Just download the app, set up your account, and start browsing with enhanced security and privacy.