By Justin Oyaro . 27 January 2024
Cybersecurity Expert
Miklos Zoltan
Fact-Checked this
Internet Protocol Security (IPSec) is a suite of protocols usually used by VPNs to create a secure connection over the internet.
The IPSec suite offers features such as tunneling and cryptography for security purposes. This is why VPNs mostly use IPSec to create secure tunnels.
IPSec VPN is also widely known as ‘VPN over IPSec.’
Quick Summary
IPSec is usually implemented on the IP layer of a network. IPSec uses two modes of operation; tunnel mode and transport mode.
Most VPN providers use the tunnel mode to secure and encapsulate the entire IP packets. Transport mode only secures the payload and not the entire IP packet.
The IPSec VPN protocol suite generally offers advanced authentication, compression, and encryption services to VPN connections.
IPSec offers the freedom of selecting algorithms, security protocols, and the mode of exchanging security keys between communication hosts.
Summary: Internet Protocol Security (IPSec) is a comprehensive suite of protocols commonly utilized by VPNs to establish secure internet connections. It is known for its robust features, including tunneling capabilities and advanced cryptography.
IPSec operates in two primary modes: tunnel mode, which encapsulates the entire IP packet, and transport mode, which only encrypts the payload of the IP packet.
The suite offers a range of advanced services such as authentication, data compression, and encryption. This flexibility extends to the choice of algorithms, security protocols, and methods for key exchange, allowing for tailored security setups.
Components of IPSec include the Authentication Header (AH) for packet integrity and source authentication, Encapsulating Security Payload (ESP) for encryption and optional authentication, Internet Security Association and Key Management Protocol (ISAKMP) for establishing security associations and cryptographic keys, and IP Payload Compression Protocol (IPComp) for data compression.
IPSec’s operational process involves establishing secure communication channels, key negotiation and exchange, data transmission in a secure manner, and eventually terminating the session once the communication is complete.
What Is IPSec?
IPSec suite of VPN protocols includes Authentication Header (AH), Encapsulated Security Payload (ESP), Internet Security Association and Key Management Protocol (ISAKMP), and IP Payload Compression (IPComp).
- Authentication Header (AH): AH offers data origin authentication of IP packets (datagrams), guarantees connectionless integrity, and gives protection against replay attacks (thanks to the sliding window technique). The AH also offers significant authentications for both IP headers and upper-layer protocols.
- Encapsulating Security Payload (ESP): ESP is responsible for offering authentication, integrity, and confidentiality of data. ESP also provides payload confidentiality and message authentication within the IPSec protocol suite.
In tunnel mode, it encapsulates the entire IP packet, while only the payload is protected in transport mode. - Internet Security Association and Key Management Protocol (ISAKMP): ISAKMP is tasked with Security Associations (SAs) – a set of pre-agreed keys and algorithms used by parties when establishing a VPN tunnel. These include Kerberized Internet Negotiation of Keys (KINK) and Internet Key Exchange (IKE and IKEv2).
- IP Payload Compression (IPComp): IPComp is a low-level compression protocol that reduces the size of IP packets, thereby improving the communication levels between two parties. This is useful when communication is overly slow, for instance, congested links.
IPComp doesn’t offer security and must be used with AH or ESP over VPN tunnels.
IPSec VPN Operation Modes
Let’s look at how the two IPSec VPN modes compare:
IPSec Tunnel Mode
VPN encryption in tunnel mode encapsulates every outgoing packet with new IPSec packets using ESP. Tunnel mode also uses AH to authenticate the server-side.
Hence, IPSec uses tunnel mode on secure gateways like a firewall that links the two communicating parties.
Transport Mode
Transport mode encrypts and authenticates the IP packets sent between two communicating parties.
As such, transport mode is often reserved for end-to-end communications between parties, considering it doesn’t change the IP header of the outgoing packets.
Cryptographic Algorithms for IPsec
IPSec relies on secure algorithms that conform to confidentiality, integrity, and authenticity.
They include:
- Authentication algorithms such as RSA, PSK, and Elliptic Curve cryptography.
- Symmetric encryption algorithms such as AES-CBC and GCM, HMAC-SHA, TripleDES, and ChaCha20-Poly1305.
- Key exchange algorithms such as Elliptic Curve Diffie-Hellman and the Diffie-Hellman key exchange.
How Does IPSec Work?
Below is a general step-by-step outline of how IPSec works.
Usually, the process starts with hosts (communicating parties) establishing that incoming or outgoing packets need to use IPSec.
If the packets trigger IPSec policies, then the process continues as follows:
- Negotiation and Key exchange: This step includes host authentication and policies to be used. In the first phase, the hosts create a secure channel. Negotiations are done using either the main mode (for greater security) or the aggressive mode (for faster IP circuit establishment).
All hosts agree on an IKE for setting up the IP circuit in the main mode. In aggressive mode, the initiating host presents the IKE for setting up the IP circuit, and the other host agrees.
In the second phase, the hosts negotiate and agree on the type of cryptographic algorithms to be used during the session. - Transmission: This involves the exchange of data between the hosts. Usually, IPSec breaks data into packets before it’s sent over the network. The packets include several segments like the payload and headers. IPSec also adds trailers and other segments that contain authentication and encryption details.
- End of transmission: This is the last step, and it involves the termination of the IPSec secure channel. Termination occurs when data exchange is complete or the session has timed out. Cryptographic keys are also discarded.
IPsec VPN vs. SSL VPN
Besides IPSec VPN, many of the best VPN providers can also use SSL VPN to secure your connection over the internet. Depending on the level of security needed, VPN providers can implement both or choose one over the other.
SSL VPNs rely on the Transport Layer Security (TLS) protocol. Unlike IPSec, which works on the IP layer, TLS works on the transport layer. Thus, the security and applications of IPSec VPN and SSL VPN vary.
With IPSec VPN, your traffic is secure as it moves to and from private networks and hosts; in a nutshell, you can protect your entire network. Thus, IPSec VPN is reliable for IP-based uses and applications.
SSL VPN protects the traffic between remote users. In most cases, SSL VPNs work with hosts that support browser-based applications.
Frequently Asked Questions
Some people found answers to these questions helpful
What network port does IPSec often use?
IPSec often uses UDP port 500 for ISAKMP and UDP port 4500 to pass through firewalls, NAT is allowed.
Which one is better, SSL VPN or IPSec VPN?
The applications of each protocol set them apart. For instance, IPSec VPN lets users remotely access the entire network and its apps. However, SSL VPN allows users remote tunnel access to particular apps on the network.
Can IPSec be hacked?
Internet Protocol Security (IPSec) is widely regarded as secure. However, according to Snowden leaks, NSA targeted the IPSec encryptions by inserting various vulnerabilities. According to other reports, hackers can break some of IPSec’s encryption. It all depends on the implementation used.
What are the uses of IPSec?
Internet Protocol Security (IPSec) is a protocol suite that secures packet-level traffic over a network. You can use IPSec to secure information between two parties. Furthermore, IPSec is responsible for data integrity, originality, authentication, and confidentiality.
Is IPSec the same as VPN?
Simply put, Internet Protocol Security (IPSec) is like a traditional VPN technology. The protocol suite was first launched in 1990 and has received significant upgrades since, making it a widely used protocol in the VPN industry.
Tech and Cybersecurity ExpertJustin is a cybersecurity expert with over 5 years of experience that stretches across various technological realms. He is a valued expert in cybersecurity, Cord-cutting, Cryptocurrency, Smartphones, How-tos and other related fields.Justin has written authoritative content for multiple Cybersecurity websites and blogs on matters regarding privacy and security. He also does pen test on tech products such as smartphone applications and websites.When he is not writing, Justin spends most of his time on the latest tech trends and Android Studio.Justin is also a degree holder in Information Systems Technology and Master of Computer Applications (M.C.A.), University of Adelaide.
