Internet Protocol Security (IPsec) is a robust suite of open standards developed by the Internet Engineering Task Force (IETF) to secure communications over IP networks. Unlike TLS-VPNs which operate higher up in the OSI model, IPsec operates at the Network Layer (Layer 3). This allows it to protect all traffic passing between two endpoints without requiring application-level awareness, making it completely transparent to individual applications.
Core Cryptographic Functions of IPsec
IPsec protects traffic by providing four essential cryptographic services:
- Confidentiality: Encrypting the payload so unauthorized parties cannot read the data.
- Data Integrity: Ensuring that packets are not modified during transmission.
- Origin Authentication: Verifying that the packet was sent by the authentic sender.
- Anti-Replay Protection: Preventing hackers from intercepting and retransmitting packets later to exploit the session.
Core Protocols: AH vs. ESP
IPsec achieves security through the use of two core headers:
- Authentication Header (AH - Protocol 51): Provides data integrity, data origin authentication, and anti-replay protection. Crucially, AH does not encrypt data; packets are transmitted in cleartext. Because AH authenticates the entire IP packet header, it is incompatible with Network Address Translation (NAT) devices.
- Encapsulating Security Payload (ESP - Protocol 50): Provides payload encryption, data integrity, origin authentication, and anti-replay services. Since it encrypts the IP payload, ESP is the foundation of secure corporate VPN tunnels.
Tunnel Mode vs. Transport Mode
IPsec can be deployed in two distinct encapsulation modes depending on the routing architecture:
| Feature | Transport Mode | Tunnel Mode |
|---|---|---|
| Primary Use | Host-to-Host (Endpoints directly) | Gateway-to-Gateway (Site-to-Site) |
| IP Header Encryption | Only payload encrypted; original header exposed | Entire original IP packet encrypted; new header added |
| NAT Traversal | Usually unsupported without UDP wrapping | Supported using NAT-T (UDP Port 4500 encapsulation) |
IPsec Packet Encapsulation (Tunnel Mode)
In Tunnel Mode, the entire original IP packet (including the source/destination headers) is encrypted and placed inside a new outer packet. This is the primary mode used for secure Site-to-Site VPNs.
How the Encapsulation Layout Works:
- New Outer IP Header: A new header containing the public gateway IP addresses of the two VPN routers. This is used to route the packet across the public internet.
- ESP Header: Contains the Security Parameters Index (SPI) and Sequence Number, which the receiving gateway uses to identify the correct decryption keys.
- Encrypted Payload: The original packet header (containing internal local IPs) and the actual data payload are encrypted, hidden from anyone sniffing traffic.
- ESP Trailer & Auth: Padding to match encryption block sizes, followed by the Integrity Check Value (ICV) used to authenticate that the packet was not altered.
The Exchange Phases: Establishing the Tunnel
Before any encrypted data can travel, the two endpoints must negotiate a **Security Association (SA)**. This happens in two phases using the **Internet Key Exchange (IKE)** protocol:
- IKE Phase 1 (ISAKMP SA): Endpoints negotiate encryption parameters (like AES or ChaCha20) and run a Diffie-Hellman key exchange to establish a secure control channel. They authenticate each other using pre-shared keys or digital certificates.
- IKE Phase 2 (IPsec SA): Inside the secure control channel established in Phase 1, they negotiate the final data encryption keys that will be used to protect the actual user traffic.
