Virtual SAN Node Communication

Our Network Security Administrator gave us some guidance on how to architect the Physical Infrastructure to provide Network Security for Virtual SAN. What she has not yet clarified for us is which Virtual SAN node talks to which Virtual SAN node, and over which IPs, Multicast Groups and ports.

Each Virtual SAN node has one of three roles: Master, Backup and Agent. Among the Master’s responsibilities is disseminating Virtual SAN Metadata (the Virtual SAN Datastore). The Backup keeps an eye on the Master and is ready to take over if the Master becomes unavailable. Every other Virtual SAN node in the same Virtual SAN cluster is an Agent.

So which Virtual SAN nodes chat with others Virtual SAN nodes? VMware has provided some of that information but the information is not granular enough to deploy an effective Network Security policy. Thus, without further ado, here is what the Network Security Administrator discovered while doing some packet captures of Virtual SAN traffic with no powered on Virtual Machines (think of this as Virtual SAN Control Plane traffic).

Master

The Master communicates with every other member of the same Virtual SAN cluster. One of the communications takes place over Unicast. The Master will send Unicast traffic to all Agents and the Backup over TCP port 2233. The interesting part here is that the Master has two Unicast flows going with each other node. One flow is initiated by the Master using the destination TCP port 2233. Other nodes, also using the destination TCP port 2233, initiate the other Unicast flow towards the Master.

Below is some output of two Unicast flows between the same two nodes. Both egress packets were captured in the Master.

Id = 71 Source = 10.154.7.40 Destination = 10.154.7.43 Captured Length = 162 Packet Length = 162 Protocol = TCP Date Received = 2016-02-02 12:47:07 +0000 Time Delta = 1.404654979705811 Information = 12873 -> 2233 ([ACK, PUSH], Seq=4133924354, Ack=1209013051, Win=4097) Id = 79 Source = 10.154.7.40 Destination = 10.154.7.43 Captured Length = 322 Packet Length = 322 Protocol = TCP Date Received = 2016-02-02 12:47:10 +0000 Time Delta = 4.489483118057251 Information = 2233 -> 20082 ([ACK, PUSH], Seq=3609429705, Ack=981228717, Win=4096)

The Master also communicates with everyone (Agents and Backup) using the Agent Group Multicast Address with a destination UDP port of 23451. This communication is used to update Metadata with all Virtual SAN cluster members. The default Agent Group Multicast Group is 224.2.3.5 Both the Agent Group MG and destination UDP port can be changed in the cli using the command:

esxcli vsan network ipv4 set -d [Agent Group MG] -p [Agent Group MG UDP Destination Port]

The Master listens to both the Agent Group MG and the Master Group MG.

Backup

The majority of the Backup Unicast communication takes place with the Master, although some infrequent Unicast traffic can flow between the Backup and some Agents. The Backup communicates with the Master using the dual-flow Unicasts using TCP port 2233.

The Backup also communicates with everyone (Agents and Master) using the Agent Group Multicast Address with a destination UDP port of 23451. Minor detail: the Backup is also kind of an Agent.

The Backup also communicates with the Master using the Master Group Multicast Address with a destination UDP port of 12345. The default Master Group Multicast Group is 224.1.2.3. Both the Master Group MG and destination UDP port can be changed in the cli using the command:

esxcli vsan network ipv4 set -u [Master Group MG] -o [Master Group MG UDP Destination Port]

The Backup listens to both the Agent Group MG and the Master Group MG.

Agents

The Agents communicate via Unicast with both the Master and the Backup, although the communication with the Backup is infrequent.

The Agents will send updates to the Agent Group Multicast Address with a destination port of 23451.

The Agents will listen to the Agent Group MG.

Powered On Virtual Machines
When Virtual Machines are powered on in the Virtual SAN Datastore (think Virtual SAN Data Plane), the above traffic flows continue (the Control Plane doesn't stop). In addition, all members of the Virtual SAN cluster that have a powered on Virtual Machines in the Virtual SAN Datastore will send Unicast packets, on source TCP port 2233, to all other nodes that have a copy of the Virtual Machines' VMDKs.

Thus from a security perspective, every Virtual SAN node in the same Virtual SAN cluster is expected, at some point, to communicate via Unicast on TCP port 2233 with all other Virtual SAN nodes.

Elver's Opinion: The biggest security risk with Virtual SAN is unauthorized access to the Virtual SAN Datastore. Since access to the Virtual SAN Datastore takes place over TPC 2233, blocking that port from unauthorized end-points will substantialy help reduce the risk of a breach. At a minimum, 1) the Virtual SAN Layer 2 domains should have direct connectivity locked down to only the Virtual SAN VMkernel port of the Virtual SAN nodes in the same Virtual SAN cluster and 2) an Access List restricting flows over TCP 2233 should be placed in the Virtual SAN segments' default gateways.