While configuring Dell PowerStore 1000T for NVMe over TCP, I opted to use two separate storage networks for availability purposes, similar to 2 storage fabric used in a Fibre Channel SAN. So each hosts has 2 NVMe over TCP NICs and each one connects to a separate switch. Each switch only has the VLAN of one of the two storage networks.
For this to work (due to the fact you can only configure the same subnet on the same front-end port for each node) this can only be done when in my scenario, port 2 of both Node A and Node B connect to the first switch, that only has storage network 1, and port 3 of both Node A and Node B connect to the second switch, that only has storage network 2.
So far so good, but as soon as I connected both ports 2 to switch 1 and ports 3 to switch 2, the PowerStore showed me an error: “Appliance port pairs are connected to the same data switch” Which is by design in my case and according to Dell’s best practices to use 2 storage networks for NVMe ….
While architecting an environment with a Dell PowerStore for a customer, I noticed some interesting details in the “Dell PowerStore: Best Practices Guide”
First a part about the on-board mezzanine card:
This means the ports on the mezzanine card have about 63.04 Gbits/sec of bandwidth available. When using 4 ports at 25Gbit/sec, we can not use the full bandwidth these ports offer. So what about the I/O module slots?
This part is about the I/O module slots? Slot 0 is 16-lane PCIe Gen3 which gives you about 126 Gbit/sec, which should be sufficient for the 4 port 25Gbit/sec card …
But ….. Regarding the I/O module:
So the 4 x 25Gbit/sec I/O module itself is 8-lane PCIe Gen3 …. So even in slot 0 you can not use the full bandwidth of this card ….
Since PowerStore OS there is a new 2-Port Ethernet card that supports speeds of up to 100 Gb/s. This 100 GbE card is supported on PowerStore 1000-9200 models in I/O Module 0 slot, the 16-lane PCIe Gen3 slot, which is still limited to 126 Gbit/sec ….
If you want to combine ethernet connectivity and Fibre Channel, it gets even more interesting, since the preferred slot for the 4 port 32Gbit/sec FC I/O module is slot 0 …. In slot 0 it can run at full capacity. When adding more FC ports to slot 1, the combined ports bandwidth is again limited to 63.04 Gbits/sec
So when designing a high bandwidth PowerStore environment, keep this in the back of your mind ….