NEW PCI-E 4.0 PLATFORMS

We went from a bandwidth of 32 GB/s to 64 GB/s.

But not only this has grown, so has memory access: from 4 channels at 2666 Mhz, through 6 channels at 2933 Mhz, up to 8 channels at 3200 Mhz. That is, put by processor and in GT/s (Gigatransactions per second), we have gone from the limit of 10,4 GT/s to 25,6 GT/s.

As is logical, this needed to go hand in hand, from disk speed and this has also increased in SSD disks. They no longer have the limitation of mechanics and can go to the speed of a persistent memory chip (without data loss), passing the limitation of the SATA bus, which could only deliver 6 Gb/s. Therefore, obtain disks with a maximum of 100.000 IOPS, at 32 Gb/s, which allow disks in Gen 3 of up to one million IOPS (Input Output per second) and up to 2 million IOPS in Gen 4.

In addition, logically, these speeds can be enhanced through RAID technologies, which allow multiple disks to be grouped and thus obtain more performance and/or added security in RAID 0, 1, 10, 5 and 6. In this sense, there have been various approximations:

1º By Intel, with the VROC, which allowed the processor to manage RAID disks in a hardware/software environment, from Gigabyte's own bios and with an additional license.

In principle, only 2 NVME SSD disks and with an expander up to 24 disks, with the consequent load on the processor and sacrifice of control lanes. We already saw this in the 2nd scalable generation or known as CascadeLake and it has been improved in the 3rd generation or IceLake.

2º By Broadcom, owner of the classic technology of LSI Raid controllers, which have a specialized RISC processor, which downloads the processor. These controllers until now were only capable of handling SATA3 and SAS3 drives. The new controllers that are being launched now allow NVME drives to be handled, up to a maximum of 4 drives directly, being able to do RAID 0, 1 and 5. Logically, the new RISC processor is more powerful and will be able to continue handling SAS/SATA drives and /or NVME disks, with certain limitations.

3 º NVIDIA has already launched this technology and the platforms will see the light for the first Q1 (Feb-Apr 2022, the company has lagged the quarters with respect to the natural ones), a new concept. The recent T1000 GPU has been redesigned and renamed SupremeRAID™ SR-1000 to be able to use it as a RAID controller, under a new concept that they have coined and which has been called GRAID SupremeRAID™. This innovative solution unleashes 100% of the performance of NVMe SSD drives, without sacrificing data security.

While traditional RAID technologies generated bottlenecks in SSD drives, the new GRAID solution develops a new hardware/software system, which overcomes this limitation, in NVME SSD U.2 (2,5") and PCI drives. -E 4.0.

GRAID SupremeRAID works by installing a virtual NVMe controller in the operating system (available on Windows Server and major LINUX distributions) and integrates a high-performance PCIe 4.0 device that the processor can handle as a single device without sacrificing performance. , so important in HPC systems.

This technology is already present in the SIE Ladón All-flash 4.0, which allows offering up to 285 TB net, with a performance of up to 6 million IOPS and 100GB/s of network throughput. It is based on the platform R282-Z9G, that can accommodate up to 20 NVME U.2 PCI-E 4.0 disks and up to 128 cores with two AMD Milan processors and a network outlet, based on the Mellanox MCX653105A-EFAT card, which can deliver 4.0 Gb/PCI-E 100 bus s, either in Infiniband HDR 100 Gb/s technology or Ethernet at the same speed.

At this point, with equipment with up to 8 A100 80 GB memory cards per NVLINK bus at 600 Gb/s or the new equipment with up to 10 A40 48 GB cards, both focused on Deep Learning (training) and Artificial Intelligence. For many of you, the question will arise that SIE Ladon systems in many cases are not isolated machines and are integrated within a cluster.

Indeed, this bandwidth at the internal level of the machines forces us to consider how to interconnect them.

First of all, the bet of Sistemas Informáticos Europeos, as an NVIDIA Preferred Partner is for Mellanox HDR Infiniband technology at 100 and 200 Gb/s, but also compatible with Ethernet, to be able to integrate these solutions within classic networks.

Infiniband involves communication packets, with headers and tails, much smaller than the Ethernet protocol, which makes it possible to parallelize processes between machines. Likewise, it offers a latency, which is in the order of magnitude, of that of 2 processors that are on the same board.

However, in order to carry out processes without bottlenecks, for example in the case of GPUs, it has been necessary for the internal NVLINK bus of the equipment to have been extended between machines through the NVSWITCH protocol. This is possible through the Broadcom PCIe Switch hub, as can be seen in the following schematic.

Here you can see, how we can sacrifice one of the PCIe 4.0 x16 slots, to instead of putting a GPU, put a Mellanox HDR card at 200 Gb/s and connect through the Infiniband network, GPUs from different machines, with much less load for the computer's main AMD processor.

All this technology can be seen more clearly in the following NVIDIA video, where this new NVSWITCH technology is very well explained.

However, we are still in the same pitfall and it is like interconnecting all this new speed with our classic Ethernet networks.

Well, first of all, we have a switch with 32 ports with technology linux cumulus. On this technology, they have a very interesting page by Eduardo Collado where this new open standard is explained very well.

As I was saying, we can have Mellanox Ethernet trunks up to 100 and 200 Gb/s, for those teams that need high-speed access to high-performance All-Flash 4.0 solutions.

In addition, SIE offers Mellanox HDR cards 100 Gb/s dual port, where one of the ports can negotiate the Infiniband protocol and the other Ethernet protocol, with the consequent slot and investment savings.

In a single PCI-E 4.0 x16 connectivity card, which can have two slots of this type, to connect simultaneously with the two processors of the machine and avoid bottlenecks.

To distribute less expensive connectivity to the rest of the network, we have the model DXS 3610 54T-SI from the manufacturer D-LINK.

It has 6 100 Gb/s Ethernet ports and that our company can interconnect without problem via QSFP+ with the previously mentioned Mellanox HDR 100 cards.

This innovative switch, with a price of around €7.000 (much cheaper than competitive solutions and with a lifetime warranty), allows these 6 ports to be used for two purposes:

1º It allows up to 16 equal switches to be interconnected in a redundant ring of up to 200 Gb/s, without bottlenecks, stable and manageable by a single IP. For this we use only 2 of the 6 mouths that the switch has.

2º The remaining 4 can be used to connect with equipment that has the dual-port Mellanox HDR card, using both in Ethernet mode up to 200 Gb/s.

In addition, these ports are also compatible with the 40 Gb/s Ethernet standard, which allows them to be used with much lower cost cards and QSFP+ connectivity, such as the Gigabyte CLN4752 card, equipped with the Intel® XL710-BM2 chip and dual port at this speed.

Logically, each one of these switches has another 48 10GbaseT ports (in copper), which allows the distribution of the network, using the cabling already available.

This cabling, if it is category 5e, can be extended up to 50 meters and if it is category 6, up to 100 meters. This is very easy to use and connect, since it is made of copper.

In this way, we can have up to 48 free 100/40 Gb/s Ethernet ports and 576 10GbaseT ports, managed as if they were a single device and with a lifetime guarantee.

So compared to expensive rack switch solutions, they are a competitive advantage. Also, D-Link D-View 7 software and future version 8 (free up to 25 devices). This allows us to visualize and manage this switch stack, as well as others distributed in the network, both at 10G and gigabit.

Currently, all D-Link switches, which SIE sells for clusters and its professional solutions, are manageable and support this software, in addition to many of the WIFI 6 access points, which already support up to gigabit bandwidth.

We have achieved this by being one of the main Nvidia Preferred Partner of reference in Spain