![[NCSA]](http://vmi.ncsa.uiuc.edu/images/footer-ncsalogo.gif){kind=small} |
|
 |
VMI and MPICH-VMI Bandwidth
The VMI bandwidth data graphed below for each kind of interconnect was measured using the bandwidth benchmark installed in the benchmarks directory of VMI install tree. bandwidth measures the through-put of an interconnect for a given message size s using the VMI messaging interface by starting a clock and streaming n messages, each of size s, to its peer bandwidth process running on a different host. The peer process, after receiving n messages, sends back an acknowledgement. After the acknowledgement is received, bandwidth stops the clock and computes the time spent on streaming n*s bytes of data. This time is divided by n to get the throughput. For each value of bandwidth plotted on the graphs below for a given message size, a total of 1024 message were streamed over the interconnect. bandwidth can use both the stream and RDMA messaging interface of VMI for communications. For the graphs plotted below, stream interface has been used for messages less then 16k. For messages over 16k RDMA has been used.
The MPICH-VMI bandwidth data was measured using bandwidth_mpi installed in the tools directory of MPICH-VMI install tree. bandwidth_mpi measures the bandwith the same way as bandwidth except that it uses MPI's send/recv interface to communicate data. bandwidth_mpi can run in both blocking (MPI_Send/MPI_Recv) and non-blocking (MPI_Isend/MPI_Irecv) modes to measure the bandwidth.
If you look at the bandwidth curves for MPICH-VMI, you will notice they knee a a little at 16k message size before ascending again. This knee occurs because MPICH-VMI, by default, switches from eager protocol to rendezvous protocol for sending/receiving data at 16k (The default message length over which rendezvous is used can also be set by the user on command line). In eager protocol, the sender sends the data immediately. If a receive is not posted by the destination, some space is allocated by the messaging layer to store the message. In rendezvous protocol, however, the sender sends the data only after a corresponding receive has been posted by the receiver. So every rendezvous send/recv requires a handshake between the sender and the receiver that causes some overhead. The knees in the graphs are caused by this overhead. Higher the latency of the inteconnect, greater is the overhead. That is why Infiniband, with the smallest latency among the interconnnects used, is affected the least due to this overhead, while TCP and Myrinet with relatively higher latencies show a greater affect. The adverse affect of latency over bandwidth would have been worse if rendezvous had been implemented using stream instead of RDMA in MPICH-VMI. Unlike stream, that is used to implement eager protocol, RDMA is more efficient for large messages, since it avoids a memory copy by depositing data directly into the receive buffer of the receiver. Using RDMA, therefore, tends to mitigate the adverse affect of latency on bandwidth in rendezvous protocol.
IA-32 Bandwith Measurements
To meaure bandwidth between IA-32 machines, we used 2 Dell PowerEdge 1750 servers with 2 hyperthreaded Intel Xeon 3.06 GHz processors (3 GB RAM, 512k L2 cache) running Linux 2.4.20-30smp kernels for all bandwidth measurements.
For Myrinet, both machines had LANai 10 on 64 bit PCIX 100MHz bus with gm-2.0.11 driver.
|
 |
For Infiniband, both machines had InfiniHost MT23108 HCA (a1 silicon) on 64 bit PCIX 133MHz bus with thca-3.1.
|
 |
For Ethernet, Broadcom Gigabit Ethernet NIC with bcm5700 driver was used.
|
 |
IA-64 Bandwith Measurements
To meaure bandwidth between IA-64 machines over Myrinet, we used 2 Intel Itanium 2 duals with 1.3 Ghz processors (4 GB RAM) running Suse SLES8 system with 2.4.21 kernel. Infiniband, GigE and shared memort measurements were done with 2 Intel Itanium 2 duals with 900 Mhz processors (2 GB RAM) running 2.4.21 kernel.
For Infiniband, both machines had InfiniHost MT23108 HCA (a1 silicon) on 64 bit PCIX 133MHz bus with thca-3.1.
|
|
For Myrinet, both machines had LANai 10 on 64 bit PCIX 100MHz bus with gm-2.0.8 driver.
|
 |
Broadcom Gigabit Ethernet NIC was bcm5700 driver was used for Gigabit ethernet.
|
|
Shared memory.
|
|
![[NCSA]](http://vmi.ncsa.uiuc.edu/images/newlogo.gif){kind=small}
|
Contact
webmaster
|
