We used 2 Dell PowerEdge 1750 servers with 2 hyperthreaded Intel Xeon 3.06 GHz processors (3 GB RAM, 512k L2 cache) running Linux 2.4.20-30smp kernels for all bandwidth measurements with PMB.

Each machine had InfiniHost MT23108 HCA (a1 silicon) on 64 bit PCIX 133MHz bus with thca-3.1 for Infiniband and LANai 10 on 64 bit PCIX 100MHz bus with gm-2.0.11 driver for Myrinet.

Msg Size (Bytes) Myrinet Latency (usec) TCP Latency (usec) Infiniband Latency (usec) 0 8.1 60.05 5.37 1 8.29 58 5.42 2 8.17 57.94 5.38 4 8.28 57.98 5.55 8 7.93 58.17 5.33 16 8.08 58.33 5.41 32 8.36 58.56 6.35 64 8.63 61.83 6.51 128 9.04 60.5 6.78 256 9.94 64.68 7.32 512 11.56 72.96 8.3 1024 15.31 86.89 10.36 2048 22.48 112.87 12.22 4096 32.51 165.2 16.21 8192 53.15 281.27 24.17 16384 100.19 477.95 41.6 32768 166.3 774.78 60.23 65536 299.07 868.1 97.88 131072 563.76 1404.57 173.45 262144 1093.74 2504.61 415.18 524288 2154.96 4631.62 627.66 1048576 4274.14 9016.31 1238.97 2097152 8519.93 17410.48 2473.25 4194304 16996 34434.4 5496.45
Msg Size (Bytes) Myrinet Latency (usec) TCP Latency (usec) Infiniband Latency (usec) 0 10.53 76.69 5.79 1 11.04 74.56 5.8 2 10.96 73.45 5.78 4 11.13 72.8 5.9 8 11.05 74.51 5.8 16 10.99 74.75 5.94 32 11.1 75.2 6.75 64 11.2 76.73 6.89 128 11.82 79.22 6.97 256 12.28 83.28 7.53 512 13.98 87.75 8.53 1024 17.23 103.34 10.6 2048 25.49 122.59 12.51 4096 38.19 173.73 18.16 8192 58.84 286.01 31.61 16384 115.98 897.9 61.07 32768 186.68 1013.46 98.93 65536 316.94 1562.4 174.54 131072 584.04 2272.94 326.46 262144 1112.82 3925.84 629.54 524288 2176.82 7237.74 1238.2 1048576 4311.33 14727.53 2450.18 2097152 8566.4 30543.05 4919.55 4194304 17117.5 60269.5 9955.6
Msg Size (Bytes) Myrinet Latency (usec) TCP Latency (usec) Infiniband Latency (usec) 0 10.53 76.97 5.75 1 10.61 84.89 5.7 2 10.65 73.03 5.74 4 10.68 72.52 5.74 8 10.65 72.49 5.87 16 10.71 72.79 5.92 32 10.87 73.29 6.67 64 10.84 74.76 6.83 128 11.56 76.98 7 256 12.11 80.98 7.59 512 13.71 87.03 8.59 1024 17.13 102.36 10.56 2048 25.2 122.11 12.64 4096 37.99 186.31 17.78 8192 58.59 284.18 52.29 16384 116.52 1038.93 61.5 32768 182.03 1019.26 99.15 65536 315.35 1573.22 174.61 131072 600.61 2652.98 328.63 262144 1113.71 4818.84 628.82 524288 2174.84 9030.36 1235.07 1048576 4300.27 17614 2448.62 2097152 8985.6 34632.83 4907.53 4194304 17105.81 67963.41 9940.05
Msg Size (Bytes) Myrinet Latency (usec) TCP Latency (usec) Infiniband Latency (usec) 0 18.73 95.87 11.24 1 18.84 96.21 11.5 2 18.83 96.34 11.41 4 18.78 96.27 11.38 8 18.81 96.25 11.33 16 18.81 96.47 11.39 32 18.89 97.06 12.9 64 19.05 98.9 12.89 128 19.18 100.4 13.22 256 19.79 104.45 13.88 512 20.97 112.9 15.1 1024 24.65 136.73 18.28 2048 35.29 138.55 21.46 4096 54.33 217.48 34.35 8192 97.08 391.92 56.11 16384 226.68 1015.34 120.21 32768 358.78 1421.01 195.86 65536 624.59 2623.98 346.87 131072 1157.33 4505.25 650.39 262144 2220.68 8528.48 1255.18 524288 4343.19 16620.89 2468.82 1048576 8662.19 32379.72 4897.59 2097152 17240.38 62717.04 9782.4 4194304 34221.1 126071.56 19866.44

To meaure bandwidth between IA-64 machines over Myrinet, we used 2 Intel Itanium 2 duals with 1.3 Ghz processors (4 GB RAM) running Suse SLES8 system with 2.4.19 kernel. GigE measurement were done with 2 Intel Itanium 2 duals with 900 Mhz processors (2 GB RAM) running 2.4.21 kernel.

For Myrinet, we used LANai 10.0 on 64 bit PCIX 100MHz bus with gm-2.0.8 driver. For Gigabit Ethernet, we had Broadcom Gigabit Ethernet adapter with bcm5700 driver.

Msg Size (Bytes) Myrinet Latency (usec) TCP Latency (usec) Infiniband Latency (usec) Shmem Latency (usec) 0 11.43 73.35 8.04 1.63 1 12.29 72.54 8.22 1.68 2 12.48 73.59 8.25 1.75 4 12.48 72.42 8.14 1.78 8 12.15 73.71 8.29 1.72 16 12.96 72.57 8.9 1.73 32 13.27 74.93 9.11 1.79 64 12.73 73.65 9.25 2.1 128 13.23 77.87 9.76 2.27 256 14.26 80 10.28 2.55 512 16.67 89.6 11.62 3.08 1024 20.14 103.04 13.51 3.69 2048 28.15 122.5 15.76 4.75 4096 39.4 147.67 19.96 6.64 8192 63.54 171.59 28.83 10.28 16384 119.89 405.27 54.78 20.29 32768 189.2 539.36 76.96 33.96 65536 332.14 849.93 121.14 61.58 131072 617.02 1530.84 210.28 116.11 262144 1181.63 2793.1 387.92 228.78 524288 2330.24 4882.39 743.69 724.71 1048576 4562.07 9352.01 1453.71 1711.19 2097152 9073.9 18279.95 2903.43 3772.4 4194304 18321.3 36126.51 5774.45 6947.45
Msg Size (Bytes) Myrinet Latency (usec) TCP Latency (usec) Infiniband Latency (usec) Shmem Latency (usec) 0 14.17 114.23 8.57 2.07 1 14.84 112.33 8.83 2.17 2 13.97 99.84 8.95 2.19 4 14.88 95.2 8.81 2.29 8 16.12 97.39 9 2.22 16 14.12 95.62 9.56 2.25 32 16.06 98.18 9.86 2.3 64 15.42 94.47 9.8 2.84 128 16.76 101.91 10.38 2.97 256 18.55 104.17 10.82 3.32 512 20.81 113.22 12.57 3.85 1024 23.07 124.98 14.09 4.49 2048 31.91 155.59 17.77 5.73 4096 45.36 169.74 22.69 8.14 8192 70.12 185.27 33.64 12.86 16384 137.99 439.16 78.38 25.42 32768 217.35 601.62 119.78 43.66 65536 362.95 1563.15 201.82 80.26 131072 696.82 2714.58 367.26 152.74 262144 1307.09 4947.29 695.85 317.81 524288 2426.44 9367.86 1358.75 1379.69 1048576 4691.18 17769.85 2683.75 3053.48 2097152 9311.04 35701.85 5359.9 6393.35 4194304 19711.8 62048.01 10631.2 12945.32
Msg Size (Bytes) Myrinet Latency (usec) TCP Latency (usec) Infiniband Latency (usec) Shmem Latency (usec) 0 14.61 97.07 8.99 2.46 1 15.38 95.65 9.14 2.47 2 13.92 97.56 9.05 2.52 4 16.75 95.72 9.1 2.6 8 14.86 97.53 9.05 2.5 16 13.96 95.68 9.93 2.52 32 14.99 98.64 9.91 2.71 64 15.73 97.58 10.13 3.19 128 15.89 100.84 10.56 3.45 256 15.99 104.9 11.26 3.76 512 19.26 113.36 12.6 4.55 1024 21.89 128.46 14.61 5.38 2048 31.74 150.85 17.2 6.77 4096 44.88 170.37 22.73 9.58 8192 72.62 180.56 33.52 14.52 16384 141.06 597.55 77.21 27.75 32768 213.23 888.08 118.25 47.37 65536 365.31 1557.59 200.83 87.47 131072 658.52 2687.27 365.51 166.97 262144 1237.03 4914.76 694.97 332.14 524288 2445.89 9421.34 1357.82 1267.91 1048576 4662.65 18213.49 2680.54 2985.53 2097152 9256.12 35647.72 5314.8 6259.65 4194304 18379.4 63960.35 10627.4 12609.05
Msg Size (Bytes) Myrinet Latency (usec) TCP Latency (usec) Infiniband Latency (usec) Shmem Latency (usec) 0 23.35 141.06 15.49 3.98 1 24.38 139.48 15.53 4.17 2 24.74 141.35 15.56 4.29 4 24.44 141.33 15.57 4.36 8 28.07 139.6 15.64 4.23 16 26.69 141.89 17.11 4.31 32 25.03 140.35 17.21 4.51 64 25.47 143.67 17.49 5.13 128 25.1 144.93 17.98 5.65 256 27.65 150.37 18.74 6.06 512 27.66 158.27 20.08 6.98 1024 30.48 136.73 23.43 7.89 2048 44.01 168.73 27.8 10.49 4096 66.87 236.95 37.84 15.96 8192 116.09 281.9 59.46 27.57 16384 270.06 923.39 160.13 63.07 32768 437.77 1787.94 245.2 111.31 65536 726.42 3067.43 414.45 206.99 131072 1310.2 5324.52 744.3 395.26 262144 2617.53 9784.23 1409.75 1089.17 524288 4875.97 18707.89 2745.47 2774.09 1048576 9670.91 36524.75 5395.29 6017.49 2097152 19320.05 71885.38 10702.43 12203.07 4194304 37614.31 125865.45 21416.95 24526.85

Back to the VMI2 web page.

Contact webmaster with questions regarding this page. Last updated March 23, 2005. All rights reserved. ©2002, 2003, 2004 Board of Trustees of the University of Illinois.