We used 2 Dell PowerEdge 1750 servers with 2 hyperthreaded Intel Xeon 3.06 GHz processors (3 GB RAM, 512k L2 cache) running Linux 2.4.20-30smp kernels for all bandwidth measurements with PMB.

Each machine had InfiniHost MT23108 HCA (a1 silicon) on 64 bit PCIX 133MHz bus with thca-3.1 for Infiniband and LANai 10 on 64 bit PCIX 100MHz bus with gm-2.0.11 driver for Myrinet.

Msg Size (Bytes) Myrinet Bandwidth (MB/sec) TCP Bandwidth (MB/sec) Infiniband Bandwidth (MB/sec) 0 0 0 0 1 0.12 0.02 0.18 2 0.23 0.03 0.35 4 0.46 0.07 0.69 8 0.96 0.13 1.43 16 1.89 0.26 2.82 32 3.65 0.52 4.81 64 7.08 0.99 9.38 128 13.5 2.02 18.01 256 24.57 3.77 33.35 512 42.23 6.69 58.83 1024 63.79 11.24 94.3 2048 86.88 17.3 159.77 4096 120.14 23.64 240.98 8192 146.99 27.78 323.19 16384 155.95 32.69 375.56 32768 187.92 40.33 518.88 65536 208.98 72 638.54 131072 221.73 89 720.67 262144 228.57 99.82 602.15 524288 232.02 107.95 796.61 1048576 233.97 110.91 807.12 2097152 234.74 114.87 808.65 4194304 235.35 116.16 727.74
Msg Size (Bytes) Myrinet Bandwidth (MB/sec) TCP Bandwidth (MB/sec) Infiniband Bandwidth (MB/sec) 0 0 0 0 1 0.09 0.01 0.16 2 0.17 0.03 0.33 4 0.34 0.05 0.65 8 0.69 0.1 1.32 16 1.39 0.2 2.57 32 2.75 0.41 4.52 64 5.45 0.8 8.86 128 10.33 1.54 17.51 256 19.88 2.93 32.43 512 34.93 5.56 57.27 1024 56.67 9.45 92.11 2048 76.61 15.93 156.19 4096 102.27 22.48 215.08 8192 132.77 27.32 247.18 16384 134.72 17.4 255.85 32768 167.4 30.83 315.88 65536 197.2 40 358.09 131072 214.03 54.99 382.9 262144 224.65 63.68 397.11 524288 229.69 69.08 403.81 1048576 231.95 67.9 408.13 2097152 233.47 65.48 406.54 4194304 233.68 66.37 401.78
Msg Size (Bytes) Myrinet Bandwidth (MB/sec) TCP Bandwidth (MB/sec) Infiniband Bandwidth (MB/sec) 0 0 0 0 1 0.18 0.02 0.33 2 0.36 0.05 0.66 4 0.71 0.11 1.32 8 1.43 0.21 2.59 16 2.84 0.42 5.14 32 5.6 0.83 9.13 64 11.26 1.63 17.79 128 21.1 3.17 34.8 256 40.17 6.03 64.34 512 70.94 11.21 112.52 1024 113.69 19.07 184.75 2048 154.89 31.97 308.36 4096 205.64 41.92 439.4 8192 266.52 54.97 298.57 16384 268.15 30.08 507.85 32768 343.31 61.31 630.04 65536 396.25 79.44 715.56 131072 416.15 94.19 760.69 262144 448.91 103.72 794.71 524288 459.75 110.67 809.58 1048576 465.08 113.51 816.56 2097152 445.1 115.45 814.75 4194304 467.61 117.66 804.55
Msg Size (Bytes) Myrinet Bandwidth (MB/sec) TCP Bandwidth (MB/sec) Infiniband Bandwidth (MB/sec) 0 0 0 0 1 0.2 0.04 0.32 2 0.41 0.08 0.66 4 0.81 0.16 1.32 8 1.62 0.32 2.65 16 3.24 0.63 5.28 32 6.46 1.26 9.33 64 12.81 2.47 18.74 128 25.43 4.86 36.47 256 49.34 9.35 69.49 512 93.13 17.29 127.86 1024 158.37 28.56 211.87 2048 221.27 56.34 361.2 4096 287.52 71.8 452.42 8192 321.84 79.73 555.41 16384 275.69 61.55 519.19 32768 348.37 87.95 637.59 65536 400.22 95.27 719.72 131072 432.01 110.94 768.23 262144 450.3 117.21 796.05 524288 460.44 120.28 809.35 1048576 461.77 123.52 816.21 2097152 464.02 127.5 817.12 4194304 467.21 126.86 804.73

To meaure bandwidth between IA-64 machines over Myrinet, we used 2 Intel Itanium 2 duals with 1.3 Ghz processors (4 GB RAM) running Suse SLES8 system with 2.4.21 kernel. GigE measurement were done with 2 Intel Itanium 2 duals with 900 Mhz processors (2 GB RAM) running 2.4.21 kernel.

For Myrinet, we used LANai 10 on 64 bit PCIX 100MHz bus with gm-2.0.8 driver. For Gigabit Ethernet, we had Broadcom Gigabit Ethernet Adapter with bcm5700 driver.

Msg Size (Bytes) Myrinet Bandwidth (MB/sec) TCP Bandwidth (MB/sec) Infiniband Bandwidth (MB/sec) Shmem Bandwidth (MB/sec) 0 0 0 0 0 1 0.08 0.01 0.12 0.57 2 0.15 0.03 0.23 1.09 4 0.31 0.05 0.47 2.15 8 0.63 0.1 0.92 4.44 16 1.18 0.21 1.71 8.84 32 2.3 0.41 3.35 17.03 64 4.79 0.83 6.6 29.06 128 9.23 1.57 12.51 53.8 256 17.12 3.05 23.76 95.72 512 29.29 5.45 42.01 158.48 1024 48.49 9.48 72.26 264.69 2048 69.39 15.94 123.91 411.36 4096 99.15 26.45 195.7 588.34 8192 122.96 45.53 270.95 759.93 16384 130.33 38.55 285.22 770.16 32768 165.17 57.94 406.04 920.31 65536 188.18 73.54 515.94 1014.95 131072 202.59 81.65 594.45 1076.54 262144 211.57 89.51 644.45 1092.78 524288 214.57 102.41 672.32 689.93 1048576 219.2 106.93 687.89 584.39 2097152 220.41 109.41 688.84 530.17 4194304 218.33 110.72 692.71 575.75
Msg Size (Bytes) Myrinet Bandwidth (MB/sec) TCP Bandwidth (MB/sec) Infiniband Bandwidth (MB/sec) Shmem Bandwidth (MB/sec) 0 0 0 0 0 1 0.06 0.01 0.11 0.44 2 0.14 0.02 0.21 0.87 4 0.26 0.04 0.43 1.67 8 0.47 0.08 0.85 3.43 16 1.08 0.16 1.6 6.78 32 1.9 0.31 3.1 13.29 64 3.96 0.65 6.23 21.51 128 7.28 1.2 11.76 41.14 256 13.16 2.34 22.56 73.56 512 23.46 4.31 38.85 126.96 1024 42.33 7.81 69.31 217.26 2048 61.21 12.55 109.89 340.86 4096 86.12 23.01 172.19 479.6 8192 111.42 42.17 232.24 607.6 16384 113.23 35.58 199.35 614.58 32768 143.78 51.94 260.89 715.84 65536 172.2 39.98 309.69 778.68 131072 179.39 46.05 340.36 818.36 262144 191.26 50.53 359.27 786.64 524288 206.06 53.37 367.99 362.4 1048576 213.17 56.28 372.61 327.5 2097152 214.8 56.02 373.14 312.83 4194304 202.92 64.47 376.25 308.99
Msg Size (Bytes) Myrinet Bandwidth (MB/sec) TCP Bandwidth (MB/sec) Infiniband Bandwidth (MB/sec) Shmem Bandwidth (MB/sec) 0 0 0 0 0 1 0.12 0.02 0.21 0.77 2 0.27 0.04 0.42 1.51 4 0.42 0.08 0.83 2.94 8 1.02 0.16 1.68 6.09 16 2.17 0.32 3.06 12.12 32 4.03 0.62 6.13 22.52 64 7.71 1.25 11.97 38.29 128 15.3 2.42 22.98 70.64 256 30.4 4.65 43.1 129.73 512 50.25 8.6 77.33 214.63 1024 88.88 15.2 133.21 363.3 2048 122.63 25.89 226.79 576.56 4096 173.65 45.84 342.41 815.83 8192 214.89 86.53 465.94 1075.72 16384 221.49 52.29 404.34 1126.16 32768 293 70.38 528.31 1319.4 65536 342.01 80.24 622.01 1429 131072 379.47 93 683.74 1497.23 262144 403.94 101.7 718.91 1505.32 524288 408.57 106.11 736.4 788.67 1048576 428.7 109.76 745.62 669.85 2097152 431.96 112.15 752.38 638.9 4194304 435.24 125.02 752.32 634.42
Msg Size (Bytes) Myrinet Bandwidth (MB/sec) TCP Bandwidth (MB/sec) Infiniband Bandwidth (MB/sec) Shmem Bandwidth (MB/sec) 0 0 0 0 0 1 0.16 0.03 0.24 0.91 2 0.31 0.05 0.49 1.78 4 0.62 0.11 0.98 3.5 8 1.09 0.22 1.95 7.22 16 2.28 0.43 3.56 14.16 32 4.88 0.87 7.07 27.09 64 9.57 1.7 13.93 47.58 128 19.4 3.37 27.07 86.39 256 35.15 6.49 52 161.04 512 70.47 12.33 97.01 279.58 1024 127.91 28.55 166.57 494.97 2048 177.21 46.3 280.31 745.03 4096 233.41 65.92 412.5 979.01 8192 269.05 110.83 524.88 1133.35 16384 231.37 67.67 390.22 991.03 32768 285.46 69.91 509.66 1122.96 65536 344.11 81.49 603.14 1207.78 131072 381.55 93.9 671.55 1264.99 262144 381.98 102.19 709.2 918.08 524288 410.12 106.89 728.21 720.92 1048576 413.56 109.49 741.27 664.72 2097152 414.02 111.26 747.24 655.57 4194304 425.3 127.09 746.97 652.01

Back to the VMI2 web page.

Contact webmaster with questions regarding this page. Last updated March 23, 2005. All rights reserved. ©2002, 2003, 2004 Board of Trustees of the University of Illinois.