CUDA bandwidth test (PCIe 1.0, x8 vs. x16)

Just got my new GeForce 9800GT (Club3D 9800GT Eco). Though my mainboard (Asus P5W DH Deluxe) does not have enough physical PCIe lanes to run two cards at x16 (and no PCIe 2.0, which would allow the same bandwidth for x8 as earlier revisions did for x16), I decided to put it into the second slot and let it run together with my 7900GT at x8. Because I was really interested in the real – not only theoretical – bandwidth difference, I just tried the bandwidth test from the CUDA SDK. I have not even looked at the code, so I don’t know how meaningful this really is, but maybe it’s interesting nevertheless:

9800GT as primary and only card (x16) vs. 7900GT as primary (display-responsible), 9800GT as secondary card (both at x8):

• Host to device bandwidth: 1920,3 MiB/s vs. 1315,5 MiB/s (x8 yields 68.5% of x16 performance)
• Device to host bandwidth: 1460,4 MiB/s vs. 1338,5 MiB/s (x8 yields 91,7% of x16 performance)
• Device to device bandwidth: 37211 MiB/s vs. 37868 MiB/s (x8 yields 101,8% of x16 performance, though this should be within measuring tolerance)

Note that theoretical x16 bandwidth should be about 4000 MiB/s :O

So I had to decide what was more important to me: Crashing my machine all the time because of my bad CUDA development skills while running at full speed or using the 9800GT as the secondary card and accept the bandwidth drop. Thinking about it, most of the time bandwidth will probably not be the most important thing in applications I am going to develop, but they should be much more compute-intensive. Option #2 is the choice for me!

Also, let me say that in almost all example applications no difference was noticeable between the both setups, because most of them are probably also not that bandwidth demanding. Running the 9800GT as a CUDA-only card has one more advantage: CUDA-applications will stop crashing the compositing system in KDE4 :)

Update: I noticed there are different modes for the bandwidth test. I will probably test these later this week, just because I’m interested :D ..could well be that the x16 bandwidth can be utilized better.