Guide FPGA Watercooling guide

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#1
I have been answering lots of FPGA watercooling questions on Discord lately, so writing it all up here:

List of places to buy components:
https://store.mineority.io/
https://shop.fpga.guide/

Amazon
http://www.frozencpu.com/
https://modmymods.com/
http://www.sidewindercomputers.com/
http://www.performance-pcs.com/
http://www.koolertek.com/

Watercooling Components:

FPGA Block: Cools the FPGA die and power components. It is possible with some DIY work to use a CPU waterblock,
but most people will buy one specific to their FPGA.

VCU1525/BCU1525/XBB1525 - TUL makes the reference waterblock for these.
The block is a full cover block and covers the entire pcb.
TUL.png

They are available from the FPGA Guide Shop (Shipping soon, $200) and the Mineority store (Shipping later, $150)
Fpga cooling is copper, VRM cooling is anodized aluminum.
Block is high restriction, use a pump with good head pressure, especially if putting multiple in series.

DimasTech is also releasing a waterblock for the BCU/VCU, similar to the TUL block, but all COPPER. :)

Preorder price is currently $200, Shipping estimate is December.
http://www.dimastechusa.com/dimastech-xmv-cool-waterblock-for-fpga-board-xilinx-vcu1525

Bittware CPV13 - The CPV13 will be offered with a waterblock direct from Bittware.
This is a 2 part water block, 1 part to cool the fpga die, and another to cool the VRMS.
CPV13.png

The liquid cooled option is $250 extra, or $350 extra if you are hosting your card (but you don't need this guide if you are hosting your card :D )

The F100 also has a waterblock option for £240, but it has not be pictured yet.
Unknown what type of water block it will be.
F100.jpg

Pumps: Pumps water (or other fluid) through the loop. Pumps have a few specs we care about:

1) Flowrate or GPM the number of gallons the pump can move per minuite when nothing is restricting the flow. (Actual flow rate will be much lower than the rating when going through all the parts of the loop)

2) Head pressure or the how high (in Feet or meters) your pump can pump water up in the open air before the flow rate stops.

3) Voltage and current, most pumps will run on 12Vdc, but some are 24vdc, and others are 120/240vac.

Most people will want to focus more on getting a pump with a higher Head pressure, especially If you plan to run multiple FPGA's in series,
because your loop will be very restrictive.

Pumps can be ran in parallel or series,
by running 2 pumps series you double the Head pressure, but don't improve the max flow rate.
running 2 pumps in parallel doubles the max flowrate, but doesn't improve the head pressure.

If a watercool pump stops working it will almost always fail "open" meaning having 2 pumps in series gives you redundancy.
But 2 pumps in parallel gives you no redundancy:

Pumps.png

The issues with parallel can be avoided with a one way check valve.

There are lots of choices in pumps, I'm going to cover a few of the best ones:

Swiftech MCP50X - 3.8 GPM - 15FT head pressure - Runs on 12v

1536886518508.png

$70 - Great all around pump, has above average head pressure. I recommend this one to a lot of people.

DDC 3.2/DDC 3.0/MCP355/MCP35X/MCP350 - these very similar to the MCP50X,
but they run hotter and need a heatsink, and have slightly lower head pressure, and are usually more expensive.

If available in your area, I would recommend the MCP50X instead of any of these pumps.

1536886870562.png

Cost from $70 to $110, depending on the model.

MCP655/D5 - 5.5GPM - 13Ft Head pressure - 12v or 24v (24 not required for full speed)

Another good pump, Higher maximum flow, but lower head pressure compared to the MCP50x

1536887372913.png

There are a few models of this pump, the cheapest is the MCP655B, which does not offer speed adjustment and is fixed at ~90%
There are also models with a switch on the pack to set the speed, and other models can be speed controlled via a computers pwm fan header.
$70 - $120

Radiator: Just like your car, the radiator is the part that moves the heat from the water to the air.

There are a ton of radiator models, so I'm not even going to try to cover all of them, instead I'll just tell you what to look for.

**Update - make sure your radiator has a Copper or brass core, not aluminum.
(Aluminum fins are fine, just the part that touches the water needs to be copper/brass)

Thermaltake apparently makes a lineup of aluminum radiators, they should be avoided.
Also most automotive radiators are aluminum, so avoid those too.

1) Size - radiators are typically designed for 120mm or 140mm fans, a 240 radiator is 120x240mm designed for 2x 120mm fans
Radiators are commonly sold in sizes: 120, 240, 360, 480, and 140, 280, 420, and 560
There are also a few 1080 radiators, which are 360x360 and support 9 120mm fans.

2) thickness - radiators come in a range of thicknesses, thinner radiators work better for slower quieter fans
Thicker radiators can dissipate more heat, but need stronger, louder fans.

3) FPI - fins per inch. FPI works the same way as thickness, the more fpi, the more heat it can disapate,
but you also need strong louder fans for higher fpi radiators:
1536887989331.png

Barbs: barbs connect your parts to your tubing. All computer watercooling components use the same thread for barbs G1/4
computer watercooling typical uses either 1/2in tubing or 3/8 tubing.
The difference between 1/2 and 3/8 is minimal, but 1/2 isn't any more expensive, so I recommend you go with 1/2
I'm skipping compression fittings for now, they are quite a bit more expensive, and not really necessary. (but they do look pretty)

1536889223882.png

Tubing: Tubes to connect all the different parts of your watercooling system.
Typical tubing is flexible and can be cut with Scissors or a razor blade.
You want to focus on the inside diameter ID of your tubing and match that to your barbs size.

There are lots of options for tubing as well, below are 2 good options:

If your loop is indoors and doesn't get any sunlight, clear tubing is fine:
PrimoFlex LRT is my recommendation, you can get it on amazon in 10ft packs.

If your loop is exposed to sunlight, you should consider something that doesn't let light through (Prevents growth)
Tygon R3400 is a good option.

Hose Clamp: Keeps your hose from coming off your barb and spraying water everywhere.
Get them on amazon, or at your local auto part store.
1536888808964.png

Reservoir: not super important, the size of your reservoir has nothing to do with your temperatures, it is just there to make filling and draining your loop easier.
Pick one you think looks good and isn't too expensive.
1536888972897.png
 

Attachments

Last edited:
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#2
Part 2 - Recommended Setups: (1-8 FPGAs)

Radiator:
We want approximately 350W of cooling for VU9P based FPGAs (VCU1525/BCU1525/XBB1525/F100)
and the larger VU13P (CPV-13) will need 500W of cooling.

Shoot for at least 120mm worth of radiator per 175W if you want your system to be quiet.
For example if you have 3 BCU1525's and 1 CPV-13 you need 1550W of cooling, or 3 thin 360mm radiators.
http://www.frozencpu.com/products/1...Series_Triple_120mm_-Black.html?tl=g30c95s161

If you are going with loud fans and thicker / higher FPI radiators you can get away with close to double that ~300W
So for the same 1550W of cooling you need 2 thick 360mm radiators.
http://www.frozencpu.com/products/2...HX-360XC_No_Nozzles_.html?tl=g30c95s161#blank

And going with more radiators isn't going to hurt, and may drop your temps by a few degrees extra.

Pump:
The ideal pump setup is currently unknown because the blocks haven't been tested yet,
But my recommendation is for 1-3 FPGAs get a single MCP50X, for 4-8 get a pair of MCP50X in series.
https://www.amazon.com/Swiftech-MCP...ie=UTF8&qid=1536890822&sr=8-1&keywords=mcp50x

Barbs:
Each part in the loop needs 2 barbs, most parts don't include them. For our example loop with 4 fpgas, 2 pumps, 3 radiators and 1 reserviour we need 20 barbs. Any G1/4 1/2in barb should do fine
https://www.amazon.com/Bitspower-Fitting-Tubing-Sparkle-4-pack/dp/B017JQ3TLE/ref=sr_1_15?ie=UTF8&qid=1536890953&sr=8-15&keywords=1/2+g1/4+barb

Tubing:
The ID inside Diameter of your tubing needs to match your barb size 1/2in typically.
Try to visualize where everything will go and think about how much tubing you will need for your setup, then double it to be safe. You don't really want to run out to Home Depot because you ran out of tubing.
https://www.amazon.com/PrimoFlex-Ad...TF8&qid=1536891015&sr=8-2&keywords=lrt+tubing

Hose Clamps:
You need 1 per barb, the hose clamps need to be sized according to the OD outside diameter of your tubing, 3/4 based on the above tubing
https://www.amazon.com/Anpro-Stainl...TF8&qid=1536891280&sr=8-7&keywords=hose+clamp

Reservoir:
Whatever you think looks good, this one is pretty cheap, so that is what I'm using for this example.
https://www.amazon.com/Maximized-Co...91334&sr=8-2&keywords=reservoir+water+cooling

Fluid:
Plain distilled water is actually one of the best non toxic coolants on the planet (how convenient :D )
You can find distilled water at your local Walmart/grocery store.
You will want to add a small amount of treatment to the water to pervent stuff from growing in it.
(The LRT tubing above includes such a treatment "Liquid Utopia".

Update: The TUL BCU waterblock has mixed metals, so Liquid Utopia is a better option than pt nuke or a silver kill coil.
 
Last edited:
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#3
Part3 - Assembly and Testing

Plan out your loop, where everything is going to go.
Try not to put any Liquid tubing connections too close to each other,
tight bends with hoses can kink. If this is unavoidable, you can buy plasic/metal 90/45 degree parts.

As far as liquid cooling components go the order is mostly not important.
except the pump should go right after/below your radiator.
This makes filling your loop with water a lot easier.
In general it is better for your reservoir to be at a higher point in your loop, but not required.
You can also add a T-splitter and a valve at the bottom of your loop to make draining easy.

6 BCU loop.png

Barbs - when installing barbs on all your parts, you want them tight, but not too tight.
I usually screw them on as tight as I can by hand, then get and then get them a little bit tighter with a small wrench.
Your barbs are usually metal, but the water block they screw into are often plastic/acrylic/acetal any of which could crack if you way over tighten them.

Hose camps - assuming you have a metal barb, you can go very tight with hose clams.
But if you tightening down on something plastic, same as before, don't over tighten or you will crack the barb.

Install everything where you want it, I usually do all the tubing last, since it gets in the way more than anything else.
Cut each piece of tubing to the right length and install.

Fill your loop with your chosen coolant, pour it in the reservoir and let flow into everything.
Water usually won't be able to fill the whole loop on its own, power up your pump separate from your computer,
So it can push out all the air, keep topping off the reservoir until the air bubbles are gone.

Check each fitting/connection for leaks.
If you find one, it may be tempting to try to tighten it in the loop, but I have found this rarely works,
and it is usually best to take that part apart to fix the leak properly.

A water leak has almost no chance of damaging your components when the are powered OFF.
This is why running the pump separate from the rest of your computer is important.
If something does get wet, let it dry for longer than you think it needs, often taking whatever got wet apart will help in drying.
Rubbing alcohol can also be used to help get water out of something that got wet.

I would usually let a water loop run for 24 hours before powering it on, but I was personally to impatient to wait that long for leak testing lol.
 
Last edited:

Soul of Jacobeh

Administrator
Staff member
Administrator
Joined
Sep 10, 2018
Messages
27
Likes
18
#4
I truly hate that swiftech pump, but I'm convinced mine's damaged.

Thank you very much for these guides. This is very helpful and informative.
 
Last edited:

Morybundus

Newcomer
Member
Joined
Sep 15, 2018
Messages
4
Likes
0
#5
did you measure outgoing temperature? I was thinking about the heat exchanger water water for my whirlpool or boiler : O (with two bcus)
 

Soul of Jacobeh

Administrator
Staff member
Administrator
Joined
Sep 10, 2018
Messages
27
Likes
18
#6
I'm not sure if it was discussed, but it's worth covering:

Do not mix aluminum components with other metals in the same loop. Even ones with a zinc coating can degrade and begin galvanic corrosion in the absence of a powerful inhibitor in the coolant.
 

Q600

Newcomer
Member
Joined
Sep 15, 2018
Messages
4
Likes
1
#7
OK - let's see. I'm waiting for 2xBCU1525. I'm working on serial connected water cooling. The radiator I choose is Black Ice Nemesis GTR (140-600W/280-1300W/420-2150W/560-2800W).

My plan is to connect everything in serial. It will be: Reservoir/Pump->BCU1->Radiator1->BCU2->Radiator2.

Does it make sense?

If yes - which radiator shoud I use? 2 x 280 should be enough?

On the other hand - If I choose option with one big radiator (Reservoir/Pump->BCU1->BCU2->Radiator) - 560 Would be enough, right? But there's one question I can't answer - how many degrees more will be on the output of second BCU?
 
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#8
Both options sound fine.
Going with 2 rads will lower your 2nd fpga by about 1 degree C, depending on pump/flowrate.

What fans are you going to use?
 

Q600

Newcomer
Member
Joined
Sep 15, 2018
Messages
4
Likes
1
#9
Noctua 140mm NF-A14 IndustrialPPC-2000 - 1st time one side, later will use push/pull. But I think it won't be much less degrees
 
Last edited:
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#11

valthonis

Newcomer
Member
Joined
Oct 6, 2018
Messages
10
Likes
1
#12
I'm going to be running 8 BCU. I was thinking of dropping everything into one case. From reading your guide it sounds like that might not be a good idea. I'm now leaning towards running 2 cases with 4 BCU in each one. Do you have any case recommendations? I'd like to be able to rack mount the cases.
 
Last edited:
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#13
It looks like server cases have room for 3x 120mm fans in the front and 3x 120mm in the back.
I would think 2 high end 360 radiators with PFC1212DE fans would be ok for 2800W.

or you could use external radiators and put all 8 in the same case.
 

valthonis

Newcomer
Member
Joined
Oct 6, 2018
Messages
10
Likes
1
#14
It looks like server cases have room for 3x 120mm fans in the front and 3x 120mm in the back.
I would think 2 high end 360 radiators with PFC1212DE fans would be ok for 2800W.

or you could use external radiators and put all 8 in the same case.
Any specific case you recommend? I was looking at this one from thermaltake one as well instead of a server case:

https://www.thermaltake.com/Chassis/Super_Tower_/Core/C_00002895/Core_WP200/Design.htm

As for radiators, the ones you listed at the beginning would work for me? Thanks for your help appreciate it!
 

valthonis

Newcomer
Member
Joined
Oct 6, 2018
Messages
10
Likes
1
#16
I have two of those hydra II cases right now. I have 60 ASICS and 4 GPU rigs, 8 cards each. Three of which are in server cases like the hydra II and hydra III (4U) and the last one is in an open air frame. Ideally, I'd like to bypass risers. So I'm looking into mobo's that have 4 PCIe x16 slots spaces out evenly to fit 4 cards without any issues. With that thermaltake case I linked, it has room for two builds in the chasis so I can just use that one case for all 8 cards but on 2 mobos.
 
Last edited:
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#17
Ah risers-less makes sense now.
Should work fine, and that case looks like it will hold several 480mm radiators.
 

valthonis

Newcomer
Member
Joined
Oct 6, 2018
Messages
10
Likes
1
#18
Cool. Yeah, I was thinking of running 4 480mm rads in the build. Any recommendation on the pumps?
 
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#19
4x 480's sounds good to me.

I have 6 BCU's coming and have 2 MCP50X Pump, I think 2 will be fine for you too
but until someone actually tries running a several of these water blocks in the same loop we won't know for sure.
 

valthonis

Newcomer
Member
Joined
Oct 6, 2018
Messages
10
Likes
1
#20
4x 480's sounds good to me.

I have 6 BCU's coming and have 2 MCP50X Pump, I think 2 will be fine for you too
but until someone actually tries running a several of these water blocks in the same loop we won't know for sure.
Cool. Thanks again for all the info. Hopefully, we'll get some more info soon.
 
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#21
Update: The TUL BCU waterblock has mixed metals in the water path (Copper and Alu),
so pain Distilled water + pt nuke or a silver kill coil may not be good enough.

The aluminum is anodized, so in theory it is fine, but I'm recommending a corrosion inhibitor in addition to a biocide for BCU loops.
I'll be trying Liquid Utopia + distilled water.

Premixed fluids are another option, but I don't have any personal experience with them.
 

peter

Newcomer
Member
Joined
Oct 10, 2018
Messages
7
Likes
0
#24
Looking for advice on connecting fans to motherboard.
I will be running 8 fans, 2 radiators and 2 pumps.
The motherboard is an Asus 270A, its has the following 4pin connections;
CHA_Fan_1
H_AMP_Fan
AIO_Pump
I was thinking of running four fans each off the top two with this splitter;
https://www.ekwb.com/shop/ek-cable-splitter-4-fan-pwm-extended
And the two pumps off the AIO_Pump connection with a 2 way splitter.

Is this the right approach?
Or is something this this better? - a hub that controls all eight fans powered by SATA
https://www.newegg.com/global/ie/Product/Product.aspx?Item=N82E16811999309
 
OP
OP
justinjja

justinjja

Member
Member
Joined
Sep 13, 2018
Messages
28
Likes
18
#25
What fans and pumps?
I wouldn't want to pull much more than an amp from a regular fan header.
The pump header should be designed to handle a few amps.

The EK splitter looks good, it powers the fans via molex and just sets PWM/RPM via the Motherboard.

Silverstone is also fine, does the same thing as above.
Unless you are running delta/nidec/other really powerful fans, 8 of those could pretty easily over load a sata connector.
 
Last edited:
Top Bottom