Kickoff! Custom GPU Crypto mining Rig - RX Vega

CryptoBiker

Newcomer
Member
Joined
Sep 30, 2018
Messages
6
Likes
6
#1
Here I am on the FPGA forums and I find the GPU section totally empty... Therefore the post being titled "kickoff".
I'm a bit puzzled, for some reason here on the FPGA forums no one is into GPUs :unsure: So I'll be the wierdo and post up the first GPU thread.

So this is just going to be a build overview of my RX Vega GPU mining rig.
20180528_215309.jpg


I first decided to tip toe into crypto mining about a year ago. Given at the time GPU prices were ridiculous, I started with just one GPU. I knew that I wanted to mine XMR and did plenty of research before getting into mining. I settled on buying my first GPU and chose the AMD's RX Vega 56 because of its complete dominance over the cryptonight algorithm thanks to the HBM2 memory. I bought one MSI blower card on ebay for somewhere near $1000. :sick:

IMG-20180213-WA0012.jpeg IMG-20180213-WA0014.jpeg

As you all know how this process goes, it wasn't long before I had to have more GPUs on this little starter rig of mine.
I ordered 3 more and got them setup and was starting to learn the how to manage all of this stuff and get my rig sipping power and writing bat files to keep her online 24-7
IMG-20180226-WA0006.jpeg

I was learning a good deal by this point and really enjoying building this rig as I used to build my own PCs all the time when I was younger going back about 20 years I built them for myself as well as all of my friends and family. However, it had been over 10 years since I had put together a computer of any kind and it took me a while to get the cob webs out. But I was getting familiar with everything again and learning all the dark corners of mining $#!+coins of every kind. Mostly focusing on cryptonote coins of course, but I did tryout some other mining software and algorithms.

This is when I started aggressively making "best offers" on ebay at any Vega card that had that purchase option, I started collecting more and more cards until they were more than would even fit on my rig frame and the extra PSU extra cards were precariously set about around the perimeter of rig on my garage shelves. I really wish i had picture of that fire hazard. I promise it would have made you laugh. It was getting to be spring time and now I was learning what over a thousand watts of computer power could do to turn my garage into a sauna.

THIS, this was when the idea for my water cooled setup was born. I decided I was going to design a better frame for these cards and I was going to water cool them and mount the radiator outside. :D This is how I would get my rig cleaned up, and turn my garage from and oven, back into a usable space again.

Goals were:
1. water cooling with remote radiator
2. spend as little money as possible
3. get the rig into as compact as space as possible

I settled on using an inexpensive 14U wall-mount cabinet as the new chassis. A 1U sliding drawer would hold the motherboard and the GPUs.

20180526_213213.jpg

In the above pic you can see where I cut fabricated brackets and panel to hold the riser cards. Also I cut holes in the shelf and mounted 140mm fans that would blow air up around the bare pcbs of the GPUs. The holes on the left are to give extra ventilation for the motherboard.


to be continued....
I fabricated a couple of other brackets to hold the mobo on the shelf.
IMG-20180530-WA0060.jpg

IMG-20180530-WA0054~2.jpg


The motherboard is a asus mining expert and the fans have since been upgraded to 38mm thick silverstone fans that put out 170cfm.

I also heavily modified the 14U cabinet by cutting 6 holes in the top for fans and I cut nearly the entire floor out of it and put it up on caster wheels. I then completely closed off all of small ventilation perforations up the sides of the cabinet and at the top with foil HVAC tape. I kept the ones at the bottom of the case only. The remaining vent holes and the large opening in the bottom floor were all then covered with air filter material.
The theory in doing all of this is to have a large amount of air moved through the cabinet and have complete bottom to top upward air movement through the chassis with filtered air.

IMG-20180530-WA0070.jpg
Then came the fun of tearing down a whole bunch of blower cards and figuring out how to water cool them without spending $2000 on fancy water blocks and expensive PC radiators, pumps, and fans.

This is the first card I striped down.
20180517_052159.jpg

I collected all sorts of various heat-sinks from amazon and ebay to come up with what I needed to cool all of the MOSFETs and VRMs.
Some of the ones in the above picture were upgraded to larger ones I came across at a later date.

I wanted all of the heat-sink fins running in the same direction as the air would flow over the card so it did take me a while to find what I needed and in the end I even had to buy some large square shaped heat sinks and cut strips off of them to make what I wanted.
I bought 40x40mm aluminum waterblocks that were about $5 a piece. There were some on ebay that were a bit cheaper but I opted for ones that had a surface treatment so they hopefully won't corrode when paired with a copper radiator. We will see?
Now before someone starts telling me how wrong I did things, and that I can't use the aluminum with copper in the same system....
Don't tell me that I can't have aluminum and copper together in the same loop, because, I can and I did. :cool::ROFLMAO: Ya know, I just live a little on the wild side over here sometimes. You are more than welcome to buy me some fancy copper water blocks if you're so inclined.



Here is a couple of pics that show the cheesy little water block, the different heat sinks, and the little custom hold down brackets I fabricated.

20180517_052400.jpg
IMG-20180528-WA0000 (2).jpg
These pics are of the night I shut down my rig and stripped the rest of the cards and loaded them into their new home.

20180528_001135.jpg
20180528_001131.jpg
20180528_014207.jpg
20180528_115626.jpg
20180528_132809.jpg
Packing everything into the 14U was a bit of a challenge since I didn't have any real estate to mount anything to the floor or the roof due to the way I wanted the air flow to be.
These pictures show the way the 1U sliding shelf pulls out to help with access

20180528_130619.jpg
20180528_125430.jpg
Everything in this beast is powered by Titanium power supplies. The motherboard and 3 GPUs are powered by a 750W EVGA Ti, and then two high efficiency HP 750W titanium server PSUs power the remainder of the GPUs and all of the fans.
I couldn't use another 1U shelf to hold all the PSUs or it would have messed with my vertical wind tunnel design I had going here. So I just fabricated some cheesy braces to hold them up high and to the very sides of the cabinet so that they wouldn't block the airflow.

20180527_234805.jpg

The thing on the right with the orange power cords fastened to it is a 1U 220v remote switched PDU I grabbed up super cheap on ebay. I modified the brackets of the PDU so it would hang down in the front of the cabinet instead of mounting horizontally. This allowed the clearence I needed for the way the PSUs were installed.
There is not a single thing in this entire system that runs on 110v. Even the radiator fan and water pump I used are high efficiency 220v industrial stuff. Which I'll bet by now you've already guessed I sourced used, for cheap, on Ebay! (y):ROFLMAO:
note: all PSU are well under loaded for peak efficiency. If I get around to adding the last 3 GPUs, 1 will go on the one server PSU that is powering all of the auxiliary fans, and the other 2 will go on the other HP PSU.


Since I used a fairly large water pump, you'll see in the next picture the water manifold that branches the water off into 4 loops. The system currently has 3 loops running, with 3 GPUs per water loop. The 4th loop is capped off for the last 3 GPUs I'll probably never get around to installing. :confused:
20180528_153850.jpg
IMG-20180530-WA0078.jpg

The water lines have extra length and loop down and around under the sliding shelf that holds the GPUs to allow for the shelf to still move in and out even when the system in running.
This is the radiator that currently sits about 20 feet away from the rig. It's mounted in the wall of my garage, and expells the massive amounts of heat those GPUs generate out through the wall to the exterior of my home.

Screenshot_20180612-084701_QuickPic.jpg

And lastly I don't have a good pic of the pump on hand but you can see it in this mess of a picture from the night I was testing out my plumbing design.

20180527_033924.jpg

Don't judge my crazy mess. I had a lot going on that night and got hosed down by lines popping off multiple times. It started to turn into a real S-show after everything was wet(including me) and I just wanted to get my design figured out and get done.
:cool:
So that is mostly it.
She now sits in the garage(not making it hot anymore) and doesn't require any air conditioning even when hot summer days get over 100f here.


Main hardware Specs are as follows:

14U wall-mount cabinet for chassis
Asus b250 mining expert mobo
celleron CPU 7th gen G3930 @ 2.9Ghz
8gb ddr4 2400 Vengence LXP ram
240GB SanDisk SSD
N Oranie Ver09s riser cards (amazon)
4x AMD RX Vega 64 GPUs
5x Vega 56s
EVGA 750W 80+ TITANIUM Modular PSU
2x HP 700287-001 DL160 G8 750W Common Slot Titanium Hot Plug PSUs
ServerTech Sentry CW-8H2-C20M Switched PDU
fan controller and splitters were sourced from amazon.
C13-C14 power cords sourced from amazon
radiator and pump are industrial surplus from Ebay. Previous use was for a water cooling loop used to cool a Co2 laser (that's right, I have laser been $hit on my mining rig)


Software:

Windows 10 pro
XMR-stak only
It runs strictly on the cryptonight algorithm(or a varient) only.

She pulls ~7amps of 220 according to the PDU. (aprox. 1500w)


Thanks for checking out my build,

-Biker



PS. I am going to start a thread soon on the Threadripper system I am building for my BCU1525, which is going to be mostly all assembled from used bull-S from ebay as well.... Stay Tuned




PSSssss... more pics that I just think are sexy below
IMG-20180530-WA0075.jpg

IMG-20180530-WA0072.jpg
IMG-20180530-WA0077.jpg
IMG-20180530-WA0083.jpg
WHAT CHA GOT ON MY WIRE MANAGEMENT o_O
IMG-20180530-WA0062.jpg
IMG-20180530-WA0048.jpg
IMG_-ao39ft (3).jpg
YUP.jpg




:eek:(y):cool::geek::geek::geek:

Biker, Over & Out
 
Last edited:

NWhiskeyTango

Newcomer
Member
Joined
Sep 15, 2018
Messages
14
Likes
3
#2
Excellent GRADE A++ write up! I recognize a couple of your pics from else where (where I remain in the shadows) and would like to respect the name change. Garage, sauna, firehazard, custom waterloops...I know these experiences as well. Would like to give you a +1, as YOU inspired me a while back to make some changes to remove some of these hazards as well (still cleaning up though). Also, Welcome to the forum. Can't wait to see the next (BCU) build posts!
 

BDF_

Newcomer
Member
Joined
Sep 13, 2018
Messages
17
Likes
9
#3
Laughing my butt off when I saw that the water blocks cost $5 each, I was wondering how you did this without taking out a loan. Looks like a bitch to service, but I'd bet its one hell of a performer and compact/quiet too. Should have no problem keeping the HBM in the 40s or lower to crank up those hashes.
 
OP
OP
CryptoBiker

CryptoBiker

Newcomer
Member
Joined
Sep 30, 2018
Messages
6
Likes
6
#4
NWhiskeyTango,
hey thanks a ton for the kind words. This was a fun project and now the family doesn't complain about the fan nose and heat, and it's much safer all being on a proper 220v supply.


BDF my man,
Happy to supply you a laugh. Yup, I am a certified cheap ass. Not counting the power supplies and fans, just all the watercooling gear... I only spent like $200 upgrading to water πŸ€‘

Growing up with a broke single mom taught me a lot about fixing things and making stuff 'just work'. Give me some 5 minute epoxy, zip ties, and a roll of foil tape. I can fix it, or make that thing do stuff no-one ever thought possible. πŸ˜‚ and no regular duct tape, that's for beginners


Dream
Ah πŸ’©, you warned me!
I didn't fully understand that the post would be lost too, I was just thinking I would have to reregister. It's not the end of the world if it's too much for you guys to restore it later. I have the text backed up. I would just have to insert the photos, it wouldn't be too awful.
I learned my lesson big time once, years back when I was doing a crazy detailed write up on a custom AR 10 build I did. I had been typing away off and on nearly all day. I was taking careful a close-up pics of custom drilling and TIG welding; all with extra detailed descriptions and measurements... Then this stupid firearms forum just threw it all in three trash when I clicked post .... 😑
Boom, my login was timed out and I lost all of it. I'm getting pissed off right now just thinking back to it LOL.
So now if I spend more than two minutes typing I do it in a word processor then copy paste it to a forum box. Shoot I'm so scarred from that, if I even type a two sentence reply from my phone I'll 'select all' and 'copy' real quick before I send it. πŸ˜– sheeeez, I've got issues
Oh, and NWhiskeyTango,

I dig your handle, LoL
That's what my 16yo daughter named her new pet bearded dragon. I still laugh a little to myself every time I see that thing.
 

Soul of Jacobeh

Administrator
Staff member
Administrator
Joined
Sep 10, 2018
Messages
27
Likes
18
#5
I'm taking efforts to ensure no new posts are lost. Lotta manual labor likely, but I'd like to save you all the trouble.

Additionally, I like the pics. Let's see if we can use the thumbnail insert option where possible. They're always clickable for a full-size. The full-size in-line pics are a little hard on the load-times :p. I'll look at a plugin for resizing content in-line :).
 
OP
OP
CryptoBiker

CryptoBiker

Newcomer
Member
Joined
Sep 30, 2018
Messages
6
Likes
6
#6
You got it, when I do my FPGA build, I will try to behave myself. You and dream already know I'm not the best at following rules, the way I'm always rambling on off-topic in the Discord servers
 
Top Bottom