chris-allen-lane.com

Building a Cryptocurrency Mining Rig - Part 4

tags: +blockchain +diy +hardware

This is Part 4 in a series on building a cryptocurrency mining rig.

Previously, I designed and built the miner chassis, corrected motherboard BIOS settings, and optimized my mining strategy. Now, in Part 4, I’ll discuss how I attempted to:

  1. Overclock GPUs to increase the mining hashrate
  2. Decrease power consumption
  3. Increase overall hardware utilization

Before discussing optimizations, let’s review the baseline benchmarks.

Benchmarks

These were the top-line benchmarks after adopting the revised mining strategy:

  1. Measured at the wall using a Kill-a-watt.
  2. As reported by nvidia-smi.
Mh/s (ETC)Watts 1GPU temp (C) 2
16091060-72

I drilled down into the power consumption using a Kill-a-watt, nvidia-smi output, and some simple math:

  1. As reported by `nvidia-smi`.
  2. Measured at the wall using a Kill-a-watt.
  3. Chassis fans were being run at maximum speed.
  4. Values were rounded to first decimal place.
ComponentWattsPercent 4
GPUs777 185.4
Motherboard / CPU12613.8
Chassis Fans7 2, 30.8
Total910 2100

It’s also worth noting an externality that wasn’t captured above: the rig made my apartment appreciably warmer, which in turn caused me to run the air-conditioner more frequently. I didn’t attempt to quantify the impact that had on my energy bill, but I imagine it was non-trivial.

GPU Overclocking

I began the optimization effort by attempting to overclock the GPUs. (EVGA GPUs are “overclocked out-of-the-box”, but it’s usually possible to push them harder.) Internet research indicated that there are two overclocking methods available on Linux:

  1. via global settings specified by nvidia-settings
  2. via application-specific settings specified by nvidia-smi

I first attempted to overclock using nvidia-settings, but quickly ran into trouble. I discovered that nvidia-settings will not detect a GPU unless:

  1. It is attached to a monitor
  2. The X server is running

2 was annoying, because I intended to run the rig mostly headless. 1 was particularly troublesome, however, because I began using integrated graphics for video after encountering BIOS issues in Part 2. Because I hadn’t connected monitors to my GPUs, nvidia-settings simply wouldn’t detect them.

(It’s worth noting that reddit users actually overcame this problem by attaching fake monitors to the GPUs via xorg.conf, but I had lost interest in nvidia-settings entirely at this point.)

Having given up on nvidia-settings, I turned my attention to nvidia-smi.

Per the NVIDIA documentation, it should require two steps to set application clocks:

  1. Determine supported clock speeds: nvidia-smi -q -d SUPPORTED_CLOCKS
  2. Set the desired clock speeds: nvidia-smi -ac $VRAM,$GPU

However, this simply didn’t work on my system:

chris@cryptos:~$ nvidia-smi -q -d SUPPORTED_CLOCKS

==============NVSMI LOG==============

Timestamp                           : Wed Sep 13 15:18:47 2017
Driver Version                      : 375.66

Attached GPUs                       : 6
GPU 0000:01:00.0
    Supported Clocks                : N/A

GPU 0000:02:00.0
    Supported Clocks                : N/A

GPU 0000:04:00.0
    Supported Clocks                : N/A

GPU 0000:05:00.0
    Supported Clocks                : N/A

GPU 0000:07:00.0
    Supported Clocks                : N/A

GPU 0000:08:00.0
    Supported Clocks                : N/A

chris@cryptos:~$ nvidia-smi -ac 3505,1455
Setting applications clocks is not supported for GPU 0000:01:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 0000:02:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 0000:04:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 0000:05:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 0000:07:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 0000:08:00.0.
Treating as warning and moving on.
All done.

Others appeared to have encountered this problem as well, and some even suggested that application clocks are not supported on “consumer” cards. (I never determined if this was true.)

I never discovered the failure’s root cause. Instead, I decided that small performance gains were not worth a large headache, and gave up on overclocking the GPUs.

GPU Undervolting

With overclocking a bust, I decided to try undervolting instead.

On Linux, undervolting is managed using nvidia-smi. Contrary to overclocking, undervolting proved to be surprisingly simple:

sudo nvidia-smi -pl $WATTS

This instructs the GPUs to self-tune to comply with a power budget of $WATTS. (I don’t know how the GPUs manage that internally.)

Initially, each GPU consumed approximately 130 watts. I used trial-and-error to incrementally reduce the allowed wattage, while monitoring the effect on hashrate.

85 watts proved to be the “sweet spot”. At 85 watts, each GPU continued to hash at its maximum rate - it simply did so more efficiently. With 6 GPUs, that adjustment saved 270 watts without introducing a performance penalty.

With the wattage dialed in, I next wrote a small shell script to spare me from having to apply these settings manually when starting the miner:

#!/bin/sh

# configure GPU changes to persist across reboots
sudo nvidia-smi -pm ENABLED

# restrict to 85W per card (seems to be the sweet-spot)
sudo nvidia-smi -pl 85

# mine!
ethminer \
  --cuda \
  --farm-recheck 3000 \
  -S my-pool.org:19999 \
  -O 0xmy-wallet-address.my-worker

The miner ran noticeably cooler after making these changes.

Motherboard BIOS Adjustments

With the large, obvious power savings made in the GPUs, I turned my attention to smaller optimizations in the motherboard BIOS. My strategy was simply to turn off all functionality that I wasn’t using with the hope of saving a few watts.

I booted into BIOS and made the following changes:

Settings\Advanced\PCI Subsystem Settings

Settings\Advanced\Integrated Peripherals

Settings\Advanced\Super IO Configuration\Serial(COM) Port 0 Configuration

Settings\Advanced\Super IO Configuration\Parallel(LPT) Port Configuration

Overclocking\CPU Features

These changes saved about 2 watts in total (as measured out of the wall).

Fans

The rig now ran much cooler, so I was able to turn the chassis fans down to their lowest setting. This saved 2 watts.

After undervolting the GPUs, I probably could have removed the chassis fans entirely. I opted to keep them, though, preferring to spend a few extra watts to keep the GPUs as cool as possible. (The fans may also prove necessary with other computing I intend to do in the future, like hash-cracking.)

Final Benchmarks

This was the new top-line after making the above changes:

Mh/s (ETC)WattsGPU temp (C)
16058555-68

These were the new drilled-down numbers:

  1. Chassis fans were being run at minimum speed.
  2. Values were rounded to first decimal place.
ComponentWattsPercent 2
GPUs51087.9
Motherboard / CPU7012.1
Chassis Fans5 10.9
Total580100

Optimizations reduced power consumption by 36% without negatively impacting performance. Additionally, the rig now ran entirely silently, and didn’t noticeably heat my apartment. I was very happy with those results.

As an aside, I was surprised by the power savings within the Motherboard/CPU as a result of the GPU adjustments. My guess was that the CPU began consuming less power due to receiving fewer GPU interrupt requests.

Increasing Utilization

I’ll use this rig for purposes beyond mining. At a minimum, I intend to use it to run the following:

The new hardware will also enable me to experiment with CUDA programming, tensorflow, hashcat and RainbowCrack, as well as parallelize some video-processing work that I do on occasion. I will “get my money’s worth” out of this hardware even if it doesn’t make me an Ethereum millionaire.

Final Thoughts

I’m really happy with how this project turned out. It gave me hands-on experience with Fusion 360 and my Shapeoko, lead me to discover OpenBeam, and gave me a platform for experimenting with new tech. It may even make some money in the crypto markets - we’ll see.

So, I happily declare this project a success, and I hope you enjoyed the build. If you have questions or comments, tweet to me at @chrisallenlane.

Update: 2 Sep 2017

This project was featured in Hack-a-Day.

Update: 5 Oct 2017

The Gainesville Hackerspace kindly invited me to talk about this build. My talk is posted in-full in Part 5, along with an accompanying image gallery.

Proceed to Part 5 ยป