Windows Home Server is undeniably awesome. We've shown you how to build one of your own, install a ton of awesome third-party add-ins, and setup video streaming services. Unfortunately, WHS doesn't work with OS X's kick-ass Time Machine backups, at least not out of the box.
iTimeMachine installed on your Mac. iTimeMachine is a handy donateware utility that lets you trick OS X into thinking that network drives are actually USB-connected drives, and thus eligible to use for Time Machine. Before you can make the image you need for Time Machine to work, you need to collect a little info.
Before you continue, you'll need to know your Mac's hostname and the Ethernet ID of your wired Ethernet connection, which is sometimes called the MAC address. You can find the hostname by opening Terminal and typing hostname -s, and find the MAC address by opening Network settings in System Preferences, selecting Ethernet and then clicking Advanced. Then click the Ethernet tab and look for the Ethernet ID.
Multimedia-PCs for figuring this one out. You need to open Disk Utility on your Mac (it's in the utilities folder), select your main hard drive, then go to File, New and select Blank Disk Image. You can save the file on your desktop, but the file name needs to be YourMacsName_EthernetIDMinusColons. For example, mine would be Wills-MacBook_d49a200a3e0a. It's really important that you maintain case and remove colons from the Ethernet ID. Before you can create the file, you need to adjust a few options. Set the Image Format to "sparse bundle disk image," and increase the size until it's larger than the amount of data on your Mac. I'm using about half of my 250GB drive, so I set the image to be 300GB, which should give me plenty of room to grow. Make sure you have iTimeMachine running and enabled, and hit Create.
Assuming Disk Utility is done with its business, copy the new image file to the root of the share you just created on the WHS box. Then, open System Preferences and click on Time Machine. Click Select Disk and browse to the newly created network share. Select it, and your first backup should start in the next few minutes. It's a good idea to run the first backup while your Mac is wired into the LAN--you'll need to move a fair amount of data. After the first backup is complete, you can back up over wireless without any problems.