Nexus devices aren't just cool mobile devices sold for competitive prices, they are also perfect platforms to tinker with Android. Google doesn't throw up any roadblocks between you and the internal workings of Android. With a few quick commands and downloads, you can gain root access and start having fun. That's not to say the process is without risk--things can always go wrong.
We're going to go over how to root a Nexus device, what kinds of errors you might encounter, and what you can do to modify the system once you're done.
Step 1: Prepare
When you receive your new Nexus device, be that the new Nexus 7, or the due-for-a-replacement Nexus 4, you'll want to unlock the bootloader. I always recommend those interested in modding unlock the bootloader because it's low risk, and you don't want to do it after you've set up the device. Unlocking the bootloader does a factory reset for security reasons. Before we can do that, some setup is needed.
First, get the Java JDK from Oracle and install it on your system. Be aware, some Windows environments will require you to install the 32-bit version of the JDK for the Android SDK to see it. Next, install the Android SDK from Google's developer portal. There is a ZIP package and an installer. You can use either one, but the installer is more foolproof.
Run the Android SDK Manager, and install the updated tools and the Google USB driver. Now, plug in your Android device, and hopefully the device will display an RSA fingerprint authorization. That will let you know the computer and Android device can see each other via the SDK. If you don't get that, you may need to manually point your operating system at the right driver. In Windows this is done through the device manager. The USB drivers are in the \sdk\extras\google\usb_driver\i386 folder.
You're going to use a command line to do all the dirty work here, but it isn't bad. There are toolkits that can automate the process, but it's best to do this yourself so you're familiar with the tools. If something gets mucked up, toolkits might not be any help. Head into your SDK directory, and go to the platform-tools folder. You'll see a few files, including one called ADB and another called fastboot.
Confirm you're ready to go by opening a command prompt or terminal from that folder, and run the command adb devices. Your Nexus should be listed as a serial number. If it doesn't work, check out your drivers again--something may be wrong there.
Step 2: Unlock the Bootloader
With your Nexus plugged in via USB, type adb reboot bootloader in your command prompt. Your device should reboot itself into bootloader mode. You can do this manually by turning the phone or tablet off, then starting it up with the down volume toggle pressed.
When the bootloader is up, again type into your computer fastboot oem unlock. Note that we're using fastboot instead of ADB while in the bootloader. Your device will produce a scary-looking warning informing you of warranty voiding, and system bricking. Fear not--you can relock later if you like. Select the Yes option with the volume toggle, and hit power to select. The device will reboot, and it is now unlocked.
Step 3: Rooting
Now, if you're ready for the serious business, there is again some setup you need to do. Android 4.3 changed the way the platform works, which make root access a little more tricky. Android now uses the nosuid flag on the system directory. Not all root packages work right now, but SuperSU from Chainfire does. It runs a superuser daemon at boot to handle root access. Grab the latest version of SuperSU from Chainfire's site and transfer it to your Android device's internal storage.
Next, we're going to need new recovery software for the Android device. A recovery is a dedicated partition on the internal storage that runs a recovery console. The stock recovery is basic by design--it's really only good for deleting user data if a system is unbootable. We need something with more advanced features, and there are two options.
ClockworkMod is probably the more popular option. This comes from noted Android developer Koushik Dutta in basic, and touch-enabled versions. The other big name recovery right now is Team Win Recovery Project (TWRP). With Android 4.3 as it currently stands, TWRP is the better option. CWM needs a few settings tweaks to work properly, so we're going to use TWRP from here on out.
You'll need to get the right recovery image for your device. Make sure you verify you have the right variant of the device by checking the hardware name (this is different from the model). For example, the 2013 Nexus 7 is Flo and the Nexus 4 is Mako. You can look up the other Nexus devices easily. Download the recovery image file and place it in the folder where ADB and fastboot are on your computer.
Backup anything you have on your device just in case something goes wrong. When you're ready to go, plug the USB in, and issue the adb reboot bootloader command. This time when you're in the bootloader you have a decision to make. You can try to boot the recovery, which doesn't make any permanent changes to the system, of you can flash it which replaces the stock recovery, but offers more functionality.
I've had issues booting recoveries on the new Nexus 7, but your mileage may vary. To try booting use fastboot boot recovery TWRPFILENAME.img and press enter. If your phone or tablet does things and flashes the TWRP logo, you're good. If that doesn't work, or you just want to go all out, use fastboot flash recovery TWRPFILENAME.img.
Either way, you're going to end up on a screen with a grid of large buttons. TWRP supports touch, which is nice. So, we need to flash the root SuperSU package that was placed on the device earlier. Press the install button in TWRP and navigate to your ZIP file. Tap it, and confirm the flash. The process should be complete in no more than a few seconds. If everything went well, the recovery will say 'successful' at the bottom of the script window.
Reboot the device and check for the SuperSU app in your app drawer. Sometimes it doesn't show up after the flash for some reason, but root access is working. Just grab SuperSU from Google Play, and open it. It will tell you immediately if root access is working or not.
Step 4: Start Tinkering
So now you've got root access, and that opens up a lot of possibilities. We're not here to talk about just using root apps--that's old hat. If you're using a Nexus device, you might as well take advantage of the openness if you're going to start messing about.
There are two general categories of system-level mods you can do. A ROM is a replacement operating system based (probably) on some version of Android. There are several big-name options as far as ROMs go like CyanogenMod and Paranoid Android. These ROMs include advanced features and optimizations for specific devices. The update cycle is also much faster with these ROMs than almost any device.
You can also install a new kernel, either in conjunction with a ROM or on its own. A kernel is the piece of software that manages input/output requests from the OS. All operating systems have kernels, and Android runs the Linux kernel. Installing a tweaked version of a kernel can change the way the device behaves at a low level.
Let's talk about a few kernels and what they can do for you first. To install a kernel, you have two options. An app like Falshify can handle all the installation for you from within Android. If you find a kernel you want to download manually, you can boot into recovery and install with the same option we used to install the root package above.
The Franco kernel is a popular option as it is updated frequently, has plenty of features, and is backed by an enthusiastic developer community. It doesn't support all Nexus devices, but many a Nexus 4 users swears by it. There is also a handy Franco Kernel Updater app.
The Motley kernel is another popular option, but again, you'll have to check to see if your specific device is supported on XDA. This one has a lot of battery saving and tweaking features. It doesn't have quite the same community behind it, but Motley implements the best features from a few different kernels. Lastly, the Faux kernel has a ton of hardware tweaks, and can even turn off thermal control on some devices. It has its own cadre of dedicated fans.
There are three well-known ROMs that even non-modders have probably heard of, but there are plenty more out there. The big three are CyanogenMod, AOKP, and Paranoid Android. All three of these ROMs have fast update cycles, and dedicated communities of users and developers. They're also based on the open source version of Android. You may also have to flash a build of Google apps separately to get the Android features you're familiar with, but that's easy to find--the developers usually link to the files.
CyanogenMod is a very well-run project. They listen to the community, and are in the process of developing new security measures to make CM the most secure consumer ROM out there. It has robust theming support and great update mechanisms. It's also clean and fast. CyanogeMod's nightly builds are very stable and support a wide range of devices. Or you can just wait for monthly or stable builds.
Paranoid Android is getting a lot of attention these days for its innovative ideas. It has a multitasking/notification interface known as Halo. It's a take on Facebook's Chat Heads. Instead of jumping from one app to the next, you can check all your notifications in a floating icon. You can manage notifications without even going into the notification drawer, and pull up apps in floating windows. It's a power user's dream.
AOKP is a good option if you want a speedy, configurable ROM without a lot of setup. It doesn't have any giant headlining features like Halo or CM's extreme configuration. Although, the Ribbon Quick Launch system is handy -- it list of custom app shortcuts accessible with an edge gesture.
When you decide on a ROM, you can flash it with an app like ROM Manager, but it's just as easy to do it yourself. Just drop it on your SD card, go into recovery and install it like the ZIPs above. It's probably a good idea to go into the recovery's Wipe menu first and clear out the Dalvik cache and regular cache. Installing a ROM will wipe all data from your phone, so be cautious.
So now you've got it all pretty much figured out. You're unlocked, rooted, and maybe even running some custom kernels and ROMs. Just make sure you use your recovery to make a backup of your device once you've got it the way you like it. That will let you revert almost any damage you might do.