Blu-ray, disks, however, are not supported directly by this Handbrake/VLC combo, and that makes things frustrating. Typically you need to first rip the Blu-ray into some format that Handbrake will understand and then you can convert/compress from there.
MakeMKV is one such program that will do that Blu-ray conversion. What many users don't realize, though, is that MakeMKV can be invoked on-the-fly to allow Handbrake to rip and convert/compress Blu-ray disks directly.
To do this, you first need to download and put Handbrake, VLC and MakeMKV into your Mac's Applications folder. The first two are completely free, the last is a paid app but Mike Chen, MakeMKV's author, has said that all of its features will remain free while in beta. You can get an updated key at any time during this extended beta directly from MakeMKV (or, you know, you can buy it and support its development <nudge>).
Once you've copied MakeMKV to your Applications folder, you need to point the OS at a special library baked inside of MakeMKV. This is the magic that will let Handbrake see your Blu-ray disks directly. The instructions (from this MakeMKV thread, also penned by Mike Chen) are to go to the Terminal and type the following (each of the 4 commands on their own separate line, followed by return):
cd ~
mkdir -p ~/lib
ln -s /Applications/MakeMKV.app/Contents/lib/libmmbd.dylib ~/lib/libaacs.dylib
ln -s /Applications/MakeMKV.app/Contents/lib/libmmbd.dylib ~/lib/libbdplus.dylib
So that you know what you're asking your computer to do, the 'cd' command just changes your directory (with ~ meaning “home”), the 'mkdir' command just creates a new folder and the 'ln -s' command creates a symlink (similar to a shortcut) to make these libraries visible in a spot that OS X would look.
That's it! From then on Handbrake will attach to your Blu-ray disks via these MakeMKV libraries, and you can use it just like you would with any other disk.
If your Mac doesn't already have a Blu-ray drive (and most don't, since none come from the factory that way), I've had 100% success with this USB Blu-ray player for US$42 from Amazon. Low price, works great. It's even Prime-eligible!
(As always, check your local laws before ripping disks. Here at TMO Towers in the US it has yet to be tested to see whether or not one actually has the right to back up a movie that one purchased).
Torn paper/disk image courtesy of Shutterstock.