To set this up on iOS, what you’ll need to do is go to the Messages app, tap on the conversation you’d like to mute, and then touch “Details” in the upper-right corner. On the following screen, scroll down and turn on the slider next to “Do Not Disturb.”
On the Mac, it’s very similar—open the Messages app, select the conversation in question, and click on “Details” in the upper-right. Again, you’ll turn on “Do Not Disturb.”
Under both iOS 8 and Yosemite, you’ll see a tiny moon appear next to the conversation you’ve muted to remind you that Do Not Disturb is on for it.
Oh, and this setting doesn’t sync, so you’ll need to configure Do Not Disturb on each device you’ve got.
Finally, it’s good to know how this behaves on iOS versus OS X. On your iPhone or iPad, you’ll still see banners or alerts for the messages that come in to that conversation (assuming that’s what you’ve got configured in Settings> Notifications> Messages), but you won’t get any sound effects. On the Mac, though, turning this on will mean you won’t see any type of notification, visual or audible, when someone texts you in that conversation. Weird, huh? Anyone want to guess at Apple’s motivation for doing it this way?