Rare Earth by Peter Ward is what you're after here. I took an elective in college that effectively was reading a bunch of space science (and history, it was odd) and discussing. This one caught me off guard but was a decent breakdown of a possible answer to Fermi.
I don't necessarily agree with the supposition, mainly because it still comes from a place of specifically carbon-based life as the end goal. But they do lay out reasoning in an easy to understand way that was super neat to learn.