ok this seems silly, I never thought I would ask about something like this. Is "Mike's brother's book" the correct way to reference the book of Mike's brother? Or should it be entirely avoided by saying "the book of Mike's brother"?
Both are grammatically correct. Though "Mike's brother's book" is kind of clunky, you can layer as many of those as you want without breaking any rules. Personally, if I was writing this into a paragraph, I'd try to avoid the situation altogether by mentioning the name of Mike's brother beforehand, so when I bring up the book, I can just say "Dave's book" or whatever his brother's name is.