It's been said sever times here already; just going to reinforce it. Do what makes you happy, just realize in about 20 years or so, when your <knee, back, elbow> starts to nag you on cold, wet nights from all the fun you're about to have; that pipe, book, and warm fire are going to seem like a little slice of Heaven.
Pretty much lived at the gym until I turned 45 or so, then started to slow down. Picked up the pipe at 50 (for real). No guarantee it will happen like that for you, but it might. Keep the pieces you love, you'll want them later.
Sorely missed would be an understatement...