You aren't the only one.
I have a wife who not only understands, but supports my racing habit. I know racers who have wives who don't want to hear about it at all. Some of them are to the point where they are only allowed to work after the wife goes to sleep, or is not home.
It would be hard for me to do that. I would get no sleep. Once I go downstairs, I am down there for hours. If I waited till Mrs Mojo went to sleep, I would see sunrises before I am done sometimes.