Parental relationships. Overthinking?

This is more of a psychological query. we r in a nuclear setup. husband goes to office at 8am, returns by 8.30-9.00pm. in the meantime, my 22 month old toddler is entirely with me. now my issue is, child goes to everyone in the family and extended family and spends time..but not with his father. he can only spend a maximum of 10 minutes, no matter what. I have tried explaining my husband to spend some "quality" ( uninterrupted, exclusive) time with son whenever he can, even if it is for 5-10 minutes every day or call from office at lunch break if he gets late. he gets irritated when the topic comes up. he says he is too tired from office to think about all that, he wants to relax when he comes home. my fear is that if this continues, there will be a lifelong distance between the father and son, which I don't want. I have mentioned it to my immediate inlaws, they say  I am overthinking and maybe it happens to girl children, but boy children don't care. am I really overthinking?