Unshapely body even after good BMI.

I'm a 23 year old male. 177 cm in height, weighing 64 kg. An ideal BMI of 20. I have quite thin arms and wrists (like, 2x1 inch diameter of wrists). And yet, I have too much fat on my chest and stomach. My belly protrudes a good 2" to the front, and ~1" on each of the sides too (comparing with lower chest/hips). Kind of a mini hourglass figure, but in males, and widest at the abdomen. Should mention, I am not a very muscular guy. Just the average joe. I'm not very sure what to do. I am considering gynecomastia for the breast tissue (which I do have), but what about the belly fat? Gym instructors looks at my frail body and laughs when I say I need to lose weight. I'm not very sure of what I can do here. Do I just lose weight now so I can get in shape? Or combine it with muscle gain? Or is there a possibility of some other issue?