Gamekeeper's thumb. If i feel the pain?

I twisted my knee and feel on my thumb 3 weeks back.MRI of the knee showed sprain (sprain pattern with interstitial tear of acl&patella but attachment intact) which is fairly stable now. X-ray on the thumb showed avulsion fracture on the left side of MCP joint.I consulted 2 orthopedists(1 said surgery is required for ucl, other said that stability exists so it'll heal non surgically) I was wearing a thumb spica for 3 weeks. Now the doctor is telling me to go for physiotherapy but I still see swelling and numbness on the thumb. I tried grabbing small things with the thumb, I feel slight pain. How to know if it's fracture pain/stiffness because I have never had a fracture before? Should I try to break stiffness with small exercise even if I feel the pain?