Concealed Carry (CCW) and basketball shorts?

I just recently got my CCW and I'm also a pretty athletic guy and wear athletic shorts a lot.. Can I carry a concealed weapon SAFELY wearing basketball shorts? ( im concerned it may fall out, at a horrible time like in a store or a mall, etc.) I have a concealed clip holster, but I'm still a little concerned. Any ideas?