Why is My Rescue Schnoodle Afraid of Me?

by LT

I adopted a rescue Schnoodle last week. They think he is around two years old. On the way home he jumped out of the car and ran away.
Fortunately he jumped into a lady's car and she brought him back. He seams to favor women but is afraid of men. He loves my wife and follows her around constantly but is very timid around me, (it's supposed to be "my dog").
He avoids me at all costs and will go in his cage and stay if my wife is not around. If I approach him he cowers in the corner or does what ever he can to get away.
We have had Schnauzers and poodles before and this has never happened. Is this something he will grow out of in a couple of weeks or could it be a permanent thing? I am not sure of his history or if there was abuse.
Do you have any suggestions on how to win him over. I tried the treat thing and he will not take it from me unless my wife is nearby. I tried ignoring him thinking he will eventually warm up.
I hate to give him back to the shelter cause he seams like a great dog, but I want him to be happy and not afraid all of the time. He would have it made and become very spoiled if he would just give me a chance.
Suggestions welcomed.
Thanks