Why doesn't he want me?

I don't think your mate has extinguished the old wick on the candle, not just yet. He has some phobia when it comes to you. I think it is his ex. I believe that has been, all along, the reasoning behind his avoiding with you. I still think he is unsure about your relationship. Yes, he probably does love you for who you are, but when it comes to demonstrative actions he is unable to prove it. The solution is to move on and find another. If he has little concern for your satisfaction then why should you concern yourself with his at your expense?