For many men, having their prostate removed brings into question their whole sense of what it is to be a man. If he is pushing you away, he might be frightened that you aren't going to stay around, frightened that he will never get an erection again, angry that this has happened to him, struggling with incontinence, worried that if he cuddles you you might want or expect more ...
Putting the shoe on the other foot, I suspect that many women who have a breast or breasts removed will avoid intimacy for a while (or perhaps forever) because they feel their body has let them down, that their partner will recoil in horror, because they are unsure whether they are still a woman and attractive if they don't have the thing that identifies them as a woman. It is no different for a man with prostate cancer.