Sorry to hear this - you've been through enough. I haven't been through radiotherapy yet, so this is something else to be on my guard against. I was offered adjuvant radiotherapy but decided to wait, on the advice of my oncologist, as my PSA is so low (0.014). In June, I asked if I could now have adjuvant and they actually refused, saying that things were going so well for me they didn't want to put me through radiotherapy. They pointed out the risk of physical damage, other side effects and the risk of other cancers in about 20 years' time. They told me it was best put off with my young age for as long as I can. I hope they're right.
So, based on their reluctance to blast me, I'm not surprised you've suffered a bit of damage.