I am assuming you are metastatic, if you aren't then radiotherapy could probably cure you and that treatment should be persued.
Excluding prostate cancer, the life expectancy of a man who has already reached 82 is another 8 years. I don't know whether you are above or below average health for your age, that will make a difference to your life expectancy.
Life expectancy for metastatic prostate cancer if being treated is also about 8 years.
So it is likely you will die at 90 and it is about 50/50 whether it will be prostate cancer or something else that kills you.
If you stop treatment then your life expectancy probably drops to about 4 years. You will almost certainly die from prostate cancer.
If you stopped HT now you would probably live a few more years with no symptoms, but at sometime you probably would develop symptoms and they can be quite severe, urinary tract problems, brittle bones, spinal chord compression leading to paralysis. Other people have been known to have no symptoms until the week before they die.
It may be that restarting HT once you develop symptoms will put the cancer on hold for many years and you will recover from any symptoms you have developed. I think that would be a risky path, but if you hate HT it mighty be a good choice.
I can't imagine your oncologist would be happy with a unilateral decision to stop HT, their job is to stop you dieing from cancer. A discussion about intermittent hormone therapy may be the way to start the conversation.