You certainly can stop HT early, although at a slightly increased risk of cancer recurrence. A number of men on here have done just that - as you say, quality of life is an important consideration. Another option would be to switch to an HT drug such as bicalutimide, which blocks the reception of testosterone rather than its production; I’ve been on bicalutimide for 8 months now and have minimal side-effects.
Have a chat with your oncologist at your review meeting and discuss the options. The oncologist will make recommendations, but the choice to accept them or not is yours.
All the best,
Chris