You've only given a little of your diagnosis, but what you gave matches mine, and you've gone for same treatment as me (except my place did the brachytherapy after the external beam radiotherapy, but that makes no difference). Full details in my profile.
If the brachytherapy is HDR (High Dose Rate), you probably want to take the following week off just because you may need to use the toilet frequently and might leak, but this is usually temporary and recovers quickly. Many people work through the radiotherapy, although a small number may find they don't want to. You can get substantial fatigue during the second half of the radiotherapy and for a couple of weeks afterwards.
Wishing you well.