I'm going to say no you shouldn't be worried, but that is because I like things clear in black and white. I would argue if the definition of recurrence is 2.1 then anything under 2.1 is good.
I can see there is an argument that it is increasing, but for all we know it will level off and possibly fall again like it has already done at least once.
I am now a couple of years post treatment and things are going fine. My threshold for recurrence is 2.1 but I am told if it got there no treatment would be contemplated until it got to 10. you are 80 at the moment, you have already survived 16 years, and I would cautiously say that even if the cancer returned, it could be kept under control for another decade.
One thing I would say is that your GP and no one else is going to be paying much attention to your PSA and they will be using guidelines which say anything below 4.0 is fine. The fact you are keeping track of your PSA over many years is good; because if it goes over 2.1 you want to be referred straight back to oncology, and not be involved in an argument about whether it has to get to 4.0. The threshold for you is 2.1 anything less is good: anything more needs a referral.