As Bri has said, the ultra sensitive test (to 2 or even 3 decimal places) has been brought into disrepute, so much so that in the area where I live the hospitals have stopped doing it. The problem with these tiny numbers is that the same sample can be tested twice on the same day on the same machine and give two different results. We actually tried it with one of my dad's PSA tests - one sample tested at 2 labs gave significantly different numbers.
However, the trend you are seeing is consistent rather than random so yes, you may be looking at a small number of cells left behind in the prostate bed. Do you have a written copy of the pathology results after the op? If not, can you get a copy from the GP? Things like whether or not your other half (OH) had positive margins or signs of perineural invasion (PNI) are important.
My OH had very similar results to yours but was in denial for quite a while and refused to accept that he needed salvage treatment until about 2 years post-op. He had the radiotherapy and some hormone treatment which brought the PSA back down to undetectable for another 5 years.
So yes you may have a biochemical recurrence but no, don't be scared.
"Life can only be understood backwards; but it must be lived forwards." Soren Kierkegaard