You should go back to your doctor with your concerns, and he should refer you to urology, because it will need some tests and imaging your doctor can't do, such as a flow rate test, flexible cystoscopy, and maybe a scan. Given your genetic history, I would ask for an mpMRI scan. Not all prostate cancers generate raised PSA, but BPH and prostatitis could be causes too.
It might be that you need some treatment for outflow obstruction (narrowing somewhere along the path).
One thing to note is that men pee at different speeds at different times of day. You will pee slower at night and first thing in the morning because the part of your autonomous nervous system which opens the internal sphincter takes longer to wake up than your conscious nervous system and will still be half asleep. However, if you start getting an outflow obstruction, you are also more likely to notice it first at these times.
If you are peeing slowly, you might stop peeing before you emptied your bladder (peeing is timed, it doesn't run until the bladder is empty). However, it is important to empty your bladder each time to prevent UTI's. So I would suggest you try double voiding - wait 30-60 seconds after peeing and try peeing again (by relaxing, not forcing) to help make sure you have emptied your bladder.
The irritants you mention (caffeine, fizzy etc.) tend to cause bladder spasms in people who are sensitive to them (which is only some people), and that results in urgency, not the symptoms you describe.