I was wondering about Soya. It's high in Flavononids, which are similar enough to estrogens to be used by the body as an estrogen.
On the one hand, estrogens can suppress Testosterone, which someone who's already low in Testosterone might be a bit concerned about. (It will depend on why your Testosterone is low, and I don't know if you've had an Andrologist investigate that.)
On the other hand, osteoporosis is caused by lack of estrogens (which is mainly manufactured from Testosterone in men which is why lack of Testosterone causes osteoporosis in men), and the Flavononids can help compensate for lack of estrogens.
I don't know the relative balance of the two opposing effects.
With regards to calcium, there isn't agreement if it's a contributory cause of prostate cancer or not. The proposed mechanism is that it reduces the amount of available Vitamin D in your body, and lack of Vitamin D during puberty does have a good correlation with prostate cancer in later life. If there is any link with calcium, it might be only during puberty, or it might be mitigated by making sure your Vitamin D is well topped up, but I'm guessing here.