As has been said, brain mets are unusual; but untreated skull mets will - eventually - grow into the brain. Even before that, if they grow on the inside of the skull, they are a 'space occupying lesion' - they are taking space reserved for the brain, and therefore apply pressure on the brain. Just like many brain tumours, which don't actually infiltrate the brain, but cause symptoms just by 'being there'.