You need something that improves eye relief, also known as eyepoint.
There are probably add-on lenses available that reduce magnification and let you see the whole frame from farther away. Canon sells one, I'm not sure it fits on the Nikon viewfinder mount thing. This is Canon's one.
EDIT: You might want to try the magnifying eyepiece DK-21m; it'll magnify it but your worry is not about glasses but actually extending the eyepiece towards your eye more.