The devil is in the details here. This is in reality an interplay of the origin of the input electric field, the position of the aperture and whether or not this is a far-field imaging system (z >>> d), or a lens/mirror imaging system (z ~ focal length). If it is the latter, then the position of the lens with respect to the aperture also plays a part.
I think on of the things that might mess you up is that people say that the electric field in the far field is proportional to the fourier transform. And this is true! But there are a couple phase factors that are usually neglected because they don't matter for the intensity patter. However, these are exactly the phase factors that would help you here. I would suggest looking up the full equation for the Fraunhofer diffraction pattern. You can find it in Fourier Optics by Goodman (https://www.amazon.com/Introduction-Fourier-Optics-Joseph-Goodman/dp/0974707724). I don't have access to it at the moment because I am traveling. Are you part of an university? They might have a copy of it. There are of course also pdf versions floating around on the web.