There is also supposed to be a rubber "plug" under the cap - might be missing. Let me see if I can find a picture.
Edit: This plug here is what should be under the cap.
Amazon link here