I use VirtualDub regularly, and if you're looking for quality over anything else, then I don't think there are any alternatives that would make the file size that much smaller, besides the Lagarith Lossless codec. This codec only reduces the GB/hr. threshold by about 15-20 gigabytes.
If you're looking for a codec that creates a smaller file size, but don't care so much about high quality video, then I would suggest getting the x264vfw codec. I would have to advise, though, that you cannot remaster anything once you've captured with this codec, as the artifacting is massively significant at that point. If you're only using VirtualDub to only capture tapes and nothing else, then I think you would be all set!