I point my finger squarely at SMB (Or samba). In the iteration you have, it treats network drives like local drives, so it doesn't just try to read the filename and -size, but it also tries to read the first couple of bytes of the file to determine what it might contain (if it's a media file, it tries to read the header, so it can say how many minutes of music and at what bitrate it is, for example).
Even though it's a problem that can be solved by upgrading to windows 7, I would suggest that you rather use some specialized software.
Lanshark and P300 are P2P filesharing for local area networks. They might solve your problems.