Difficult to tell.
The best thing, is to monitor net traffic and from which programs the traffic comes from. Even better, is to put the home server (if you do not need it to be facing the internet) on a private network, only accessible by the machines that need access.
If it needs to face the internet, make sure only services needed are started, only open necessary ports on the firewall etc. Never install 3rd party software you do not know.
Also make sure you run software like The Rootkit Hunter and OSSEC
The problem is if you dont have any linux experince doing forensics or linux security you are most likely not going to find what is happening. Its very easy to hide things in linux.
You can do some basic netstat and ps -aux to see what connections are active and what processes are running. You can try to run something like the rootkit hunter on the box
In terms of services you can use the netstat command to first check if any services are listening that shouldn't be:
netstat -tulpn
Check out rkhunter it will detect a lot of the common rootkits and do a basic sanity check on your system binaries.
If you want to go further with a file-level IDS AIDE might be a good bet - it keeps a database of checksums of system binaries and compares it to see if anything has changed.
Depends what you're trying achieve. As other people have mentioned already we run ClamAV on mail that we handle, but that's to protect our Windows users.
But what we do now run on our public facing servers is Rootkit Hunter which is as close as I've seen to antivirus software for Linux itself.
Also found this: http://rkhunter.sourceforge.net/
Just reacting to the windigo thing that has happened recently, would be nice to set up some kind of malware detection that would hopefully catch this kind of thing
There's quite a few tools that do this and more. I use these.
Lynix https://github.com/CISOfy/lynis
Rootkit Hunter http://rkhunter.sourceforge.net/
OSSEC is really good for monitoring and alerting on file changes. http://ossec.github.io/