You can set up a Domain Controller and File Server with Linux and Samba 4. I am not sure how advisable that is if you have nobody in house with Linux experience. I usually use Debian as Distribution but anything that was intended for server use should be fine.
When you use Samba and set it up correctly you can use the default windows tools to manage the Active Directory and the File share access. So day to day stuff won't really differ much from Windows, but i am not sure how much powershell you can use to script it. All my scripts that use the AD are on Linux.
If you want an easy Linux option you can look into univention corporate server
I looked at this quite a few years back and they did offer a lot of functionality that was fairly easy to manage. I know they have a free core license now but they changed their license model from when I tested them I think. If you have to use the paid license you will likely not save a lot of money compared to MS though. They had pretty steep prices and it may not be worth it if you just need Active Directory and File Servers.
While I am generally for Linux you should consider a few things:
Linux has a lower initial investment but may be just as expensive or more expensive in the long run. If you need help from a contractor or MSP, the ones that do Linux are often more expensive.
If you are not comfortable with the Idea of using the command line to accomplish your tasks, Windows is often the better solution. (Though Windows has been getting more command line focused in the last years)
The real benefit of Linux is that you can usually easily migrate to other Software since the open nature of Linux does not lock you into one specific system. And Licensing is often a lot easier especially with very strict Distributrutions such as Debian. Microsoft Licensing can be a real headache especially if you do not want to pay to much and if you use more than just the AD/File Share.
Hey Chris,
If you are only planning for a small DC for your homelab I would go with a virtual machine. For just couple of instances you really don't need to bother with a bare metal machine and that way it is far easier to do snapshots or backups.
Turst me on this, you need at least semi-regular backups of that machine. It can really throw a wrench into your week when your DC just dies one day and you can't just reload a previous state. Or when you f something up in the configuration and can't just go back.
On a side-node: If you aren't already sworn to the Microsoft overlords, have a look at Univention's UCS. It's a great piece of open source software and won't cost you a dime. They might not invite you to as many conference trips as microsoft does, but it never let me down and plays so good with mircosofts clients and structure (thanks to samba), you can literally use the AD tools from microsoft to manage it.
I use Univention Corporate Server (UCS) it is Debian-based and it works remarkably well for Active Directory with Multiple AD Servers as well (active backup servers). It has more functionality than just an AD clone. It also has DHCP, DNS, and several/many free or paid add-ons. It works well enough that my Windows VMs and workstations do not appear to be able to tell it is not connected to Windows Active Directory - including Exchange, SharePoint, Dynamics, MS SQL Server, etc. My company is a Microsoft Partner and while we have no problem supporting clients with Microsoft products, we use Microsoft as little as possible in house due to Linux and FOSS fulfilling our needs better, cheaper, and usually faster. Finding and evaluating UCS was the last piece to move away from MS internally.
I came across to this german distribution that is designed to be a microsoft server replacement.
https://www.univention.com/products/ucs/
Tried zentyal and broke it million of times. Univention seems to be pretty solid so far.