What is Reddit's opinion of
Chef?
From 3.5 billion Reddit comments
16 reviews of this app found across Reddit:
10 Think "man, this sucked, I should learn about Vagrant and Chef/Puppet"
(or system configuration management and provisioning in general)
As far as automation the two games in town are puppet and chef. Each have their pros and cons but they basically scratch the same itch: getting a system configured properly.
An extremely useful introductory exercise would be to use the tool of your choice to configure your ideal workstation. Templatize your configuration files and have them deployed automatically, have the software installed, configure your network, and do any and all configuration via puppet. Try to never log into the system, except to watch the puppet runs and ensure they're doing everything you expected them to do.
In the situations where the domain specific languages fall short, write scripts to fill in the gaps. ONLY write pure-scripts if you're absolutely certain there's no sanctioned way of doing it though. Scripting is a great skill, but the most important thing to learn is when a tool is the right fix for a given issue or if there are better options.
For this I'd use Puppet: http://puppetlabs.com/
Create a puppet master server and assign it a static IP or good DNS resolvable. Build out your AMI's to start up Puppet which checks in with the master and then builds out however you want it to.
Alternatively, take a look at Chef: http://www.opscode.com/chef/
But basically what you want has already been written.
Puppet (http://docs.puppetlabs.com/) and Chef (http://www.opscode.com/chef/) are configuration management tools that allow you to manage a group of servers based on rules that you create. As an example you can create a Puppet "manifest" describing your Apache server configuration and then configure dozens of them automatically. They make managing large amounts of Unix systems way less resource-intensive, and things will generally be more stable as you'll be able to roll out changes in an automated, staged fashion much easier.
The right way to do this IMHO is to use a config management tool. I suggest you checkout something like puppet or chef. There are other tools out there, but these two are definitly the heavy weights (and what I would recommend to somebody starting out.)
Also, check out vagrant. It's a great way to build-test your puppet/chef code :)
Not to split hairs, but it sounds like your VPS company is provisioning them (VM creation, OS, basic software) - what you seem to be asking about is configuration and package management.
Have you looked into Puppet and/or Chef?
We've recently gone through this architecture debate at my job. We needed to create a scalable platform that could easily handle as much traffic as heroku gets. The system below is what we came up with.
If you want to keep a fully managed hosting system, instead of going with Heroku (recommended), then you'll want to look at some of the newer tools on the market currently. Heroku uses a custom version of Chef with the Knife tool to do there remote management and deploy logic. I'd recommend Chef, unicorn, Nginx, combined with a hosting platform with a great scalabale architecture and api (we use EC2 boxes), combined with a master controller interface for grouping the servers, creating app groups, user management, etc...
Pick up Chef then: http://www.opscode.com/chef/ Or Puppet: http://projects.puppetlabs.com/projects/puppet
If you can grok Puppet, PM me in a couple months and I'll give you a referral to my current employer. They've got a really good track record of working with people with disabilities. They usually offer relocation assistance.
If you're looking to manage config files, I guess you could use git by itself. The real way to do it though would be using system configuration management tools like puppet or chef (and back up the configuration repository using git). There is a bit of a learning curve to overcome to implement them, but in the end you will have a much smoother configuration change management and server deployment.
I learned the hardway that AWS "tiny"/"free" option cant handle chef11 server. You need to do at least 2 Gigs VM or the "Mini" option.
Just a heads up if you want to play with it.
5000 machines? You need a platform for that. Both of these CM tools have good Windows support:
http://www.opscode.com/chef/ http://saltstack.com
More work now, yes. But problems like this will become trivial next time.
Chef + PXEBoot = Bootstrapping, IPs, DNS etc.
Forward and reverse DNS for everything - improves diagnostics, helps prevents any emails the system sends you (cron, OMSA, insights, Logwatch, whatever) from getting spammed
For a long while I was quick to say that anyone using the word "cloud" is trying to sell you something you can already get for free, but as I've learned more about the capabilities of OpenStack and Chef I've realized that the idea of a private cloud means that you have a management layer that handles provisioning VM instances almost automatically. So for sysadmins, the difference between a data center and a private cloud is that a data center houses servers that run private cloud software, basically allowing you to run an Amazon EC2-type service on your internal network.