If the team is willing to pair program, they need to start. That makes code reviews much faster, easier, and higher quality.
You'll need to get your WIP down so one way to do this is to freeze all coding and focus the entire team on getting the current PRs merged. If that sounds crazy go read The Phoenix Project.
If you have all devs stop coding in the middle of their feature, you risk them losing all the context they built up over the past X days and will need to recover that once the freeze is over. Two solutions come to mind. First, have each dev spend as much time as they need to document their current state so that they can jump right back in once the freeze is over. Or second, everyone keeps coding until finished with their current thing and then they don't do any further coding until all PRs are merged.
My god... you need to write a book like the The Phoenix Project but like... you know. Your life haha.
Wow ok thats what I was looking for. The Kobayashi Maru style of everything in my life. But here we are now a fantastic community of people that self-regulate and have I think a genuine interest in learning and tech possibly.
My final question my good sir is what was the time frame with which you started and ended where you are now? This is only for reference.
Absolutely read The Phoenix Project. It will describe half the projects you inherit over your career and tell you exactly how they need to be put back on the rails. Also it's just a great read.
I was going to suggest a copy of this https://www.amazon.co.uk/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 as it's a fun book but also educational getting into a programming career... Jeez though... Not worth £35!! There must be some cheaper places for it
Not sure if it counts as an old software book, but The Phoenix Project is one of my picks for this that I haven't already seen posted in this thread. I'd call it an "IT DevOps thriller" which seems like an odd category for a book, but that really describes it well. If you're ever looking to un-fuck your DevOps and deployment practices then this is quite possibly the best book you can get in my opinion.
The phoenix project is a sort of homage to the goal. Can absolutely recommend it.
> > "This book is a gripping read that captures brilliantly the dilemmas that face companies which depend on IT, and offers real-world solutions. As Deming reminds us, 'It is not necessary to change. Survival is not mandatory.' The Phoenix Project will have a profound effect on IT, just as Dr. Goldratt's book The Goal did for manufacturing."
>>-- Jez Humble, co-author of the Jolt award-winning book Continuous Delivery and Principal at ThoughtWorks Studios
and:
> > "This book is the modern day version of The Goal. Today, our constraints aren't robots inside our factories, but it's how we manage technologies like Tomcat and Java that power our most critical projects and applications. This book continues the journey that began with Shewhart, Deming, Ohno and Dr. Goldratt, and shows us how to diminish our modern constraints to help the business win."
>>-- John Willis, VP Client Services and Enablement, enStratus, Host of "DevOps Cafe"
> source: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
Thanks for that! I'm trying to have a wide breadth of understanding so I can better know where I should dive deep.
I should have mentioned that I really dislike when companies hire for "DevOps" positions, I completely agree that administration and the ideals/methodology of DevOps aren't exclusive to those positions, and all admins should take note. The book "The Phoenix Project" was really helpful in understanding this whole idea that Dev and Ops should be companions and not at odds with each other.
Unfortunately, there seem to be a lot of folks hiring for specifically DevOps roles. I really dislike that idea, but it feels like a new niche is forming for these web-scale companies. Not sure if that's the case or not. Either way, I'll definitely need to learn these things for any job I'll have in the future. Would you say that I should balance my attention between mastering the minutia of Linux/Unix and learning these tools/ideas for CI/CD, Immutable infrastructure.. all the things normally associated with the "DevOps" mind frame?
It might be slightly unrelated but I think you might benefit from/enjoy reading this book:
http://www.amazon.ca/The-Phoenix-Project-Helping-Business/dp/0988262592
** The author basically describes a very similar situation and how they put in processes to manage changes and be a more effective team.
Grab the Phoenix book: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
> "can't we just install Google Tag Manager" - it's just a block of code in the <head> tag
​
Things your dev will probably be thinking of:
​
What's the performance like for this? Will this block the page load? Is this important? What kind of info is going to collect? Is this complaint with our privacy policy and how we collect info on our users?
​
"Iit's just a block of code in the <head> tag" ... hosted where? From Google? Do we operate in China? Will we ever? Will their CDN work? Can we afford to be reliant on their CDN / resource? Does it work in all the browsers we support?
​
Depending on the scope and reach, this could have a lot of stuff.
​
​
> "can't we install this countdown pixel on one of our servers? it's just a block of php code" link
​
Do the servers run PHP? Should they? If not, would we need to install it? Who is going to keep that server patched? If it does, where do we put it? What are you using it for, emails?
​
> "can't we just have read only access to the database? just certain tables? such as a category data, so we can count/sum/group-by categories - what about a staging database - can't we use that?"
​
Maybe a replica. As you've been told, you can't just run random queries since you could hurt performance without proper scheduling and permissions. This takes time. Privacy issues might prevent you from handling it without lawyers involved.
​
Who now has to manage your access? How are the credentials issue? Are you machines secured enough to handle the data or is someone going to click a random link and now get backdoored and now some competitor has access to your info? Oops!
Aside from that...
> To the marketing team, everything sounds easy "just copy paste this script"
​
Sometimes it is. Often times, it is not. That being said, a lot of stuff can be very simple especially if it's a one off. You should organize time with your teams to get this kind of stuff. Where I work, we do help out with these requests but they are put in queue. However, you have to prove it's really going to help the business. You can't be wasting expensive resources chasing things that not going to return some kind of value for the business.
​
If you REALLY want to understand, reach this cliche book: https://www.amazon.ca/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 Judging by how you are talking, I am sure it will resonate with you.
I wouldn't say it's the exact same, but I still lump it in a nice to read category:
This topic is actually covered quite a bit in The Phoenix Project: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
Classic Brent. For anyone who hasn't done so already, read The Phoenix Project (or listen to it on Audible). It's great.
https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
I'm going to second that recommendation. DevOps is a really versatile role and you'll want to make sure that you have a solid understanding of the scope involved so that you can confidently set expectations when applying for positions.
It's become a bit of a buzzword in the last year, but for a good reason. It's pretty much essential for agile development and overlaps strongly with architecture / infrastructure development.
There are two books by the same team of authors I strongly recommend reading, including non-referral amazon links below.
For a book on computer architecture starting from first principles there's Digital Design and Computer Architecture. Or in a similar vein but if you want to do some actual really fun programming there's https://www.nand2tetris.org/ which is a course teaching you how to build tetris starting from logic gates and circuits.
And then I might suggest reading a book about software development/DevOps. The Phoenix Project is recommended a lot and is a pretty quick read.
Can I also ask what a Project Owner is? What kind of work do you do?
The Pheonix Project for some reading related to what its like working in a software org and the good and bad practices in it.
As others have said... DevOps really isn't "Developer and Operations". It denotes a philisophy. I highly recommend you read The Phoenix Project if you haven't yet to get an better understanding of just what DevOps means and can offer.
Also, having been an DevOps for the last 22 years (yes, since before it had the name) I can tell you I have done my fair share of programming... however, the last 5-6 years or so I do a lot less, and this is the way this industry is going. There are amazing tools and frameworks that do most of our job now (eg: Kubernetes, Terraform, Cloud Providers Managed Services, etc). With these tools, good DevOps practitioners don't need to code, we just configure various json/yaml files and let frameworks and tools do all the magic. The most programming I do these days is random open-source projects and the occasional script for whatever CI/CD framework I'm currently using.
Con respecto a lo primero, Google Cloud funciona casi como una Startup (a otra escala por supuesto). Por lo tanto, muy poca burocracia, muy horizontal la organización, dentro de lo posible.
Con respecto a lo segundo, hay varias formas de verlo. Google tiene muchos productos y creo que 9 de ellos tienen más de mil millones (1 billion) de usuarios. Cuando se está en esa situación, es necesario priorizar recursos que siempre, incluso en Google, son escasos. También hay decisiones estratégicas que determinan en qué se quiere invertir y en qué no. Este tipo de consideraciones determina, al menos en algunos casos, en qué productos Google decide o no invertir a futuro.
Aparte de eso, Google tiene una cultura de apostar por grandes cosas (10X thinking) y por fallar seguido, aprender y seguir adelante. Si sabes lo que DevOps es como cultura organizacional, es algo así. Te recomiendo leer The Phoenix Project si te interesan esos temas.
The Phoenix Project
It was like therapy for my last job.
Start with reading The Phoenix Project and SRE books. The tools don’t matter as much. You can build good systems and bad systems on any CI tooling. Good luck!
There's a lot to unpack, and lots for me to share on many points here, so lets dive in... first let me share some personal thoughts I think are relevant about DevOps...
First, I'll tell you that every company does DevOps wildly differently, and (opinionatedly, generally) wrong. I've been consulting DevOps since before it had a name, and it is painful almost anywhere you go. If it's painful, it's wrong. If it's not easy and/or using standard and common tools in this field, again it's wrong. An healthy DevOps practice in a company in my experience would be that the DevOps team needs to be responsible for the foundation for everything infra/ci-cd/deploy/etc. However, depending on your engineering/dev team's capabilities and interest, your DevOps team doesn't necessarily need to "own" everything however. You can share ownership of things relating to the services they support.
I'll leave you on one last comment, that a great DevOps team sets up foundations for everything, making it easy, automatic, and simple for engineering teams to do anything requiring little if any further support from an DevOps person. This should include things like creating a new service, an new repo, creating new dependence asset(s) on a cloud provider, building images/artifacts, deploying artifacts, etc. You should be the author of the "templates/examples" of how to do everything for your company.
>am I out of my mind to request that our team owns the terraform and we provide the developers everything up until their runtime environment
No, this is not unreasonable, this is actually preferred, as you, DevOps, are the "experts" in these things. This doesn't mean you should wall it off and not let the devs submit patches/updates/etc. The devs will know their application the best, if they suddenly need to open up a port in a firewall they should be able to do it and not be bottlenecked by your DevOps team. You (DevOps) should still be reviewing their MR/PR's to ensure they aren't doing something silly before Terraform applies.
>Also we as a team decided that we do not want to manage things like 3rd party SaaS services but rather let the developers write the terraform to allow other developers consume these sources.
Are your devs experts in application and infrastructure, security, Terraform and best-practices? I expect not, and as such your devs can help co-own and support your Terraform, but you should in the end be responsible for and manage the entirety of it. If you do not get a better handle on your Terraform, it will become a tangled web, then your company would hire someone like me to un-tangle the mess you've let them make.
>Also out terraform is written in such a way that it is almost completely application centric instead of infrastructure centric. We run GKE, and all our terraform is basically driven around the K8 namespace, the other actual infrastructure is driven deep into modules that is being called from the same local paths, so when you make a change to the module, it either gets merged in without an apply or you get blocked by every other PR and the poor PR management driven with a combination of Atlantis and terraform cloud executing every remote state 4x to actually perform a single apply. ( this is driven by the remote backend they are using for their state). Part of why I am asking is the terraform repo is basically a giant module without very many remote modules
Without seeing it to judge I would say this sounds like an unholy mess already. This sounds like what I have to walk into and untangle on a regular basis. If you, as a team, don't have a ton of experience working with Terraform especially at scale and over time, I highly recommend either hiring an expert to come in and assess/recommend how to pivot off of this spider web. If you're not interested in doing that, please look into the standards that Gruntworks has established and documented.
Gruntworks uses the concept of micro-stacks, which they didn't invent it was thought up years earlier iterated upon by all of us DevOps folks at the time sharing this best-practice. This concept being that you really want to reduce the complexity of each stack, making it simpler to iterate on, much shorter time to plan/apply, so you create a very tiny folder that only has like 1-3 major things it does. Usually you'd create a micro-stack per-application eg terraform/dev/us-west3/widgets
For the above example, lets say your "widgets" application uses S3 and RDS in this folder you'd create an RDS instance named the application name, a security group which you'll need to be able to connect to that rds instance, an S3 bucket, and an IAM role for your application to use allowing access to that s3 bucket only. Now, if your "widgets" team needs to iterate on their cloud-related dependancies (eg: add an SQS topic, or adjust IAM permissions) they can, and they can do so very simply. There's only a few files in this folder, and it follows a sane and easy-to-understand folder hierarchy as well. This makes is "easy" for anyone to come in and support this application.
>And we ask developers to do a lot of the heavy lifting when they don’t really care to. This is at a startup btw.
This is a really bad mentality and approach to DevOps. Your developers aren't experts at the wide-range of things that a DevOps person would normally be. Your developers aren't security experts, or automation experts, or cloud experts, or Terraform experts. Why are you making them do something they have no idea how to do?
I'll leave on some thoughts, that really you, as a DevOps person, should be enabling and empowering your teams to go fast and not have to think about or worry about all the "fiddly" things. Your devs shouldn't have to go learn the intricacies of Helm, Kubernetes, Terraform, your CI language, etc. If you set and own the foundations and examples of all of these, it should make your devs able to do their job easily, without having to go spend days/weeks learning how to use each these tools.
Finally, I highly recommend you read the book The Phoenix Project and ideally also get a few of your peers both in your department and in other departments to read it. It will shine a light on what you should be aiming for and it makes DevOps really understandable and digestible and relatable. Best of luck!
>Manifesto for DevOps
Usually The Phoenix Project is the prescribed text here
>Something that summarizes SRE principles
How about the Manifesto for SRE (Hint: SRE != DevOps)?
Read The Phoenix Project? This novel borrows the concept of the theory of constraints from The Goal.
In The Phoenix Project, Brent is the constraint. He seems to be involved in every project, and is the first person people go to. Moral of the story is, constraints are valuable and need to be protected from extraneous demands.
Sounds like a great opportunity - good luck!
I think you're one of the characters in the Phoenix Project.
I almost wonder if you could do a project for yourself to sell them on disaster proofing their company?
If you have any free time, think of a way sell them on upgrading slowly. Maybe find a good article on a company that went through a disaster and how much money it costs them to finally get to industry standards.
I've seen a few companies go through audits and our IT departments had to step up security, documentation, process changes and SLAs. This is more work, but also improves the company reliability.
Just reading the Phoenix Project helped me realize burdens in IT in all industries but also how great a company can run if IT is truly valued.
https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
Read the phoenix project and memorize the three ways https://www.amazon.com/dp/0988262592/ref=cm_sw_r_cp_apa_i_rk5nFb4G92P90
So.. this is a tough question because it's only a) from my perspective b) based on working in an enterprise and c) depends on your background. In the broadest sense:
Ok, you can start with some reading in books.
A suggestion: Scrum a Pocket guide - Gunther Verheyen
If you like a good novel (about DevOps): https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
​
If you going for training, go for PSM 1 first (prefer that your boss is paying :)). This will give you a lot of background theory on Scrum, specially if you participate in a class. If being Product Owner is more suiting late, you have the advantage of having the theoretical background already.
​
Good luck in your journey! :)
Cheers, Johan
Read https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 and share it with your team.
That's a great book! I also recommend this book /u/sudz3 → The Phoenix Project
Read this if you haven't already:
https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592/
Very good read, also recommend the follow up DevOps Handbook:
https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/
The Pheonix Project is a wonderful read and even has a gripping story. I think it was my first audible book.
> trilogy
The one I read was a single book.
http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592
Just read The Phoenix Project: http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592
It should get you started and has links to further material
I might also suggest some of the best practices guides, such as this one for EC2.
Welcome aboard! Feel free to tear into things. If they don't turn out right the first time, you can tear it down and start again in a matter of minutes.
Oh, look into leveraging other resources such as S3 and application services such as SQS, RDS, DynamoDB. Can help your teams from having to reinvent the wheel for app-related services.
Have you ever read The Phoenix Project? That book changed my view on dev ops stuff. You're story sounds like that book.