The Jez Humble / David Farley book on Continuous Delivery is a must read from a standpoint of teams that deliver solutions in an automated way. More oriented towards software developers than operations / IT but really a must read for both types of folks for us to all come together as "DevOps".
edit: Amazon Link: https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
If you're a professional in the field, you should have been aware of CI/CD for a logn while.
CI/CD is considered a pretty basic feature of software dev pipelines for 10+ years.
The Fowler book on it was published in 2010, but it existed way before that already. https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912/ref=asc_df_0321601912/
Well. I started with this bad boy: https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
But looking at YouTube clips about "GitOps", CI CD etc Will get you far. It is about understanding the benefits of it and how to implement it. There are tons of build servers out there and they all work abit differently.
Try using free ones and set Up a pipeline to automatic build and test your Code. Then ADD more steps to analyze your Code and deploy IT.
This is going to sound a little sarcastic, but go on more interviews and you'll get the list pretty quickly. There's a lot of truth to it though. Interviewers ask the same stuff, so it's a smart move to do a few throwaway interviews with companies you don't like early on in the process just to warm yourself up.
Other than that, stay abreast of the buzzwords. Microservices, devops, react, redux, vue.js, es6. All of those are going to get you points in a job interview right now.
Here's a book I recommend: https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
Thanks. It wasn't easy and took us a little bit of time to get here, but we had to automate everything , add more automated testing and less manual testing and lastly invest heavily in cloud infrastructure and configuration management.
Once all those pieces fell into place ( thorough months of hard work) our cadence improved without causing burnout. Our devs work at the same pace and know that if they finish a feature today it's out the door tomorrow.
Oh and during the prod deployments , we canary and monitor for exceptions etc. This is a big part of this.
Go check out the book 'continuous delivery' by Jez Humble . It's the CI bible
Ultimately we want to move to CD and multiple prod pushes per day (safely).
Edit: here's the book http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
Can somebody please recommend some literature on how to architect a good build system? Or even on how to design applications so that they are easy to build and deploy?
Googling for that topic is rather frustrating; most hits are rather specific to a tool or a language; others is marketing for proprietary tools.
I liked Continuous Delivery by Humble and Farley, but they basically already assume a good build system, and only briefly describe some good properties of one.
If you compare the literature on build systems to the literature on software architecture in general, it's pathetic.
What I really want is something structured, starting from common goals (automated builds, repeatable builds, good error messages, performance) and challenges (diverse environments, partial builds, ...), and the proceed to common solutions, design patterns, declarative vs. imperative, all the stuff that you usually discuss for "normal" software architecture.
> Can you explain a bit more about 1 and 2 ? Also, what does "experience with monitoring" mean in technical terms?
For 1, I mean that you should be comfortable writing Puppet/Chef/Ansible/Salt/CFEngine 'code' so that you never need to log into a server to take it from an empty slate to being in production. This may mean using something like Salt-Cloud to join a new server to the master when it's spun up, or for a RedHat-based machine using kickstart to start the configuration process.
For 2, you should be familiar with software like Jenkins, Bamboo, Travis CI, etc, and how that fits into modern software development workflows. Read this book.
Experience with monitoring means that you should be familiar with using one or more monitoring platforms (Nagios, New Relic, etc etc) and customising them to monitor a specific application (likely developed in-house).
Honestly, if you interviewed well (i.e. gave the impression that you were motivated and I thought that you'd fit into the team), I'd consider you for a junior position based on your post. I'm working as a consultant now though, not a team leader, so I have nothing to offer sorry!
It seems as though you're focused on a particular method for your deployment strategy right now and it may be useful to take a step back and see if your requirements have already been met by recommended patterns.
What you're describing in the above is an integration workflow, and there are many well tested strategies already out there for integrating code (depending on your language/framework, this work may already have been done for you). Most importantly, these strategies rarely suggest cyclical actions like your FTP^(*) transfer from html back to "Remote1".
Ideally, you want your flow to look something like the following:
git repository -> integration steps -> build -> build artifacts -> artifact repository -> deployment tool -> deployed code
Note how this is a unidirectional flow, with nothing being pushed back into the Git repo. You may need to pull artifacts in from multiple build pipelines depending upon your requirements, but if you see the flow being reversed (committing back to git), you should carefully consider why you're doing so and ensure that there's not a better solution^(†). Additionally, while you may very well not need some of the previously listed steps, you should understand what each of them is and make an informed decision about whether or not it should be included.
I would strongly recommend spending some time diving in to Continuous Integration and Continuous Delivery/Deployment (CI/CD) and the patterns used within those methodologies, keeping in mind that CI/CD does not start with automation, but with understanding your code and requirements. One book I would strongly suggest is Continuous Delivery by Jez Humble and David Farley (https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912). It's starting to show its age a bit, but the fundamental ideas are still there and there's a lot of value to be gained from reading it.
Focus your reading on the what and the why before you move on to the how. There are tons of articles out there on how to build fast and efficient CI/CD pipelines, but if you don't have a solid understanding of why the pipeline should be there in the first place and what value it adds, it's easy to see articles of this nature as pointless exercises or end up building something that doesn't really fit your requirements.
​
* This is a warning sign, by the way. If you find yourself thinking that FTP is the solution, consider it a red flag; step back and re-evaluate.
† There are some common patterns which result in committing code back to a git repository, but they solve rare use cases and tend to generate a significant amount of debate. In particular, some say that generated code should be committed back to a repository. As an alternative, I would suggest considering generated code to be a build artifact and storing either it or resulting binaries in an artifact repository rather than trying to commit back to git.
Ideal resting strategy is explained here (preferably with some exploratory testing on top, but they require time):
http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html http://martinfowler.com/bliki/TestPyramid.html
This pyramid is easy to achieve in applications with architecture based on micro services (Netflix, Amazon).
The problem is with huge, monolithic applications which doesn't have separate components communicating via REST API. In that case you have to rely on GUI level testing (Seleniums) which require a lot of maintenance work.
There is tendency in the software engineering community to deliver software almost immediately on production. This makes changes delta a lot smaller (fewer changes in release = less risk of bugs in release). This enforces strong emphasis on testers to automate everything which is possible and test on production.
You can (and should really, if you consider a career in engineering) find explanations of those ideas in this book https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
*Jez Humble and Dave Farley.
Here: http://www.amazon.co.uk/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
A book that may be of interest:
https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
Merge to main triggers pipeline, automated task in pipeline execute, and application deployed. Sounds like you winning to me.
As far as editing the pipeline file and triggering the pipeline. Make a new branch, modify as needed, and when finished merge into main. Azure DevOps has the ability to manually trigger a pipeline from any branch.
For expanding the pipeline. Read about continuous integration (CI). The purpose of CI is to automate testing. Write your tests and have the pipeline run them. Add security scanning so vulnerabilities can be addressed before the code is deployed. Add any task that your team does manually to ensure quality code. The goal is to automate for consistent results every time. I personally like David Farley’s philosophy on software development.
https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
For more learning use the below URL. Azure DevOps has a generous free tier. Go provision a new organization with a personal account and just use the product.
DevOps is a culture and not a set of technologies. Understanding what problems DevOps solves will help you pick the technologies that support your DevOps culture. I personally like David Farley and his philosophy on software development.
https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
https://www.amazon.com/Modern-Software-Engineering-Better-Faster-ebook/dp/B09GG6XKS4
As for books on CI/CD pipelines, there probably not many. You better off going through that vendor’s specific training. For example, Microsoft learn for Azure pipelines.
Have you considered reading https://www.amazon.co.uk/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912 (continuous deployment is an obvious extrapolation of that).
Automated testing is not the same as test automation. What you are doing is record and playback of the UI for testing the full application on a Functional Acceptance Test, User Acceptance Test or even End to End Integration Test level. It is very hard to make it a good return on investment because it needs coding, configuring, maintenance, reruns, fixes (you know those brittle tests?) and it will break a lot on the simplest things.
Test automation is something you need to adopt with your team and it works best bottom up. Look for the Test Automation Pyramid (Martin Fowler is a great resource). Unless you especially test the GUI, you should not touch the GUI with automated tests. Start with unit tests, then integration tests in the small, continue with tests that only test the javascript libraries with test libraries like Mocka. Functional testing should be done on the code through other interfaces, preferably APIs on webservices. Data tests should be done with an interface that allows bulk upload: If you need to do test how transactions are processed by your system, you don't want to enter those one-by-one through the GUI; you want to upload them in bulk in CSV/XML/JSON format, preferably through a web api.
When you test functionality that is testing if webpage elements are correctly wired to the backend, you should not do it on the full application, but on a mocked backend (look for TestCafe, it has this build in). For functional testing pages that are represented by a web page, there should be at least an API (like a webservice that does the same), and should be a deliverable by the developers.
I know it is a harsh message, but automating tests on the GUI can be a pitfall that can turn teams or management away form test automation because you are proofing them that test automation doesn't work.
Cooperate with your developers. Tell them what you want to test and they can help you out with building the correct interfaces for you, because that is what they do and like: building stuff.
Let them realize that testing is not "checking if they made an error somewhere", but that testing helps to give them confidence in deploying code faster because when they make an error it got signaled by a breaking test.
Read the pages from https://martinfowler.com/, get hooked on the stuff from Jez Humble, his Continuous Delivery book (https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912) explains lots of software development principles in a clear way.
I would use one of free CI (Jenkins or TeamCity) and make it build a complete war on some basis (you can setup the build trigger) and write automated scripts/plugin to upload war to your sever and redeploy to JBoss server.
Here's presentation on this matter: http://www.slideshare.net/jezhumble/continuous-delivery-5359386
Very nice book from Dave Farley: http://www.amazon.com/gp/product/0321601912