Hands-On

Continuous Integration and Delivery Build and release quality software at scale with Jenkins, Travis CI, and CircleCI

By Jean-Marcel Belmont

www.packt.com

FOR SALE IN INDIA ONLY

Hands-On Continuous Integration and Delivery

Build and release quality software at scale with Jenkins, Travis CI, and CircleCI

Jean-Marcel Belmont

BIRMINGHAM - MUMBAI

Hands-On Continuous Integration and Delivery Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Gebin George Acquisition Editor: Rohit Rajkumar Content Development Editor: Ronn Kurien, Devika Battike Technical Editor: Aditya Khadye Copy Editor: Safis Editing Project Coordinator: Jagdish Prabhu Proofreader: Safis Editing Indexer: Priyanka Dhadke Graphics: Tom Scaria Production Coordinator: Aparna Bhagat First published: August 2018 Production reference: 1270818 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78913-048-5

www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content

PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors About the author Jean-Marcel Belmont is a software engineer with a passion for automation and continuous integration. He is heavily involved in the open source community and frequently writes workshops on many different software development topics. He maintains a couple of meetup groups where he advocates clean code patterns and software craftsmanship. I would like to first and foremost thank my loving and patient wife, Christine, my son, Michael, and my daughter, Gabriella, for their support, patience, and encouragement throughout the long process of writing this book. Thanks also to my close friends and family for their encouragement throughout the book.

About the reviewers Hai Dam works as a DevOps Engineer in Netcompany, Denmark. His DevOps toolchain are Jenkins, CircleCI, ELK, AWS, and Docker.

Craig R Webster has delivered projects for clients of all sizes, ranging from small startups such as Orkell, Picklive and Tee Genius, through Notonthehighstreet, the UK Government, and the BBC. With more than 15 years of experience developing and deploying web applications, delivery pipelines, automation, and hosting platforms, Craig is able to take point on any technical project and ensure it is delivered on time, and on budget.

Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents Preface

1

Chapter 1: CI/CD with Automated Testing Business scenario

Manual processes – a hypothetical scenario Employee frustration Introducing automation Developer productivity Breaking down communication barriers Creating an environment of collaboration

Summary Questions Further reading Chapter 2: Basics of Continuous Integration Technical requirements What is CI? What is a software build anyway? CI process steps in a nutshell The value of CI Mitigating risks by utilizing CI

But it works on my machine! Database synchronization A missing deployment automation phase Late discovery of defects Test coverage not known Code coverage tools

Lack of project visibility

Software builds at source code check-in

What is a software build again? Scripting tool Performing single command builds Building your software in a nutshell Separating build scripts from your IDE Software assets should be centralized Creating a consistent directory structure Software builds should fail fast Building for any environment

Small build and large build breakdown CI build practices Private build Usage of CI server

7 7 8 10 13 18 21 26 27 27 27 29 30 30 30 31 31 32 32 32 32 33 33 33 34 34 34 34 35 35 40 40 41 43 44 44 45 45 45

Table of Contents

Manual integration builds Running fast builds

Summary Questions Further reading Chapter 3: Basics of Continuous Delivery Technical requirements Problems delivering software

What do we mean by delivering software? Common release anti-patterns Deploying software manually

Deployment automation to the rescue

Manual configuration management

Configuration management automation

How a production environment differs from other environments

The production environment should be the same as the staging environment

How to conduct a software release Frequent releases Automated releases

The benefits of automation in delivering software Team empowerment Error reduction Stress reduction

Configuration management

What does configuration management mean anyway? Version control Example properties file Version control management tools Version control practices Conducting software check-ins often Writing descriptive and meaningful commit messages

Dependency management

Example Node.js dependency file and Gopkg.toml dependency file Managing software components

Software configuration management

Configurability and flexibility concepts Specific types of configuration Configuration management across applications

Environment management

Manual environment setup Important configuration information for environments Containerized environments

Deployment pipeline

What is a deployment pipeline? Deployment pipeline practices

Building binaries once Deployment should be done the same way in every environment

[ ii ]

46 46 47 47 47

49 49 50 50 50 50 51 51 52 52 52 52 52 53 53 53 53 54 54 54 54 55 55 56 56 56 56 57 58 58 59 59 60 60 60 60 61 62 62 62 62 63

Table of Contents

Commit stage – first step of the deployment pipeline

Test gates

Acceptance tests build stage Manual testing Nonfunctional testing

Release preparation

Automating release processes Conducting rollbacks

Deployment scripting

Overview of build tools Deployment scripting concepts

Writing a script for each stage in the deployment pipeline Every environment should use the same scripts The deployment process should not change on each run

Deployment scripting best practices

Testing only known good foundations Testing the environment configuration Using relative paths Removing manual processes

Deployment ecosystem

Infrastructure tooling Cloud providers and tooling

Summary Questions Further reading Chapter 4: The Business Value of CI/CD Technical requirements Problems with communication

Miscommunication of requirements Lack of proper documentation Timezone differences Lack of trust and mutual respect Cultural differences and language barriers Long feedback loop cycles

Communicating pain points to team members Waiting for requirements information Undocumented steps in a deployment pipeline Keys to the kingdom to select few Too many communication channels Pain-Driven Development (PDD)

Sharing responsibilities among different teams Rotating team members Asking for feedback on development practices Creating cross-functional teams

Knowing your stakeholders

[ iii ]

63 64 64 64 64 65 65 65 66 66 66 67 67 67 68 68 68 68 68 69 69 69 70 70 70

71 71 72 72 73 73 74 74 75 75 76 76 77 77 78 78 79 80 80 81

Table of Contents

Project managers Executive leadership team End users

Demonstrating why CI/CD is important Metrics and reporting Educating leadership on automation

Getting approval for CI/CD from your stakeholders Starting a skunkworks project Starting CI/CD on your local machine Company presentation Lunch and learn

Summary Questions Further reading Chapter 5: Installation and Basics of Jenkins Technical requirements Windows installation Prerequisites to installing Jenkins Finding your version of Windows Installing Java

Windows installer Installing Jenkins in Windows Running the Jenkins Installer in Windows

Installing Jenkins with the Chocolatey Package Manager

Starting and stopping Jenkins in Windows in Command Prompt

Linux installation

Installing Jenkins on Ubuntu Starting the Jenkins service in Ubuntu Opening network traffic firewalls Unlocking Jenkins for the first login

macOS installation

Jenkins downloads package Unlocking Jenkins for the first login Installing Jenkins via Homebrew

Running Jenkins locally Creating a new item Console Output

Managing Jenkins

Configuring environment variables and tools Configuring a job to poll the GitHub version control repository

Summary Questions Further reading Chapter 6: Writing Freestyle Scripts

[ iv ]

81 81 82 83 83 84 85 85 85 86 86 87 87 87 89 89 89 89 90 90 91 92 92 94 95 95 96 96 97 98 100 100 104 107 108 108 112 113 116 117 120 120 121 123

Table of Contents

Technical requirements Creating a simple freestyle script Jenkins dashboard navigation Jenkins login screen Jenkins dashboard

Adding a new build job item Build configuration options

Configuring a freestyle job General Source Code Management Build Triggers Build environment Build Post-build actions

Adding environment variables

Global environment variable configuration EnvInject Plugin

Debugging issues with a freestyle job Build Project View Debugging issues with freestyle script

Summary Questions Further reading Chapter 7: Developing Plugins Technical requirements Jenkins plugins explained

Why are plugins useful? Jenkins plugin documentation Installing plugins in Jenkins

Building a simple Jenkins plugin Java installation Maven installation instructions Windows installation

Maven installation through the Chocolatey package manager Maven installation through the Maven source code Environment variables setup for the Windows operating system

macOS installation

Maven installation through the Homebrew package manager

Unix installation

Maven installation through the apt-get package manager

Jenkins plugin development

Maven settings file HelloWorld Jenkins plugin Folder layout explanation Jenkins plugin source code explanation

[v]

123 123 123 124 124 124 126 126 126 128 130 131 132 133 135 135 136 138 139 141 142 142 142 143 143 143 144 144 144 145 145 146 146 146 147 147 148 148 149 149 150 150 151 154 155

Table of Contents

Building a Jenkins plugin Installing a Jenkins plugin

Jenkins plugin ecosystem List of available plugins

Summary Questions Further reading Chapter 8: Building Pipelines with Jenkins Technical requirements Jenkins 2.0

Why move to Jenkins 2.0? Installing the Blue Ocean plugin on an existing instance Installing the Blue Ocean plugin through a Jenkins Docker Image Docker prerequisites Installing Docker images

Accessing Blue Ocean Jenkins view

The Jenkins pipeline

Creating a Jenkins pipeline

Providing a personal access token for GitHub Picking your GitHub organization Picking the GitHub repository

Creating a pipeline with the pipeline editor

Navigating in Jenkins Blue Ocean Pipelines view Pipeline detail view Pipeline build view Pipeline stage view Other views in Jenkins pipelines

Pipeline Syntax

Pipeline Editor Jenkinsfile

Pipeline keyword Agent keyword Stages keyword

Pipeline Syntax documentation

Summary Questions Further reading Chapter 9: Installation and Basics of Travis CI Technical requirements Travis CI introduction Comparing Travis CI and Jenkins

Travis CI prerequisites

Creating a GitHub account

[ vi ]

157 158 159 159 160 161 161 163 163 163 164 164 165 165 166 169 170 171 172 174 174 175 180 180 182 182 183 184 184 185 185 185 186 186 186 186 186 187 189 189 190 190 190 191

Table of Contents

Creating a Travis CI account Adding an SSH Key to your new GitHub Account

Adding a simple Travis CI YAML configuration script Travis CI YML script contents Adding a Github repository into your Travis CI account Travis CI job log

Travis CI script breakdown

Select a programming language Selecting infrastructure

Virtual image with Ubuntu Precise (12.04) infrastructure Default infrastructure Virtual Image with Ubuntu Trusty (14.04) infrastructure Container based infrastructure macOS infrastructure

Build customization

The build life cycle

before_install step install step before_script step script step after_script step List of build life cycle

Build failure behavior Installing a secondary programming language for a CI build Example Travis CI YML script with multiple languages

Docker in Travis CI

Example Travis CI YML script with a Dockerfile

GUI and headless browsers in Travis CI

Example Travis yml script with headless Chrome, Puppeteer, and the Jest Testing Library

Summary Questions Further reading Chapter 10: Travis CI CLI Commands and Automation Technical requirements Travis CLI installation Windows installation Linux installation macOS installation

Travis CLI commands Non-API commands

Printing help information Printing version information

API commands

Logging in to Travis CI Displaying the current access token Logging out of Travis CI Displaying accounts information

[ vii ]

195 198 200 200 200 203 204 204 205 205 205 205 205 206 206 206 206 206 207 207 207 207 208 208 208 209 209 210 211 213 213 213 215 215 216 216 218 218 220 220 221 222 222 222 223 224 224

Table of Contents Displaying help information for Travis commands

Interactive console session Printing API endpoint Information Conducting live monitoring with all the CI builds that are currently running Initiating a Travis CI API call API V3 REST call with curl

Printing out important system configuration information Listing all the repositories that the currently logged in user has access to Initiating synchronization with Travis CI for any new or outdated repositories in GitHub lint - a Travis YML script Obtaining current build information for the organization or user Finding the currently logged-in user information

Repository commands

Displaying the most recent build information for each branch in Git version control Listing cache information for all the repositories Deleting cache information for a given repository Enabling a repository in Travis CI Disabling a repository in Travis CI Cancelling the latest build in Travis CI Encrypting environment variables or deployment keys Adding environment variables in Travis CI

Encrypting a file Listing environment information Setting an environment variable Deleting an environment variable Clearing out all environment variables List history information for recent builds Initializing Travis CLI on a project Printing out CI build log information Opening the Travis web interface for the project Printing out public key information for the repository Restarting the latest CI build in Travis CI Printing out current build requests in Travis CI Printing out particular repository settings Configuring Travis CI add-ons Displaying general information for the current CI build Listing SSH key in Travis CI Displaying status information for the current build

Travis CI options for Pro version and Enterprise version Option to display the information for Pro version Option to display information for Enterprise version

Summary Questions Further reading Chapter 11: Travis CI UI Logging and Debugging Technical requirements Travis web client overview [ viii ]

225 226 226 226 228 231 232 233 234 234 235 235 236 236 236 236 237 237 237 238 238 239 240 240 240 241 241 242 243 243 244 245 245 246 246 247 247 248 248 248 249 249 249 250

251 251 252

Table of Contents

Main dashboard overview Job log overview

Debugging build locally with Docker Running build in debug mode

Getting an API token from the profile page

Getting access to a token using Travis CLI

Obtaining a job ID from the build log Obtaining the job ID from the URL in the View config button Obtaining the job ID via an API call to the /builds endpoint API call to start a build job in debug mode Fetching an SSH session for the debug mode Travis debug mode convenience Bash functions tmate shell session actions

Travis Web UI logging

Steps that Travis CI takes to protect your environment-specific variables

Travis CI deployment overview and debugging Supported Providers in Travis CI Heroku setup in Travis CI Debug failure in Travis YML script

Summary Questions Further reading Chapter 12: Installation and Basics of CircleCI Technical requirements CircleCI Comparing CircleCI and Jenkins CircleCI prerequisites Creating a GitHub account Creating a Bitbucket account Creating a CircleCI account

Setting up CircleCI in GitHub Setting up CircleCI in Bitbucket

Setting up a new Java project in Bitbucket with CircleCI build

CircleCI configuration overview

Concept overview of CircleCI configuration Adding source files to the new repository CircleCI build job for the new repository

Summary Questions Further reading Chapter 13: CircleCI CLI Commands and Automation Technical requirements CircleCI CLI installation [ ix ]

252 254 257 259 259 259 260 260 261 262 262 263 265 266 266 267 267 268 270 272 272 272 273 273 274 274 274 274 275 278 282 288 290 296 296 296 298 299 299 299 301 301 302

Hands-On

Continuous Integration and Delivery Hands-On Continuous Integration and Delivery starts with the fundamentals of continuous integration (CI) and continuous delivery (CD) and where it fits in the DevOps ecosystem. You will explore the importance of stakeholder collaboration as part of CI/CD. As you make your way through the chapters, you will get to grips with Jenkins UI, and learn to install Jenkins on different platforms, add plugins, and write freestyle scripts. Next, you will gain hands-on experience of developing

plugins with Jenkins UI, building the Jenkins 2.0 pipeline, and performing Docker integration. In the concluding chapters, you will install Travis CI and CircleCI and carry out scripting, logging, and debugging, helping you to acquire a broad knowledge of CI/CD with Travis CI and CircleCI. By the end of this book, you will have a detailed understanding of best practices for CI/CD systems and be able to implement them with confidence.

Things you will learn: •

Install Jenkins on multiple operating systems

•

Use CircleCI CLI jobs and work with pipelines

•

Work with Jenkins freestyle scripts, pipeline syntax, and methodology

•

Automate tasks using CircleCI CLI and learn to debug and troubleshoot

•

Explore Travis CI build life cycle events and multiple build languages

•

Learn open source tooling such as Git and GitHub

•

Master the Travis CI CLI (command-line interface) and automate tasks with the CLI

•

Install Docker and learn concepts in shell scripting

www.packt.com

FOR SALE IN INDIA ONLY