Keras Deep Learning Cookbook

Over 30 recipes for implementing deep neural networks in Python

Rajdeep Dua and Manpreet Singh Ghotra FOR SALE IN INDIA ONLY

www.packt.com

Keras Deep Learning Cookbook

Over 30 recipes for implementing deep neural networks in Python

Rajdeep Dua Manpreet Singh Ghotra

BIRMINGHAM - MUMBAI

Keras Deep Learning Cookbook Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Amey Varangaonkar Acquisition Editor: Karan Jain Content Development Editor: Karan Thakkar Technical Editor: Sagar Sawant Copy Editor: Safis Editing Project Coordinator: Nidhi Joshi Proofreader: Safis Editing Indexer: Pratik Shirodkar Graphics: Jisha Chirayil Production Coordinator: Aparna Bhagat First published: October 2018 Production reference: 1301018 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78862-175-5

www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content

Packt.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors About the authors Rajdeep Dua has over 18 years experience in the cloud and big data space. He has taught Spark and big data at some of the most prestigious tech schools in India: IIIT Hyderabad, ISB, IIIT Delhi, and Pune College of Engineering. He currently leads the developer relations team at Salesforce India. He has also presented BigQuery and Google App Engine at the W3C conference in Hyderabad. He led the developer relations teams at Google, VMware, and Microsoft, and has spoken at hundreds of other conferences on the cloud. Some of the other references to his work can be seen at Your Story and on ACM digital library. His contributions to the open source community relate to Docker, Kubernetes, Android, OpenStack, and Cloud Foundry.

Manpreet Singh Ghotra has more than 15 years experience in software development for both enterprise and big data software. He is currently working at Salesforce on developing a machine learning platform/APIs using open source libraries and frameworks such as Keras, Apache Spark, and TensorFlow. He has worked on various machine learning systems, including sentiment analysis, spam detection, and anomaly detection. He was part of the machine learning group at one of the largest online retailers in the world, working on transit time calculations using Apache Mahout, and the R recommendation system, again using Apache Mahout. With a master's and postgraduate degree in machine learning, he has contributed to, and worked for, the machine learning community.

About the reviewer Sujit Pal works at Elsevier Labs, a research and development group within the ReedElsevier PLC Group. His interests are in information retrieval, distributed processing, ontology development, natural language processing, and machine learning, and codes in Python, Scala, and Java. He combines his skills in these areas in order to help build new features or feature improvements in different products across the company. He believes in lifelong learning and blogs about his experiences at sujitpal.blogspot.com.

Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents Preface

1

Chapter 1: Keras Installation Introduction Installing Keras on Ubuntu 16.04 Getting ready How to do it...

Installing miniconda Installing numpy and scipy Installing mkl Installing TensorFlow Installing Keras Using the Theano backend with Keras

Installing Keras with Jupyter Notebook in a Docker image Getting ready How to do it...

Installing the Docker container Installing the Docker container with the host volume mapped

Installing Keras on Ubuntu 16.04 with GPU enabled Getting ready How to do it...

Installing cuda Installing cudnn Installing NVIDIA CUDA profiler tools interface development files Installing the TensorFlow GPU version Installing Keras

Chapter 2: Working with Keras Datasets and Models Introduction CIFAR-10 dataset How to do it...

CIFAR-100 dataset How to do it...

Specifying the label mode

MNIST dataset

How to do it...

Load data from a CSV file How to do it...

Models in Keras – getting started Anatomy of a model Types of models

7 7 7 8 8 8 8 9 10 11 13 14 14 14 14 15 16 16 17 18 19 19 19 20 21 21 21 22 23 24 24 25 25 26 26 26 26 27

Table of Contents

Sequential models How to do it...

Create a Sequential model Compile the model Train the model Evaluate the model Predict using the model Putting it all together Model inspection internals Model compilation internals Initialize the loss Model training Output of the sample

Shared layer models

Introduction – shared input layer How to do it... Concatenate function

Keras functional APIs

How to do it... The output of the example

Keras functional APIs – linking the layers How to do it...

Model class

Image classification using Keras functional APIs How to do it...

Chapter 3: Data Preprocessing, Optimization, and Visualization Feature standardization of image data Getting ready How to do it...

Initializing ImageDataGenerator

Sequence padding Getting ready How to do it...

Pre-padding with default 0.0 padding Post-padding Padding with truncation Padding with a non-default value

Model visualization Getting ready How to do it...

Code listing

Optimization Common code for samples Optimization with stochastic gradient descent Getting ready How to do it...

[ ii ]

28 28 28 30 30 30 30 31 32 33 34 35 36 36 36 37 37 39 39 41 41 41 42 42 42 47 47 47 48 49 50 50 50 51 51 51 52 52 52 53 53 54 54 55 55 55

Table of Contents

Optimization with Adam

58 59 59 62 63 63 64 66 67 67

Getting ready How to do it...

Optimization with AdaDelta Getting ready How to do it...

Adadelta optimizer

Optimization with RMSProp Getting ready How to do it...

Chapter 4: Classification Using Different Keras Layers Introduction Classification for breast cancer How to do it...

Data processing Modeling

Full code listing

Classification for spam detection How to do it...

Data processing Modeling

Full code listing

Chapter 5: Implementing Convolutional Neural Networks Introduction Cervical cancer classification Getting ready How to do it…

Data processing Modeling Predictions

Digit recognition Getting ready How to do it… Modeling

Chapter 6: Generative Adversarial Networks Introduction GAN overview

Basic GAN

Getting ready How to do it...

Building a generator Building a discriminator Initialize the GAN instance Training the GAN

[ iii ]

71 71 72 72 73 76 78 80 81 81 83 85 89 89 90 90 91 91 96 100 102 102 103 104 109 109 110 110 111 111 111 113 114 115

Table of Contents

Output plots Average metrics of the GAN

Boundary seeking GAN Getting ready How to do it...

Generator Discriminator Initializing the BGAN class Boundary seeking loss

Train the BGAN Output the plots

DCGAN

Iteration 0 Iteration 10000 Metrics of the BGAN model Plotting the metrics

Getting ready How to do it... Generator

Summary of the generator Training the generator

Discriminator

Build the discriminator Summary of the discriminator Compile the discriminator

Combined model - generator and discriminator Train the generator using feedback from a discriminator Putting it all together The output of the program Average metrics of the model

Chapter 7: Recurrent Neural Networks Introduction The need for RNNs

Simple RNNs for time series data Getting ready

Loading the dataset

How to do it…

Instantiate a sequential model

LSTM networks for time series data LSTM networks LSTM memory example Getting ready How to do it...

Encoder LSTM configuration and model Train the model Full code listing

Time series forecasting with LSTM [ iv ]

117 119 119 121 121 121 122 123 125 125 127 127 128 128 129 129 130 130 131 132 132 133 133 134 135 135 136 138 138 140

141 141 141 142 143 143 145 150 154 154 155 156 156 156 157 159 160 162

Table of Contents

Getting ready

Load the dataset

How to do it…

Instantiate a sequential model Observation

Sequence to sequence learning for the same length output with LSTM Getting ready How to do it…

Training data Model creation Model fit and prediction

Chapter 8: Natural Language Processing Using Keras Models Introduction Word embedding Getting ready How to do it...

Without embeddings With embeddings

Sentiment analysis Getting ready How to do it… Full code listing

Chapter 9: Text Summarization Using Keras Models Introduction Text summarization for reviews How to do it…

Data processing Encoder-decoder architecture Training

See also

Chapter 10: Reinforcement Learning Introduction The CartPole game with Keras How to do it...

Implementing the DQN agent The memory and remember The replay function The act function Hyperparameters for the DQN DQN agent class Training the agent

Dueling DQN to play Cartpole Getting ready

DQN agent

[v]

162 162 164 169 173 173 174 174 175 177 178 181 181 182 182 183 183 186 189 190 191 195 197 197 198 198 199 203 205 207 209 209 210 211 211 212 213 214 214 215 216 218 220 221

Table of Contents init method

Setting the last layer of the network Dueling policy

Init code base BoltzmannQPolicy

Adjustment during training

Sequential memory

How to do it...

Plotting the training and testing results

221 222 222 222 224 224 225 226 227

Other Books You May Enjoy

231

Index

235

[ vi ]

Preface Keras has quickly emerged as a popular deep learning library. Written in Python, it allows you to train convolutional as well as recurrent neural networks with speed and accuracy. The Keras Deep Learning Cookbook shows you how to tackle different problems encountered while training efficient deep learning models, with the help of the popular Keras library. This book covers installing and setting up Keras, while also demonstrating how you can perform deep learning with Keras in the TensorFlow, Apache MXNet, and CNTK backends. From loading data to fitting and evaluating your model for optimal performance, you will work through a step-by-step process to tackle every possible problem faced while training deep models. You will implement convolutional and recurrent neural networks, adversarial networks, and more with the help of this handy guide. In addition to this, you will learn how to train these models for real-world image and language processing tasks. By the end of this book, you will have a practical, hands-on understanding of how you can leverage the power of Python and Keras to perform effective deep learning.

Who this book is for Keras Deep Learning Cookbook is for you if you are a data scientist or machine learning expert who wants to find practical solutions to common problems encountered while training deep learning models. A basic understanding of Python and some experience in machine learning and neural networks is required for this book.

What this book covers Chapter 1, Keras Installation, covers various installation and setup procedures, as well as

defining various Keras configurations.

Chapter 2, Working with Keras Datasets and Models, covers using various datasets, such as

CIFAR10, CIFAR100, or MNIST, and many other datasets and models used for image classification.

Chapter 3, Data Preprocessing, Optimization, and Visualization, covers various preprocessing

and optimization techniques using Keras. The optimization techniques covered include TFOptimizer, AdaDelta, and many more.

Preface Chapter 4, Classification Using Different Keras Layers, details various Keras layers, for

example, recurrent layers, and convolutional layers.

Chapter 5, Implementing Convolutional Neural Networks, teaches you convolutional neural

network algorithms in detail, using the example of cervical cancer classification and the digit recognition dataset.

Chapter 6, Generative Adversarial Networks, covers basic generative adversarial networks

(GANs) and boundary-seeking GAN.

Chapter 7, Recurrent Neural Networks, covers the basics of recurrent neural networks in

order to implement Keras based on historical datasets.

Chapter 8, Natural Language Processing Using Keras Models, covers the basics of NLP for

word analysis and sentiment analysis using Keras.

Chapter 9, Text Summarization Using Keras Models, shows you how to use Keras models for

text summarization when using the Amazon reviews dataset.

Chapter 10, Reinforcement Learning, focuses on formulating and developing reinforcement

learning models using Keras.

To get the most out of this book Readers should have some basic knowledge of Keras and deep learning.

Download the example code files You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you. You can download the code files by following these steps: 1. 2. 3. 4.

Log in or register at www.packt.com. Select the SUPPORT tab. Click on Code Downloads & Errata. Enter the name of the book in the Search box and follow the onscreen instructions.

[2]

Preface

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of: WinRAR/7-Zip for Windows Zipeg/iZip/UnRarX for Mac 7-Zip/PeaZip for Linux The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Keras-Deep-Learning-Cookbook. In case there's an update to the code, it will be updated on the existing GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https:/​/​github.​com/​PacktPublishing/​. Check them out!

Download the color images We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http:/​/​www.​packtpub.​com/​sites/​default/​files/ downloads/​9781788621755_​ColorImages.​pdf.

Conventions used There are a number of text conventions used throughout this book. CodeInText: Indicates code words in text, database table names, folder names, filenames,

file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Finally, we save all the reviews into a pickle file." A block of code is set as follows: stories = list() for i, text in enumerate(clean_texts): stories.append({'story': text, 'highlights': clean_summaries[i]})

[3]

Keras Deep Learning Cookbook

Keras has quickly emerged as a popular deep learning library. Written in Python, it allows you to train convolutional as well as recurrent neural networks with speed and accuracy. The Keras Deep Learning Cookbook shows you how to tackle different problems encountered while training ef cient deep learning models, with the help of the popular Keras library. Starting with installing and setting up Keras, the book demonstrates how you can perform deep learning with Keras in the TensorFlow. From loading data to tting and evaluating your model for

optimal performance, you will work through a step-by-step process to tackle every possible problem faced while training deep models. You will implement convolutional and recurrent neural networks, adversarial networks, and more with the help of this handy guide. In addition to this, you will learn how to train these models for real-world image and language processing tasks. By the end of this book, you will have a practical, hands-on understanding of how you can leverage the power of Python and Keras to perform effective deep learning.

Things you will learn: •

Install and con gure Keras in TensorFlow

•

•

Master neural network programming using the Keras library

Work with various datasets and models used for image and text classi cation

•

•

Understand the different Keras layers

•

Use Keras to implement simple feed-forward neural networks, CNNs and RNNs

Develop text summarization and reinforcement learning models using Keras

www.packt.com

FOR SALE IN INDIA ONLY