Machine Learning with Core ML An iOS developer's guide to implementing machine learning in mobile apps

By Joshua Newnham

FOR SALE IN INDIA ONLY

www.packt.com

Machine Learning with Core ML

An iOS developer's guide to implementing machine learning in mobile apps

Joshua Newnham

BIRMINGHAM - MUMBAI

Machine Learning with Core ML Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Amey Varangaonkar Acquisition Editor: Tushar Gupta Content Development Editor: Karan Thakkar Technical Editor: Sagar Sawant Copy Editor: Safis Editing Project Coordinator: Nidhi Joshi Proofreader: Safis Editing Indexer: Tejal Daruwale Soni Graphics: Tania Dutta Production Coordinator: Arvindkumar Gupta First published: June 2018 Production reference: 1260618 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78883-829-0

www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content

PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors About the author Joshua Newnham is a technology lead at a global design firm, Method, focusing on the intersection of design and artificial intelligence (AI), specifically in the areas of computational design and human computer interaction. Prior to this, he was a technical director at Masters of Pie, a virtual reality (VR) and augmented reality (AR) studio focused on building collaborative tools for engineers and creatives. First and foremost, I would like give thanks to my incredible wife and son for their extraordinary support, encouragement, and inspiration throughout this book and life in general. Thank you both. Writing a book is no small undertaking, and without the team at Packt continuously refining the work, you would likely be reading 400+ pages of late night ramblings. So, a big thanks to the team for helping me make this happen.

About the reviewer Shilpa Karkeraa is a leading solution expert and the founder CEO of Myraa Technologies, an Artificial Intelligence Solutions Company. From being an independent entrepreneur to a hands-on developer, she has been an innovator with cutting edge technologies. Prior to Myraa, she spearheaded as a Team Lead—Data Engineering Group of a Bay Area start-up to a Top Corporates Financial Services Firm in India to being an architect at a Singaporean B2C company. She is an active global technology speaker. She hopes to commercialize research and innovation to touch human lives effectively! Reviewing this book was indeed a page turner along with an excelling keyboard cruncher. Thanks Packt Publishing for an amazing experience with the book. Cheers to the author for his writing streak over the beauty of design and detail, to my dear friend Tanvi Bhatt and to my technologists at Myraa Technologies for the hands-on implementations.

Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents Preface Chapter 1: Introduction to Machine Learning What is machine learning? A brief tour of ML algorithms

Netflix – making recommendations Shadow draw – real-time user guidance for freehand drawing Shutterstock – image search based on composition iOS keyboard prediction – next letter prediction

A typical ML workflow Summary Chapter 2: Introduction to Apple Core ML Difference between training and inference Inference on the edge A brief introduction to Core ML Workflow

Learning algorithms

Auto insurance in Sweden Supported learning algorithms

Considerations Summary Chapter 3: Recognizing Objects in the World Understanding images Recognizing objects in the world Capturing data Preprocessing the data

Performing inference Summary Chapter 4: Emotion Detection with CNNs Facial expressions Input data and preprocessing Bringing it all together Summary Chapter 5: Locating Objects in the World Object localization and object detection Converting Keras Tiny YOLO to Core ML

1 7 8 11 11 15 20 23 26 27 29 29 31 34 35 37 37 45 46 47 49 50 55 55 65 76 77 79 80 84 106 115 117 118 129

Table of Contents

Making it easier to find photos Optimizing with batches Summary Chapter 6: Creating Art with Style Transfer Transferring style from one image to another A faster way to transfer style Converting a Keras model to Core ML Building custom layers in Swift Accelerating our layers Taking advantage of the GPU

Reducing your model's weight Summary Chapter 7: Assisted Drawing with CNNs Towards intelligent interfaces Drawing Recognizing the user's sketch Reviewing the training data and model Classifying sketches Sorting by visual similarity

Summary Chapter 8: Assisted Drawing with RNNs Assisted drawing Recurrent Neural Networks for drawing classification Input data and preprocessing Bringing it all together Summary Chapter 9: Object Segmentation Using CNNs Classifying pixels Data to drive the desired effect – action shots Building the photo effects application Working with probabilistic results Improving the model Designing in constraints Embedding heuristics Post-processing and ensemble techniques Human assistance

Summary Chapter 10: An Introduction to Create ML A typical workflow Preparing the data [ ii ]

137 167 171 173 174 180 183 194 204 205 209 214 217 218 218 234 235 237 253 269 271 271 273 280 296 309 311 312 316 321 340 340 341 342 343 343 344 345 345 347

Table of Contents

Creating and training a model

351 355 356 356 357 358

Model parameters Model metadata Alternative workflow (graphical)

Closing thoughts Summary Other Books You May Enjoy

359

Index

363

[ iii ]

Preface We are living on the verge of a new era of computing, an era where computers are becoming more of a companion than a tool. The devices we carry in our pockets will soon better understand our world and us a lot better, and this will have a profound impact on how we interact with and use them. But right now, a lot of these exciting advancements are stuck in the labs of researchers and not in the hands of designers and developers, making them usable and accessible to users. This is not because the details are locked away; on the contrary, in most cases they are freely available. This gap is somewhat due to our contentment with sticking to what we know, having the user do all the work, making them tap on the buttons. If nothing else, I hope this book makes you curious about what is out there and how it can be used to create new experiences, or improve existing ones. Within the pages of this book, you will find a series of examples to help you build an understanding of how deep neural networks work and how they can be applied. This book focuses on a set of models for a better understanding of images and photos, specifically looking at how they can be adapted and applied on the iOS platform. This narrow focus of image-based models and the iOS platform is intentional; I find that the visual nature of images makes the concepts easier to, well, visualize, and the iPhone provides the perfect candidate and environment for experimentation. So, as you go through this book, I encourage you to start thinking about new ways of how these models can be used and what new experiences you could create. With that being said, let's get started!

Who this book is for This book will appeal to three broad groups of people. The first are intermediate iOS developers who are interested in learning and applying machine learning (ML); some exposure to ML concepts may be beneficial but are not essential as this book covers the intuition behind the concepts and models used throughout it.

Preface

The second group are those who have experience in ML but not in iOS development and are looking for a resource to help them to get the grips with Core ML; for this group, it is recommended to complement this book with a book that covers the fundamentals of iOS development. The last group are experienced iOS developers and ML practitioners who are curious to see how various models have been applied in the context of the iOS platform.

What this book covers Chapter 1, Introduction to Machine Learning, provides a brief introduction to ML, including

some explanation of the core concepts, the types of problems, algorithms, and general workflow of creating and using a ML models. The chapter concludes by exploring some examples where ML is being applied.

Chapter 2, Introduction to Apple Core ML, introduces Core ML, discussing what it is, what it

is not, and the general workflow for using it.

Chapter 3, Recognizing Objects in the World, walks through building a Core ML application

from start to finish. By the end of the chapter, we would have been through the whole process of obtaining a model, importing it into the project, and making use of it. Chapter 4, Emotion Detection with CNNs, explores the possibilities of computers

understanding us better, specifically our mood. We start by building our intuition of how ML can learn to infer your mood, and then put this to practice by building an application that does just that. We also use this as an opportunity to introduce the Vision framework and see how it complements Core ML. Chapter 5, Locating Objects in the World, goes beyond recognizing a single object to being

able to recognize and locate multiple objects within a single image through object detection. After building our understanding of how it works, we move on to applying it to a visual search application that filters not only by object but also by composition of objects. In this chapter, we'll also get an opportunity to extend Core ML by implementing customer layers. Chapter 6, Creating Art with Style Transfer, uncovers the secrets behind the popular photo

effects application, Prisma. We start by discussing how a model can be taught to differentiate between the style and content of an image, and then go on to build a version of Prisma that applies a style from one image to another. We wrap up this chapter by looking at ways to optimize the model.

[2]

Preface Chapter 7, Assisted Drawing with CNNs, walks through building an application that can

recognize a users sketch using the same concepts that have been introduced in previous chapters. Once what the user is trying to sketch has been recognized, we look at how we can find similar substitutes using the feature vectors from a CNN. Chapter 8, Assisted Drawing with RNNs, builds on the previous chapter and explores

replacing the the convolution neural network (CNN) with a recurrent neural network (RNN) for sketch classification, thus introducing RNNs and showing how they can be applied to images. Along with a discussion on learning sequences, we will also delve into the details of how to download and compile Core ML models remotely. Chapter 9, Object Segmentation Using CNNs, walks through building

an ActionShot photography application. And in doing so, we introduce another model and accompanying concepts, and get some hands-on experience of preparing and processing data. Chapter 10, An Introduction to Create ML, is the last chapter. We introduce Create ML, a

framework for creating and training Core ML models within Xcode using Swift. By the end of this chapter, you will know how to quickly create, train, and deploy a custom models.

To get the most out of this book To be able to follow through the examples in this book, you will need the following software: macOS 10.13 or higher Xcode 9.2 or higher iOS 11.0 or higher (device and simulator) For the examples that are dependent on Core ML 2, you will need the following software: macOS 10.14 Xcode 10.0 beta iOS 12 (device and simulator)

[3]

Preface

It's recommended that you use https:/​/​notebooks.​azure.​com (or some other Jupyter notebook service provider) to follow the examples using the Core ML Tools Python package, but those wanting to run locally or train their model will need the following software: Python 2.7 Jupyter Notebooks 1.0 TensorFlow 1.0.0 or higher NumPy 1.12.1 or higher Core ML Tools 0.9 (and 2.0 for Core ML 2 examples)

Download the example code files You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you. You can download the code files by following these steps: 1. 2. 3. 4.

Log in or register at www.packtpub.com. Select the SUPPORT tab. Click on Code Downloads & Errata. Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of: WinRAR/7-Zip for Windows Zipeg/iZip/UnRarX for Mac 7-Zip/PeaZip for Linux The code bundle for the book is also hosted on GitHub at https:/​/​github.​com/ PacktPublishing/​Machine-​Learning-​with-​Core-​ML. In case there's an update to the code, it will be updated on the existing GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https:/​/​github.​com/​PacktPublishing/​. Check them out!

[4]

Preface

Download the color images We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http:/​/​www.​packtpub.​com/​sites/​default/​files/ downloads/​MachineLearningwithCoreML_​ColorImages.​pdf.

Conventions used There are a number of text conventions used throughout this book. CodeInText: Indicates code words in text, database table names, folder names, filenames,

file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "At the top of the class, we have the VideoCaptureDelegate protocol defined." A block of code is set as follows: public protocol VideoCaptureDelegate: class { func onFrameCaptured( videoCapture: VideoCapture, pixelBuffer:CVPixelBuffer?, timestamp:CMTime) }

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold: @IBOutlet var previewView:CapturePreviewView! @IBOutlet var classifiedLabel:UILabel! let videoCapture : VideoCapture = VideoCapture()

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Select System info from the Administration panel." Warnings or important notes appear like this.

Tips and tricks appear like this.

[5]

Machine Learning with Core ML (specifically iOS). You'll learn to implement Core ML for visual-based applications using the principles of transfer learning and neural networks. Having got to grips with the basics, you'll discover a series of seven examples, each providing a new use-case that uncovers how machine learning can be applied along with the related concepts.

Core ML is a popular framework by Apple, with APIs designed to support various machine learning tasks. It allows you to train your machine learning models and then integrate them into your iOS apps.

Machine Learning with Core ML is a fun and practical guide that not only demystifies Core ML but also sheds light on machine learning. In this book, you'll walk through realistic and interesting examples of machine learning in the context of mobile platforms

By the end of the book, you will have the skills required to put machine learning to work in their own applications, using the Core ML APIs

Things you will learn: •

Understand components of an ML project using algorithms, problems, and data

•

Create and optimize custom layers for unsupported layers

•

Master Core ML by obtaining and importing machine learning model, and generate classes

•

Apply CoreML to image and video data using CNN

•

Prepare data for machine learning model and interpret results for optimized solutions

•

Learn the qualities of RNN to recognize sketches, and augment drawing

•

Use Core ML transfer learning to execute style transfer on images

www.packt.com

FOR SALE IN INDIA ONLY