Alexander Sosnovshchenko

Machine Learning with Swift

Artificial Intelligence for iOS

FOR SALE IN INDIA ONLY

Machine Learning with Swift

Artificial Intelligence for iOS

Alexander Sosnovshchenko

BIRMINGHAM - MUMBAI

Machine Learning with Swift Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Veena Pagare Acquisition Editor: Vinay Argekar Content Development Editor: Mayur Pawanikar Technical Editor: Dinesh Pawar Copy Editor: Vikrant Phadkay, Safis Editing Project Coordinator: Nidhi Joshi Proofreader: Safis Editing Indexer: Pratik Shirodkar Graphics: Tania Dutta Production Coordinator: Arvindkumar Gupta First published: February 2018 Production reference: 1270218 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78712-151-5

www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content

PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors About the author Alexander Sosnovshchenko has been working as an iOS software engineer since 2012. Later he made his foray into data science, from the first experiments with mobile machine learning in 2014, to complex deep learning solutions for detecting anomalies in video surveillance data. He lives in Lviv, Ukraine, and has a wife and a daughter. Thanks to Dmitrii Vorona for moral support, invaluable advice, and code reviews; Nikolay Sosnovshchenko and Oksana Matskovich for the help with pictures of creatures and androids; David Kopec and Matthijs Hollemans for their open source projects; Mr. Jojo Moolayil for his efforts and expertise as a contributing author and reviewer; and my family for being supportive and patient.

About the reviewers Jojo Moolayil is an artificial intelligence, deep learning, and machine learning professional with over 5 years of experience and is the author of Smarter Decisions – The Intersection of Internet of Things and Decision Science. He works with GE and lives in Bengaluru, India. He has also been a technical reviewer about various books in machine learning, deep learning, and business analytics with Apress and Packt. I would like to thank my family, friends, and mentors.

Cecil Costa, also known as Eduardo Campos in Latin American countries, is a EuroBrazilian freelance developer who has been learning about computers since he got his first PC in 1990. Learning is his passion, and so is teaching; this is why he works as a trainer. He has organized both on-site and online courses for companies. He is also the author of a few Swift books. I’d like to thank Maximilian Ambergis for creating the delete key; it has been very useful for me!

Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents Preface

1

Chapter 1: Getting Started with Machine Learning What is AI? The motivation behind ML What is ML ? Applications of ML

7

Digital signal processing (DSP) Computer vision Natural language processing (NLP) Other applications of ML

Using ML to build smarter iOS applications Getting to know your data Features Types of features Choosing a good set of features

Getting the dataset Data preprocessing

Choosing a model Types of ML algorithms Supervised learning Unsupervised learning Reinforcement learning Mathematical optimization – how learning works Mobile versus server-side ML Understanding mobile platform limitations

Summary Bibliography Chapter 2: Classification – Decision Tree Learning Machine learning toolbox Prototyping the first machine learning app

8 9 10 11 11 12 12 13 13 15 15 16 17 19 21 21 22 22 23 25 27 28 30 31 31 33 34 35

Table of Contents

Tools Setting up a machine learning environment

IPython notebook crash course Time to practice Machine learning for extra-terrestrial life explorers Loading the dataset Exploratory data analysis Data preprocessing Converting categorical variables Separating features from labels One-hot encoding Splitting the data

36 37 39 41 42 43 44 49 49 50 50 51

Decision trees everywhere Training the decision tree classifier Tree visualization Making predictions Evaluating accuracy Tuning hyperparameters Understanding model capacity trade-offs

How decision tree learning works Building a tree automatically from data Combinatorial entropy Evaluating performance of the model with data Precision, recall, and F1-score K-fold cross-validation Confusion matrix

51 52 53 56 57 59 60 61 62 62 66 67 68 69

Implementing first machine learning app in Swift Introducing Core ML

70

Core ML features Exporting the model for iOS Ensemble learning random forest Training the random forest Random forest accuracy evaluation Importing the Core ML model into an iOS project Evaluating performance of the model on iOS

71

[ ii ]

70 71 73 73 74 76 77

Table of Contents Calculating the confusion matrix

79

Decision tree learning pros and cons

82

Summary

83

Chapter 3: K-Nearest Neighbors Classifier Calculating the distance DTW Implementing DTW in Swift

85 86 88 91

Using instance-based models for classification and clustering People motion recognition using inertial sensors Understanding the KNN algorithm Implementing KNN in Swift

93 94 96 98

Recognizing human motion using KNN Cold start problem Balanced dataset Choosing a good k

100 102 104 104

Reasoning in high-dimensional spaces KNN pros KNN cons Improving our solution Probabilistic interpretation More data sources Smarter time series chunking Hardware acceleration Trees to speed up the inference Utilizing state transitions

105 106 107 107 107 108 108 108 108 109

Summary Bibliography

109 109

Chapter 4: K-Means Clustering Unsupervised learning K-means clustering Implementing k-means in Swift

111 111 112 114

Update step Assignment step

116 117

Clustering objects on a map

119

[ iii ]

Table of Contents

Choosing the number of clusters K-means clustering – problems K-means++ Image segmentation using k-means Summary Chapter 5: Association Rule Learning Seeing association rules Defining data structures Using association measures to assess rules Supporting association measures Confidence association measures Lift association measures Conviction association measures

Decomposing the problem Generating all possible rules Finding frequent item sets The Apriori algorithm Implementing Apriori in Swift Running Apriori Running Apriori on real-world data The pros and cons of Apriori Building an adaptable user experience Summary Bibliography Chapter 6: Linear Regression and Gradient Descent Understanding the regression task Introducing simple linear regression Fitting a regression line using the least squares method

121 123 125 128 130 131 132 132 134 135 137 137 138 139 140 141 142 143 144 145 147 148 149 150 151 152 152

Where to use GD and normal equation Using gradient descent for function minimization

156 157 158

Forecasting the future with simple linear regression

161

Feature scaling Feature standardization

162 162

Multiple linear regression

165

[ iv ]

Table of Contents

Implementing multiple linear regression in Swift Gradient descent for multiple linear regression Training multiple regression Linear algebra operations

Feature-wise standardization Normal equation for multiple linear regression

Understanding and overcoming the limitations of linear regression

Fixing linear regression problems with regularization Ridge regression and Tikhonov regularization LASSO regression

ElasticNet regression

166 168 169 170 172 174 174 178 178 179 179

Summary Bibliography

181 181

Chapter 7: Linear Classifier and Logistic Regression Revisiting the classification task Linear classifier Logistic regression

183 183 184 185

Implementing logistic regression in Swift

187

The prediction part of logistic regression Training the logistic regression Cost function

188

Predicting user intents

189 190 192

Handling dates

192

Choosing the regression model for your problem Bias-variance trade-off Summary Chapter 8: Neural Networks What are artificial NNs anyway? Building the neuron

194 194 195 197 198 198

Non-linearity function

201 202 203

Step-like activation functions Rectifier-like activation functions

Building the network Building a neural layer in Swift Using neurons to build logical functions [v]

204 205 206

Table of Contents

Implementing layers in Swift Training the network

208 208

Vanishing gradient problem Seeing biological analogies

209 209

Basic neural network subroutines (BNNS) BNNS example

210 211

Summary

214

Chapter 9: Convolutional Neural Networks Understanding users emotions Introducing computer vision problems Introducing convolutional neural networks Pooling operation Convolution operation Convolutions in CNNs

215 216 217 219 220 221 226

Building the network

227

Input layer Convolutional layer Fully-connected layers Nonlinearity layers Pooling layer Regularization layers

228 228 230 230 230 231 231 231

Dropout Batch normalization

Loss functions Training the network Training the CNN for facial expression recognition Environment setup Deep learning frameworks Keras

232 233 233 234 235 235

Loading the data Splitting the data Data augmentation Creating the network Plotting the network structure

236 238 239 240 242

[ vi ]

Table of Contents

Training the network Plotting loss Making predictions Saving the model in HDF5 format Converting to Core ML format Visualizing convolution filters Deploying CNN to iOS Summary Bibliography Chapter 10: Natural Language Processing NLP in the mobile development world Word Association game Python NLP libraries Textual corpuses Common NLP approaches and subtasks

243 245 245 247 247 247 250 252 252 253 254 255 258 258 259

Tokenization Stemming Lemmatization Part-of-speech (POS) tagging Named entity recognition (NER) Removing stop words and punctuation

260

Distributional semantics hypothesis Word vector representations Autoencoder neural networks Word2Vec Word2Vec in Gensim Vector space properties iOS application

268

Chatbot anatomy Voice input NSLinguisticTagger and friends Word2Vec on iOS Text-to-speech output UIReferenceLibraryViewController

275

[ vii ]

262 263 263 266 267 268 269 270 273 273 274 275 277 279 280 281

Table of Contents

Putting it all together

282

Word2Vec friends and relatives Where to go from here? Summary Chapter 11: Machine Learning Libraries Machine learning and AI APIs Libraries General-purpose machine learning libraries AIToolbox BrainCore Caffe Caffe2 dlib FANN LearnKit MLKit Multilinear-math MXNet Shark TensorFlow tiny-dnn Torch YCML

283 285 285 287 287 288 289 292 293 293 294 294 294 295 295 295 296 296 297 297 298 298

Inference-only libraries

299

Keras LibSVM Scikit-learn XGBoost

300

NLP libraries

301

Word2Vec Twitter text

301

300 300 301

301

Speech recognition

302

TLSphinx OpenEars

302 302

Computer vision

302

[ viii ]

Table of Contents

OpenCV ccv OpenFace Tesseract

303 303 304 304

Low-level subroutine libraries

304

Eigen fmincg-c IntuneFeatures SigmaSwiftStatistics STEM Swix LibXtract libLBFGS NNPACK Upsurge YCMatrix

305 305 305 306 306 306 307 307 308 308 308

Choosing a deep learning framework Summary Chapter 12: Optimizing Neural Networks for Mobile Devices Delivering perfect user experience Calculating the size of a convolutional neural network Lossless compression Compact CNN architectures SqueezeNet MobileNets ShuffleNet CondenseNet

309 309 311 312 312 315 317 317 318 318 318

Preventing a neural network from growing big Lossy compression Optimizing for inference

319 320 321 321 322 322 323 324

Network pruning Weights quantization Reducing precision Other approaches Facebook's approach in Caffe2

[ ix ]

Machine Learning with Swift Machine learning as a field promises to bring increased intelligence to the software by helping us learn and analyse information efficiently and discover certain patterns that humans cannot. This book will be your guide as you embark on an exciting journey in machine learning using the popular Swift language.

Things you will learn:

• Learn rapid model prototyping with Python and Swift

• Deploy pre-trained models to iOS using Core ML

We'll start with machine learning basics in the first part of the book to develop a lasting intuition about fundamental machine learning concepts. We explore various supervised and unsupervised statistical learning techniques and how to implement them in Swift, while the third section walks you through deep learning techniques with the help of typical real-world cases. In the last section, we will dive into some hard core topics such as model compression, GPU acceleration and provide some recommendations to avoid common mistakes during machine learning application development.

• Find hidden patterns in the data using unsupervised learning

• Get a deeper understanding of the clustering techniques

• Learn modern compact architectures of neural networks for iOS devices

• Train neural networks for image processing and natural language

By the end of the book, you'll be able to develop intelligent applications written in Swift that can learn for themselves.

www.packtpub.com

processing

FOR SALE IN INDIA ONLY