Adding Machine Learning to iOS Apps

Use case: using machine learning and iPhone’s camera, identify certain types of objects in real-time.

Steps:

  1. create a cNN (convolutional neural network) in python with a ML (machine learning) package called Keras with TensorFlow backend.
  2. convert the newly created cNN to a format that can be used by iPhones by leveraging the Core ML iOS package.
  3. run the converted Core ML model on the iPhone to make predictions on what the phone’s camera is viewing.
  4. save the distribution of predicted probabilities from the Core ML model and send to API created with Firebase which is a BaaS (backend as a service).
  5. return the custom output based on API back to iPhone

Technologies/languages used:  Keras, python, iOS swift, Core ML, Firebase SDK, javascript

How the models and algorithms worked together:

I ended up creating combination Core ML models, in which loosely based on OOD (object orientated design).  How this works, is 1st model identifies domain, if greater than 50% probability, it would then call the next model, and if that is greater than 50% probability, which then call the API to get a returned result.

High-level diagram of how iOS can use models to make useful prediction:

Here is the iOS swift function to call both models.

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

// initial model always called for domain detection
guard let model_one = try? VNCoreMLModel(for: imagenet_ut().model) else { return }
let request = VNCoreMLRequest(model: model_one) { (finishedRequest, error) in
guard let results = finishedRequest.results as? [VNClassificationObservation] else { return }
guard let Observation = results.first else { return }
DispatchQueue.main.async(execute: {
confidence_one = Int(Observation.confidence * 100)
})
}
guard let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
// executes request
try? VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:]).perform([request])

// ADDING 2ND MODEL TO RUN AT SAME TIME AS 1ST MODEL

// model for issue 1

let chosen_all_model = getAllModel(cise: self.Model.cise)

guard let model_two = try? VNCoreMLModel(for: chosen_all_model) else { return }

let request_two = VNCoreMLRequest(model: model_two) { (finishedRequest, error) in
guard let results = finishedRequest.results as? [VNClassificationObservation] else { return }
guard let Observation = results.first else { return }

DispatchQueue.main.async(execute: {
_confidence_two = Int(Observation.confidence * 100)

}})}

In the complete iOS project, I have 12 models available to run based on various context of what the user was doing. The API create on Firebase could handle various combinations of calls based on these contexts, and send back a useful prediction to the User.