Deploying to Core ML

Object detector models created in Turi Create can easily be deployed to Core ML.

Deployment for iOS 12 and macOS 10.14 (Turi Create 5)

With Turi Create 5.0+ and starting in iOS 12, macOS 10.14 you can directly integrate object detector models via the Vision Framework.

You can export to Core ML in Turi Create 5 as follows:

model.export_coreml('MyDetector.mlmodel')

let mlmodel = MyDetector()
let visionModel = try VNCoreMLModel(for: mlmodel.model)
let objectRecognition = VNCoreMLRequest(model: visionModel,
                                        completionHandler: { (request, error) in
    guard let results = request.results else { return }

    for case let foundObject as VNRecognizedObjectObservation in results {
        let bestLabel = foundObject.labels.first! // Label with highest confidence
        let objectBounds = foundObject.boundingBox // Normalized between [0,1]
        let confidence = foundObject.confidence // Confidence for the predicted class

        // Use the computed values.
        print(bestLabel.identifier, confidence, objectBounds)
    }
})
objectRecognition.imageCropAndScaleOption = .scaleFill

For more details on the integration with Core ML and a sample app to get you started, please look at the the article on Recognizing Objects in Live Capture. Please note that the demo in the above link uses a normalized confidence which enforces that the score across all classes sum to 1.

Note: The bounding box object VNRecognizedObjectObservation.boundingBox has a different definition from the one used for Turi Create. First, the location is defined by the lower-left corner of the bounding box instead of the center. Secondly, the coordinate space has origin located in the lower-left corner of image. Third, the location, width, and height for the bounding box are all normalized between [0,1] by the dimensions of the images.

Note: Only models that were exported with non-maximum suppression (the default behavior in Turi Create 5.0+) will work with this example app. Older models or models that specify include_non_maximum_suppression=False will not give results as VNRecognizedObjectObservation objects and instead as VNCoreMLFeatureValueObservation objects; these are silently ignored in this example app. To learn how to work with VNCoreMLFeatureValueObservation objects, please continue reading.

Deployment for iOS 11 and macOS 10.13

To deploy the object detector in apps for iOS 11 and macOS 10.13, you need a few extra steps (these have been incorporated into iOS 12 and macOS 10.14).

You can export to Core ML in Turi Create 5 as follows:

model.export_coreml('MyDetector.mlmodel', include_non_maximum_suppression=False)

and with Turi Create 4 you can do

model.export_coreml('MyDetector.mlmodel')

This Core ML model takes an image and outputs two matrices of floating point values, confidence and coordinates. The first one has shape N-by-C, where N is the maximum number of bounding boxes that can be returned in a single image, and C is the number of classes. If you index this at (n, c), you get the confidence of the n:th bounding box for class c. The other output, coordinates, is N-by-4 and contains [x, y, width, height] coordinates for each bounding box. The coordinates are expressed relative to the original input size as values between 0 and 1. This is because it does not know the size before it was resized to fit the neural network. To get pixel values as in Turi Create, you have to multiply x and width by the original width y and height by the original height. Note that x and y refer to the center of the box.

Drag and drop MyDetector.mlmodel into your Xcode project and add it to your app by ticking the appropriate Target Membership check box. An arrow next to MyDetector should appear:

Useful meta data is stored inside the model, such as class labels:

let mlmodel = MyDetector()
let userDefined: [String: String] = mlmodel.model.modelDescription.metadata[MLModelMetadataKey.creatorDefinedKey]! as! [String : String]
let labels = userDefined["classes"]!.components(separatedBy: ",")

The order of labels corresponds to the confidence output. The meta data also contains a third type of threshold that we have yet to discuss: non_maximum_suppression_threshold:

let nmsThreshold = Float(userDefined["non_maximum_suppression_threshold"]!) ?? 0.5

Before we discuss how to use this threshold, we must first make a prediction.

Prediction

Making a prediction is easy. Starting in iOS 13 we can now include confidence thresholds directly in the VNCoreMLModel. For an example of how to include thresholds in a project utilizing the Vision framework, see here. First, we need to setup a subclass of MLFeatureProvider:

class ThresholdProvider: MLFeatureProvider {

    open var values = [
        "iouThreshold": MLFeatureValue(double: 0.3),
        "confidenceThreshold": MLFeatureValue(double: 0.2)
    ]

    var featureNames: Set<String> {
        return Set(values.keys)
    }

    func featureValue(for featureName: String) -> MLFeatureValue? {
        return values[featureName]
    }
}

After we've specified one or more thresholds, we assign them to the model object and make a prediction:

let model = try VNCoreMLModel(for: mlmodel.model)

model.featureProvider = ThresholdProvider()

let request = VNCoreMLRequest(model: model, completionHandler: { [weak self] request, error in
    self?.processClassifications(for: request, error: error)
})
request.imageCropAndScaleOption = .scaleFill

We use .scaleFill that stretches the image into the native input size of the model. If you use .centerCrop or .scaleFit, it will be a bit trickier to correctly map the bounding box coordinate system to the original input image.

From the request results we get two MLMultiArray instances, coordinates and confidence. For easier handling, we will convert these results into an array of the following struct:

struct Prediction {
    let labelIndex: Int
    let confidence: Float
    let boundingBox: CGRect
}

While building an array of these, we might as well trim the list of predictions by enforcing a minimum confidence threshold. This threshold is entirely up to you and your user experience and corresponds to the confidence_threshold parameter in predict inside Turi Create:

let results = request.results as! [VNCoreMLFeatureValueObservation]

let coordinates = results[0].featureValue.multiArrayValue!
let confidence = results[1].featureValue.multiArrayValue!

let confidenceThreshold = 0.25
var unorderedPredictions = [Prediction]()
let numBoundingBoxes = confidence.shape[0].intValue
let numClasses = confidence.shape[1].intValue
let confidencePointer = UnsafeMutablePointer<Double>(OpaquePointer(confidence.dataPointer))
let coordinatesPointer = UnsafeMutablePointer<Double>(OpaquePointer(coordinates.dataPointer))
for b in 0..<numBoundingBoxes {
    var maxConfidence = 0.0
    var maxIndex = 0
    for c in 0..<numClasses {
        let conf = confidencePointer[b * numClasses + c]
        if conf > maxConfidence {
            maxConfidence = conf
            maxIndex = c
        }
    }
    if maxConfidence > confidenceThreshold {
        let x = coordinatesPointer[b * 4]
        let y = coordinatesPointer[b * 4 + 1]
        let w = coordinatesPointer[b * 4 + 2]
        let h = coordinatesPointer[b * 4 + 3]

        let rect = CGRect(x: CGFloat(x - w/2), y: CGFloat(y - h/2),
                          width: CGFloat(w), height: CGFloat(h))

        let prediction = Prediction(labelIndex: maxIndex,
                                    confidence: Float(maxConfidence),
                                    boundingBox: rect)
        unorderedPredictions.append(prediction)
    }
}

This gives us an array of predictions (unorderedPredictions), still in no particular order. This array contains more predictions than returned by predict in Turi Create, since we are still missing a post-processing step called non-maximum suppression.

Non-maximum suppression

This step is performed automatically by predict in Turi Create, but it is not performed inside the Core ML inference model, so we will have to add it ourselves. The model is prone to predict multiple similar predictions associated with a single object instance. Here are the results of the two dogs without non-maximum suppression:

The algorithm is simple: Start by taking your highest-confidence prediction and add it to your final list of predictions. Check the IoU (see Evaluation) between it and and all the remaining predictions. Remove (or suppress) any prediction with an IoU above a pre-determined threshold (the nmsThreshold we extracted from the meta data). Repeat this procedure, now excluding predictions that you have already added or removed. Here is a reference implementation:

# Array to store final predictions (after post-processing)
var predictions: [Prediction] = []
let orderedPredictions = unorderedPredictions.sorted { $0.confidence > $1.confidence }
var keep = [Bool](repeating: true, count: orderedPredictions.count)
for i in 0..<orderedPredictions.count {
    if keep[i] {
        predictions.append(orderedPredictions[i])
        let bbox1 = orderedPredictions[i].boundingBox
        for j in (i+1)..<orderedPredictions.count {
            if keep[j] {
                let bbox2 = orderedPredictions[j].boundingBox
                if IoU(bbox1, bbox2) > nms_threshold {
                    keep[j] = false
                }
            }
        }
    }
}

The intersection-over-union can be computed as:

public func IoU(_ a: CGRect, _ b: CGRect) -> Float {
    let intersection = a.intersection(b)
    let union = a.union(b)
    return Float((intersection.width * intersection.height) / (union.width * union.height))
}

The array predictions is now the final array of predictions corresponding to what predict returns in Turi Create.