代码之家 › 专栏 › 技术社区 › Swift Rabbit

结合CoreML和ARKit

coreml arkit xcode9-beta ios11 swift

Swift Rabbit · 技术社区 · 7 年前

我试图在我的项目中使用给定的 接收v3 website .

我从ARKit的标准模板开始(Xcode 9 beta 3)

在viewDelegate的末尾,我写道:

sceneView.session.delegate = self

然后扩展viewController以符合ARSessionDelegate协议(可选协议)

// MARK: ARSessionDelegate
extension ViewController: ARSessionDelegate {

    func session(_ session: ARSession, didUpdate frame: ARFrame) {

        do {
            let prediction = try self.model.prediction(image: frame.capturedImage)
            DispatchQueue.main.async {
                if let prob = prediction.classLabelProbs[prediction.classLabel] {
                    self.textLabel.text = "\(prediction.classLabel) \(String(describing: prob))"
                }
            }
        }
        catch let error as NSError {
            print("Unexpected error ocurred: \(error.localizedDescription).")
        }
    }
}

起初我尝试了那个代码,但后来注意到《盗梦空间》需要一个Image类型的像素缓冲区&书信电报;RGB,<299299>。

https://github.com/yulingtianxia/Core-ML-Sample )

func resize(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
    let imageSide = 299
    var ciImage = CIImage(cvPixelBuffer: pixelBuffer, options: nil)
    let transform = CGAffineTransform(scaleX: CGFloat(imageSide) / CGFloat(CVPixelBufferGetWidth(pixelBuffer)), y: CGFloat(imageSide) / CGFloat(CVPixelBufferGetHeight(pixelBuffer)))
    ciImage = ciImage.transformed(by: transform).cropped(to: CGRect(x: 0, y: 0, width: imageSide, height: imageSide))
    let ciContext = CIContext()
    var resizeBuffer: CVPixelBuffer?
    CVPixelBufferCreate(kCFAllocatorDefault, imageSide, imageSide, CVPixelBufferGetPixelFormatType(pixelBuffer), nil, &resizeBuffer)
    ciContext.render(ciImage, to: resizeBuffer!)
    return resizeBuffer
}

不幸的是,这还不足以让它发挥作用。这就是捕捉到的错误:

Unexpected error ocurred: Input image feature image does not match model description.
2017-07-20 AR+MLPhotoDuplicatePrediction[928:298214] [core] 
    Error Domain=com.apple.CoreML Code=1 
    "Input image feature image does not match model description" 
    UserInfo={NSLocalizedDescription=Input image feature image does not match model description, 
    NSUnderlyingError=0x1c4a49fc0 {Error Domain=com.apple.CoreML Code=1 
    "Image is not expected type 32-BGRA or 32-ARGB, instead is Unsupported (875704422)" 
    UserInfo={NSLocalizedDescription=Image is not expected type 32-BGRA or 32-ARGB, instead is Unsupported (875704422)}}}

如果有更好的建议将两者结合起来,我洗耳恭听。

编辑 :我也试过了 resizePixelBuffer

编辑2 :因此,我将像素格式更改为kCVPixelFormatType_32BGRA(与在resizePixelBuffer中传递的pixelBuffer格式不同)。

let pixelFormat = kCVPixelFormatType_32BGRA // line 48

apple developers forum .

编辑3 :所以我尝试实施rickster解决方案。与inceptionV3配合得很好。我想尝试一个功能观察(VNClassificationObservation)。此时,它无法使用 TinyYolo .边界错误。试着弄明白。

1 回复 | 直到 7 年前

rickster 7 年前

不要自己处理图像以将其提供给核心ML。使用 Vision This one )Vision采用ML模型和几种图像类型中的任何一种( including CVPixelBuffer )并自动将图像获取到合适的大小、纵横比和像素格式,以便模型进行评估,然后给出模型的结果。

var request: VNRequest

func setup() {
    let model = try VNCoreMLModel(for: MyCoreMLGeneratedModelClass().model)
    request = VNCoreMLRequest(model: model, completionHandler: myResultsMethod)
}

func classifyARFrame() {
    let handler = VNImageRequestHandler(cvPixelBuffer: session.currentFrame.capturedImage,
        orientation: .up) // fix based on your UI orientation
    handler.perform([request])
}

func myResultsMethod(request: VNRequest, error: Error?) {
    guard let results = request.results as? [VNClassificationObservation]
        else { fatalError("huh") }
    for classification in results {
        print(classification.identifier, // the scene label
              classification.confidence)
    }
}

看见 this answer