In VideoEffect, can't get pixelBuffer from CIImage.

[jexwang]

In the VideoTrackScreenObject makeImage(_ renderer:) function, I noticed that you applied transformed(by:) to resize the image for performance optimization, but this causes the CIImage to lose its pixelBuffer data.
Some beauty SDKs require a pixelBuffer to render frames, so I'm wondering if it's possible to preserve the pixelBuffer inside the CIImage?
Here's a related discussion for reference:
https://stackoverflow.com/questions/72425288/ciimage-cvpixelbuffer-is-nil-after-scaling-down

[cookesan]

VideoEffect is intentionally a Core Image stage, so it receives a CIImage, not the original CVPixelBuffer.

The render path creates a CIImage from the incoming pixel buffer, applies transforms/effects as CIImage operations, and later renders the final image back into a pixel buffer for output. By the time your VideoEffect runs, the API contract is the image pipeline, not the capture buffer.

So the usual fix is to implement the effect as a Core Image operation or filter. If you need direct CVPixelBuffer access for something that cannot be expressed as a CIImage transform, that belongs earlier/lower in the capture/render pipeline rather than inside VideoEffect.

[jexwang]

Hi,

Are there any sample projects I can reference? I couldn't seem to find a relevant demo in the example projects.

In an older version, I was passing a pixelBuffer to another SDK to apply face adjustments on the user

override func execute(_ image: CIImage, info: CMSampleBuffer?) -> CIImage {
    guard let sampleBuffer = info,
          let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
    else { return image }

    let processedPixelBuffer = effectProvier.processPixelBuffer(pixelBuffer)
    return CIImage(cvPixelBuffer: processedPixelBuffer)
}

but I'm not sure if this approach is still correct in the current version. If it's no longer the right way to do it, I honestly have no idea where to start with the changes.

Any guidance would be greatly appreciated. Thank you so much!

[cookesan]

I do not know of a sample project that does exactly this.

Based on the current API shape, VideoEffect is a CIImage effect stage. The older CMSampleBuffer / CVPixelBuffer approach only fits if you are inside a lower-level capture/render hook that still has the sample buffer. For a face-adjustment SDK that requires CVPixelBuffer, I would look for an interception point before the CIImage effect stage, or convert the processed pixel buffer back into a CIImage; I would not expect VideoEffect itself to be the direct pixel-buffer integration point.