[Public WebGL] OES_EGL_image_external Extension proposal

Byungseon Shin [email protected]
Tue Jan 10 09:16:10 PST 2017


Hi Mark,

Thanks for the in-depth feedback.
I agree your opinions and have changed the logic by your guide.

I have created a new patch and please review it again.
https://github.com/KhronosGroup/WebGL/pull/2243/commits/c8193b5f1ffb37881bb815bd98b4da91e516828d

About the OES_EGL_image_external_essl3, we will follow up soon.

Kind regards,
Byungseon Shin

On Tue, Jan 10, 2017 at 7:05 PM Mark Callow <[email protected]> wrote:

> On Jan 10, 2017, at 11:54, Byungseon Shin <[email protected]> wrote:
>
> On Tue, Jan 10, 2017 at 10:27 AM Mark Callow <[email protected]> wrote:
>
>
> Thanks for the update. Unfortunately it does nothing to settle my previous
> concerns.
>
> On the native side EGL_image and its companion OES_EGL_image_external are
> designed only for static images such as something drawn with OpenVG.
>
>
> I could not find such a restriction. Please see the Android Video
> Streaming use cases.
> <
> https://developer.android.com/reference/android/graphics/SurfaceTexture.html
> >
>
>
> I see the disconnect. My comments have been referring to the native
> OES_EGL_image_external OpenGL ES extension. The API added by that
> extension only supports binding an EGLImage to a texture. An EGLImage is a
> single image.
>
> You are referring to the underlying external texture object. Yes an
> external texture as defined by OES_EGL_image_external can be the consumer
> endpoint for either an EGLImage or an EGLStream or indeed Android’s
> SurfaceTexture. But different API functions are used to hook up EGLStreams
> or SufaceTextures.
>
> An external texture maps only a single image. It is really important to be
> clear about this. SurfaceTexture is a stream, almost certainly
> implemented, in most cases, using EGLStream. It says
>
> When updateTexImage()
> <https://developer.android.com/reference/android/graphics/SurfaceTexture.html#updateTexImage%28%29>
> is called, the contents of the texture object specified when the
> SurfaceTexture was created are updated to contain the most recent image
> from the image stream. This may cause some frames of the stream to be
> skipped.
>
>
> updateTexImage() is to be called by the app every frame.
>
> Your WebGL version of OES_EGL_image_external modifies EGLImageTargetTexture2DOES
> to accept an HTMLVideoElement instead of an EGLImage. Thus you are,
> confusingly, changing a call that in the native extension took a single
> image, into one that takes a stream of images. You are stepping into the
> world of EGLStream but avoiding its API.
>
> The example code shows EGLImageTargetTexture2DOES being called just once
> at the start of the program. This is equivalent to calling
> SurfaceTexture.attachToGLContext. There is no step to latch a video frame
> when drawing a WebGL frame, no equivalent to updateTexImage. It just does
> WebGL rendering every frame.
>
> So you are relying on unspecified magic to ensure that the mapped frame is
> never being updated while the WebGL application is sampling it. You are
> also concealing from the app which video frame it will be sampling.
>
> I maintain that you need to stop relying on magic and hiding what’s
> happening. Internally here has to be a buffered stream to ensure there is
> always a complete frame when the WebGL app is sampling thus avoiding
> tearing. (The alternative is locking on a single image buffer which is
> likely to stall both the video decoder and the graphics pipeline causing
> horrendous performance.) The buffering should be exposed and put under the
> control of the application.
>
> The extension needs updateTexImage-like functionality. Once you have that
> then your new WebGLVideoFrameInfo can return information about the
> currently mapped frame.
>
> If you are trying to mimic SurfaceTexture then you may also need to handle
> cases where the video size or orientation can change between frames. This
> is handled in SurfaceTexture by requiring the app to query the texture
> transform to be used with the texture. I don’t know if such changes can
> occur in HTMLVideoElements.
>
>
> EGL_KHR_stream (and family) + EGL_KHR_stream_consumer_gltexture are
> designed for dynamic images, i.e. sequences of image frames, . The stream
> provides buffering necessary to accommodate different frame rates of the
> video decoder and the application’s GL rendering and to avoid having to
> lock either to ensure half-completed video frames are not rendered.
>
>
> Again, not all embedded GPU vendors support EGL_KHR_Stream.
>
>
> Do they all support Android implementing SurfaceTexture on them? If so
> they support something very like EGLStream.
>
>
> One more thing. To support this on WebGL 2 (ESSL 3.x) you will need a
> WebGL wrapper for OES_EGL_image_external_essl3
> <https://www.khronos.org/registry/gles/extensions/OES/OES_EGL_image_external_essl3.txt> as
> well.
>
> Regards
>
>
>     -Mark
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://khronos.org/pipermail/public_webgl_khronos.org/attachments/20170110/7e264dc1/attachment.html>


More information about the public_webgl mailing list