[Public WebGL] OES_EGL_image_external Extension proposal

Mark Callow [email protected]
Tue Jan 10 02:05:32 PST 2017

> On Jan 10, 2017, at 11:54, Byungseon Shin <[email protected]> wrote:
> On Tue, Jan 10, 2017 at 10:27 AM Mark Callow <[email protected] <mailto:[email protected]>> wrote:
> Thanks for the update. Unfortunately it does nothing to settle my previous concerns.
> On the native side EGL_image and its companion OES_EGL_image_external are designed only for static images such as something drawn with OpenVG.
> I could not find such a restriction. Please see the Android Video Streaming use cases.
> <https://developer.android.com/reference/android/graphics/SurfaceTexture.html <https://developer.android.com/reference/android/graphics/SurfaceTexture.html>>

I see the disconnect. My comments have been referring to the native OES_EGL_image_external OpenGL ES extension. The API added by that extension only supports binding an EGLImage to a texture. An EGLImage is a single image.

You are referring to the underlying external texture object. Yes an external texture as defined by OES_EGL_image_external can be the consumer endpoint for either an EGLImage or an EGLStream or indeed Android’s SurfaceTexture. But different API functions are used to hook up EGLStreams or SufaceTextures.

An external texture maps only a single image. It is really important to be clear about this. SurfaceTexture is a stream, almost certainly implemented, in most cases, using EGLStream. It says

When updateTexImage() <https://developer.android.com/reference/android/graphics/SurfaceTexture.html#updateTexImage%28%29> is called, the contents of the texture object specified when the SurfaceTexture was created are updated to contain the most recent image from the image stream. This may cause some frames of the stream to be skipped.

updateTexImage() is to be called by the app every frame.

Your WebGL version of OES_EGL_image_external modifies EGLImageTargetTexture2DOES to accept an HTMLVideoElement instead of an EGLImage. Thus you are, confusingly, changing a call that in the native extension took a single image, into one that takes a stream of images. You are stepping into the world of EGLStream but avoiding its API.

The example code shows EGLImageTargetTexture2DOES being called just once at the start of the program. This is equivalent to calling SurfaceTexture.attachToGLContext. There is no step to latch a video frame when drawing a WebGL frame, no equivalent to updateTexImage. It just does WebGL rendering every frame.

So you are relying on unspecified magic to ensure that the mapped frame is never being updated while the WebGL application is sampling it. You are also concealing from the app which video frame it will be sampling.

I maintain that you need to stop relying on magic and hiding what’s happening. Internally here has to be a buffered stream to ensure there is always a complete frame when the WebGL app is sampling thus avoiding tearing. (The alternative is locking on a single image buffer which is likely to stall both the video decoder and the graphics pipeline causing horrendous performance.) The buffering should be exposed and put under the control of the application.

The extension needs updateTexImage-like functionality. Once you have that then your new WebGLVideoFrameInfo can return information about the currently mapped frame.

If you are trying to mimic SurfaceTexture then you may also need to handle cases where the video size or orientation can change between frames. This is handled in SurfaceTexture by requiring the app to query the texture transform to be used with the texture. I don’t know if such changes can occur in HTMLVideoElements.

> EGL_KHR_stream (and family) + EGL_KHR_stream_consumer_gltexture are designed for dynamic images, i.e. sequences of image frames, . The stream provides buffering necessary to accommodate different frame rates of the video decoder and the application’s GL rendering and to avoid having to lock either to ensure half-completed video frames are not rendered.
> Again, not all embedded GPU vendors support EGL_KHR_Stream.

Do they all support Android implementing SurfaceTexture on them? If so they support something very like EGLStream.

One more thing. To support this on WebGL 2 (ESSL 3.x) you will need a WebGL wrapper for OES_EGL_image_external_essl3 <https://www.khronos.org/registry/gles/extensions/OES/OES_EGL_image_external_essl3.txt> as well.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://khronos.org/pipermail/public_webgl_khronos.org/attachments/20170110/2fa89319/attachment.html>

More information about the public_webgl mailing list