[Public WebGL] OES_EGL_image_external Extension proposal

Florian Bösch [email protected]
Fri Jan 6 13:56:12 PST 2017


Would it be possible to inform the decoder of a desired time so it can pick
the best matching frame?

Although we don't have control over when a frame is going to be presented,
we have a pretty good idea of when it is going to be on screen (assuming we
make the vsync), and from that we could relatively easily derive a "I'd
like a video frame closest to that time".

On Fri, Jan 6, 2017 at 9:10 PM, Byungseon Shin <[email protected]> wrote:

> First of all, I missed to reply-all, so I am adding public-webgl alias
> again.
>
> Second, I agree with your point. I just was trying to explain that
> application can control the rendering logic.
>
> We can propose alternative approach by making explicit calling of
> EGLImageTargetTexture2DOES.
> Then, application can decide if new frame will be used or not.
>
> var update = function(){
>   updateTexture(); // ext.EGLImageTargetTexture2DOES
>   render();
>   requestAnimationFrame(update);
> }
> requestAnimationFrame(update);
>
>
> On Sat, Jan 7, 2017 at 4:46 AM Florian Bösch <[email protected]> wrote:
>
>> That's not how WebGL works, this is:
>>
>> var update = function(){
>>   render();
>>   requestAnimationFrame(update);
>> }
>> requestAnimationFrame(update);
>>
>> If you block in the JS thread on purpose you achieve nothing because no
>> matter how long you block, the frame is only going to be presented at the
>> next animationFrame composit, which is synced to vsync (or g-sync, or
>> direct-present sync, etc.)
>>
>> You do run a risk though of missing the sync, in which case you drop a
>> frame, which is bad (jitter/glitching/nausea (in HMD users) etc.).
>>
>> On Fri, Jan 6, 2017 at 8:43 PM, Byungseon Shin <[email protected]>
>> wrote:
>>
>> As in the proposal, application can skip calling of render function which
>> will call swapBuffers as following:
>> ----------------------------------------------------
>> update();
>> while (true) {
>>    ...
>>     // check the gap between frame time of WebGL and Video
>>     if ( gap < allowable_delay)
>>         render()
>> }
>>
>> On Sat, Jan 7, 2017 at 4:36 AM Florian Bösch <[email protected]> wrote:
>>
>> Sure, but WebGL applications have no control over and can't influence
>> (beyond not being late) when they're presented on screen. They do not
>> control the swapBuffers command.
>>
>> On Fri, Jan 6, 2017 at 8:32 PM, Byungseon Shin <[email protected]>
>> wrote:
>>
>> Hi Florian,
>>
>> To control the sync, I think, application can use the gap between Video
>> frame time and WebGL frame time.
>>
>> Kind regards,
>> Byungseon Shin
>>
>> On Sat, Jan 7, 2017 at 4:14 AM Florian Bösch <[email protected]> wrote:
>>
>> Unless I'm mistaken, knowing the time alone isn't going to help that much.
>>
>> The dynamic_texture extension attempted to solve this problem with
>> setting a presentation time for a frame. I'm not sure how realistic it is
>> to try to time presentation time, so that might have been overly ambitious.
>>
>> On Fri, Jan 6, 2017 at 8:01 PM, Byungseon Shin <[email protected]>
>> wrote:
>>
>> Hi WebGL working group members,
>>
>> We have updated proposal by adding following function :
>> - Provides time of frame, texture width and height of HTMLVideoElement's
>> EGLImage.
>>
>> Please take a look at the pull request:
>> <https://github.com/KhronosGroup/WebGL/pull/2243>
>>
>> Kind regards,
>> Byungseon Shin
>>
>>
>>
>> On Tue, Jan 3, 2017 at 8:48 AM Byungseon Shin <[email protected]>
>> wrote:
>>
>> Hi Florian,
>>
>> We are working on update which will cover dynamic_texture concerns.
>> I plan to upload new version on github for reviewing by this week.
>>
>> Happy 2017 year!
>> Byungseon Shin
>>
>> On Tue, Jan 3, 2017 at 1:58 AM Florian Bösch <[email protected]> wrote:
>>
>> What is the status of the https://www.khronos.org/
>> registry/webgl/extensions/proposals/OES_EGL_image_external/ extension?
>> Is work on the spec still going on or is this the final version that takes
>> into account dynamic_texture concerns?
>>
>> On Wed, Oct 12, 2016 at 3:59 AM, Byungseon Shin <[email protected]>
>> wrote:
>>
>> Hi Ken,
>>
>> Thanks for the feedback.
>> Of course, we are willing to devote time and eager to enhance WebGL video
>> rendering performance.
>> We prefer to use our proposal as a base but we could revise
>> dynamic_texture as well.
>>
>> Kind regards,
>> Byungseon Shin
>>
>>
>> On Wed, Oct 12, 2016 at 8:48 AM Kenneth Russell <[email protected]> wrote:
>>
>> Hi Byungseon,
>>
>> Thanks for putting together this extension proposal.
>>
>> In conversations with groups within Google that are trying to do 360
>> degree video rendering in WebGL, they need more information than your
>> extension would provide; specifically, they would need the exact timestamp
>> of the frame currently being rendered, as well as other per-frame metadata
>> like the current width and height of the texture (for variable bitrate
>> video streams).
>>
>> Mark Callow's WEBGL_dynamic_texture extension proposal
>> https://www.khronos.org/registry/webgl/extensions/
>> proposals/WEBGL_dynamic_texture/ provides the controls needed. However,
>> per working group discussions, the current extension's a little too
>> complicated. I think if that extension were simplified a little bit that it
>> would provide all of the performance benefits yours offers, and the
>> controls that are known to be necessary.
>>
>> Do you think you'd be willing to devote some time to either extending
>> your proposal (I can provide specific feedback) or editing down Mark's
>> proposal to handle these use cases? I'd prefer to edit down
>> WEBGL_dynamic_texture, if you're willing.
>>
>> Thanks,
>>
>> -Ken
>>
>>
>>
>> On Tue, Oct 11, 2016 at 8:27 AM, Byungseon Shin <[email protected]>
>> wrote:
>>
>> Hi Corentin,
>>
>> On Tue, Oct 11, 2016 at 11:16 PM Corentin Wallez <[email protected]>
>> wrote:
>>
>> The problem with using an EGL_image type of extension is that the texture
>> data store becomes shared between the new texture and the object it was
>> created from. This is great when the application controls both sides and
>> can ensure read and writes are probably synchronized, but in the case of
>> video here, one side would write without the application being able to
>> control synchronization.
>>
>> The standard way to do this type of video / texture binding in EGL is to
>> use EGL_KHR_stream
>> <https://www.khronos.org/registry/egl/extensions/KHR/EGL_KHR_stream.txt> which
>> is a much more complex and might not be an improvement over what developers
>> can do with the current APIs.
>>
>>
>> As in proposal revision#3 and as Florian already mentioned,
>> Current proposal support "bind once and update implicitly" concept :  "
>> ext.EGLImageTargetTexture2DOES(ext.TEXTURE_EXTERNAL_OES, videoElement);
>> that the texture is "bound" to the video and no further calls need to be
>> emitted (as would presently be the case)"
>>
>> In a that sense, we could abstract the details implementation into
>> MediaPlayer Side like synchronizing issues.
>>
>> Even EGL_KHR_stream provides more details synchronization issues but
>> still lot's of GPU driver and Video Decoder need to support the latest
>> specification.
>> But OES_EGL_image_external extension is already supported by most of the
>> GPU drivers and Video decoders.
>>
>> And one of the performance bottleneck is to converting EGLImage input to
>> TEXTURE_2D to handled by WebGL application.
>>
>>
>> On Tue, Oct 11, 2016 at 6:54 AM, Byungseon Shin <[email protected]>
>> wrote:
>>
>> Hi Florian, Maksims,
>>
>> Thanks for the feedback.
>>
>> I have updated proposal to explain how application works.
>> Please see the attached updated proposal.
>>
>> To render an update frame, we need to call ext.EGLImageTargetTexture2DOES
>> to bind newly generated texture buffer from video decoder.
>>
>> By supporting OES_EGL_image_external extension, application can use
>> direct texture when video driver provides EGLImage like OpenGL ES natvie
>> applications.
>>
>> Our proposal is just focusing on extending format of texture compatible
>> with OpenGL ES extension and does not conflict with previous proposals.
>>
>> So, we provides a simplest way to adopt OES_EGL_image_external extension.
>>
>> By using references implementations of browser, WebGL renders video more
>> than 10 times faster than TEXTURE_2D format with Full HD resolution output.
>>
>> Kind regards,
>> Byungseon Shin
>>
>> On Tue, Oct 11, 2016 at 4:51 PM Maksims Mihejevs <[email protected]>
>> wrote:
>>
>> Worth mentioning that in web there are multiple sources that have
>> independent redraw mechanics, that includes video as well as canvas
>> elements. It might make sense to have single extension for providing direct
>> access to those sources without need to re-upload it.
>>
>> I have assumptions that internally in browsers, both video and canvas
>> have their buffers, so it might lead to very similar implementation as from
>> internal side as well as from webgl api side.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://khronos.org/pipermail/public_webgl_khronos.org/attachments/20170106/16e58d37/attachment.html>


More information about the public_webgl mailing list