<div dir="ltr">Hi Mark,<div><br></div><div>Thanks for the in-depth feedback.</div><div>I agree your opinions and have changed the logic by your guide.</div><div><br></div><div>I have created a new patch and please review it again.</div><div><a href="https://github.com/KhronosGroup/WebGL/pull/2243/commits/c8193b5f1ffb37881bb815bd98b4da91e516828d">https://github.com/KhronosGroup/WebGL/pull/2243/commits/c8193b5f1ffb37881bb815bd98b4da91e516828d</a><br></div><div><br></div><div>About the OES_EGL_image_external_essl3, we will follow up soon.</div><div><br></div><div>Kind regards,</div><div>Byungseon Shin</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jan 10, 2017 at 7:05 PM Mark Callow <<a href="mailto:khronos@callow.im">khronos@callow.im</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg">On Jan 10, 2017, at 11:54, Byungseon Shin <<a href="mailto:sun.shin@webkit.org" class="gmail_msg" target="_blank">sun.shin@webkit.org</a>> wrote:</div></blockquote></div></div><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Tue, Jan 10, 2017 at 10:27 AM Mark Callow <<a href="mailto:khronos@callow.im" class="gmail_msg" target="_blank">khronos@callow.im</a>> wrote:<br class="gmail_msg"></div></div></div></div></blockquote></div></div><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="gmail_msg" style="word-wrap:break-word"><div class="gmail_msg"><br class="gmail_msg"></div></div><div class="gmail_msg" style="word-wrap:break-word"><div class="gmail_msg">Thanks for the update. Unfortunately it does nothing to settle my previous concerns.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">On the native side EGL_image and its companion OES_EGL_image_external are designed only for static images such as something drawn with OpenVG.</div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I could not find such a restriction. Please see the Android Video Streaming use cases.</div><div class="gmail_msg"><<a href="https://developer.android.com/reference/android/graphics/SurfaceTexture.html" class="gmail_msg" target="_blank">https://developer.android.com/reference/android/graphics/SurfaceTexture.html</a>></div></div></div></div></blockquote></div></div><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><div class="gmail_quote gmail_msg"></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div>I see the disconnect. <span style="font-family:HelveticaNeue" class="gmail_msg">My comments have been referring to the native </span><span style="font-family:HelveticaNeue" class="gmail_msg">OES_EGL_image_external </span><span style="font-family:HelveticaNeue" class="gmail_msg">OpenGL ES extension. The API added by that extension only supports binding an EGLImage to a texture. An EGLImage is a single image.</span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg">You are referring to the underlying external texture object. Yes a</span>n external texture as defined by <font face="HelveticaNeue" class="gmail_msg">OES_EGL_image_external can be the consumer endpoint for either an EGLImage or an EGLStream or indeed Android’s SurfaceTexture. But different API functions are used to hook up EGLStreams or SufaceTextures.</font></div><div class="gmail_msg"><font face="HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></font></div><div class="gmail_msg"><font face="HelveticaNeue" class="gmail_msg">An external texture maps only a single image. It is really important to be clear about this. </font><span style="font-family:HelveticaNeue" class="gmail_msg">SurfaceTexture is a stream, almost certainly implemented, in most cases, using EGLStream. It says</span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></span></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px" class="gmail_msg"><div class="gmail_msg">When <code class="gmail_msg"><a href="https://developer.android.com/reference/android/graphics/SurfaceTexture.html#updateTexImage%28%29" class="gmail_msg" target="_blank">updateTexImage()</a></code> is called, the contents of the texture object specified
 when the SurfaceTexture was created are updated to contain the most recent image from the image
 stream.  This may cause some frames of the stream to be skipped.</div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><font face="HelveticaNeue" class="gmail_msg">updateTexImage() is to be called by the app every frame.</font></div><div class="gmail_msg"><font face="HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></font></div><div class="gmail_msg"><font face="HelveticaNeue" class="gmail_msg">Your WebGL version of </font><span style="font-family:HelveticaNeue" class="gmail_msg">OES_EGL_image_external</span><span style="font-family:HelveticaNeue" class="gmail_msg"> modifies </span>EGLImageTargetTexture2DOES to accept <span style="font-family:HelveticaNeue" class="gmail_msg">an HTMLVideoElement instead of an EGLImage. Thus you are, confusingly, changing a call that in the native extension took a single image, into one that takes a stream of images. You are stepping into the world of EGLStream but avoiding its API.</span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg">The example code shows </span>EGLImageTargetTexture2DOES<span style="font-family:HelveticaNeue" class="gmail_msg"> being called just once at the start of the program. This</span><span style="font-family:HelveticaNeue" class="gmail_msg"> is equivalent to calling SurfaceTexture.attachToGLContext.</span><span style="font-family:HelveticaNeue" class="gmail_msg"> </span><span style="font-family:HelveticaNeue" class="gmail_msg">There is no step to latch a video frame when drawing a WebGL frame, no equivalent to updateTexImage. It just does WebGL rendering every frame.</span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg">So you are relying on unspecified magic to ensure that the mapped frame is never being updated while the WebGL application is sampling it. You are also concealing from the app which video frame it will be sampling.</span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><font face="HelveticaNeue" class="gmail_msg">I maintain that you need to stop relying on magic and hiding what’s happening. Internally here has to be a buffered stream to ensure there is always a complete frame when the WebGL app is sampling thus avoiding tearing. (The alternative is locking on a single image buffer which is likely to stall both the video decoder and the graphics pipeline causing horrendous performance.) The buffering should be exposed and put under the control of the application.</font></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg">The extension needs updateTexImage-like functionality. Once you have that then your new WebGLVideoFrameInfo</span><span style="font-family:HelveticaNeue" class="gmail_msg"> can return information about the currently mapped frame.</span></div><div class="gmail_msg"><span style="font-family:HelveticaNeue" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><font face="HelveticaNeue" class="gmail_msg">If you are trying to mimic SurfaceTexture then you may also need to handle cases where the video size or orientation can change between frames. This is handled in SurfaceTexture by requiring the app to query the texture transform to be used with the texture. I don’t know if such changes can occur in HTMLVideoElements.</font></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"></div></div><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="gmail_msg" style="word-wrap:break-word"><div class="gmail_msg">EGL_KHR_stream (and family) + EGL_KHR_stream_consumer_gltexture are designed for dynamic images, i.e. sequences of image frames, . The stream provides buffering necessary to accommodate different frame rates of the video decoder and the application’s GL rendering and to avoid having to lock either to ensure half-completed video frames are not rendered.</div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Again, not all embedded GPU vendors support EGL_KHR_Stream.</div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg">Do they all support Android implementing SurfaceTexture on them? If so they support something very like EGLStream.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">One more thing. To support this on WebGL 2 (ESSL 3.x) you will need a WebGL wrapper for <a href="https://www.khronos.org/registry/gles/extensions/OES/OES_EGL_image_external_essl3.txt" class="gmail_msg" target="_blank">OES_EGL_image_external_essl3</a> as well.</div><br class="gmail_msg"><div class="gmail_msg">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class="gmail_msg">Regards</div></div></div><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg">    -Mark</div></div></div></blockquote></div>