[Public WebGL] WEBGL_texture_from_depth_video extension proposal

Kenneth Russell [email protected]
Mon Nov 10 11:43:51 PST 2014

On Thu, Nov 6, 2014 at 2:29 AM, Florian Bösch <[email protected]> wrote:
> On Thu, Nov 6, 2014 at 9:08 AM, Kenneth Russell <[email protected]> wrote:
>> The reason to expose an extension from my point of view is (a) to allow
>> detection of support for uploading depth videos to WebGL and (b) to provide
>> a single, efficient upload path that users can rely on.
> a) Is support depending on anything else than the browser implementing it?

The browser will need to add support both for depth streams and the
WebGL integration. These two parts are likely to land at different
times. I think it's important to send a clear message to developers
when all of the functionality is available.

> b) Is that goal best served with an extension?

It's the only way I can think of to support detection of this feature.
If there's a way to do so in the Media Capture API that would be fine
too. Suggestions welcome.

> Since depth texture streaming interacts with various other specifications
> (canvas, typed arrays and webgl 1, 2, 2.1, 3, depth textures, floating point
> textures and <video>), it would seem to me the most consistent and easiest
> to use entry point for any discovering support and consistently supporting a
> feature would be the media capture depth specification, rather than modify
> half a dozen other specifications and add pieces to it.

I agree in general. If it's possible to incorporate this sort of
feature detection into the Media Capture spec that would be fine.
Perhaps Rob, Ningxin or someone else from that group can comment.

>>> 3) The style of how to support the upload isn't very friendly to users
>>> because it requires manual conversion in the fragment shader. I'd much
>>> rather see float/half-float texture formats being satisfied, where the
>>> browser knows about the surface format of the depth texture, and converts it
>>> (on GPU, without blocking operation) to the specified floating point format
>>> for upload. This'd have the advantage of offering users a transparent
>>> support for depth images, regardless of their underlying format.
>> Yes, with WebGL 1.0 the conversion is a little painful. This will become
>> trivial with WebGL 2.0 and the availability of R16UI / R16I textures. I'd
>> like to focus on getting WebGL 2.0 out the door, but we'd appreciate any
>> concrete suggestions you have for the Media Capture folks about better data
>> upload formats to use with WebGL 1.0.
>> -Ken
> There's some concern for me for backwards/forwards compatibility and
> extension proliferation. The current extension specification defines as only
> supported format upload format UNSIGNED_SHORT_5_6_5.
> What if...
> A new device appears that delivers depth fields as UNSIGNED_BYTE -->
> WEBGL_texture_from_depth_video_unsigned_byte?
> A new device appears that delivers depth fields as UNSIGNED_INT ->
> WEBGL_texture_from_depth_video_unsigned_integer?
> A new device appears that delivers depth fields as Float ->
> WEBGL_texture_from_depth_video_float32?
> A new texture format should be supported (float) ->
> WEBGL_texture_from_depth_video_to_float?
> A new texture format should be supported (depth_component) ->
> WEBGL_texture_from_depth_video_depth_component?
> WebGL 2.0's texture formats should be supported ->
> WEBGL_texture_from_depth_video2?
> WebGL 2.1's texture formats should be supported ->
> WEBGL_texture_from_depth_video2_1?
> etc.
> I'd like depth upload to work for the following WebGL 1 formats:
> gl.RGB, gl.UNSIGNED_BYTE -> greyscale 0 - 255
> gl.RGBA, gl.UNSIGNED_BYTE -> greyscale 0 - 255, alpha 1
> gl.RGBA, gl.FLOAT -> greyscale 0 - 1 (32-bit float), alpha 1
> gl.RGB, gl.FLOAT -> greyscale 0 - 1 (32-bit float)
> gl.LUMINANCE, gl.UNSIGNED_BYTE -> 0 - 255
> gl.LUMINANCE, gl.FLOAT -> 0-1
> gl.RGBA, ext.HALF_FLOAT_OES -> 0-1 (16-bit float), alpha 1
> gl.RGB, ext.HALF_FLOAT_OES -> 0-1 (16-bit float)
> ext.DEPTH_COMPONENT, gl.UNSIGNED_INT -> 0-2^32-1
> ext.DEPTH_COMPONENT, ext.UNSIGNED_INT_24_8_WEBGL -> 0-2^24-1, stencil 0
> I'd also like to see depth uploads work for the about 30 internal formats
> that is has, including things that where in extensions in WebGL 1 and are
> now core in WebGL 2.
> I'd also like to see depth uploads work with any of the changed texture
> formats coming to WebGL 2.1 and WebGL 3.0.
> And I'd like this to work without having to introduce 3 dozen new extensions
> to make it happen in the years to come...

It'll only be efficient to upload depth videos to WebGL textures using
the internal format which avoids converting the depth values during
the upload process. That's why UNSIGNED_SHORT_5_6_5 was chosen as the
single supported format for uploading this content to WebGL 1.0. It's
not desirable for either the browser implementer or the web developer
to support uploading depth videos to lots of random texture formats if
they won't be efficient. The Media Capture group should comment on
what formats depth cameras tend to output, and are likely to output in
the future.


You are currently subscribed to [email protected]
To unsubscribe, send an email to [email protected] with
the following command in the body of your email:
unsubscribe public_webgl

More information about the public_webgl mailing list