Skip to content

Drawing the Camera

This article directly follows on from Pipelines and Camera Processing, we highly recommend reading through that first.

The platform provides the most recent camera frame as a WebGL texture that you can draw to the screen. To make this texture available, call the following function after your frameUpdate() call:

pipeline.cameraFrameUploadGL();

Once uploaded, there are two ways to draw the camera:

  1. Using the cameraFrameDrawGL convenience function
  2. Manually drawing a full-screen quad with WebGL

You can use the following function:

pipeline.cameraFrameDrawGL(renderWidth: number, renderHeight: number, mirror?: boolean)

It will automatically draw the camera to the screen as a full screen quad. Please note this function modifies some GL state during its operation so you may need to reset the following GL state if you use it:

GL stateExample
The currently bound texture 2D is set to nullgl.bindTexture(gl.TEXTURE_2D, null)
The currently bound array buffer is set to nullgl.bindBuffer(gl.ARRAY_BUFFER, null);
The currently bound program is set to nullgl.useProgram(null)
The currently active texture is set to gl.TEXTURE0gl.activeTexture(gl.TEXTURE0)
These features are disabled:gl.SCISSOR_TEST, gl.DEPTH_TEST, gl.BLEND, gl.CULL_FACE

Alternatively, you can draw the camera manually to a full screen quad. To do so, you can use the following functions:

pipeline.cameraFrameTextureGL() : WebGLTexture | undefined: returns a WebGLTexture containing the current camera image (or undefined if none are available) pipeline.

cameraFrameTextureMatrix(renderWidth : number, renderHeight : number, mirror?: boolean) : Float32Array: pass in your canvas’ width and height and it returns a 4x4 column-major matrix that you can use to transform the UV coordinates of the following quad:

QuadUV Coordinates
Vertex 0-1, -1, 0
UV 00, 0
Vertex 1-1, 1, 0
UV 10, 1
Vertex 21, -1, 0
UV 11, 0
Vertex 31, 1, 0
UV 11, 1

Here’s an example vertex shader to show how this can be accomplished:

attribute vec4 position; // Bound to a buffer with the vertex data above
attribute vec4 texCoord; // Bound to a buffer with the UV data above
uniform mat4 texTransform; // Set to the matrix returned by cameraFrameTextureMatrix(...)
varying vec4 texVarying; // Used to pass the UV coordinates to the fragment shader
void main()
{
gl_Position = position;
texVarying = texTransform * texCoord;
}

And the corresponding fragment shader:

varying vec4 texVarying; // The UV coordinate from the vertex shader
uniform sampler2D texSampler; // Bound to the texture returned by cameraFrameTextureGL()
void main()
{
gl_FragColor = texture2DProj(texSampler, texVarying);
}