Pipelines and Camera Processing
In the Zappar package, a pipeline is used to manage the flow of data coming in (i.e. the frames) through to the output from the different tracking types and computer vision algorithms. While most projects will only need one pipeline, it is possible to create as many as you like. Each pipeline can only have one active source of frames (i.e. one camera, or one video), so if you'd like to simultaneously process frames from multiple sources then you'll need a pipeline for each.
We have provided a default
ZapparCamera implementation that acts as a wrapper around a single pipeline. This camera manages a list of listeners (tracking targets) and ensures that both the Zappar library and its own pipeline are initialized before tracking targets themselves initialize.
Constructing and Managing Pipelines
To create and initialize a pipeline you should call the following function:
IntPtr pipeline = Z.PipelineCreate();
Constructing the Camera
Zappar cameras act as inputs to a particular pipeline. To create a camera source, call the following function:
bool useFrontFacingCamera = false; string device = Z.CameraDefaultDeviceId(useFrontFacingCamera); IntPtr camera = Z.CameraSourceCreate(pipeline, device);
If you'd like to start the user-facing 'selfie' camera, set
useFrontFacingCamera = true.
The library needs to ask the user for permission to access the camera and motion sensors on the device.
To do this, you can use the following function to show a built-in UI informing the user of the need and providing a button to trigger the underlying platform's permission prompts.
To request permissions using a default UI you should call:
This will trigger the native permissions prompt on both iOS and Android, and will display a default UI on WebGL. Calling
bool permissionGranted = Z.PermissionGrantedAll();
will return true once the user has granted the relevant permissions.
Starting the Camera
Once the user has granted the necessary permissions, you can start the camera on the device with the following functions:
Note that if you are not using OpenGL as your rendering engine then
Z.PipelineGLContextSet(...) will result in a no-op and can be omitted.
To switch between the front- and rear-facing cameras during your experience, just create a camera source for each and call
Z.CameraSourceStart(...) as appropriate.
Processing Camera Frames
Call the following functions to populate texture data in the render loop of a Zappar camera:
Z.Process(pipeline); Z.PipelineFrameUpdate(pipeline); Z.CameraFrameUpload(pipeline);
This will upload a frame for rendering, and will update the internal state of any associated tracking targets.
In the render loop, be sure to update both the projection and texture matrices for the camera:
int width = Screen.width; int height = Screen.height; Matrix4x4 projection = Z.PipelineProjectionMatrix(pipeline, width, height); Matrix4x4 textureMatrix = Z.PipelineCameraFrameTextureMatrix(pipeline, width, height, useFrontFacingCamera); GetComponent<Camera>().projectionMatrix = projection; material.SetMatrix("_nativeTextureMatrix", textureMatrix);
CameraMaterial material object is provided in the Assets/Materials folder that can be assigned to the
material object referenced above. This material must use a custom shader of type
Unlit/CameraBackgroundShader, an implementation of which is available in the 'Assets/Shaders' folder.
The current frame texture can then be attached to the material by calling:
Texture2D texture = Z.PipelineCameraFrameTexture( pipeline ); if (texture != null) material.mainTexture = texture;
Enabling Realtime Reflection
The package provides an automatically generated environment map that's useful if you're using materials that support reflections . The map uses the camera feed to create an approximate environment that can add some realism to your scene.
To enable the realtime reflection for your reflective materials, follow the following process:
- Make sure realtime reflection is enabled from Unity project settings.
Realtime Reflection Probes.
Realtime Refectionsfrom menu
Zappar/Editor/OpenUARSettings. This will add a new layer
ZapparReflectto your project.
- Right-click on
ZapparCameraRear(Camera object containing
ZapparCamerascript) and select
- Adjust the
ZapparReflectionProbeproperties according to your project needs.