Unity Google VR Video Async Reprojection

Qué es Video Async Reprojection?

Async Reprojection Video is a layer (referred to as an “external surface”) that an application can use to feed video frames directly into the async reprojection system. The main advantages to using the API are:

1. Sin la API, el video se muestrea una vez para convertirlo en el búfer de color de la aplicación, luego se vuelve a muestrear el búfer de color para realizar la corrección de la distorsión. Esto introduce artefactos de doble muestreo. La superficie externa pasa el video directamente al compositor EDS, por lo que solo se muestrea una vez, lo que mejora la calidad visual del video.

2. Al usar la API de superficie externa, la velocidad de frames de video está desacoplada de la velocidad de frames de la aplicación. La aplicación puede tomar 1 segundo para renderizar un nuevo frame y el único resultado es que el usuario verá barras negras cuando mueve su cabeza, el video seguirá reproduciéndose normalmente. Esto debería reducir significativamente los fotogramas de video caídos y mantener la sincronización AV

3. La aplicación puede marcar que quiere reproducir video DRM, y la API creará una ruta protegida que muestra videos protegidos y mantiene la velocidad de fotogramas de Reproyección Asíncrona.

Problemas conocidos:

1. Cuando se usa la reproyección de Video Async, la cámara debe comenzar en el origen (0,0,0). Pueden ocurrir errores si la posición de la cámara no está ajustada a 0,0,0.

2. No hay una interfaz de C # públicamente accesible para la reproyección Async. La API pública es solo Java.

Habilitar la reproyección de video asíncrono

Async Video Reprojection forma parte de la configuración del dispositivo Daydream VR.

Click the grey arrow next to Daydream and then check the “Enable Video Surface” box to enable use of the Async Video Reprojection feature

Select the “Use Protected Memory”option ONLY if you require memory protection for all of your content as enabling this means that it is enabled for the lifetime of the application.

Documentación de la API

To take advantage of the Google VR API you will need to extend the UnityPlayerActivity. For more information, see documentation on Extending the UnityPlayerActivity

Because Java plug-ins cannot directly access Objects in your scene, you will need to provide a simple API to your C# code that will allow you to pass a transform to the Java side as well as to tell your Java code when to start rendering.

Note: This code is not complete. It contains no implementation of a video player as that is a client specific implementation detail. It also doesn’t have any playback controls, which would have to be implemented as objects in the scene and actions on those objects would need to call into Java.

Para obtener información sobre el uso de Java en Unity y la extensión de UnityPlayerActivity, consulte la documentación sobre Desarrollo de Android en Unity.

For information about the Google Video Async Reprojection system, refer to the Android Developer Network documentation on Video Viewports.

Java sample code:

package com.unity3d.samplevideoplayer;

import com.unity3d.player.GoogleVrVideo;

import com.unity3d.player.GoogleVrApi;

import android.app.Activity;

import android.os.Bundle;

import android.util.Log;

import android.view.Surface;

public class GoogleAVRPlayer implements GoogleVrVideo.GoogleVrVideoCallbacks {

    private static final String TAG = GoogleAVRPlayer.class.getSimpleName();

    private MyOwnVideoPlayer videoPlayer;

    private boolean canPlayVideo = false;

    private boolean isSceneLoaded = false;

    // API you present to your C# code to handle initialization of your

    // video system.

    public void initVideoPlayer(UnityPlayerActivity activity) {

     // Initialize Video player and any other support you need…

     // Register this instance as the Google Vr Video Listener to get

     // lifetime and control callbacks.

         GoogleVrVideo gvrv = GoogleVrApi.getGoogleVrVideo();

         if (gvrv != null) gvrv.registerGoogleVrVideoListener(this); 

    }

 

    // API you present to your C# code to start your video system

    // playing a video.

    public void play()

    {

           if (canPlayVideo && videoPlayer != null && videoPlayer.isPaused())

            videoPlayer.play();

    }

    // API you present to your C# code to stop your video system

    // playing a video

    public void pause()

    {

            if (canPlayVideo && videoPlayer != null && !videoPlayer.isPaused())

            videoPlayer.pause();        

    }

    // Google Vr Video Listener

    @Override

    public void onSurfaceAvailable(Surface surface) {

     // Google Vr has a surface available for you to render into.

     // Use this surface with your video player as needed.

         if (videoPlayer != null){

            videoPlayer.setSurface(surface);

            canPlayVideo = true;

            if (isSceneLoaded)

            {

                videoPlayer.play();

            }

          }

    }

    @Override

    public void onSurfaceUnavailable() {

     // The Google Vr Video Surface is going away. You need to remove

     // it from anything you have holding it and stop your video player.

        if (videoPlayer != null){

            videoPlayer.pause();

            canPlayVideo = false;

          }

     }

    @Override

    public void onFrameAvailable() {

     // Handle Google Vr frame available callback

    }

}

Unity C# Sample code:

using System;

using System.Collections;

using System.Collections.Generic;

using System.Text;

using UnityEngine;

public class GoogleVRVideo : MonoBehaviour {

 private AndroidJavaObject googleAvrPlayer = null;

 private AndroidJavaObject googleVrVideo = null;

 void Awake()

 {

    if (googleAvrPlayer == null)

    {

      googleAvrPlayer = new AndroidJavaObject("com.unity3d.samplevideoplayer.GoogleAVRPlayer");

    }

    AndroidJavaObject googleVrApi = new AndroidJavaClass("com.unity3d.player.GoogleVrApi");

    if (googleVrApi != null) googleVrVideo = googleVrApi.CallStatic<AndroidJavaObject>("getGoogleVrVideo");

 }

 void Start()

 {

  if (googleVrVideo != null)

  {

   // We need to tell Google VR the location of the video suface in

   // world space. Since there isn't a way to get at that info from

   // Java, we can do it here and then pass the calculated matrix

   // down to the api we expose on our UnityPlayerActivity subclass.

   Matrix4x4 wm = transform.localToWorldMatrix;
   
   wm = Camera.main.transform.parent.worldToLocalMatrix * wm;
   
   wm = wm * Matrix4x4.Scale(new Vector3(0.5f, 0.5f, 1));


   // Convert 4x4 Row Ordered matrix into a 16 element column ordered

   // flat array. The transposition is to make sure that the matrix is

   // in the order that Google uses and the we flatten it to make passing

   // over the JNI boundary easier. The complication being that you have to

   // then convert it back to an 4x4 matrix on the Java side.

   float[] matrix = new float[16];

   for (int i = 0; i < 4; i++)

   {

    for (int j = 0; j < 4; j++)

    {

     matrix[i * 4 + j] = wm[j,i];

    }

   }

   googleVrVideo.Call("setVideoLocationTransform", matrix);  

  }

  

  if (googleAvrPlayer != null)

  {

    AndroidJavaClass jc = new AndroidJavaClass("com.unity3d.player.UnityPlayer");

    AndroidJavaObject jo = jc.GetStatic<AndroidJavaObject>("currentActivity");

    googleAvrPlayer.Call("initVideoPlayer", jo);

    googleAvrPlayer.Call("play");

  }

 }

}

• 2017–08–03 Page published with limited editorial review
• Added in 2017.2 NewIn20172