Processing for Android

Introduction to Cardboard

Get started with Google Cardboard to write VR apps using Processing.

(This tutorial is applicable only to version 4+ of the mode)

Google Cardboard

The Google Cardboard platform lets you use your smartphone as a quick entry point to Virtual Reality. The Android mode supports Google Cardboard, and you should be able to write 3D sketches that run on the phone as Cardboard apps in stereo mode and respond to the head movement. If you first need to get started with 3D in Processing, check this tutorial out.

Requirements

You need an compatible phone with Android 4.1 or higher and a gyroscope sensor for head tracking, and a Cardboard viewer. If you don't have a viewer, you can still run the sketch in monoscopic mode, which will be explained below.

If you are using verion 4.0-beta3, all additional packages needed to build cardboard apps are bundled into the mode, so there are no additional requirements.

If you happen to be using an older beta release of the mode, then you will need to install the processing-cardboard library separately. It provides specialized stereo and monoscopic renderers that work with the phone sensors to automatically update Processing's camera. This library is not available in the Contributions Manager, meaning that you need to install it manually. To do so, first download the latest library zip package from here, uncompress it and copy the cardboard folder into the libraries subfolder inside your sketchbook folder, and then restart Processing.

Getting started

After installing the Cardboard library, you need to select the Cardboard option in the Android menu to make sure that your sketch is built as a Cardboard app:

Cardboard selection

A basic sketch for Cardboard needs to include the library import, and the STEREO renderer:


import processing.cardboard.*;

void setup() {
  fullScreen(PCardboard.STEREO);
}

void draw() {
}

With this code, you should get an empty stereo view on your phone:

Stereo view

The stereo renderer works by drawing each frame twice, once for each eye. You can know what eye is being drawn in each call of the draw() function using the eyeType variable in the Cardboard renderer:


import processing.cardboard.*;

void setup() {
  fullScreen(PCardboard.STEREO);
}

void draw() {
  PGraphicsCardboard pgc = (PGraphicsCardboard)g;
  if (pgc.eyeType == PCardboard.LEFT) {
    background(200, 50, 50);
  } else if (pgc.eyeType == PCardboard.RIGHT) {
    background(50, 50, 200);
  } else if (pgc.eyeType == PCardboard.MONOCULAR) {
    background(50, 200, 50);
  }
}

Left and right eyes

You can notice that eyeType could also be MONOSCOPIC, which happens if you use the MONO renderer instead of STEREO. With the monoscopic renderer, the frame is drawn just once but the camera still responds to the movements of the phone.

There is nothing special to add 3D objects to the scene, you simply use the same functions for drawing 3D primitives and shapes with the P3D renderer. You can also add textures, lights, and shader-based effects, like in the following sketch (full code available here):


import processing.cardboard.*;

PShader toon;
boolean shaderEnabled = true;  

void setup() {
  fullScreen(PCardboard.STEREO);  
  noStroke();
  fill(204);
  toon = loadShader("ToonFrag.glsl", "ToonVert.glsl");
}

void draw() {
  if (shaderEnabled == true) shader(toon);   
  background(80); 
  directionalLight(204, 204, 204, 0, 0, -1);
  translate(width/2, height/2);
  sphere(500);
}  

void mousePressed() {
  if (shaderEnabled) {
    shaderEnabled = false;
    resetShader();
  } 
  else {
    shaderEnabled = true;
  }
}

Toon shading example

Creating a more complex 3D scene

In this example, we will create a 3D scene with a few more objects. Let's start by defining a 2D grid as a reference.

Because performance is very important in VR to keep the framerate as high as possible and avoid simulation sickness in the user, we could rely on PShape objects to store static geometry and so avoid re-generating it in every frame:


import processing.cardboard.*;

PShape grid;

void setup() {
  fullScreen(PCardboard.STEREO);
  
  grid = createShape();
  grid.beginShape(LINES);
  grid.stroke(255);
  for (int x = -10000; x < +10000; x += 250) {
    grid.vertex(x, -1000, +10000);
    grid.vertex(x, -1000, -10000);
  }
  for (int z = -10000; z < +10000; z += 250) {
    grid.vertex(+10000, -1000, z);
    grid.vertex(-10000, -1000, z);      
  }  
  grid.endShape();  
}

void draw() {
  background(0);
  translate(width/2 - 1000, height/2, 500);
  shape(grid);
}

Now we can add some objects! In order to optimize performance, we can group several 3D shapes inside a single PShape group, which is faster than handling each object separately, like so:


import processing.cardboard.*;

PShape grid;
PShape cubes;

void setup() {
  fullScreen(PCardboard.STEREO);
  
  grid = createShape();
  grid.beginShape(LINES);
  grid.stroke(255);
  for (int x = -10000; x < +10000; x += 250) {
    grid.vertex(x, -1000, +10000);
    grid.vertex(x, -1000, -10000);
  }
  for (int z = -10000; z < +10000; z += 250) {
    grid.vertex(+10000, -1000, z);
    grid.vertex(-10000, -1000, z);      
  }  
  grid.endShape();  
  
  cubes = createShape(GROUP);
  for (int i = 0; i < 100; i++) {
    float x = random(-1000, +1000); 
    float y = random(-1000, +1000);
    float z = random(-1000, +1000);
    
    float r = random(50, 150);
    PShape cube = createShape(BOX, r, r, r);
    cube.setStroke(false);
    cube.setFill(color(180));
    cube.translate(x, y, z);
    cubes.addChild(cube);
  }
}

void draw() {
  background(0);
  ambientLight(150, 150, 150);
  pointLight(255, 255, 255, 0, 0, 0);
  translate(width/2 - 1000, height/2, 500);
  shape(cubes);
  shape(grid);
}

The final result should look like this, depending on your viewpoint:

Grid and boxes