A few months ago I purchased a Vuse XR, along with an Oculus Quest 2, and Davinci Resolve Studio (free Speed Editor).

Oculus Quest 2, Vuze XR, DaVinci Resolve Studio

 The Vuse XR (upper right) is a Stereoscopic 3D digital camera that captures VR180 at 5.7K (5760x2880). It is a great camera  for making 3D videos and Panoramics.

I purchased these devices in the hope of making stereoscopic video of my artistic process....

HOWEVER.....

I have an old Acer Nitro laptop, and the first thing it told me was that...my hardware is not up to spec. Meaning PCVR games, such as Minecraft, were once again outside my reach. Apparently, my GPU (Radeon RX 560x) wasn't enough..along with my ram.

When I tried to edit my content using the headset, I ran into a lot of issues with DaVinci Fusion crashing...apparently this is normal. So I couldn't edit using the VR headset.

Sadly I put everything away for a few months....

--------------------------------------------------

Ram Upgrade

I upped my Ram to 32 Gigs, and I was able to use Fusion without a hick-up. I quickly grabbed some footage and rendered a VR180 Video using Fusion Titles.

Link Here: https://youtu.be/2tOfG4N-B2w

------------------------------------------------

Now that I can do something...

I could at least make and edit VR180 footage.

Ergo, I could now consider the possibility of producing using content featuring my artistic process.

I also wanted to incorporate a 3D environment into my work space, meaning I wanted to use virtual sets for some 3D presentations....

MINECRAFT 1.17.1

 I recall back in the day, that Minecraft had great video capture mods, so you can record yourself in game. I came across the Replay Mod, which allows you to record game sessions.  You can then edit the session and place a camera where you like. The Mod allows you to record in any resolution and at different aspects, including stereoscopic and spherical.  

HYPOTHESIS:

1. The Vuze XR has an IDP  0.065 meters.

2. The RePlay Mod has the ability to absolute position a camera and render in spherical 360 with an FOV of 360.

Would it be possible to create a Virtual Set in Minecraft in Stereoscopic 3D with Correct IDP 0.065) ?

Would it be possible to film myself or object in stereoscopic 3D and Mask, Crop, or Rotoscope the environment out with only the subject remaining?

Would it be possible to use the Cropped Vuse XR video (VR180) , to mask the Rendered Minecraft VR180/360 Virtual Set, thereby creating a Mixed reality presentation in Stereoscopic VR180 or VR360?

-------------------------------------------------

The Process:

 Every process and workflow is different, and perspective on that mythology is varied. The following process is what I used to achieve an end result. 

I'll begin with Minecraft.

Note: If you love Minecraft and would like to see what your creation would look like in VR , but don't have the system to play the PCVR version, then this may help you.

------------

What you will Need

- Minecraft 1.17.1 (java edition) : https://minecraft.net

- CurseForge Launcher : https://www.curseforge.com/

- Fabric API : https://www.curseforge.com/minecraft/mc-mods/fabric-api/files

- RePlay Mod : https://www.replaymod.com/

-------------------------------------

Launch and Play Minecraft. The RePlay Mod will record your session. You can optionally start or stop your recording at any time.

Once you have created your virtual environment or VR360 scene in Minecraft, you can begin your Replay recording. Once you're done with your Minecraft session, click on the Replay button. This will load an instance of Minecraft with a recording of your game play . 

The Coolest part of this mod is that you can view yourself at any angle for that particular game session.  

 Before we continue, some notes about VR180/360 Video. 

1.  MOTION CAUSES NAUSEA - If you want moving video, I would suggest VR360(flat). If you have watched a VR video, the motion of the camera will play with your sense of balance, and some people don't react well.  Keep your camera in a fixed Position. 

2.  Objects in the viewer should be 1 meter away (relative distance). Anything closer causes our eyes to go cross-eyed, causing fatigue. 

3.  IPD, Extremely important consideration using mixed reality. Know the IDP of your camera (0.065). 

4. Measure, measure, measure.....Minecraft Blocks are 1m. Everything is relative to that.

-----------------

Replay Mod

In the replay mod, 

  • position your camera to  approximately where you want to view.
  • Set your keyframes.
  • Set your second set of key frames.
  • Set your render session (Equirectangular with FOV of 180 or 360), name it Left
  • Repeat the render session, name it Right
  • This should create two in-Mod recording sessions at an Equirectangular Render with a FOV of 180 or 360, at a resolution appropriate to your Camera (I set my session to FOV 180 at 2880x2880 for Left and Right)

Something to consider

 This process is difficult to articulate without screenshots. The process involves altering the functionality of a Mod 

If you are familiar with Mod development then the following should be understandable;

  • Use the F3 in game to know your positioning.
  • Note your Pitch and Yaw
  • Note your X , Y , Z

1 Minecraft Block = 1 meter

X and Z are Horizontal parallel to the Horizon

Y is Vertical

It's important to face the block at a perpendicular angle (0,90,180,270,360)

Here are some Coordinates as an Example,

(23.23423,34.23431,459.23423)

(178.3,-1.2)

The ReplayMod considers these as: "camera:rotation\":[178.2,-1.2] , "camera:position\":[23.23423,34.23431,459.23423]

Vuze XR Camera Setup:

I set the Camera to have a Height of 172 cm, approx. eye level.

1.72 meters = 1.72 Minecraft Blocks

Vuze XR IDP: 65 millimeters = 0.065 Minecraft Blocks.

Mod Alteration Part

The ReplayMod creates a file for your session.

You will need to unpack the session and change the renderQueue.json 

Edit the json file (Intimidating text file)

There is a lot of script in here, and if you have a code editor, then use it, otherwise you will see a lot of intimidating  case-sensitive  lines of code.

If you setup your session correctly, you should have two in-ReplayMod sessions in the json file.

You want to look for the following (example):

{\"camera:rotation\":[178.2,-1.2],\"camera:position\":[23.23423,34.23431,459.23423]}

Left: This is your first video, and we will use that as a basis to set the second camera, but first.

Correct your Camera Rotation and Position (unless you like Math)

\"camera:rotation\":[180,0] - Facing Perpendicular, and parallel to  Horizon 

Repositioning Math

\"camera:position\":[23.23423,34.23431,459.23423]}

X: 23.23423 --> X: 23.5 / now the center of the block

Y: 34.23431 --> Y: is your Camera Height: in my example, the  Horizon is 33.375. I added 1.7, resulting in 35.075 (relative height of my Vuze XR)

Z: 459.23423 --> Z: 459.5 / now the center of the block

Left Eye should have the following set for each keyframe: {\"camera:rotation\":[180,0],\"camera:position\":[23.5,35.075,459.5}

Right Eye: Right should be setup in consideration of the 0.065 offset.

{\"camera:rotation\":[180,0],\"camera:position\":[23.565,35.075,459.5}

I offset the Camera 0.065 Minecraft Block on the X axis, which parallels the same IDP of the Vuze XR.

Edit as needed and save your renderQueue.json file

Repack the Replay session file.

What I do is unpack the session, then alter the json, save the file.

I then drop the json file into the saved session using Winrar.

Back to Minecraft.

Launch the Game, Open your Replay session and Render your files.

The end result is that you will have a nice square 2880x2880 video for each eye.

In your  Favorite video editor (I Use Davinci Resolve), create a timeline with a resolution of 5760 by 2880.

Drop your Videos and offset each video 1440 pixels (+/-)

Render as 5760x2880, which will give you a VR180 video at 5.7 K.

You should be able to view your rendered VR180 video in your own VR device.

--------------------------------------------------------------------

Example Video:

The example video was rendered at scale of 10 (ten times the normal size)

I represented 10 Minecraft Blocks as 1 meter, and set my ReplayMod virtual cameras  appropriately 

This resulted in the following Video: https://youtu.be/Iiet33UVHG8

-----------------------------

Virtual Set

Nothing closer than 1 meter (actual and relative)

Spent $24 on my Green screen using Table cloths.

Made sure I was 1 meter away in my presentation, this would make me appear to be at least 10 Minecraft blocks away from the Camera.

I made the ReplayMod recording in consideration of where my live action  character would be.

Camera to Subject: 1 meter, at a height of 1.7 meters

Virtual Camera to Subject Area: 10 Minecraft Blocks, at a Height of 17 Blocks, with a Camera Seperation of 0.65 Blocks.

I used Davinci Resolve to remove the background from the Vuze XR footage. 

Then added the footage to the VR180 timeline.

This resulted in a relative position of both camera, resulting in a Stereoscopic Mixed-Reality represention of live-actor in a virtual Minecraft world at 10 scale.

---------------------------------------------

Conclusion:

Mixed Reality Presentations are possible using Stereoscopic Video Cameras and Virtual Environments, like Minecraft.

Exact relative positioning is nessasary. The Virtual Camera must have a parallel representation in the Virtual World, as it would in the Real  World. 

I will be making more Mixed Reality videos featuring the items I make here.

Thank you for consideration

Sincerely,

Dave Krouse

------------------------------------

Contact: promaker.ca@gmail.com

Web: https://promaker.ca