HoloLens Rubik’s Cube Solver

Andre Fuller, Spencer Wood, Parker Nokes, Jake Fairbairn

Introduction

Have you ever tried to solve a Rubik’s Cube with a guide and failed or had a bad experience? We have too. For this reason we chose this project to address that problem and create a simpler way to get help solving the cube.

Project Description

Our project is an app for the Microsoft HoloLens that can scan a Rubik’s Cube and solve it step by step with directions projected onto the cube itself. This app is the solution to the problem of having to constantly switch focus from cube to guide when learning how to solve it.

Outcomes

The outcomes that we were aiming for included an engaging and immersive experience throughout use, successful solving of the Rubik’s Cube no matter the shuffle, and an intuitive interface so the user knows what to do and how to do it.

Design Components

The design components to reach these outcomes include a cube scanner, which is one of the most integral parts for the app to solve the user’s specific cube and shuffle, the solving algorithm, and on-screen feedback, which will show the user what moves to make and how to make them.

Implementation

Cube Scanner in Unity

We first prototyped the OpenCV scanner in Python for proof-of-concept and then ported the script into C# so it would be compatible with the HoloLens. The main challenges we faced were the differences in OpenCV’s API and the strongly-typed behavior of C#. At this point, we also added additional logic to help prevent double-scanning the same square.

K-means color clustering

Lighting conditions and variance in Rubik’s cube colors meant that our solution would have to accommodate variance in scanned RGB values. For example, red squares would not always be RGB(255,0,0) but rather something close to that. We decided to use a K-Means clustering algorithm to group the squares based on their RGB value. K-Means doesn’t provide out-of-the-box constraining of cluster size so we developed a custom K-Means clustering function that would restrict the size of the clusters to nine squares. This solution allows Rubik’s Cubes of any color to be used in our application.

Virtual Cube based on real cube

We had previously followed a tutorial where we built a virtual Rubik’s Cube in Unity which we added to our project to be used as the “guide” cube for scanning and solving steps. This virtual cube will receive the scanned square colors and map them to create a virtual replica of the physical cube. Once scanning is completed, the virtual cube will rotate to show the steps the user should take to solve the physical cube.

Solving algorithm and voice commands

From the Unity Rubik’s Cube project, we also used Kociemba’s solving algorithm which typically finds solutions with less than 21 moves. We formatted the scanned colors and their positions to interface with this algorithm and were able to successfully return a list of moves for solving the cube. The virtual cube then performs these moves on cue with voice commands from the user so that they can move forward or backwards through the solving steps as needed.

Deploying onto HoloLens

The HoloLens uses different libraries to access each camera frame compared to a traditional webcam. It took a significant amount of work to plug in with those libraries and convert our existing code over to the HoloLens. An additional challenge was spacing and sizing all of the elements together. How we saw these elements on our computers was different once deployed to the HoloLens.

Evaluations

Our evaluations focused on determining how intuitive and effective our scanning and solving solutions were for our users. Two rounds of evaluations were done, with the first focused on just scanning and the second on scanning and solving. Measurements for the intuitiveness and effectiveness of the tasks were gathered using surveys, as these would indicate if the hands-on purpose of the application was successful.

Analyses

From our evaluations, we uncovered several strong trends about the user’s experiences:

• 100% of users that tested written instructions said they either had difficulty interpreting the movement of the cube or that a scan had actually taken place.
• Replacing written instructions with audio cues and commands improved the perceived intuitiveness of the application by 33%.
• Users who had used an AR application at least once before were 75% more likely to report that they enjoyed completing the tasks.

Lessons Learned

• Visual directions are clearer than written when performing hands-on tasks. Written instructions distracted users from the task and required them to shift their attention back and forth.
• Voice commands were more effective than gestures/buttons when performing hands-on tasks. Keeping both hands on the Rubik’s Cube allowed users to be more immersed in the tasks and improved ease of completion.
• The AR environment improves the learning experience. Immersion in an AR environment helped users persevere through difficulties.

References

Unity Rubik’s Cube Project
https://www.megalomobile.com/lets-make-and-solve-a-rubiks-cube-in-unity/

Kociemba’s Solving Algorithm
https://www.speedsolving.com/wiki/index.php/Kociemba's_Algorithm