# Tracking movements over a projected 3D plane

I recently updated my virtual rubik’s cube applets with a new functionality. It is now possible to twist a cube by dragging the mouse cursor over the stickers of a cube.

For example, dragging the mouse cursor over the green sticker as indicated by the arrows below twists the upper layer or the back layer of the cube:

This looks simple, but involves a lot of math.

If seen from the front, the green sticker would be square. Since the sticker has been projected by a virtual camera on the computer screen, its shape has been distorted into a trapezoid.

To track mouse movements over such a projected plane, I have performed the following steps – as illustrated in the picture below:

**Identify the location of the mouse cursor on the view plane (blue cross) and of the face of the 3D model which has been rendered at this location (blue triangle).**Most 3D engines support this directly. For example, the idx3d engine which I used here, associates each rendered pixel to the triangles of a face by means of an ID buffer.

The blue triangle belongs to the right face of the cube. Since the blue triangle has been projected on the view plane using a perspective transformation, the relationship between parallel lines has been lost and its angles have been distorted.**Reverse the perspective distortion caused by the camera lens**

(yellow cross and triangle).Since we got hold of the triangle of the 3D model, we can compute its true location in 3D space (yellow triangle). We can also get hold of the virtual camera used by the 3D engine. This allows us to apply the inverse of the perspective transformation to the mouse location. As a result we get the mouse coordinates in 3D space (yellow cross).

**Rotate the plane of the triangle into the view plane (bright cyan cross and triangle).**

For computing the transformation matrix which rotates the yellow triangle into the view plane, I found this very helpful post from Joe Ante in a mailing list.

As a result we get a version of the triangle in 2D coordinates (bright cyan triangle). The mouse coordinates are now in 2D coordinates on this plane (bright cyan cross).

*Instead of transforming the plane of the triangle into the view plane, we could have chosen to transform it into 3D world coordinates. But this not always useful, because in 3D world coordinates, the plane could be facing in a direction which makes it hard to track movements on it. The red line shows where our triangle is located in world coordinates. It is lying on the z-x plane, so we can only see one edge of the triangle.***Bring the first edge of the triangle in parallel with the x-axis (dark cyan cross and triangle).**

I did this by computing a 2D transform matrix, which brings the angle from the first to the second vertice of the triangle to 0°.

As a result, we consistently get the same 2D coordinates for the mouse cursor regardless of the location of the model in the 3D world and regardless of the location and orientation of the virtual camera.

I hope this post is useful for others who need to track movements over projected 3D planes. Let me know, if there is a simpler solution than the one I have described here.

Comments are closed.

Nice work, I think this is the best cube in java i saw so far.