Attaching a hand model to your hands

Question

Created 1d

Replies 1

Boosts 0

Participants 2

Hi, we have been working on an application that attaches a hand model to the users hands. Apple provides an animating hand models in visionOS project that is a useful starting point.

https://developer.apple.com/documentation/visionOS/animating-hand-models-in-visionOS

We have been trying to create our own hand model to attach but have had some issues with how it is attaching to the hand. For our hand model we want to include the forearm all the way up to the users elbow. I have attached a sample project of what our code currently looks like so you can run it. Just select show immersive space to attach the models. The left hand model is the space glove that we were trying to mirror. The right hand model is our model that we have been using. I have mapped each of the joints to the pertaining joint name on our model.

The first issue we are having seems to be based around the placement of the forearm. It attaches itself at the wrist.

The second issue seems to be around rotation.

Our team is looking for some guidance on what needs to change in order to map this model correctly. Thanks in advance!

Boost

Answer 1

Vision Pro Engineer OP

Apple

4h

Hi @lane174

I cloned your GitHub repo and both models appear to be the gloves that ship with the sample. Did you forget to push the right hand model?

Issue 1: Forearm placement attaching at the wrist

This is likely because your model's origin isn't positioned at the wrist. When you call glove.transform = Transform(matrix: handAnchor.originFromAnchorTransform), ARKit places your model so its origin sits at the wrist joint. If your model's origin is set at the forearm instead, the entire model, including the forearm, will be shifted up the arm rather than aligning correctly with the hand.

You have two options to fix this:

Option A: Adjust programmatically

If you don't have access to the source model, you can offset it in code:

let modelContainer = Entity()
modelContainer.addChild(yourHandModel)

// Offset the model so its wrist joint aligns with the container's origin.
// The value should equal the distance from your model's current origin to its wrist joint.
yourHandModel.position.z = distanceFromOriginToWrist // Adjust as needed

// Apply the hand anchor transform to the container instead.
modelContainer.transform = Transform(matrix: handAnchor.originFromAnchorTransform)

Option B: Fix the origin in your 3D software

The cleaner solution is to reposition the model's origin to the base of the wrist directly in your 3D authoring tool.

Issue 2: Rotation

The rotation issue likely stems from how joint rotations are being applied. The sample uses parentFromJointTransform, which provides rotations relative to each joint's parent. This is critical for maintaining correct kinematic chain behavior.

Your model's joint hierarchy must match ARKit's hand skeleton structure exactly:

Joints must be in the same order as handSkeleton.allJoints
Each joint's rotation is relative to its parent, not world space
The kinematic chain flows from wrist > finger bases > fingertips

Make sure that:

Your USD model has exactly 27 joints matching ARKit's structure
You're using parentFromJointTransform (relative rotations), not anchorFromJointTransform (absolute transforms)
Your model's rest pose matches ARKit's expected rest pose, fingers pointing along the negative z-axis, palm facing down along the negative y-axis

Debugging tips:

Print the joint names from your model and compare them against handSkeleton.allJoints to verify order
Start by testing with just the wrist and one finger to isolate the problem

1