Notebook source code:
notebooks/14_real_world_applications__hand_poses_analysis_in_kendall_shape_space.ipynb
Run it yourself on binder
Classifying hands poses with Kendall shape spaces#
Lead author: Nina Miolane.
In this tutorial, we show how to use geomstats to perform a shape data analysis. Specifically, we aim to study the difference between two groups of data: - hand poses that correspond to the action “Grab”, - hand poses heads that correspond to the action “Expand”.
We wish to investigate if there is a difference in these two groups.
The hand poses are represented as the coordinates of 22 joints in 3D:
Setup#
In [1]:
import os
import sys
import warnings
sys.path.append(os.path.dirname(os.getcwd()))
warnings.filterwarnings("ignore")
In [2]:
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import geomstats.visualization as visualization
import geomstats.backend as gs
import geomstats.datasets.utils as data_utils
from geomstats.geometry.pre_shape import PreShapeSpace, KendallShapeMetric
visualization.tutorial_matplotlib()
INFO: Using numpy backend
Hands shapes#
Load the dataset of hand poses, where a hand is represented as a set of 22 landmarks - the hands joints - in 3D.
The hand poses represent two different hand poses: - Label 0: hand is in the position “Grab” - Label 1: hand is in the position “Expand”
This is a subset of the SHREC 2017 dataset [SWVGLF2017].
We load the dataset of landmarks’ sets and corresponding labels.
In [3]:
hands, labels, bone_list = data_utils.load_hands()
In [4]:
print(hands.shape)
print(labels)
(52, 22, 3)
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
We extract two hands, one corresponding to the “Grab” pose, and the other to the “Expand” pose.
In [5]:
label_to_str = {0: "Grab", 1: "Expand"}
label_to_color = {
0: (102 / 255, 178 / 255, 255 / 255, 1.0),
1: (255 / 255, 178 / 255, 102 / 255, 1.0),
}
first_grab_hand = hands[labels == 0][0]
first_expand_hand = hands[labels == 1][0]
We implement a function to plot one hand in 3D.
In [6]:
def plot_hand(hand, bone_list):
fig = plt.figure()
ax = plt.axes(projection="3d")
x = hand[:, 0]
y = hand[:, 1]
z = hand[:, 2]
sc = ax.scatter(x, y, z, s=40)
for bone in bone_list:
start_bone_idx = bone[0]
end_bone_idx = bone[1]
ax.plot(
xs=[x[start_bone_idx], x[end_bone_idx]],
ys=[y[start_bone_idx], y[end_bone_idx]],
zs=[z[start_bone_idx], z[end_bone_idx]],
)
We plot two examples of hands.
In [7]:
%matplotlib notebook
plot_hand(first_grab_hand, bone_list)
plt.title(f"Hand: {label_to_str[0]}");
In [8]:
plot_hand(first_expand_hand, bone_list)
plt.title(f"Hand: {label_to_str[1]}");
We want to investigate if there is a difference between these two groups of shapes - grab versus expand - or if the main difference is merely relative to the global size of the landmarks’ sets.
In [9]:
m_ambient = 3
k_landmarks = 22
preshape = PreShapeSpace(m_ambient=m_ambient, k_landmarks=k_landmarks)
matrices_metric = preshape.embedding_space.metric
sizes = matrices_metric.norm(preshape.center(hands))
plt.figure(figsize=(6, 4))
for label, col in label_to_color.items():
label_sizes = sizes[labels == label]
plt.hist(label_sizes, color=col, label=label_to_str[label], alpha=0.5, bins=10)
plt.axvline(gs.mean(label_sizes), color=col)
plt.legend(fontsize=14)
plt.title("Hands sizes", fontsize=14);