{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Hand gesture classification with EMG data using Riemannian metrics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lead author: Marius Guerard.\n",
"\n",
"In this notebook we are using EMG time series collected by 8 electrodes placed on the arm skin. We are going to show how to:\n",
"\n",
"- Process these kind of signal into covariance matrices that we can manipulate with geomstats tools.\n",
"- How to apply ML algorithms on this data to classify four different hand gestures present in the data (Rock, Paper, Scissors, Ok).\n",
"- How do the different methods (using Riemanian metrics, projecting on tangent space, Euclidean metric) compare to each other."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"figures/paper_rock_scissors.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Context\n",
"\n",
"The data are acquired from somOS-interface: an sEMG armband that allows you to interact via bluetooth with an Android smartphone (you can contact Marius Guerard (marius.guerard@gmail.com) or Renaud Renault (renaud.armand.renault@gmail.com) for more info on how to make this kind of armband yourself). \n",
"\n",
"An example of application is to record static signs that are linked with different actions (moving a cursor and clicking, sign recognition for command based personal assistants, ...). In these experiments, we want to evaluate the difference in performance (measured as the accuracy of sign recognition) between three different real life situations where we change the conditions of training (when user record signs or \"calibrate\" the device) and testing (when the app guess what sign the user is doing):\n",
"\n",
"- 1. What is the accuracy when doing sign recognition right after training?\n",
"- 2. What is the accuracy when calibrating, removing and replacing the armband at the same position and then testing? \n",
"- 3. What is the accuracy when calibrating, removing the armband and giving it to someone else that is testing it without calibration?\n",
"\n",
"To simulate these situations, we record data from two different users (rr and mg) and in two different sessions (s1 or s2). The user put the bracelet before every session and remove it after every session.\n",
"\n",
"Quick description of the data:\n",
"\n",
"- Each row corresponds to one acquisition, there is an acquisition every ~4 ms for 8 electrodes which correspond to a 250Hz acquisition rate.\n",
"- The time column is in ms.\n",
"- The columns c0 to c7 correspond to the electrical value recorded at each of the 8 electrodes (arbitrary unit).\n",
"- The label correspond to the sign being recorded by the user at this time point ('rest', 'rock', 'paper', 'scissors', or 'ok). 'rest' correspond to a rested arm.\n",
"- the exp identify the user (rr and mg) and the session (s1 or s2)\n",
"\n",
"Note: Another interesting use case, not explored in this notebook, would be to test what is the accruacy when calibrating, removing the armband and giving it to someone else that is calibrating it on its own arm before testing it. The idea being that transfer learning might help getting better results (or faster calibration) than calibrating on one user."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"\n",
"import geomstats.backend as gs\n",
"\n",
"matplotlib.interactive(True)\n",
"gs.random.seed(2021)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Parameters"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"N_ELECTRODES = 8\n",
"N_SIGNS = 4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The Data\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import geomstats.datasets.utils as data_utils\n",
"\n",
"data = data_utils.load_emg()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" time | \n",
" c0 | \n",
" c1 | \n",
" c2 | \n",
" c3 | \n",
" c4 | \n",
" c5 | \n",
" c6 | \n",
" c7 | \n",
" label | \n",
" exp | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 23 | \n",
" 127 | \n",
" 123 | \n",
" 128 | \n",
" 134 | \n",
" 125 | \n",
" 128 | \n",
" 130 | \n",
" 124 | \n",
" rest | \n",
" mg_s1 | \n",
"
\n",
" \n",
" | 1 | \n",
" 28 | \n",
" 126 | \n",
" 130 | \n",
" 128 | \n",
" 119 | \n",
" 129 | \n",
" 128 | \n",
" 126 | \n",
" 133 | \n",
" rest | \n",
" mg_s1 | \n",
"
\n",
" \n",
" | 2 | \n",
" 32 | \n",
" 129 | \n",
" 130 | \n",
" 127 | \n",
" 125 | \n",
" 129 | \n",
" 129 | \n",
" 127 | \n",
" 130 | \n",
" rest | \n",
" mg_s1 | \n",
"
\n",
" \n",
" | 3 | \n",
" 36 | \n",
" 127 | \n",
" 128 | \n",
" 126 | \n",
" 123 | \n",
" 128 | \n",
" 127 | \n",
" 125 | \n",
" 131 | \n",
" rest | \n",
" mg_s1 | \n",
"
\n",
" \n",
" | 4 | \n",
" 40 | \n",
" 127 | \n",
" 128 | \n",
" 129 | \n",
" 124 | \n",
" 127 | \n",
" 129 | \n",
" 127 | \n",
" 128 | \n",
" rest | \n",
" mg_s1 | \n",
"
\n",
" \n",
"
\n",
"