Towards a Non-Invasive Measurement of Human Motion,
Force, and Impedance During a Complex
Physical-Interaction Task: Wire-Harnessing
by
A. Michael West Jr.
B.S., Yale University (2018)
Submitted to the Department of Mechanical Engineering
in partial fulfillment of the requirements for the degree of
Master of Science in Mechanical Engineering
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
June 2020
○c Massachusetts Institute of Technology 2020. All rights reserved.
Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Department of Mechanical Engineering
May 8, 2020
Certified by. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Neville Hogan
Sun Jae Professor of Mechanical Engineering
Thesis Supervisor
Accepted by . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Nicolas G. Hadijconstantinou
Chairman, Department Committee on Graduate Theses
2
Towards a Non-Invasive Measurement of Human Motion, Force,
and Impedance During a Complex Physical-Interaction Task:
Wire-Harnessing
by
A. Michael West Jr.
Submitted to the Department of Mechanical Engineering
on May 8, 2020, in partial fulfillment of the
requirements for the degree of
Master of Science in Mechanical Engineering
Abstract
Despite inferior ‘wetware’ and ‘hardware’, humans out-perform robots in tasks that require
physical interaction, especially physical interaction with non-rigid objects. This performance
gap is ever-present in the manufacturing process known as wire-harnessing. Generally, on
an assembly line, robots can be used to quickly and accurately assemble a system. However,
due to the non-rigid nature of a wire harness, wire-harnessing is done manually. In an effort
to better inform robots to help humans do this task, it was proposed to study how humans
install wire-harnesses. However, current methods of studying human behavior inherently
impede the human. Thus, it was proposed to develop a method that can measure the force
and impedance of a wire-harness installation worker’s hands and arms using solely motion
information.
To do so for the human hand, the idea of using a whole-hand, synergy-based Interaction
Capture method was explored. However, current knowledge of synergies has not yet been
extended beyond reach and grasp to object manipulation. Upon trying to compare synergies
during wire-harness installation and object reach-and-grasp, it was found that only the first
synergy is common.
In arm motion, it has been previously found that humans can estimate joint stiffness
using purely visual observation of kinematic motion. However, it was unclear which mo-
tion characteristics subjects used to determine their stiffness estimates. To explore this,
new simulations were produced using modified velocity profiles. The results suggested that
path, not trajectory, information is more important to subjects when estimating stiffness.
Furthermore, upon training a machine to do this task, using supervised linear regression,
it was found that the better performing linear classifiers had a higher emphasis on position
values as opposed to velocity and acceleration, which reinforces the conclusion that path
information dominates trajectory information in the visual perception of stiffness.
Given stiffness estimates, using kinematic hand synergies and visual perception of arm
stiffness, the challenge of estimating contact force in a high degree-of-freedom space still
remains. MuJoCo was tested and validated as a viable tool for estimating contact forces in
3
multi-joint dynamics, allowing one to accurately obtain force information.
While the ability to monitor a human’s motion, force, and impedance during wire-harness
installation was not achieved, this thesis provides essential insight into how to create a system
to do so.
Thesis Supervisor: Neville Hogan
Title: Sun Jae Professor of Mechanical Engineering
4
Acknowledgments
As I reflect on the accomplishment this thesis represents, I would like to thank a few people
who have helped me reach this milestone in my academic journey.
First and foremost, I want to thank my advisor, Professor Neville Hogan. His insight in
the realm of controls and motor neuroscience is unparalleled. I firmly believe I would not
have gained as much knowledge under any other advisor. In addition to his intelligence, I am
very appreciative of his ability to teach and mentor students. Professor Hogan was always
attentive, inquisitive, and constructive when we discussed my work. Furthermore, I felt he
always considered my interests. It has truly been an honor to work with him, and I look
forward to continuing to do so.
I would also like to thank my labmates: James Hermus, Jongwoo Lee, Meghan Huber,
Davi da Silva, David Verdi, Moses Nah, Logan Leahy, Stephan Stansfield, and Kaymie
Shiozawa. They all have made our lab environment both fun and inventive. I want to
especially thank James Hermus and Meghan Huber. I was initially introduced to them as
an undergrad research assistant in the summer of 2017. Since then, they have both spent
countless hours helping me think through ideas, solve problems, and further my research.
Most importantly, it was the two of them who introduced me to this fascinating field of
research, and had me excited about becoming a full time student in the Newman Lab.
Prior to MIT, I was greatly influenced by a number of organizations during my under-
graduate career. Namely, I want to thank the National Society of Black Engineers (NSBE)
for creating a platform for black engineers to meet and inspire one another, and the MIT
Summer Research Program (MSRP) for giving me the tools I needed to confidently to pur-
sue a graduate degree at an institution like MIT. A special shout-out goes to Dean Gloria
Anglon for her tireless work in uplifting and inspiring minority graduate students.
I would like to give a special thanks to my 9𝑡ℎ grade high school English teacher, Mrs.
Villaseñor. She has always pushed me and helped me fulfill my potential as a student. Ten
years later, she is still a great mentor and friend, and took time out of her weekend to
proofread my entire thesis. You Rock Mrs. V!!
Additionally, I want to thank the Black Graduate Student Association (BGSA), Academy
5
of Courageous Minority Engineers (ACME), and Club Rugby team for allowing me to be
apart of their communities. Their support outside the lab has kept me HAPPY!
I want to thank my girlfriend, Caroline Ayinon. Her support over these last few years
has pushed me to be the best student, and person I can be. Through endless home-cooked
meals, short pep talks, and the most fun adventures, she has always found a way to keep me
both motivated and sane during some of my most trying times.
Lastly, I want to thank family. Thank you to my little brother, Amazi, who has always
found a way to keep me smiling and laughing, even from afar. Thank you to my parents.
Without my mother and father’s endless support, guidance, and mantra of, "you can do
anything you set your mind to," none of these accomplishments would have ever even been
thought of, let alone achieved.
This research was performed in the Eric P. and Evelyn E. Newman Laboratory for Biome-
chanics and Human Rehabilitation at the Massachusetts Institute of Technology. It was sup-
ported in part by a research subcontract to Qinetiq, North America, funded by the Advanced
Robotic Manufacturing Institute Project ARM-17-01-TA7. Additional funding support was
also provided by the Massachusetts Institute of Technology Office of Graduate Education
Diversity Fellowship.
6
Contents
1 Introduction 15
1.1 The Problem of Wire-Harnessing . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.2.1 Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.2.2 Mechanical Impedance . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.2.3 Force . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
1.3 Overview of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2 The Analysis of Kinematic Hand Synergies during Wire-Harness Installa-
tion 29
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.2.1 Experimental Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.2.2 Data Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.2.3 Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.3.1 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.3.2 Eigenpostures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.3.3 Reach-and-grasp vs Manipulation Correlation Coefficients . . . . . . 39
2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.4.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.4.2 The First Eigenposture . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.4.3 The Second Eigenposture . . . . . . . . . . . . . . . . . . . . . . . . 53
7
2.4.4 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.4.5 Eigenposture Comparison . . . . . . . . . . . . . . . . . . . . . . . . 54
2.4.6 Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3 Visual Perception of Arm Stiffness: Human Subject Study 59
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2.3 Simulated Arm Motions . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2.4 Task Instruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.2.5 Statistical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.2.6 Self-Report Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.3.1 Subjects’ Self-Reported Results . . . . . . . . . . . . . . . . . . . . . 72
3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.4.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.4.2 Theoretical Implications . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.4.3 Practical Implications . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4 Visual Perception of Arm Stiffness: Implementation 79
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2.1 Simulated Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2.2 Supervised Learning Algorithm . . . . . . . . . . . . . . . . . . . . . 82
4.2.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
8
5 Analysis of MuJoCo’s Contact Force Measurements 95
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3.1 One-Link Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3.2 Two-Link Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.3.3 Three-Link Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.4.1 Force and Torque Impulse at Contact Onset . . . . . . . . . . . . . . 116
5.4.2 Error in Static Zero-Force Trajectory . . . . . . . . . . . . . . . . . . 117
5.4.3 Error in Dynamic Zero-Force Trajectory . . . . . . . . . . . . . . . . 117
5.4.4 Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
6 Conclusions and Future Work 119
6.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.2.1 Whole-Hand Interaction Capture . . . . . . . . . . . . . . . . . . . . 121
6.2.2 Visual Perception of Stiffness with Eye-Tracking . . . . . . . . . . . . 121
6.2.3 Visual Perception of Stiffness Implementation . . . . . . . . . . . . . 122
6.2.4 Further Exploration of MuJoCo . . . . . . . . . . . . . . . . . . . . . 123
A Correlation Testing via Fisher Transformation 125
9
10
List of Figures
1-1 1 DOF mass-spring-damper system . . . . . . . . . . . . . . . . . . . . . . . 19
1-2 Interaction capture for the index finger metacarpal joint [Kry et al., 2006] . . 21
1-3 Santello hand synergies [Santello et al., 1998] . . . . . . . . . . . . . . . . . . 23
1-4 Soft synergy model [Bicchi et al., 2011] . . . . . . . . . . . . . . . . . . . . . 24
2-1 Mock electrical cabinet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2-2 All subjects’ first three Eigenpostures. . . . . . . . . . . . . . . . . . . . . . 40
2-3 All subject’s first Eigenposture . . . . . . . . . . . . . . . . . . . . . . . . . 41
2-4 First Eigenposture average comparison. . . . . . . . . . . . . . . . . . . . . . 42
2-5 All subject’s second Eigenposture . . . . . . . . . . . . . . . . . . . . . . . . 43
2-6 Second Eigenposture average comparison. . . . . . . . . . . . . . . . . . . . . 44
2-7 All subject’s third Eigenposture . . . . . . . . . . . . . . . . . . . . . . . . . 45
2-8 Third Eigenposture average comparison. . . . . . . . . . . . . . . . . . . . . 46
2-9 All subjects’ Eigenposture correlations between experiment . . . . . . . . . . 48
2-10 All subjects’ Eigenposture corrected correlations between experiment . . . . 50
3-1 Experimental set up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3-2 Simulated endpoint paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3-3 Simulated endpoint velocity profiles . . . . . . . . . . . . . . . . . . . . . . . 65
3-4 Group stiffness rating results. . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3-5 Subjects’ individual stiffness ratings . . . . . . . . . . . . . . . . . . . . . . . 70
3-6 Average coefficient of determination, 𝑅2, across experiment. . . . . . . . . . . 71
3-7 Features used in the subjects’ classified self report . . . . . . . . . . . . . . . 72
3-8 Subjects’ classified self report and 𝑅2 . . . . . . . . . . . . . . . . . . . . . . 73
11
4-1 Supervised learning RMS error and hypothesis weights . . . . . . . . . . . . 87
4-2 Comparison of the ratio of path to trajectory related weights and RMSE
performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4-3 Learning curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4-4 The linear classifier’s stiffness predictions . . . . . . . . . . . . . . . . . . . . 90
5-1 A three-link manipulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5-2 The desired minimum-jerk trajectory of the simulated finger . . . . . . . . . 98
5-3 One-link simulation – kinematic, force, and torque measurements . . . . . . 103
5-4 One-link simulation – torque comparison . . . . . . . . . . . . . . . . . . . . 104
5-5 One-link simulation – torque comparison RMS error . . . . . . . . . . . . . . 105
5-6 Two-link simulation – kinematic, force, and torque measurements of the MCP
joint and proximal phalange . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5-7 Two-link simulation – kinematic, force, and torque measurements of the PIP
joint and middle interphalange . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5-8 Two-link simulation – torque comparison . . . . . . . . . . . . . . . . . . . . 109
5-9 Two-link simulation – torque comparison RMS error . . . . . . . . . . . . . . 110
5-10 Three-link simulation – kinematic, force, and torque measurements of the
MCP joint and proximal phalange . . . . . . . . . . . . . . . . . . . . . . . . 111
5-11 Three-link simulation – kinematic, force, and torque measurements of the PIP
joint and middle interphalange . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5-12 Two-link simulation – kinematic, force, and torque measurements of the DIP
joint and distal interphalange . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5-13 Three-link simulation – torque comparison . . . . . . . . . . . . . . . . . . . 114
5-14 Three-link simulation – torque comparison RMS error . . . . . . . . . . . . . 115
12
List of Tables
2.1 All subject’s Eigenposture variance. . . . . . . . . . . . . . . . . . . . . . . . 37
2.2 Diagonal Eigenposture correlation coefficients . . . . . . . . . . . . . . . . . 47
2.3 Correlation coefficient confidence intervals . . . . . . . . . . . . . . . . . . . 49
2.4 Corrected diagonal Eigenposture correlation coefficients . . . . . . . . . . . . 51
2.5 Corrected correlation coefficient confidence intervals . . . . . . . . . . . . . . 51
3.1 Subject self-report encoding criteria . . . . . . . . . . . . . . . . . . . . . . . 67
5.1 Finger parameters for simulation . . . . . . . . . . . . . . . . . . . . . . . . 98
13
14
Chapter 1
Introduction
Despite inferior ‘wetware’ and ‘hardware,’ humans’ ability to perform tasks that require
physical interaction far exceeds that of robots. Humans have the remarkable ability to
dexterously manipulate complex objects, while simultaneously controlling their own high
degree of freedom (DOF) body [Kandel et al., 2000a]. For example, the human hand has
more than 20 DOF, yet it enables us to skillfully use tools, a hallmark of human behavior
[Hogan and Sternad, 2012]. Unfortunately, this sort of dexterity has not been achieved in
robots. This was clearly demonstrated at the DARPA Robotics challenge where robots failed
to complete tasks that are trivial for humans [Banerjee et al., 2015, Knoedler et al., 2015].
This gap in performance is even more pronounced in tasks that require manipulation of
non-rigid objects.
The stark difference between human and robot performance in tasks that require ma-
nipulation of non-rigid objects is clearly demonstrated in manufacturing. Generally, on an
assembly line, robots can be used to quickly and accurately assemble a system. However,
when this system requires the manipulation of a non-rigid object, the automatic (robot)
process becomes a manual (human) one.
1.1 The Problem of Wire-Harnessing
The switch from a robotic process to a manual process is seen in wire-harness installation,
a complex task in the manufacturing process for large electro-mechanical systems. A wire
15
harness is the assembly of electrical cables used in machinery, such as aircraft, automobiles,
and other systems with electrical components. Wire-harnessing is the process of installing
these wire harnesses into their respective electrical systems. The manual nature of this task
can be quite taxing for the human worker. Wire-harnessing can require a human to maintain
unnatural postures for long periods of time as they fasten the wire harness into the assem-
bly. This can lead to not only worker fatigue, but also worker injury. With injured workers,
there is a decrease in the work force, thus leading to a decrease in productivity. Amidst
the identified challenges of manual wire-harnessing, the Advanced Robotics for Manufactur-
ing (ARM) Institute Project has assembled the Robot-Assisted Wire-Harness Installation
Demonstration Effort (RAWHIDE) Project, which aims to develop a novel robotic system
that assists workers in wire-harness installation.
As previously mentioned, robotic control of non-rigid objects is quite challenging. Many
researchers have looked at this problem. Some of the earliest studies come from Hopcroft et
al., who in 1991 looked at the intricacies involved in using a robot to tie knots in rope. The
work highlights the challenges of robot perception of and interaction with non-rigid objects
[Hopcroft et al., 1991]. Much of the new research on robotic manipulation of deformable ob-
jects takes advantage of today’s fast computing processes to model the object dynamics neces-
sary for a given manipulation task (for review see Sanchez et al. 2018 [Sanchez et al., 2018]).
Furthermore, some have looked at the problem of using multiple robots when a single robot
is unable to manipulate the non-rigid objects due to size and grasp constraints, a problem
similar to that of the wire-harness installation workers [Herguedas et al., 2019]. The afore-
mentioned studies highlight the numerous approaches used to tackle this complex problem in
robotics. However, in the case of wire-harness installation, the fact still remains that despite
slower ‘wetware’ and ‘hardware,’ humans still outperform robots in this task. This paradox
motivates research that looks at how humans perform this task. Thus, the work reported
here aimed to develop a system to help one better understand the human workers’ actions
during wire-harness installation.
Specifically, this thesis attempted to ask, “What makes humans so great at complex,
physical-interaction tasks?” Behavioral Neuroscience aims to answer aspects of this question
by experimentally measuring humans’ actions as they perform a given task. A plethora
16
of behavioral studies deduce information about what humans may be doing by observing
purely kinematic motions. However, in tasks that require physical interaction, modula-
tion of neuromuscular mechanical impedance has been shown to be of great importance
[Hogan and Sternad, 2012]. To measure impedance, though, is encumbering and/or disrup-
tive [Bennett et al., 1992, Guarín and Kearney, 2017, Lacquaniti et al., 1993, Lee and Hogan, 2015,
Lee et al., 2016, Rouse et al., 2014, Rouse et al., 2013, Van De Ruit et al., 2020]. One usu-
ally uses EMG or applies a random perturbation, then measures the resultant reaction.
Furthermore, to measure force usually requires mounting instrumentation on the human or
on the object being used. This additional hardware can impede the human from doing the
original task, such as wire-harness installation. Thus, it was proposed to develop a new,
minimally-invasive method that can monitor the motion, force, and impedance of humans’
hands and arms as they perform the wire-harnessing task.
1.2 Solution
In this study, a few existing technologies were drawn upon, furthered, and combined in a
cohesive manner to measure a human operator’s hand motion, force, and impedance in a
non-encumbering way. Moreover, it extended existing knowledge of motor control. Assuming
recent advances in pose estimation algorithms can provide estimates of motion, knowledge
of upper limb motor control of the human hand and arm was used with a view to estimate
impedance based on motion measurements. Additionally, new simulation software calculated
forces of multi-joint dynamic systems using the previously mentioned motion measurements
and impedance estimates.
1.2.1 Motion
Recent advances in the application of machine learning and deep neural networks have
led to software that can identify the kinematics of systems with many degrees of free-
dom based on video data alone. For example, OpenPose is an open source software that
uses new machine learning techniques to track the kinematics of a human using video
data. It can detect the pose of multiple persons’ bodies, feet, faces, and hands in real-
17
time [Cao et al., 2017a, Cao et al., 2017b, Simon et al., 2017, Wei et al., 2016]. Detection
of hand poses is a particularly challenging problem, because too often, parts of the hand
are occluded from video by other parts of the hand. However, this challenge has been
deeply explored and many researchers have already posed viable solutions (for review see
[Erol et al., 2007, Barsoum, 2016]). DeepLabCut is another open source software that uses
machine learning algorithms to measure pose kinematics. It is advantageous as it allows the
user to define which components of a person, animal, or object to track [Mathis et al., 2018,
Nath et al., 2019]. For our purposes, it even has the capability to track the motion of the
wire harness in addition to the human. In conclusion, the technology exists to non-intrusively
measure motion (for further review see [Liu et al., 2018]). However, the problem persists in
how to use kinematics to estimate impedance; if this can be solved, the relationship between
motion and impedance can be used to estimate force.
1.2.2 Mechanical Impedance
In tasks that require physical interaction, such as wire-harness installation, there exists the
challenge of controlling both force and motion. One solution to this challenge is impedance
control. Impedance control regulates the relation between motion and force, and is often
modeled by the equation 1.1,
𝐹 = 𝑍{𝑥0 − 𝑥} (1.1)
where 𝑍{·} is the mapping from a force to a motion (i.e. the impedance). Theoretically,
this mapping may be any linear, or non-linear, operator. It was introduced by Hogan,
[Hogan, 1985c, Hogan, 1985b, Hogan, 1985a] and has since been supported as a plausible
description of human motor control.
To better understand impedance control, refer to the 1 DOF example in Figure 1-1.
Figure 1-1 shows a mass-spring-damper system whose equation of motion, in the absence of
external forces takes the form,
𝑚?̈? = 𝑘(𝑥0 − 𝑥) + 𝑏(𝑥0 − ?̇?) (1.2)
18
Figure 1-1: A diagram of a 1 DOF mass-spring-damper system. All displacements and forces
pointing rightward will be considered positive.
and the control law takes the form,
𝐹𝑎𝑐𝑡 = 𝑘(𝑥0 − 𝑥) + 𝑏(𝑥0 − ?̇?) (1.3)
where the control input, 𝐹𝑎𝑐𝑡 (actuator force), is determined by the zero-force trajectory1, 𝑥0,
and is connected to the actual trajectory, 𝑥, by a rigid body of mass, 𝑚, spring of stiffness,
𝑘, and damper of damping, 𝑏. In the Laplace domain, the forward path dynamics of this
controller are,
𝑋 𝑏𝑠 + 𝑘
= (1.4)
𝑋0 𝑚𝑠2 + 𝑏𝑠 + 𝑘
Intuitively, one can think of 𝑥0 as a planned motion that is “pulling” the actual motion, 𝑥,
via a mass-spring-damper connection.
In the presence of an external force, 𝐹𝑒𝑥𝑡, the disturbance response is modeled by equation
1.5,
𝐹𝑒𝑥𝑡
𝑍 = = 𝑚𝑠2 + 𝑏𝑠 + 𝑘 (1.5)
𝑋
where 𝑍 is the impedance. This impedance characterizes the interactive behavior of the
1The name zero-force trajectory was coined, because this is the trajectory 𝑥 would follow in the absence
of external forces.
19
object (i.e. the relation between force and motion).
When using impedance control during tasks that require physical interaction, such as
wire-harness installation, it is useful to modulate this impedance value, 𝑍. Thus, a system
that aims to quantify human actions during this task must measure impedance. Furthermore,
to use equation 1.1 to calculate force, it is necessary to understand the form of the zero-force
trajectory as well.
Interaction Capture
Understanding the importance of impedance during physical contact, Kry and Pai [Kry et al., 2006]
developed a technique called interaction capture, which uses measured motion and force to
estimate the zero-force trajectory and impedance of a finger when it comes into contact with
an object. Interaction Capture was implemented in two steps. In the first step, for a given
action, motion was captured using traditional motion capture marker systems, and force
was measured using a small six-axis force torque sensor placed at the finger tip. These sen-
sors are less invasive than traditional force measurement methods. Furthermore, given the
known kinematics of the finger, the Jacobian relating differential joint motion to differential
fingertip motion was used to calculate the joint torques corresponding to fingertip forces,
𝜏 = 𝐽𝑇𝐹 . These measurements allowed the authors to estimate the zero-force trajectory
and the impedance of equation 1.1.
To understand how interaction capture allows for simultaneous estimates of zero-force
trajectory and motion, refer to Figure 1-2. In Figure 1-2, the blue lines in the top and
bottom plots show the measured joint motion and torque, respectively, of a single joint as a
single finger moved into contact with an object. At time zero seconds, contact was detected.
This was determined by an increase in joint torque and the change in direction of joint angle.
The dashed green line shows the estimated zero force trajectory. It was approximated as
a linear fit to the motion prior to contact, and was assumed to continue linearly through
contact. With the estimated zero-force trajectory, equation 1.1 could be used to estimate
the impedance.
In the second step of interaction capture, these impedance estimates were used with mea-
sured motion to estimate contact force. Similarly, the zero-force trajectory was estimated
20
Figure 1-2: Interaction capture for the index finger metacarpal joint. Figure from Kry and
Pai [Kry et al., 2006]
based on motion exactly as described before. Thus, equation 1.1 could be used to estimate
contact force based on measured motion. Kry and Pai [Kry et al., 2006] laid the ground
work for estimating hand mechanical impedance from motion data as the hand manipulates
an object. However, the complex kinematics of the hand present a greater challenge. Fortu-
nately, many have looked at techniques to simplify the kinematics of the human hand, and
that work can be used to inform the zero-force trajectory of the human hand for whole-hand
interaction capture.
Hand synergies
Kinematic analysis of the human hand is challenging due to the 20+ degrees of freedom of
the various joints. However, knowledge of the features of human biomechanics and sensory-
motor control can simplify this problem. Specifically, one can use synergies, a coordinate-
based dimensionality reduction, to reduce the degrees of freedom in the human hand. Given
the coordinates of the human hand, 𝜃, it is likely that many of these degrees of freedom are
controlled simultaneously, based on biomechanical (and possibly neural) constraints. If so,
21
there exists a kinematic mapping such that
𝜃 = 𝜑(𝛿) (1.6)
where 𝛿(𝑡) is a neural command that has fewer dimensions than 𝜃(𝑡), and 𝜑(·) is a kinematic
mapping between 𝛿 and 𝜃. Direct access to 𝛿 is not immediately available. However, assuming
it exists, the kinematic mapping is often found via Principal Component Analysis or Singular
Value Decomposition,2 two dimensionality reduction methods.
Hand synergies were first studied by Santello et al. [Santello et al., 1998]. There, the
static hand posture (15 joints) of subjects was measured as they held their right hand as if3
to grasp and use 57 various household objects. Using principal component analysis, it was
found that the first two synergies accounted for greater than 80% of variance in all hand
postures. Specifically, the first synergy is generally described as a significant flexion and slight
adduction of the four fingers, and the second synergy is described as significant thumb inward
rotation, significant flexion of the index finger, and medium flexion of the middle finger
(Figure 1-3). Furthermore, Weiss and Flanders [Weiss and Flanders, 2004] found similar
results when asking subjects to spell the alphabet using American Sign Language. Both of
these results led to the conclusion that the control of hand posture is not regulated by the
control of each individual joint. Rather, a few synergies are used to regulate most of the
hand motion as a whole.
The study of synergies was later extended to experiments that involved object contact
and the exploration of temporal movement of the hand throughout reach-to-grasp. Mason et
al. [Mason et al., 2001] used Singular Value Decomposition to obtain the temporal evolution
of a given postural synergy throughout reach to grasp. Their methods will be detailed later
in Chapter 2. It was found that the first synergy was similar to that of PC1 in Figure
1-3, and temporal evolution of this posture was similar across both subjects and objects
grasped. Initially, subjects opened their hand to an aperture greater than the object’s size
and then closed their hand around it. These results were further analyzed by Santello et al.
2For better understanding of these mathematical techniques the reader is referred to [Smith, 1988]
3It is important to emphasize that in these experiments the object was neither physically nor virtually
present. Subjects were asked to imagine them.
22
Figure 1-3: Postural synergies defined by the first two principal components. The hand
posture at the center of the PC axes is the average of 57 hand postures for one subject. The
postures to the right and left are for the postures for the maximum (max) and minimum
(min) values of the first principal component (PC1), coefficients for the other principal
components having been set to zero. The postures at the top and bottom are for the
maximum and minimum values of the second principal component (PC2). Figure from
Santello et al. [Santello et al., 1998]
[Santello et al., 2002], who explored the temporal evolution of synergies in three experimental
conditions: (1) memory-guided movements, where the object was not shown; (2) movements
where the object was virtually shown but not physically present; and (3) movements where
the object was physically present. It was found that the first two synergies were still similar
to those of Figure 1-3, and accounted for greater than 75% of variance in hand posture.
However, in the case of physical presence of the object, the second synergy only manifested
itself after fingers came in contact with the object. Nonetheless, the results are consistent
with the hypothesis that the hand is controlled by a subspace of lower dimension than the
original 20+ degrees of freedom of the human hand.
As explained earlier, in the presence of physical contact, impedance modulation is im-
portant. In the case of grasping, this modulation is especially important as it determines
the stability of a grasp. It was found that the first few synergies can be used to establish a
stable grasp. However, without active control of the first few synergies, many higher-order
synergies are needed to establish stability [Gabiccini et al., 2011]. However, these results
23
Figure 1-4: The reference hand moves on the synergy manifold (a–d) and represents an
attractor for the real hand (e–h), which is repelled by contact forces with the object. The
resulting configuration is ultimately dictated by the hand mechanical impedance and control
stiffness. Figure from Bicchi et al. [Bicchi et al., 2011]
were only obtained through the introduction of hand mechanical impedance. The hand in
their study was in a dynamical equilibrium under two force fields. The first force field was
governed by the desired hand motion, which in this case was a given synergy, and the second
force field was governed by the object which repelled the hand from penetrating it (Figure
1-4). In the framework of equation 1.1, the synergy served as the zero-force trajectory 𝑥0.
This model is known as the soft synergy model.
Even with a model that uses a known synergy as the zero-force trajectory, the correspond-
ing impedance is still unknown. While the soft synergy model was robust to small changes in
impedance values, the uncertainty of these impedance values was highlighted as a challenge
by Gabiccini et al. [Gabiccini et al., 2011]. Some researchers [Weiss and Flanders, 2004,
Leo et al., 2016] have aimed to correlate postural synergies to muscle activation (i.e. muscle
synergies). Weiss and Flanders [Weiss and Flanders, 2004] recorded both the joint angles
and electromyographic (EMG) activities of the hand during various static grasps and hand
shapes. It was found that it is rare for a single muscle, or motor unit, to align with a single
postural or muscle synergy. While these results do not definitively correlate hand impedance
to synergies, they do lay a foundation for doing so.
In conclusion, the human hand presents the challenge of analyzing a high degree of free-
dom system. However, this challenge can be mitigated through dimensionality reduction
24
methods that lead to kinematic synergies. Additionally, these kinematic synergies can be
used to not only inform motion, but also the zero-force trajectory. If one can associate
impedance values with these kinematic synergies, using synergy-based whole-hand Interac-
tion Capture, there is enough information to solve Equation 1.1.
The work reviewed above leaves one striking gap in knowledge. The presence of synergies
has been explored in experiments that involve grasping. However, this thesis focuses on
monitoring the wire-harness installation worker performing a functional task. As of now, it
is unknown whether the current knowledge of synergies can be extended to cases of object
manipulation and tool use. This question is explored in Chapter 2 of this thesis.
Stiffness Measurements in the Upper Limb during Movement
In a similar vein as above, this work would also like to noninvasively monitor the motion,
force, and impedance of the wire-harness worker’s arm during wire-harness installation. This
raises the question, “How can one make estimates of force and impedance based on motion
data?”
Recently, Hermus et al. [Hermus et al., 2020] presented work that uses measurements
of motion and force and an estimate of mechanical impedance to calculate a zero-force
trajectory. Unfortunately, their formulation requires more than just motion measurements.
If mechanical impedance were known, using the synergy-based interaction capture described
above, one might obtain an estimate of the force at the hand (i.e. the arm’s endpoint force).
In previous work, Huber et al. found that humans could infer dynamic properties, such
as joint stiffness, from multi-joint limb motion [Huber et al., 2017, Huber et al., 2019]. In
that study, subjects observed the motion of a simulated stick-figure, two-link planar arm
on a computer screen, and then rated its stiffness on a numeric scale. To mimic aspects
of human neuromotor control, the arm movement was driven by the superimposition of a
hand-space impedance controller and a joint-space impedance [Hogan and Sternad, 2012,
Hogan, 2017]. Results showed that subjects’ stiffness ratings positively correlated with the
joint stiffness values used in the control policy, indicating that they could estimate changes in
joint stiffness. Remarkably, this was possible without force information or explicit knowledge
of the underlying limb controller. It is impossible to unequivocally quantify features such as
25
limb stiffness from motion alone. Thus, humans must have used prior knowledge to estimate
latent information from visual observation of motion. To estimate limb stiffness, for instance,
their prior knowledge had to be congruent with the relationship between limb stiffness and
motion produced by the control policy used to drive the simulated limb. However, there are
still open questions regarding the form of the prior knowledge used and how it is acquired.
One possibility is that shared resources are used for action execution and action perception.
Thus, a greater understanding of how humans perceive the motor behavior of others can
increase the knowledge of how humans control their own movements. In monitoring wire-
harness installation, knowledge of how subjects were able to do this task can better inform
how to train a machine to estimate stiffness using motion.
Prior work indicates that highly regular patterns exist in the temporal aspects of human
movement and that temporal information plays a key role in biological motion perception
and understanding. It is commonly observed that hand tangential velocity changes loga-
rithmically with hand path radius of curvature, the so-called 1/3 power law (also commonly
referred to as the 2/3 power law, depending on the formulation) [Huh and Sejnowski, 2015,
Viviani and Flash, 1995]. Humans are also sensitive to such velocity-curvature patterns
when visually perceiving and interpreting motion. For instance, humans perceive motions
that follow the 1/3 power law to be more natural [Bidet-Ildei et al., 2006] and uniform
[Viviani and Stucchi, 1992]. Humans can also anticipate the motion of a system more accu-
rately when it follows the 1/3 power law [Kandel et al., 2000b]. Maurice et al. [Maurice et al., 2018]
showed that humans can control physical interaction with a robot better when its velocity
profile follows the 1/3 power law. Consistent with these behavioral findings, Dayan et al.
[Dayan et al., 2007] found a stronger and more extensive neural response, especially in motor-
related areas, when humans perceived motion that followed the 1/3 power-law compared to
motion that did not. These studies emphasize the importance of the 1/3 power law in human
perception of motion. However, it is unknown whether this relation is important for human
perception of dynamics, such as stiffness. This topic will be explored in Chapters 3 and 4.
26
1.2.3 Force
Given motion measurement and estimates of impedance and zero-force trajectory, one possi-
ble challenge still emerges in using Equation 1.1 to calculate contact forces in a high degree
of freedom system, such as the human hand. Specifically, managing the numerous forces
at play during a complex manipulation task can be computationally taxing. MuJoCo is a
physics-based simulation engine specifically designed for multi-joint dynamics with contact
[Todorov et al., 2012, Todorov, 2014], and it can be used to estimate the forces evoked by a
given impedance and zero-force trajectory.
Some previous work has suggested that there may be inaccuracies in various simulation
software platforms. However, there is not a benchmark to determine which simulation soft-
ware is the best. Rather, it is up to the user to determine which software is the best for their
needs. For instance, Erez et al. [Erez et al., 2015] found that MuJoCo performs faster and
more accurately than other simulation software, such as Bullet, Havok, ODE, and Phys X, in
tasks that involve constrained systems relevant to robotics. However, in simulations of many
disconnected bodies, its performance was slower and more inaccurate. Similarly, Collins et
al. [Collins et al., 2019] used a motion capture system and a kinova 6 DOF robotic arm to
measure ground truth movement of a robotic controller, and compared those measurements
to simulations of the robot in MuJoCo, PyBullet, and V-rep. It was found that all the sim-
ulation tools were not 100% accurate, and no software consistently outperformed the others
in measures of accuracy. Horak et al. [Horak and Trinkle, 2019] compared contact models
used in robotic simulators, and emphasized the importance of knowing the advantages and
disadvantages of the contact models used in a chosen simulator. Thus, before MuJoCo can
be used to estimate contact forces in a high degree of freedom space, it is important to
understand its uses and limitations, a topic that will be explored in Chapter 5.
1.3 Overview of Thesis
This introductory chapter has motivated the need for non-encumbering system that can
monitor human motion, force, and impedance during a task that involves physical interaction.
The topic of interest here is wire-harness installation. This chapter has also provided a basis
27
for understanding the proposed solution for such a system.
Knowledge of reach and grasp synergies may be used to inform the zero-force trajectory
of the human hand and enable whole-hand interaction capture. Thus, chapter 2 reports
the results of an experiment that looks specifically at hand synergies in a task resembling
wire-harness installation. A specific aim was to determine whether synergies observed in
object manipulation were the same as those observed in reach and grasp.
Whole-hand Interaction Capture requires an estimate of mechanical impedance. Prior
work indicates that human subjects can estimate stiffness using solely motion information
[Huber et al., 2017, Huber et al., 2019]. Extending that work, chapter 3 reports a human
subject experiment that provides insight into which motion cues are more important in
estimating stiffness. The work presented there can help lead to minimally-invasive impedance
estimation.
Given humans ability to estimate stiffness using only kinematic information, it is possible
that one could teach a machine to do the same. Chapter 4 demonstrates the use of a
supervised linear regression algorithm to teach a “machine” to estimate stiffness based solely
upon motion cues.
Given estimates of the zero-force trajectory and hand stiffness, there remains the chal-
lenge of estimating force. Chapter 5 aims to alleviate the challenge of estimating the nu-
merous forces at play during a complex manipulation task. Moreover, it serves to validate
MuJoCo as a tool to estimate the contact force of a multi-joint system, by presenting a series
of finger simulations coming into contact with an object.
Finally, Chapter 6 discusses the conclusions of the present work, and suggests future
directions.
28
Chapter 2
The Analysis of Kinematic Hand
Synergies during Wire-Harness
Installation
2.1 Introduction
Wire-Harness installation presents a grave challenge in manufacturing. In a plant, where
much of the operation is done automatically by robots, wire-harness installation must be
done manually by humans, thus creating a bottleneck in the assembly line. Robots are
unable to do this task because of the dynamic complexity and non-rigidity of wire harnesses.
However, in an effort to allow robots to aid wire-harness installation workers, this thesis
aims to look at how humans currently install wire harnesses. More specifically, this thesis is
interested in making noninvasive estimates of motion, force, and impedance as wire-harness
workers do this task.
As mentioned in Chapter 1, to achieve this goal, it was proposed to use knowledge of
kinematic synergies to incorporate whole-hand Interaction Capture. Specifically, Equation
1.1 (𝐹 = 𝑍{𝑋0 − 𝑋}) was going to be solved using a measurement of motion, 𝑋, com-
bined with a knowledge or estimate of impedance, 𝑍{·}. With this measurement of 𝑋,
the zero-force trajectory could be estimated under the assumption that a kinematic synergy
29
observed during unconstrained motion would continue through contact, similar to the soft
synergy model of Bicchi et al. [Bicchi et al., 2011]. However, kinematic synergies have only
been explored in reach and grasp experiments [Santello et al., 1998, Santello et al., 2002,
Weiss and Flanders, 2004, Mason et al., 2001]; to the best of the author’s knowledge, as of
yet no one has studied kinematic synergies in object manipulation. Thus, to apply the exist-
ing literature’s knowledge of synergies to manipulation, one must determine if the synergies
in reach and grasp are the same as those during object manipulation.
This chapter is focused on comparing synergies observed during two experiments in the
context of wire-harness installation. In the first experiment, subjects reached for and grasped
a tool or object commonly found in wire-harness installation, and in the second experiment,
subjects manipulated those objects and tools to install a wire harness on a mock electri-
cal cabinet. Here, synergies were extracted using the algorithm presented by Mason et al.
[Mason et al., 2001]. This particular algorithm extracts an Eigenposture 1 – a hand posture
that determines how the joints are coordinated, and a temporal weighting – the Eigenpos-
ture’s associated evolution over time. The Eigenposture is the usual definition of a synergy
and the temporal weighting estimates the neural command (𝛿 in Equation 1.6) that drives
that synergy. Upon comparing Reach-and-Grasp Eigenpostures to Manipulation Eigenpos-
tures, it was found that only the first Eigenposture was the same across experimental tasks.
2.2 Methods
2.2.1 Experimental Task
Seven adults, with no known history of neurological or musculoskeletal problems, participated
in this study (3 women and 4 men, age ranging from 18 to 28 years old). All subjects were
right-handed. Participants were informed about the experimental procedure and agreed to
sign a consent form. All procedures were approved by MIT’s Institutional Review Board.
Each subject performed two different tasks. The first task, which will be referred to
as the Reach-and-Grasp Experiment, involved reaching, grasping, and picking up a tool.
1In accordance with Mason et al. [Mason et al., 2001], the static hand synergies presented here will be
referred to as Eigenpostures.
30
Specifically, subjects were asked to grasp a pair of scissors, a Zip tie, a screwdriver, a wire
harness with its branched ends Zip-tied, and a wire harness without its branched ends Zip-
tied. Subjects stood in front of a table with their hands initially flat on its surface. The
object was placed 30 cm in front of the subject’s right hand, and they were instructed to
reach and grasp the object as if they were going to use it. Subjects repeated this four times
for each object. The second task was designed to emulate wire-harnessing in a manufacturing
plant; subjects were required to use the tools from the first task to install a wire harness
on a “mock” electrical cabinet. This component of the experiment will be referred to as
the Manipulation Experiment. This task occurred in five individual steps. At each step,
subjects were verbally told what to do. Additionally, they were given a booklet that had
written instructions and a picture of each specific step. The steps were:
1. Zip-tie the branched ends of the wire harness at the three points denoted by blue tape.
2. Use the scissors to cut off the excess tips of the Zip ties.
3. Use the U-brackets, provided screws, and screwdriver to screw the un-zip-tied end of
the wire harness into the top of the mock electrical cabinet.
4. Route the wire harness through the red J-hooks on the mock electrical cabinet.
5. Plug the connector, denoted by the yellow tape, into the black 3D-printed socket on
the mock electrical cabinet.
The completion of these 5 tasks is shown in Figure 2-1.
2.2.2 Data Acquisition
While subjects performed these tasks, hand posture was measured using a CyberGlove (Cy-
berGlove; Virtual Technologies, Palo Alto, CA), a glove with embedded sensors that measure
joint kinematics. Specifically, the flexion of the distal interphalangeal (DIP), proximal inter-
phalangeal (PIP), and metacarpophalangeal (MCP) joints of the four fingers were measured.
Additionally, the abduction (ABD) of the four fingers at the metacarpophalangeal joints was
measured. At the thumb, the flexion at the MCP and interphalangeal (IP) joints, abduction
31
Figure 2-1: A mock electrical cabinet after subjects had finished the Manipulation Experi-
ment.
32
(ABD) at the carpometacarpal joint, and rotation (ROT) about an axis passing through
the trapeziometacarpal joint were measured. Lastly, palm arch (PA) and wrist (W) pitch
and yaw were measured. Throughout both experiments, subjects wore a Cyberglove on each
hand. However, during reach-and-grasp of the scissors, Zip tie, and screwdriver, subjects
only used one hand; reach-and-grasp of the wire harness and the manipulation tasks required
the use of both hands. Data was collected on either the right hand alone or on both hands
as deemed appropriate. In all cases, the CyberGlove collected samples at ∼200 Hz with a
spatial resolution of <0.1∘.
In the Reach-and-Grasp Experiment, subjects started with their hands flat on a table.
After three seconds of data collection, subjects were verbally instructed to grasp the object.
After 15 seconds, data collection ended. This was repeated 20 times (4 trials each for 5
different objects). In the mock wire-harnessing task, subjects started with their hand in the
same initial position, and the tools to be used were similarly placed 30 cm in front of them.
After three seconds of data collection, subjects were verbally instructed to begin the specific
step of the wire-harness assembly task. Data collection terminated when subjects deemed
they were done with the task and returned their hands to the initial position.
To detect movement onset, average joint displacement was calculated using equation 2.1.
∑︁𝑗
𝑥𝑖+1,𝑗 − 𝑥𝑖,𝑗
∆𝑥 23𝑎𝑣𝑔 = (2.1)𝑖 23
where 𝑥 is an angle measurement, 𝑖 is the time sample, and 𝑗 is a particular joint. This
average joint displacement was smoothed using a sliding window of 5 adjacent samples.
Movement was detected when the smoothed displacement was greater than 5% of its max-
imum value computed over the entire data set. Thus, in the Reach-and-Grasp Experiment,
reach began at the instant movement was detected, and ended after the last movement was
detected. After movement concluded, it was said that this was the moment when grasp
commenced. Grasp concluded with the end of data collection. Thus, the Reach-and-Grasp
Experiment was partitioned into two sets of data; the first set contained the time-sampled
measurement of 23 joint angles during reach, and the second set contained the time-sampled
33
measurement of 23 joint angles during grasp. Each partitioned set of data was then interpo-
lated to 100 bins. In accordance with the work done by Mason et al., reach and grasp were
analyzed together. So, the two data sets were recombined to form the reach-and-grasp data
of one subject during one trial, and the reach portion was evenly weighted with the grasp
portion of the data set. If the grasp portion had been analyzed individually, there would be
no temporal evolution, which would differ from the manipulation task. Furthermore, prelimi-
nary analysis of reach and grasp individually showed that they contained similar information.
In the manipulation task, movement onset was detected using the same algorithm; however,
there was no reliable method to detect when the object was grasped. The duration of each
segment of the manipulation task varied between subjects. This factor was not analyzed in
this initial study. Additionally, as each step in the task took varying amounts of time, time
was scaled differently. This was done so that each step was weighted equally during sub-
sequent analysis. Specifically, in this experiment, the data after movement detection until
the end of collection for a given step was interpolated to 1000 bins. An increased number of
bins was used, because these experiments generally took longer than the Reach-and-Grasp
Experiment.
2.2.3 Analyses
Following the work of Mason et al. [Mason et al., 2001], singular value decomposition (SVD)
was used to analyze the evolving hand postures in both tasks. In the Reach-and-Grasp
Experiment, the data for a given subject was formed into matrix 𝑋1(4000 × 23). The
columns consisted of the 23 joint angles that were measured, and the rows consisted of their
measurements across time, stacked for each object and each trial. Thus, in the Reach-and-
Grasp Experiment, matrix 𝑋1 had 4000 rows from 200 bins × 5 objects × 4 trials. In the
Manipulation Experiment, the data matrix, 𝑋2 , was computed similarly: – 𝑋2(5000 × 23)
had 5000 rows and 23 columns, where the columns represent the 23 joint angles measured,
and the rows represent the 1000 bins × 5 steps. Singular value decomposition takes the left
and right singular vectors2 of a matrix. So, 𝑆𝑉 𝐷(𝑋) = 𝑈Σ𝑉 𝑇 produces linear combinations
of hand postures in 𝑉 (23× 23), their temporal evolutions in 𝑈1(4000× 4000) or 𝑈2(5000×
2A right singular vector is an eigenvector of 𝑋𝑇𝑋, and a left singular vector is an eigenvector of 𝑋𝑋𝑇 .
34
5000), and an associated variance metric, the singular values, on the diagonal of Σ(4000 ×
23 or 5000× 23).
The singular values along the diagonal of matrix Σ provide a measure of the variance ac-
counted for by a given Eigenposture (the hand synergy) in the data. The variance accounted
for (VAF) is reported as a decimal, ranging from 0 to 1. It was computed from the values
in Σ, using equation 2.2.
𝜎2𝑗
𝑉 𝐴𝐹𝑗 = ∑︁ (2.2)𝑗
𝜎2𝑗
Here, 𝜎 is a particular singular value, denoted by the subscript 𝑗. It lies along the diagonal
of matrix Σ. The denominator in equation 2.2 is the sum of all the singular values in matrix
Σ.
Two-way analysis of variance was used to assess whether the variance accounted for was
similar across experiments and Eigenpostures. Specific interest was given to the first three
Eigenpostures, because 3 Eigenpostures consistently accounted for ∼95% of the variance, as
shown in the results section. To complement the analysis of variance, post-hoc paired-sample
t-tests were conducted, and adjusted using the Bonferroni correction,
𝛼
𝛼𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 = (2.3)
𝑐
where 𝛼 is the significance level and 𝑐 is the number of repeated tests. Here, the significance
level was set to 5% (𝛼 = 0.05).
The calculated Eigenpostures, 𝑉 , were compared across both experiment and subject.
To compare them across experiment, the product of the Reach-and-Grasp Eigenpostures and
the Manipulation Eigenpostures was computed,
𝐶 = 𝑉 𝑇2 × 𝑉1 (2.4)
resulting in a correlation matrix, 𝐶. Because an eigenvector has unit magnitude, the dot
product of two vectors computes the cosine of the angle between them, resulting in a value
ranging from -1 to 1. The cosine of the angle between two unit vectors is a geometric inter-
35
pretation of the Pearson correlation coefficient3 [Lee Rodgers and Alan Nice Wander, 1988].
Thus, the correlation matrix, 𝐶, contains the Pearson correlation coefficients between Eigen-
postures. Since singular value decomposition does not take into account the absolute direc-
tion of the eigenvector, exact anti-correlation, -1, is the same as exact correlation. In terms
of synergies, the negative of an eigenvector gives the same information as the positive. Thus,
for ease of the readers’ viewing, the magnitude of this correlation matrix is reported, resulting
in a 𝐶 matrix of values ranging from 0 to 1.
Furthermore, to test whether these Eigenpostures were statistically different from one
another, correlation testing via the Fisher transformation was conducted on the diagonal
Pearson correlation coefficients. This metric tested the null hypothesis that the correlation
coefficient, 𝑟, came from a population correlation of 0.994. Using the Fisher transformation,
a 95% confidence interval was produced for the first three diagonal correlation coefficients. If
the confidence interval of a correlation coefficient contained the value 0.99, that correlation
coefficient was not considered statistically different from 1. Thus, the two Eigenpostures
that produced that particular correlation coefficient were considered to be the same.
To compare Eigenpostures across subjects, it was important that the postures pointed
in the same direction. To correct for cases where this did not occur, a “ground truth”
Eigenposture based on data from all seven subjects was computed (i.e. 𝑋𝑔𝑟𝑜𝑢𝑛𝑑𝑡𝑟𝑢𝑡ℎ1 had
dimensions 28000×23 and 𝑋𝑔𝑟𝑜𝑢𝑛𝑑𝑡𝑟𝑢𝑡ℎ2 had dimensions 35000×23). Any Eigenposture that
had a negative correlation with the "ground truth" was multiplied by -1, so it would then
have a positive correlation with the "ground truth" Eigenposture. With all the Eigenpostures
facing the same direction, a mean and standard error from that mean was computed.
All analyses were conducted in MATLAB 2017b.
3Also known as Pearson’s r.
4The Fisher transformation takes the non-normally distributed Pearson correlation coefficients and con-
verts them into to a normally distributed function. However, that function asymptotically approaches -1
and 1. Thus, testing the null hypothesis that the sample correlation coefficient, 𝑟, came from a population
with the correlation coefficient 1 is an ill-posed problem. For more info, see Appendix A.
36
2.3 Results
2.3.1 Variance
Table 2.1 reports the variance accounted for by each of the first ten Eigenpostures for each
subject. Additionally, it reports the mean and standard error from the mean across subjects.
With just two Eigenpostures, all subjects could account for at least 90% of all hand posture
variance in both the Reach-and-Grasp and Manipulation Experiments. In the Reach-and-
Grasp Experiment, all subjects required three Eigenpostures to account for greater than 95%
of variance in hand posture, while in the Manipulation Experiment, some subjects (subjects
3, 4, and 6) required 4 Eigenpostures. Given the large amount of variance accounted for by
the first three Eigenpostures, further analysis focused on these Eigenpostures.
Subject Task E1 E2 E3 E4 E5 E6 E7 E8 E9 E10
1 1 86.4 4.8 4 1.3 1.2 0.7 0.4 0.4 0.3 0.22 91.7 2.5 1.9 1 0.8 0.6 0.5 0.3 0.2 0.1
2 1 88.7 4.7 2.9 1.4 0.7 0.5 0.3 0.3 0.2 0.12 88.7 3.8 3.3 1.2 0.9 0.6 0.4 0.4 0.2 0.2
3 1 90 5.6 1.5 1 0.6 0.3 0.3 0.2 0.2 0.12 87.5 4.8 2.7 1.2 1.1 0.8 0.5 0.4 0.3 0.2
4 1 93.3 3.1 1.6 0.8 0.5 0.2 0.2 0.1 0.1 02 87.1 3.7 2.9 1.6 1.4 1.1 0.7 0.5 0.4 0.2
5 1 88.4 4.8 2.1 1.6 1 0.7 0.4 0.4 0.2 0.12 91.2 2.5 1.6 1.3 1 0.5 0.4 0.4 0.2 0.2
6 1 88.2 6.3 2.7 1.2 0.7 0.3 0.2 0.1 0.1 0.12 88.1 4 2.6 1.6 1.1 0.7 0.7 0.4 0.2 0.2
7 1 87.2 7.4 2.7 0.7 0.6 0.5 0.3 0.2 0.1 0.12 87.5 4.9 3 1.4 1 0.5 0.5 0.3 0.2 0.2
Mean 1 88.9 5.2 2.5 1.1 0.7 0.5 0.3 0.2 0.2 0.12 88.8 3.7 2.6 1.3 1 0.7 0.5 0.4 0.3 0.2
Standard 1 0.9 0.5 0.3 0.1 0.1 0.1 0 0 0 0
Error 2 0.7 0.4 0.2 0.1 0.1 0.1 0 0 0 0
Table 2.1: All subject’s Eigenposture variance. Task 1 is the Reach-and-Grasp Experiment.
Task 2 is the Manipulation Experiment.
Across-subject two-way (2 (experiment) × 3 (Eigenposture)) analysis of variance on the
variance-accounted-for shown in Table 2.1 showed no significant effect across experiment (𝑝 =
0.652, 𝛼𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 = 0.0167). However, there was a significant effect of Eigenposture (𝑝 <
37
0.001, 𝛼𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 = 0.0167). Additionally, there was no significant interaction effect (𝑝 =
0.2553). Post-hoc paired-sample t-tests showed a significant difference of variance between
all three Eigenpostures (between Eigenposture 1 and 2, 𝑝 < 0.001; between Eigenposture
1 and 3 𝑝 < 0.001; and between Eigenposture 2 and 3 𝑝 < 0.001, 𝛼𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 = 0.0167).
Furthermore, additional post-hoc paired-sample t-tests on the across-subject mean of the
variance, accounted for by each Eigenposture, showed that the variance accounted for in
Eigenposture 2 differed significantly across the two experiments (𝑝 = 0.015, 𝛼𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 =
0.0167). This was not the case for the variance of Eigenposture 1 (𝑝 = 0.966, 𝛼𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 =
0.0167), or Eigenposture 3 (𝑝 = 0.869, 𝛼𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 = 0.0167). The second Manipulation
Eigenposture accounted for less variance than the second Reach-and-Grasp Eigenposture.
As a result, subsequent Manipulation Eigenpostures had greater variance accounted for than
their corresponding Reach-and-Grasp Eigenpostures (see table 2.1).
2.3.2 Eigenpostures
All subjects’ first three Eigenpostures can be seen in Figure 2-2. The first Eigenposture
was dominated by a strong flexion of the MCP and PIP joints. Additionally, there was a
strong associated thumb rotation. These associated movements are typical of any grasp, as
it represents the general opening and closing of the human hand. This is shown in Figure
2-3.
Furthermore, the first Eigenposture was very repeatable across subjects as the average
standard error from the mean was ∼0.7 and ∼0.6 degrees in the Reach-and-Grasp and
Manipulation experiments, respectively. This standard error is small, considering that the
joint angles reached 25 degrees in magnitude. This standard error is dominated by the palm
arch, which has a standard error of 2∘ and 1.7∘ across all subjects in the two Experiments.
Comparison of the average first Eigenposture, across all subjects for a given experiment,
shows that in manipulation, the MCP joints had greater extension by ∼2 degrees, while the
PIP joints had greater extension in the reach to grasp case by ∼2 degrees. The greatest
difference between the two experiments was seen in the palm arch, where the Manipulation
Experiment had a 3.3∘ increase. This is shown in Figure 2-4.
The second Eigenposture was dominated by a strong thumb rotation, which had average
38
values of 36 and 23 degrees for the Reach-and-Grasp and Manipulation Experiments, respec-
tively. Additionally, Eigenposture 2 detected strong flexion in the MCP of the ring and little
fingers. This is shown in Figure 2-5.
Subject variability in the second Eigenposture is greater than that of the first, as seen
by the increase in the standard error from the mean. Here, subjects have average standard
errors of 1.6 and 2.92 degrees in the Reach-and-Grasp and the Manipulation Experiments,
respectively. In the Reach-and-Grasp Experiment, the standard error was dominated by the
palm arch (∼3∘), while standard error in the Manipulation Experiment was dominated by
thumb rotation (∼8∘).
Comparison of the average subject’s second Eigenposture shows an 11∘ decrease in thumb
rotation, and a 15∘ decrease in the flexion of the index MCP joint from reach-and-grasp to
manipulation. An average 7∘ increase in the flexion of the four fingers’ PIP joints from
the Reach-and-Grasp to the Manipulation Experiment was also observed. This is shown in
Figure 2-6.
Subject variability greatly increased from Eigenposture 2 to Eigenposture 3 (Figure 2-7).
The average standard error in the Reach-and-Grasp Experiment was 2.2∘ and was dominated
by a 7∘ standard error in palm arch. The average standard error in the Manipulation
Experiment was 3∘ and was dominated by an 8∘ standard error in palm arch.
In the Reach-and-Grasp Experiment, the third Eigenposture was characterized by strong
thumb rotation, MCP extension, and PIP flexion. However, MCP extension and PIP flexion
decreased in the manipulation case, while thumb rotation increased.
In the third Eigenposture, average standard error was 3.5∘ in the Reach-and-Grasp Ex-
periment and 3.6∘ in the Manipulation Experiment. Subjects were so variable that further
analysis was deferred.
2.3.3 Reach-and-grasp vs Manipulation Correlation Coefficients
To compare Eigenpostures across the two experiments, the correlation matrix was computed
using equation 2.4, and its absolute value was plotted in Figure 2-9. The bottom right of
the figure shows the average of all the subjects’ correlation matrices. The correlation in the
first Eigenposture is high, as all subjects had a correlation greater than 0.9. Between both
39
Figure 2-2: All subjects’ first three Eigenpostures.
40
41
Figure 2-3: All subject’s first Eigenposture. Left: The Reach-and-Grasp Experiment. Right: The Manipulation Experiment.
42
Figure 2-4: Top: A comparison of the average Reach-and-Grasp first Eigenposture to the Manipulation first Eigenposture
Bottom: The absolute error between the average Reach-and-Grasp first Eigenposture and the Manipulation first Eigenposture
43
Figure 2-5: All subject’s second Eigenposture. Left: The Reach-and-Grasp Experiment. Right: The Manipulation Experiment.
44
Figure 2-6: Top: A comparison of the average Reach-and-Grasp second Eigenposture to the Manipulation second Eigenpos-
ture Bottom: The absolute error between the average Reach-and-Grasp second Eigenposture and the Manipulation second
Eigenposture
45
Figure 2-7: All subject’s third Eigenposture. Left: The Reach-and-Grasp Experiment. Right: The Manipulation Experiment.
46
Figure 2-8: Top: A comparison of the average Reach-and-Grasp Third Eigenposture to the Manipulation third Eigenposture
Bottom: The absolute error between the average Reach-and-Grasp third Eigenposture and the Manipulation third Eigenposture
experiments and across all subjects, the 23𝑟𝑑 eigenvector was always a zero vector; thus, it
was excluded.
In Table 2.2, the diagonal correlation coefficients of the first 10 Eigenpostures are re-
ported. These are the diagonal values of the graphs in Figure 2-9. If a subject performed
similarly in both tasks, then these values should show high correlations (∼1). For higher-
order Eigenpostures, they do not. Generally, as the order of the Eigenposture increased, this
correlation decreased.
Furthermore, the confidence intervals of the first three Eigenpostures for each subject
produced by Fisher’s Transformation are shown in Table 2.3. The confidence intervals that
contained 0.99, considered to be statistically equal, and the corresponding correlation coef-
ficient are highlighted in both tables. It can be seen that only the first Eigenposture values
were found to be statistically equal.
Diagonal Correlation Coefficients
Subject E1 E2 E3 E4 E5 E6 E7 E8 E9 E10
1 0.989 0.755 0.702 0.341 0.255 0.144 0.106 0.294 0.742 0.037
2 0.985 0.311 0.152 0.529 0.067 0.464 0.078 0.212 0.165 0.32
3 0.986 0.904 0.75 0.516 0.034 0.409 0.071 0.237 0.025 0.066
4 0.972 0.923 0.802 0.064 0.537 0.415 0.348 0.324 0.525 0.665
5 0.983 0.823 0.556 0.523 0.61 0.156 0.233 0.076 0.491 0.074
6 0.972 0.359 0.235 0.561 0.603 0.497 0.448 0.1 0.428 0.212
7 0.989 0.907 0.523 0.689 0 0.207 0.24 0.558 0.451 0.561
Mean 0.982 0.712 0.531 0.461 0.301 0.328 0.218 0.257 0.404 0.277
Standard
Error 0.003 0.1 0.096 0.076 0.105 0.058 0.054 0.061 0.09 0.095
Table 2.2: Diagonal Eigenposture correlation coefficients. Highlighted values are considered
statistically equal to 0.99.
It can be seen that in Figure 2-9, occasionally, the sub- or super-diagonal terms showed a
higher correlation than the diagonal term. This was most notable in Eigenpostures 2 and 3
of subjects 2 and 6. Since the Eigenpostures were ordered by variance, it is possible that the
ordering did not provide the most accurate comparison across subjects. To better compare
the Eigenpostures, Figure 2-10 plots a reordered correlation matrix. This matrix was pro-
duced using Equation 2.4 as well. However, the Manipulation Eigenpostures, in matrix 𝑉2,
were reordered such that when another Manipulation Eigenposture had a higher correlation
47
Figure 2-9: A plot of all the subjects’ Eigenposture correlations between experiment. The
bottom right plot shows the average across all subjects.
48
Confidence Intervals
Subject E1 E2 E3
1 0.9732 0.9953 0.497 0.89 0.4085 0.8644
2 0.9634 0.9936 -0.1163 0.6409 -0.2779 0.5307
3 0.9668 0.9942 0.7831 0.9587 0.4895 0.8879
4 0.9345 0.9883 0.8246 0.9672 0.5825 0.9126
5 0.9599 0.9929 0.6209 0.9221 0.1864 0.7876
6 0.9352 0.9885 -0.0624 0.6718 -0.1962 0.5901
7 0.9749 0.9956 0.7904 0.9603 0.1408 0.7692
Table 2.3: Correlation coefficient confidence intervals. Highlighted values are considered
statistically equal to 0.99.
Algorithm 1 Algorithm to appropriately reorder Eigenpostures based on best correlations.
1: Produce correlation matrix (Equation 2.4): 𝐶 = 𝑉 𝑇𝑀𝑎𝑛𝑖𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛×𝑉𝑅𝑒𝑎𝑐ℎ−𝑎𝑛𝑑−𝐺𝑟𝑎𝑠𝑝
2: get matrix dimensions: [𝑚,𝑛] = 𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛(𝐶)
3: for i = 1,2,...,n-1
4: if 𝐶𝑖,𝑖+1 > 𝐶𝑖,𝑖 OR 𝐶𝑖+1,𝑖 > 𝐶𝑖,𝑖
5: Swap 𝑉𝑀𝑎𝑛𝑖𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 columns i and i+1
6: Recompute correlation matrix: 𝐶 = 𝑉 𝑇𝑀𝑎𝑛𝑖𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 × 𝑉𝑅𝑒𝑎𝑐ℎ−𝑎𝑛𝑑−𝐺𝑟𝑎𝑠𝑝
7: return: 𝐶
with a given Reach-and-Grasp Eigenposture than that Reach-and-Grasp Eigenposture’s Ma-
nipulation complement, the two Manipulation Eigenpostures were swapped. For example,
for subject 2, it was found that the third Manipulation Eigenposture had a higher correlation
with the second Reach-and-Grasp Eigenposture than the second Manipulation Eigenposture.
So, the third Manipulation Eigenposture was re-assigned to become the second Manipulation
Eigenposture and vice-versa. The algorithm to do this is described in Algorithm 1. Figure
2-10 is supplemented by Table 2.4, where the diagonal correlation coefficients of the first ten
Eigenpostures, with this correction, are reported.
The confidence intervals of the first three Eigenpostures for each subject’s corrected
correlation produced by Fisher’s Transformation are shown in Table 2.5. The confidence
intervals that contained 0.99, considered to be statistically equal, and the corresponding
correlation coefficient, are highlighted in both tables. It can be seen that this correction did
not change which Eigenpostures were found to be statistically equal.
49
Figure 2-10: A plot of all the subjects’ Eigenposture correlations between experiment. Cor-
relations are corrected such that highest correlation lies along diagonal. The bottom right
plot shows the average across all subjects.
50
Corrected Diagonal Correlation Coefficients
Subject E1 E2 E3 E4 E5 E6 E7 E8 E9 E10
1 0.989 0.755 0.702 0.534 0.613 0.657 0.464 0.391 0.38 0.202
2 0.985 0.892 0.747 0.328 0.704 0.464 0.499 0.769 0.044 0.623
3 0.986 0.904 0.75 0.366 0.606 0.409 0.071 0.221 0.597 0.522
4 0.972 0.923 0.802 0.102 0.399 0.592 0.348 0.448 0.604 0.665
5 0.983 0.823 0.556 0.591 0.586 0.528 0.513 0.478 0.479 0.789
6 0.972 0.726 0.855 0.561 0.603 0.497 0.448 0.293 0.446 0.345
7 0.989 0.907 0.523 0.689 0.463 0.754 0.134 0.37 0.516 0.209
Mean 0.982 0.847 0.705 0.453 0.568 0.557 0.354 0.424 0.438 0.479
Standard
Error 0.003 0.03 0.047 0.075 0.039 0.045 0.068 0.066 0.072 0.087
Table 2.4: Corrected diagonal Eigenposture correlation coefficients. Highlighted values are
considered statistically equal to 0.99.
Confidence Intervals
Subject E1 E2 E3
1 0.9732 0.9953 0.497 0.89 0.4085 0.8644
2 0.9634 0.9936 0.7592 0.9537 0.4832 0.8861
3 0.9668 0.9942 0.7831 0.9587 0.4895 0.8879
4 0.9345 0.9883 0.8246 0.9672 0.5825 0.9126
5 0.9599 0.9929 0.6209 0.9221 0.1864 0.7876
6 0.9352 0.9885 0.4472 0.8759 0.6845 0.9371
7 0.9749 0.9956 0.7904 0.9603 0.1408 0.7692
Table 2.5: Corrected correlation coefficient confidence intervals. Highlighted values are con-
sidered statistically equal to 0.99.
2.4 Discussion
This study focused on comparing the Eigenpostures of two experiments. In the first exper-
iment, subjects reached for and grasped an object, and in the second experiment, subjects
manipulated those objects. It was hypothesized that the Eigenpostures from Reach-and-
Grasp would not differ from those during manipulation. The results here suggest that this
is true for the first Eigenposture, but not the higher-order ones.
51
2.4.1 Limitations
The study presented here suffers from a few limitations. Although it was intended to
broadly compare Reach-and-Grasp synergies to those used during object manipulation, this
study only considered four objects, while other synergy studies considered more than 40
[Santello et al., 1998, Santello et al., 2002, Weiss and Flanders, 2004]. Thus, it is important
to emphasize that the results presented here should be interpreted only in the context of
wire-harness installation.
Second, the Eigenpostures were compared across the two tasks, rather than on a per-
object basis. Here, the use of one tool was not specifically isolated and compared to the
reach-and-grasp of that tool; rather, the reach-and-grasp of all objects has been compared
to the manipulation of all objects. Furthermore, in the Manipulation Experiment, subjects
had to manipulate objects that were not part of the Reach-and-Grasp Experiment. For
example, in Step 3, subjects had to use U-brackets and screws to attach the wire to the
mock electrical cabinet. Unfortunately, those U-brackets and screws were not included in
the Reach-and-Grasp Experiment. However, those operations were generally conducted by
the non-dominant (left) hand, and those data were not presented in this work.
2.4.2 The First Eigenposture
Consistent with much of the synergy work first introduced by Santello et al. (see Figure 1-3)
[Santello et al., 1998], the first Eigenposture, in both of the experimental tasks presented
here, has strong flexion of the MCP and PIP joints (Figure 2-3). This result has been found
across numerous studies that look at hand posture in grasping, both with and without contact
[Mason et al., 2001, Santello et al., 1998, Santello et al., 2002]; this even includes signing in
American sign language [Weiss and Flanders, 2004]. This first synergy is generally compared
to Napier’s power grasp. A power grasp is when flexed fingers and the palm clamp down
on an object while the thumb applies counter pressure [Napier, 1956]. Typical examples
of a power grasp include gripping a hammer or holding a bottle. The presence of this
Eigenposture, in both of these experiments and prior studies, demonstrates the robustness
of these results. As this Eigenposture was found across many diverse experiments involving
52
varying task constraints, it furthers the claim that this is the result of either neural control
or biomechanics, rather than constraints produced by object physics and task requirements.
2.4.3 The Second Eigenposture
The second Eigenposture was generally dominated by a strong thumb rotation (Figure 2-
5). This too is consistent with other studies of hand postural synergies (see Figure 1-3)
[Santello et al., 1998, Santello et al., 2002, Weiss and Flanders, 2004]. In other researchers’
postural synergy experiments, the second synergy was also characterized by a slight flexion
of the index and middle finger. This typical second synergy has been compared to Napier’s
precision grasp. A precision grasp is when the fingertips, especially of the middle and index
fingers, and thumb pinch an object between them [Napier, 1956]. A typical example of a
precision grasp is holding a pencil. Napier’s precision grasp and Santello’s studies of synergies
all focus solely on grasping as opposed to manipulation. Thus, the second Eigenposture in
the Reach-and-Grasp Experiment might be expected to be more consistent with a precision
grasp as opposed to the second Eigenposture during manipulation. In the Manipulation
Experiment, the second Eigenposture had a slight flexion of the index and middle fingers
while in the Reach-and-Grasp Experiment, it did not. Additionally, the second Eigenposture
had less inward thumb rotation during manipulation than it did during reach-and-grasp.
Both of these Eigenpostures are consistent with Napier’s precision grasp. However, the
pinching in manipulation requires more flexion of the index and middle fingers and less
rotation of the thumb than pinching in reach-and-grasp. Reasons why this occurs should be
further explored.
2.4.4 Variance
The first two Eigenpostures account for >90% of the variance in grasp postures in both exper-
imental tasks. This value is significantly greater than Santello andWeiss’s 75% [Santello et al., 1998,
Weiss and Flanders, 2004]. However, this may be due to the fact that this experiment used
5 objects as opposed to their 57 and 46, respectively.
Statistically, it was shown that the second Reach-and-Grasp Eigenposture accounted for
53
significantly more variance than the second Manipulation Eigenposture (p = 0.015). Recall
that the second Eigenposture was dominated by an inward thumb rotation. Thus, one can
conclude from these results that thumb rotation was more important in grasping than in
manipulation. This is an interesting result, because it is believed that the opposable thumb
is what gives humans their unique ability to manipulate tools [Marzke, 1997, Susman, 1994,
Diogo et al., 2012].
Furthermore, it took more eigenvalues to account for the same variance in the Manipula-
tion Experiment than it did in the Reach-and-Grasp Experiment (Table 2.1). This suggests
that a greater variety of hand postures was used when manipulating a tool or object as op-
posed to reaching to pick it up. Additionally, the significantly greater variance in Reach-and-
Grasp Eigenposture 2 led to more variance accounted for by the higher-order Eigenpostures
in manipulation. This may be because manipulation requires more variable hand kinematics
than reach-and-grasp, which also suggests that high-order Eigenpostures may be more im-
portant in manipulation than in reach-and-grasp. Intuitively, this would be consistent with
the need for high dexterity in tasks that require object manipulation.
2.4.5 Eigenposture Comparison
The main goal of this chapter was to compare the Eigenpostures across tasks. To do so, Equa-
tion 2.4 was used to compute a correlation matrix comparing each subject’s Eigenpostures
(Figure 2-9). Particular focus was centered on the diagonal values of the correlation matrix.
These values are reported in Table 2.2. There it can be seen that the correlation for the
first Eigenposture is consistently high across all subjects. This result is also demonstrated in
Figure 2-4, where the subjects’ average of the first Eigenposture in both experimental tasks
is compared. Five of the seven subjects were found to have statistically equal Eigenpos-
tures. The two subjects whose Eigenpostures were not, had a confidence interval upper limit
just below the 0.99 threshold (0.9883 and 0.9885). So in the case of the first Eigenposture,
the results show that there is no difference across tasks; in both experiments, the subjects
demonstrated a motion similar to that of a power grasp.
When comparing the second Eigenposture, there was a decrease in the computed corre-
lation coefficients (Table 2.2). Here, it was concluded that subjects’ synergies are not the
54
same across experiments, as no subjects reported statistically equal Eigenpostures. How-
ever, further insight into why this may be the case came from subjects 2 and 6, who had
remarkably lower correlation coefficients between the second Eigenposture of both experi-
ments. Specifically, they had correlation coefficients of 0.31 and 0.36. However, in Figure 2-9,
the off-diagonal correlation between these two subjects’ second and third Eigenpostures was
much stronger with correlations of about 0.8. Figure 2-2 shows the first three Eigenpostures
of all seven subjects. There, it can be seen that subjects 2 and 6 did not exhibit the large
thumb rotation that generally characterized the second Eigenposture in the Manipulation
experiment. However, this large thumb rotation does appear in the third Eigenposture of
the Manipulation Experiment. Thus, it is possible that results would change if these two
labels of the Eigenpostures were switched.
Figure 2-9 and Table 2.4 show the correlation coefficients that account for a correction
described by Algorithm 1. This correction fixes the above mentioned problem by ordering
the Eigenpostures based on correlation, as opposed to variance accounted for. One can ar-
gue that since the variance accounted for in Eigenpostures 2 and 3 was small relative to the
first Eigenposture, the difference between the two variances may be a random effect. Thus,
labeling the Eigenpostures according to variance is subject to error. However, across subject
two-way analysis of variance on the two experiments and first three Eigenpostures showed a
significant effect on Eigenposture variance (𝑝 < 0.001, 𝛼 = 0.05). Moreover, post-hoc paired-
sample t-tests showed a significant difference of the variance between all three Eigenpostures
(between Eigenposture 1 and 2, 𝑝 < 0.001; between Eigenposture 1 and 3 𝑝 < 0.001; and
between Eigenposture 2 and 3 𝑝 < 0.001; 𝛼𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 = 0.0167 for all tests). Since these vari-
ances were statistically different, one cannot argue that the Eigenpostures were mislabeled
by a noisy measurement. Nonetheless, analysis of the second and third Eigenposture, after
this correction, shows that these Eigenpostures still differed across experimental tasks.
2.4.6 Implications
The motivation for this experiment was to determine if one can use knowledge of existing
synergies to inform both the zero-force trajectory, 𝑋0, and impedance mapping 𝑍{·} to solve
Equation 1.1. The results presented here suggest that the first synergy, which behaves most
55
like a power grasp, was similar in both the previously reported reach-and-grasp experiments
and the wire-harness installation task studied here. However, this was not the case for the
second and third synergies.
From the above results, one might conclude that further exploration will be required to
be able to map the higher-order synergies to an associated hand impedance. However, at
the moment, it is unclear whether or not this may be necessary. As mentioned in Chapter
1, Gabicinni et al. [Gabiccini et al., 2011] found that only the lower-order synergies needed
to be modulated to establish a stable force grasp. Additionally, they were able to do this
with varying impedance values. Since the first Eigenposture in the wire-harness installation
task presented here is the same as the first Eigenposture in the reach-and-grasp task, which
accounted for ∼89% of the variance in the hand throughout manipulation, knowledge of this
synergy may be enough to make estimates of the hand impedance. While there currently
exists no method that relates a kinematic synergy to an impedance, evidence has shown that
motion alone appears to be sufficient for humans to estimate stiffness [Huber et al., 2017,
Huber et al., 2019]. Using a kinematic synergy to estimate an impedance is a topic that
should be further explored, as this relationship is essential to implementing synergy-based,
whole-hand Interaction Capture.
2.5 Conclusion
In an effort to noninvasively measure motion, impedance, and force, it was proposed to
introduce a synergy-based, whole-hand interaction capture method. Using this method,
motion prior to contact was equated with the zero-force trajectory, and assumed to continue
through contact. However, synergies have not yet been explored in object manipulation.
Thus, this study focused on comparing Eigenpostures observed in two experiments: the
first, where subjects reached for and grasped a tool or object commonly used in wire-harness
installation, and the second, where subjects manipulated those objects and tools to install a
wire harness on a mock electrical cabinet. Upon comparing Reach-and-Grasp Eigenpostures
to Manipulation Eigenpostures, it was found that only the first Eigenposture was the same
across experimental tasks. Yet, this Eigenposture has been found to be the most important
56
for stable grasp [Gabiccini et al., 2011]. Furthermore, as this Eigenposture has been found
to account for ∼89% of variance of hand motion throughout this experiment, it is possible
that one can use hand mechanical impedance during this particular Eigenposture to estimate
the hand impedance during wire-harness installation. Thus, further analysis of the synergies
needed for stable object manipulation should be conducted.
57
58
Chapter 3
Visual Perception of Arm Stiffness:
Human Subject Study
3.1 Introduction
The interaction capture method explored in Chapter 2 requires an estimate of impedance in
order to use Equation 1.1 (𝐹 = 𝑍{𝑋0 − 𝑋}) to calculate force. How might that estimate
be obtained? Visual observation may be enough, and the evidence below supports that
conjecture.
When observing another human, one can only see their overt behavior (e.g., motion);
one cannot perceive the underlying neural inputs that generate the behavior. Yet, humans
have an astonishing ability to extract latent information by visually observing the move-
ment of others (for review, see Blake and Shiffrar 2007 [Blake and Shiffrar, 2007]). This
impressive ability has been best shown in copious point light animation studies, in which
subjects were shown the motion of only a small subset of points on the body. Even from
such sparse motion information, humans can easily determine intention from arm movement
[Pollick et al., 2001], distinguish emotion from patterns in dancing [Walk and Homan, 1984],
and identify individuals from gait patterns [Loula et al., 2005]. In the context of motor learn-
ing, Mattar and Gribble [Mattar and Gribble, 2005] showed that subjects, who observed the
motion of other humans reaching in an unseen force field, subsequently performed better
when reaching in a similar unseen force field. That particular study demonstrated humans’
59
ability to learn about novel force environments solely based on visual observation of kine-
matics during physical interaction. Huber et al. extended upon this idea and found that
humans could infer dynamic properties, such as joint stiffness, from multi-joint limb motion
[Huber et al., 2017, Huber et al., 2019]. However, there are still open questions regarding
how humans are able to make stiffness estimates based on motion. Knowledge of how sub-
jects were able to perform this task can better inform how to train a machine to estimate
stiffness using motion, which can be useful in noninvasive monitoring of wire-harness workers’
impedance during wire-harnessing.
The purpose of this chapter is to investigate the role of temporal information in the
visual perception of limb stiffness. Presented here are three new experiments in which the
velocity profile of the moving arm is distorted while keeping the motion paths the same as
in Huber et al.’s previous experiment [Huber et al., 2017, Huber et al., 2019]. As mentioned
in the introduction, it is commonly observed that hand tangential velocity changes logarith-
mically with hand path radius of curvature, the so-called 1/3 power law (also commonly
referred to as the 2/3 power law, depending on the formulation) [Huh and Sejnowski, 2015,
Viviani and Flash, 1995, Zago et al., 2018]. Humans are also sensitive to such velocity-
curvature patterns when visually perceiving and interpreting motion [Bidet-Ildei et al., 2006,
Viviani and Stucchi, 1992, Kandel et al., 2000b, Dayan et al., 2007]. Thus, to change the
velocity profile, the velocity-curvature relation was manipulated. In all three experiments,
participants increased their arm stiffness rating with the simulated elbow stiffness. More-
over, there was no significant difference in subjects’ ability to rate limb stiffness across the
experimental conditions, including Huber at al.’s original experiment with veridical veloc-
ity profiles. These results suggest that path information is the predominant factor used by
humans to visually estimate changes in limb stiffness, providing further insight into how hu-
mans interpret and learn from the motor actions of others. This, therefore, lays a foundation
for estimating stiffness based solely on geometric features.
60
3.2 Methods
3.2.1 Participants
A total of 30 participants (10 in each of 3 experiments) took part in the present study (15
males and 15 females with a mean age of 25.5 ± 5.6 years). Participants had a variety of
educational backgrounds, and none had any prior experience with the experimental task.
All participants gave written informed consent before the experiment. The experimental
protocol was reviewed and approved by the Institutional Review Board of the Massachusetts
Institute of Technology. Data of 10 participants collected as part of a prior study (Experiment
2; Huber et al. [Huber et al., 2019]), referred to here as the original experiment, was used
for comparison in the statistical analyses.
3.2.2 Experimental Protocol
In each trial, participants were instructed to observe a stick-figure display of a two-link
planar arm as it moved its endpoint along a closed path for 20s and then rate its stiffness
on a numeric scale from 1 (“least stiff”) to 7 (“stiffest”) (Figure 3-1). Note that the endpoint
path was not explicitly displayed. After submitting their rating, participants self-initiated
the next trial.
Each participant performed 30 trials. Participants were shown six unique arm motions,
each of which were simulated with a different value of elbow stiffness, repeated five times
in a blocked manner. The order was randomized within each block. After completing the
experiment, participants were asked to provide a written description of their strategy for
estimating “arm stiffness”. The whole experiment lasted approximately 20 minutes.
A custom MATLAB program (The Mathworks, Natick, MA) was used to simulate and
display the arm motions and to record participants’ stiffness ratings.
3.2.3 Simulated Arm Motions
The arm was modelled as a two-link planar manipulator moving in the vertical plane and was
driven with a controller comprised of two attractors, inspired by the proposal that human
61
Figure 3-1: Experimental set up. In each trial, participants were instructed to observe a
stick-figure display of a two-link planar arm move its endpoint along a closed path for 20s
and then rate its stiffness on a numeric scale from 1 (“least stiff”) to 7 (“stiffest”).
motor behavior is composed of dynamic primitives [Hogan and Sternad, 2012, Hogan, 2017].
The first attractor was a combination of an oscillatory primitive with mechanical impedance
in endpoint coordinates, acting to pull the endpoint along a circular path; the second was
a combination of a fixed-point primitive with mechanical impedance in joint coordinates,
acting to pull the limb to a nominal joint configuration.
The dynamics of this model were described as
𝑀(𝑞)𝑞 + 𝐶(𝑞, 𝑞)𝑞 + 𝑔(𝑞) = 𝜏 (3.1)
where 𝑞, 𝑞, 𝑞 ∈ R2×1 are the joint angular positions, velocities, and accelerations, respectively.
𝑀(𝑞) ∈ R2×2 is the inertia matrix; 𝐶(𝑞, 𝑞) ∈ R2×2 are the Coriolis and centrifugal terms;
𝑔(𝑞) ∈ R2×1 are the gravitational terms; and 𝜏 ∈ R2×1 are the controller joint torques. The
length, mass, center of mass, and moment of inertia parameters, for the two links, were
chosen to match the forearm and upper arm of an average human male as described in
62
Zatsiorsky 2002 [Zatsiorsky, 2002]. The controller joint torques 𝜏 were determined by
𝜏 = 𝐽(𝑞)𝑇𝐾𝑥(𝑥𝑟 − 𝑥)− 𝐽(𝑞)𝑇𝐵𝑥?̇? + 𝐾𝑞(𝑞 − 𝑞) (3.2)
⎣⎡ ⎤ ⎡ ⎤
𝑟
0.1 cos 20𝜋𝑡⎦ ⎣𝜋𝑥 = 3𝑟 , 𝑞𝑟 = 4⎦ (3.3)
⎡ ⎤0.1 sin
20𝜋𝑡 𝜋
3⎡ ⎤ 4 ⎡ ⎤
500 0 10 0 0 0
𝐾𝑥 = ⎣ ⎦ , 𝐵 = ⎣ ⎦ , 𝐾 = ⎣ ⎦𝑥 𝑞 (3.4)
0 500 0 10 0 𝐸
where 𝑥, ?̇? ∈ R2×1 were the endpoint (i.e., hand) positions and velocities, respectively.
𝐽(𝑞) ∈ R2×2 was the Jacobian matrix; 𝑥𝑟 was the zero-force endpoint position, which followed
a circular path; 𝑞𝑟 was the zero-force joint configuration, which was constant; 𝐾𝑥 and 𝐵𝑥
were the endpoint stiffness and damping matrices, respectively; 𝐾𝑞 was the joint stiffness
matrix; and 𝐸 ∈ R≥0 was the value in the joint stiffness matrix corresponding to the elbow
joint. The six unique arm motions were generated by setting 𝐸 = {0, 10, 20, 30, 40, 50}
Nm/rad (Figure 3-2). The range of elbow stiffness values used is similar to those reported in
human studies [Bennett et al., 1992, Lacquaniti et al., 1982, Mussa-Ivaldi et al., 1985]. The
dynamic simulation of the arm used in this study was identical to that used in Experiment
2 of the previous Huber et al. [Huber et al., 2019] study.
While all participants observed the arm move so that its endpoint followed the same six
endpoint paths (Figure 3-2), the velocity profiles along those paths varied across experiments
(Figure 3-3). Participants, in the original experiment, saw the endpoint move with a velocity
profile governed by the aforementioned dynamic simulation. In Experiment 1, the tangential
velocity of the endpoint, 𝑣, was set to a constant value of 0.185m/s for all six motions. In
Experiment 2, tangential velocity varied as a power of the endpoint path’s radius of curvature
𝑅(𝑡):
𝑣(𝑡) = 𝐾𝑅(𝑡)−1/3 (3.5)
where 𝐾 was the velocity gain tuned to match the period of arm motions in the constant
experiment. In this condition, the speed of the endpoint increased with the curvature of
its path. Note that this is the inverse of the typical power law relation between speed and
63
Figure 3-2: The six endpoint motions of the simulated arm in each experiment. Elbow
stiffness (E) was varied. During the experiments, participants only saw the moving limb and
were not shown the endpoint traces displayed here.
64
Figure 3-3: The simulated endpoint velocity profiles. In the original experiment, the velocity
profiles followed the veridical power-law relationship. In Experiment 1, the velocity profiles
were constant. In Experiment 2, the velocity profiles followed the inverse of the veridical
power-law relationship, and in Experiment 3, no particular velocity profile was followed.
curvature observed in human motor behavior (for review see Zago et al. 2018). In Experiment
3, the same temporal signal (the time series used to maintain constant tangential velocity
for endpoint path generated with 𝐸 = 50 Nm/rad) was assigned to the six different endpoint
paths.
While the period of arm motion increased slightly across stiffness conditions in Experi-
ments 1 and 2 (from 3.33 to 3.68 seconds), it was constant in the original experiment (3.33s)
and Experiment 3 (3.68s).
3.2.4 Task Instruction
Participants were intentionally not provided with any details regarding the underlying con-
troller. Prior to the start of the experiment, participants were not presented with examples
of “more” and “less” stiff arm motions. They also did not receive feedback regarding the
65
accuracy of their ratings at any point during the experiment. In the event that a partic-
ipant was unsure of what the term stiffness meant, they received the following definition:
“Stiffness is the extent to which an object resists deformation or deflection in response to an
applied force. A stiffer object has higher resistance to deflections than a less stiff object.”
This instruction was provided to 8 of the 30 participants in Experiments 1, 2, and 3.
After subjects had viewed all the simulations, they were asked to write down the strategy
they used to make their stiffness estimates.
3.2.5 Statistical Analysis
To assess whether the ratings were similar across experiments, a 4 (experiment) × 6 (joint
stiffness) × 5 (block) analysis of variance (ANOVA) was conducted on the arm stiffness
rating for each experiment. ‘Experiment’ was a between-subject factor, and ‘joint stiffness’
and ‘block’ were within-subject factors. The Greenhouse-Geisser correction was applied to
the within-subject factors.
In addition, a linear model of stiffness rating, as a function of joint stiffness, was applied
to the data for each subject. A one-way ANOVA of the model’s coefficient of determination
(𝑅2), which measures the model’s goodness of fit, was conducted with ‘experiment’ as a
between-subjects factor.
In all statistical tests, the significance level was set to 𝑝 < 0.05. Statistical analyses were
performed using SPSS Statistics for Windows, Version 24.0 (IBM Corporation, Armonk,
NY).
3.2.6 Self-Report Analysis
When asking the subjects to write down the strategy they used to make their stiffness
estimates, specific interest was taken in whether subjects used path information or temporal
information and whether they looked at the joints or the endpoint. Thus, each subject’s
response was manually codified with four binary motion features of interest–joint motion,
endpoint motion, path information, and temporal information–to quantitatively assess what
type of motion information subjects used to rate stiffness. The criteria used to set the value
66
Feature Value Criteria
Use of the words or phrases:
-"distance"
’yes’ -"displacement"
-"range of motion"
Path -"angle"
Information ’no’ Did not meet the criteria for ’yes’
Use of the words:
’yes’ -"speed"-"rate"
Temporal -"acceleration"
Information ’no’ Did not meet the criteria for ’yes’
Use of the words:
-"joint"
-"shoulder"
’yes’ -"elbow"
-"angle"
Joint A hand-drawn picture of the arm with a pointer to at least one of the joints
Motion ’no’ Did not meet the criteria for ’yes’
Use of the words:
-"endpoint"
’yes’ -"hand"
Endpoint A hand-drawn picture of the arm with a pointer to the endpoint
Motion ’no’ Did not meet the criteria for ’yes’
Table 3.1: The criteria used to encode the type of information subjects reported using to
rate stiffness.
of each binary feature as either ’yes’ or ’no’ are presented in Table 3.1.
3.3 Results
A three-way ANOVA revealed a significant effect of simulated joint stiffness [𝐹 (1.64, 59.184) =
85.22, 𝑝 = 1.27𝑒 − 16]. Across all experimental conditions, participants increased their
stiffness rating with the simulated joint stiffness used to generate the arm motion paths
(Figure 3-4). However, the remaining effects and interactions were not significant [block:
𝐹 (2.67, 96.11) = 0.35, 𝑝 = 0.77; speed profile: 𝐹 (3, 36) = 0.42, 𝑝 = 0.74; simulated joint
stiffness × block: 𝐹 (11.40, 410.36) = 1.21, 𝑝 = 0.28; simulated joint stiffness × speed profile:
67
𝐹 (4.93, 59.18) = 1.22, 𝑝 = 0.31; block × speed profile: 𝐹 (8.01, 96.11) = 0.22, 𝑝 = 0.99;
simulated joint stiffness × speed profile × block: 𝐹 (34.20, 410.36) = 0.95, 𝑝 = 0.58].
Figure 3-5 shows each individual subject’s stiffness ratings for every motion path simu-
lated with a different joint stiffness value, along with the linear model fit to each subject’s
data and the corresponding 𝑅2 value. The ability to estimate stiffness from motion varied
across subjects.
The 𝑅2 value calculated for each subject denotes the fraction of variability in the stiffness
rating measurements accounted for by variability in the independent variable, simulated joint
stiffness.
Figure 3-6 shows the average coefficient of determination, 𝑅2, of subjects across experi-
ments. A one-way ANOVA revealed that there was no significant effect of speed profile on
the 𝑅2 values [𝐹 (3, 36) = 1.04, 𝑝 = 0.39].
These results indicate that the speed profile of the arm motions had no statistically
detectable influence on participants’ stiffness rating abilities.
68
Figure 3-4: Group stiffness rating results. In all four experiments, there was a significant
positive linear effect of joint stiffness on the arm stiffness ratings. Solid lines show the
average arm stiffness rating across subjects within each experiment. The dashed line shows
the average stiffness rating across participants in all experiments. Error bars represent ± 2
standard errors.
69
Figure 3-5: All the individual subjects’ stiffness ratings across simulated joint stiffness in all
four experiments. Larger dots indicate greater response frequency. The black lines represent
a linear fit of each subject’s data. The coefficient of determination, 𝑅2, is also reported.
70
Figure 3-6: Average coefficient of determination, 𝑅2, across experiment. Error bars show ±
2 standard errors. There was no significant effect of speed profile on the 𝑅2 values.
71
Figure 3-7: A histogram of the features used in the subjects’ classified self report. (b) 𝑝
represents that path information was used; 𝑡 represents that trajectory information was used;
𝑒 represents that endpoint information was used; and 𝑗 represents that joint information was
used.
3.3.1 Subjects’ Self-Reported Results
In subjects’ self-reports of their strategy used to estimate stiffness, more subjects reported
using path information (𝑁 = 18) than temporal information (𝑁 = 7). Addtionally, more
subjects reported using joint motion (𝑁 = 16) than endpoint motion (𝑁 = 2) (Figure 3-7a).
The majority of subjects (𝑁 = 10) reported using both path information and joint motion
(Figure 3-7b). However, it is important to note that a large number of subjects (𝑁 = 9) did
not report using any of the four features.
Results of binomial regression found that the experiment did not significantly affect the
likelihood that subjects reported using any of the four motion features [path information:
𝜒2(2) = 0.84,𝑝 = 0.66; temporal information: 𝜒2(2) = 2.63,𝑝 = 0.27; joint motion: 𝜒2(2) =
1.08,𝑝 = 0.58; endpoint motion: 𝜒2(2) = 1.69,𝑝 = 0.53] (Figure 3-7b). Results of the
linear regression also found that none of the four motion features extracted from subjects
were significant predictors of a subject’s ability to rate stiffness (i.e., 𝑅2 value) [𝐹 (4, 29) =
1.04,𝑝 = 0.918] (Figure 3-8).
72
Figure 3-8: Subjects’ classified self report and the associated coefficient of determination,
𝑅2. Left: Path versus trajectory. Right: Joint versus endpoint.
3.4 Discussion
Previous work has shown that humans can estimate joint stiffness from visually observing
multi-joint limb motion [Huber et al., 2017, Huber et al., 2019]. The simulations in the pre-
vious work were generated by a model that mimics human motor control. Moreover, the
simulated motions generally follow the trend that tangential velocity increases with radius
of curvature (as seen in Figure 3-3). Given the resemblance of the simulation to neuromotor
control, it was suggested that to successfully make stiffness estimates, subjects relied on
their prior knowledge of human neuromotor control. Evidence has shown that there may be
shared resources used for both action execution and action perception, and those resources
may be affected by the power law relationship. The study presented here explored the extent
to which the power-law relationship affects the visual perception of stiffness. Moreover, this
study investigated the role of temporal information in the visual perception of limb stiffness.
Considerable research has shown that temporal information plays a key role in biologi-
cal motion perception and understanding [Bidet-Ildei et al., 2006, Viviani and Stucchi, 1992,
Kandel et al., 2000b, Maurice et al., 2018, Dayan et al., 2007]. Those studies demonstrated
that humans are sensitive to velocity-curvature patterns (specifically the 1/3 power-law)
when visually perceiving and interpreting motion. That understanding motivated three new
stiffness-perception experiments in which the velocity profile was distorted by manipulating
73
the velocity-curvature relationship of the moving arm, but keeping the motion paths the
same as in the previous experiment. In all three experiments, participants increased their
arm stiffness rating with the simulated elbow stiffness. Moreover, no significant difference in
subjects’ ability to rate limb stiffness across the experimental conditions was observed. This
remained true even when comparing the new experiments to the original experiment with
veridical velocity profiles. The results here indicate that path information is the predomi-
nant factor used by humans to visually estimate changes in limb stiffness, providing further
insight into how humans interpret and learn from the motor actions of others.
3.4.1 Limitations
While this study showed promising results, it is important to acknowledge its limitations.
The first limitation comes from using a discrete numerical scale to quantify subjects’ stiffness
ratings. It is well-known that such a quantization can lead to measurement error. In this
particular instance, the variance of error is inversely proportional to the number of rating
options (i.e. the greater the options the lower the noise). Here, subjects rated stiffness using
a seven-point Likert-type scale, which is vulnerable to such quantization error. Additionally,
the Likert-scale is vulnerable to response bias, which occurs whenever a person responds
systematically on some basis other than what the items were specifically designed to measure.
In this study, this occurred when subjects did not use the full range of the numeric scale.
In Figure 3-5, it can be seen that this occurred. Response bias can often be a result of
the central tendency bias, where subjects avoided selecting the most extreme results. Often,
subjects reported always looking for the most stiff and least stiff simulations that never came.
Both of these forms of bias worsen subject performance. However, despite these limitations,
subjects were still able to successfully perform the task.
It is also possible that since the definition of stiffness was open to interpretation by sub-
jects, this could have resulted in inter-participant variability. When subjects asked, they
were given the definition: “Stiffness is the extent to which an object resists deformation or
deflection in response to an applied force. A stiffer object has higher resistance to deflections
than a less stiff object.” This definition could have skewed subjects’ interpretations of “stiff-
ness,” leading to increased variability in the strategies used for producing stiffness ratings.
74
Only 8 subjects received the formal definition of stiffness, and statistical analysis showed
that whether or not subjects were given the formal definition had no effect on their ability
to estimate stiffness.
3.4.2 Theoretical Implications
Strictly speaking, identifying an object’s mechanical properties, such as stiffness, requires
contact. With only motion information, unambiguous measurements of stiffness are funda-
mentally impossible, since the same limb motion can be generated with an infinite number
of stiffness values. Presented here are three new experiments that show that humans can
perform this remarkable task. Subjects’ ability to estimate stiffness based solely on kine-
matic information demonstrates coupling between the output of motion behavior and the
modulation of underlying dynamics in the human neuromotor controller. Moreover, the work
presented here emphasizes the robustness of Huber et al.’s previous findings: humans can
estimate stiffness using only motion information. This observation reinforces the importance
of finding out how it’s done.
More importantly, this work demonstrates that humans can estimate stiffness from mo-
tion, despite changes in temporal information. Considerable work has been done to show that
trajectory information is important in human motion. Much of this work has focused on the
1/3 power-law, which states that during movement, the tangential velocity is proportional to
the 1/3 power of the radius of curvature ([Huh and Sejnowski, 2015],[Viviani and Flash, 1995]).
In the original Huber et al. experiment, the motion of the manipulator mimicked aspects
of human control, as it was produced by a controller that superimposed joint-space and
hand-space impedance [Hogan, 2017]. This controller produced a trajectory that generally
increased tangential velocity with radius of curvature (Original Experiment, Figure 3-3).
Despite a manipulation of this power-law in Experiments 1, 2, and 3, subjects were still able
to estimate stiffness. This observation indicates that the subjects’ prior knowledge used to
estimate stiffness was dominated by path shape, as opposed to trajectory information.
Although a one-way ANOVA revealed that there was no significant effect of speed profile
on the 𝑅2 values [𝐹 (3, 36) = 1.04, 𝑝 = 0.39], there are some trends that can be seen in
Figure 3-6. First, subjects in Experiment 1, with a constant velocity profile, performed
75
better (based on 𝑅2 value) than subjects in the original experiment, with the biological
velocity profile. That may suggest that velocity information acts as a distractor, which
further supports the point that subjects predominantly used path information to determine
their stiffness measurements. Moreover, subjects performed the worst in Experiment 3,
where the velocity profile did not follow a power-law relationship. This suggests that there
may exist a velocity profile, not explored here, that could prevent subjects from estimating
joint stiffness. Further experiments with more extreme velocity-curvature relations should be
conducted to determine if there exists a velocity profile that inhibits subjects from estimating
stiffness.
Out of all 40 subjects who conducted the 4 experiments, only 3 had a negative correlation
between their stiffness estimates and the simulated joint stiffness values. All three of those
subjects had a low coefficient of determination (𝑅2 = 0.10 in experiment 2, and 𝑅2 = 0.09
and 0.12 in experiment 3). Given the low coefficient of determination, it was concluded
that subjects did not detect some other motion characteristic in determining their stiffness
estimates; rather, they simply were unable to do the task.
If subjects with low 𝑅2 values were unable to do the task, statistical analysis on the 𝑅2
values may disguise a difference between subject performance across the experiment. To
check this, the one-way ANOVA across speed-profile was re-run, excluding subjects with
a low 𝑅2 value. Removing subjects with an 𝑅2 value less than 0.1 yielded no statistical
differences, and neither did removing subjects with an 𝑅2 value less than 0.2 or 0.3.
Subject Self-Reported Results
As mentioned in the Subject Self-Reported Results section, subjects generally reported using
path information in joint space when making estimates of stiffness. Considering that the
velocity profile changed across the four experiments, and that the joint stiffness at the elbow
was the variable being modulated, this response is intuitively consistent. However, it is
possible that subjects may be using a subconscious or implicit strategy that they are unable
to accurately articulate. Subjects did not know ahead of time that they would have to self-
report their strategy. This was done to prevent potential interference of conscious strategies.
The results in Figures 3-7 and 3-8 show that across all experiments, subjects primarily
76
looked at path information in joint space. However, linear regression found that none of the
four motion features extracted from subjects were predictors of subjects’ ability to rate stiff-
ness (i.e., 𝑅2 value) [𝐹 (4, 29) = 1.04,𝑝 = 0.918]. Looking at individual subjects’ performance
explains this. For instance, the subject in Experiment 3, who had a negative correlation and
an 𝑅2 value of 0.09, reported interest in solely path and joint information, while the only sub-
ject who reported interest in solely trajectory information (specifically speed) and endpoint
information had a coefficient of determination of 0.58 (Figure 3-5 Experiment 2). These
findings emphasize that subjects’ self-reported results should be interpreted with caution.
Moreover, further experiments should be conducted to determine which aspects of motion
subjects are using to estimate stiffness.
3.4.3 Practical Implications
Under the assumption that humans are using knowledge of their own controller to estimate
stiffness in this task, identifying conditions when humans can and cannot estimate stiffness
from motion path can be used to probe the form of the latent human neuromotor controller.
It has been shown that understanding the underlying neuromotor controller can aid in the de-
velopment of effective rehabilitation interventions [Igo Krebs et al., 1998, Krebs et al., 2002].
Thus, determining what velocity profile manipulations might prohibit humans from estimat-
ing joint stiffness is an important topic of further research.
Moreover, this work suggests a new way to determine limb stiffness based on easily mea-
surable motion characteristics. At the moment, measuring limb stiffness requires perturbing
the limb or measuring co-activation with EMG sensors and using models of how muscle
mechanical impedance varies with activation. However, knowing that humans can estimate
stiffness, just from observing motion, suggests that stiffness might be estimated from purely
visual data (e.g. using machine learning techniques, such as ‘deep neural networks’ driven
by camera-acquired data). When monitoring wire-harness installation workers, one might
quantify limb stiffness from easily accessible motion signals without impeding the worker.
77
3.5 Conclusion
This chapter showed that humans can reliably infer changes in limb stiffness from nontrivial
changes in multi-joint limb motion without force information or explicit knowledge of the
underlying limb controller. In all experiments, most participants increased their stiffness
rating with the simulated elbow stiffness. This result was robust, despite changes in the
velocity profile. Moreover, these results provide new insight into how humans interpret the
motor actions of others, suggesting that path, not trajectory, information is more important
to subjects when estimating stiffness. This exploration of how humans extract latent features
of neuromotor control, such as stiffness, from kinematics provides new insight into how
humans interpret the motor actions of others, and how it may aid the innovation of a machine
to do the same.
78
Chapter 4
Visual Perception of Arm Stiffness:
Implementation
4.1 Introduction
In tasks that require physical interaction and tool use, such as wire-harness installation, mod-
ulation of neuromuscular impedance1 has been shown to be important [Hogan and Sternad, 2012,
Hogan, 2017]. However, in practice, impedance measurement is difficult and encumbering
[Bennett et al., 1992, Guarín and Kearney, 2017, Lacquaniti et al., 1993, Lee and Hogan, 2015,
Lee et al., 2016, Rouse et al., 2014, Rouse et al., 2013, Van De Ruit et al., 2020]. One usu-
ally uses EMG, or applies a random perturbation and measures the resultant reaction. How-
ever, it has been shown that humans can observe a limb’s motion and make predictions about
its stiffness [Huber et al., 2017, Huber et al., 2019]. This is remarkable, because fundamen-
tally, physical contact is needed to estimate the underlying dynamics. Nevertheless, humans
are able to determine stiffness based purely on kinematic motion without knowledge of forces.
Chapter 3 extended Huber’s work and showed that humans are able to make stiffness esti-
mates despite manipulations of the velocity profile, suggesting that subjects used path (i.e.
displacement) information to make their estimates. Furthermore, subjects reported a higher
interest in joint motion as opposed to endpoint motion. Chapter 4 aimed to replicate the
1Mechanical impedance characterizes the dynamic relation between motion and a resultant force (i.e.
Equation 1.1) [Hogan, 1985c, Hogan, 1985b, Hogan, 1985a].
79
previous experiments using a supervised linear regression algorithm. Specifically, it explored
if a machine could learn to estimate stiffness by purely observing kinematic data, and if so,
what aspects of motion are important in determining stiffness.
Machine learning allows one to train a “machine” to make decisions, or predictions, based
on data. In simulation, data can be quickly generated in large quantities. The simulation-
based nature of the Huber et al. and Chapter 3 studies allow them to be addressed and
reproduced using machine learning. In particular, the machine can be trained using a su-
pervised linear regression algorithm. Supervised linear regression uses labeled data to find
a mapping from a given set of inputs to a given set of outputs. The study reported here
focused on finding the relation2 between motion and joint stiffness from simulated data.
To find a noninvasive minimally-encumbering way to measure joint stiffness during wire-
harness installation, a supervised learning algorithm was used to train a machine to estimate
simulated arm stiffness based on position, velocity, and acceleration information. The ma-
chine was trained using a data set that expanded on Huber et al. [Huber et al., 2019]. De-
pending on the data set it was trained on, the algorithm had a root-mean-square error that
ranged from 2 to 10 Nm/rad (4 to 20% of the maximum simulated stiffness). Furthermore,
the linear classifier that performed best had a higher emphasis on position information as op-
posed to velocity and acceleration information, which reinforced the conclusion expressed in
Chapter 3: path information dominates trajectory information in the perception of stiffness.
4.2 Methods
4.2.1 Simulated Data
To produce data, a two-link manipulator was simulated using MATLAB. The two-link ma-
nipulator was modeled to behave as a human arm moving in the frontal plane. Its motion
was governed by a zero-force trajectory, model dynamics, and command torque given by
equations 4.1, 4.2, and 4.3, respectively. This framework directly follows that of Huber et
2In machine learning, this relation is often referred to as the hypothesis. In the simple linear regression
presented here the hypotheses are numeric weightings of the features in the feature vectors
80
al. [Huber et al., 2017, Huber et al., 2019] and Chapter 3:
⎡⎣ ⎤ ⎡ ⎤𝐴 cos 2𝜋𝑓(𝑡 + 𝜑)𝑥 = ⎦ , 𝑞 ⎣𝜋4⎦𝑟 𝑟 = (4.1)
𝐴 sin 2𝜋𝑓(𝑡) + 𝑌 𝜋
4
𝑀(𝑞)𝑞 + 𝐶(𝑞, 𝑞)𝑞 + 𝑔(𝑞) = 𝜏 (4.2)
𝜏 = 𝐽(𝑞)𝑇𝐾𝑥(𝑥𝑟 − 𝑥)− 𝐽(𝑞)𝑇𝐵𝑥?̇? + 𝐾𝑞(𝑞𝑟 − 𝑞) (4.3)
𝐾 = ⎣
⎡ ⎦⎤ ⎡ ⎤ ⎡ ⎤500 0 10 0 𝑆 0
𝑥 , 𝐵 = ⎣ ⎦𝑥 , 𝐾 ⎣ ⎦𝑞 = (4.4)
0 500 0 10 0 𝐸
Here, 𝑥𝑟 is the zero-force endpoint position, and 𝑞𝑟 is the zero-force joint configuration, which
was constant. 𝐴 is amplitude, equaling 0.1 meters; 𝑓 is frequency; and 𝜑 is phase shift. 𝑌 was
an offset in the Ca√︁rtesian y-direction, and it adjusted for the different link lengths described
below (𝑌 = 𝐴 + 𝑙21 + 𝑙2
𝜋
2 − 2𝑙1𝑙2 cos , where 𝑙1 and 𝑙2 are the manipulator upperarm and4
forearm lengths, respectively). 𝐴, 𝑓 , 𝜑, and 𝑌 all guided the zero-force endpoint position.
𝑞, 𝑞, 𝑞 ∈ R2×1 are the joint angular positions, velocities, and accelerations, respectively.
𝑀(𝑞) ∈ R2×2 is the inertia matrix; 𝐶(𝑞, 𝑞) ∈ R2×2 represents the Coriolis and centrifugal
terms; 𝑔(𝑞) ∈ R2×1 is the gravitational term; and 𝜏 ∈ R2×1 is the controller joint torques. 𝑥,
?̇? ∈ R2×1 represents the endpoint (i.e., hand) positions and velocities, respectively. 𝐾𝑥 and
𝐵𝑥 are the endpoint stiffness and damping matrices, respectively; 𝐾𝑞 is the joint stiffness
matrix; and 𝐸 ∈ R≥0 is the value in the joint stiffness matrix corresponding to the elbow
joint; lastly, 𝑆 ∈ R≥0 is the value in the joint stiffness matrix corresponding to the shoulder
joint.
In total, 4050 simulations were run and then split into 6 data sets. They consisted of
3 different sets of link lengths, masses, and inertias, (values were drawn from Zatsiorsky
[Zatsiorsky, 2002] and chosen to represent the standard American child, adult male, and
adult female), 3 phase shifts (𝜑 = 0, 1/3, or 2/3), which varied the elliptical orientation,
and 3 frequency values (0.075, 0.5 and 2 rev/s), which varied speed. Lastly, for each of
those 1350 simulations, 3 experiments that varied stiffness were run. Following Huber et. al
[Huber et al., 2019] in Experiment 1, the shoulder stiffness was held constant at 1 Nm/rad,
while the elbow stiffness varied from 1 to 50 Nm/rad. In Experiment 2, the elbow stiffness
81
was held constant at 1 Nm/rad, while the shoulder stiffness varied from 1 to 50 Nm/rad. In
Experiment 3, both the shoulder and elbow stiffnesses varied simultaneously from 1 to 50
Nm/rad. From these simulations, time dependent vectors of position, velocity, speed, and
acceleration, in both Cartesian endpoint space and joint space, were produced. The joint
space data were then split from the Cartesian endpoint data. The kinematic data of each set
served as the feature vectors, and the joint and elbow stiffnesses served as the labels. As a
whole, 6 data sets of 1350 feature vectors and labels each were produced; specifically, those
sets were experiments 1, 2, and 3, each in both joint space and Cartesian endpoint space.
Initially, each feature vector consisted of a time-stamped set of positions, velocities,
and accelerations, all linearly interpolated to a set of 500 values. However, training 1500
features on a subset of 1350 training examples led to obvious structural error3. Additionally,
having such a high-dimensional feature set was computationally expensive. Thus, the feature
vector was reduced to consist of 24 values: in the joint space data, these values were the
max, min, mean, and standard deviation of the position, velocity, and acceleration of both
the shoulder and elbow joints, while in the Cartesian endpoint data, these values were the
max, min, mean, and standard deviation of the position, velocity, and acceleration of both
the x and y positions of the endpoint. Additionally, to improve computation, all data was
standardized4. Thus, there were 6 data sets of 1350 simulations, each with 24 feature vectors,
and a corresponding stiffness value. Lastly, each data set was divided into 3 random subsets,
where 60% of the original data set was training data, 20% was cross validation data, and
20% was testing data.
4.2.2 Supervised Learning Algorithm
Ultimately, a machine was trained to determine a linear mapping of features to stiffness
values. Thus, this problem was set up as a supervised linear regression problem. A su-
3Structural error occurs when the designed hypothesis class does not have the structure to perform well
on the given data. For example, training a linear hypothesis class on data produced using a sine wave will
have high structural error [Barzilay et al., 2019d].
4Standardizing data (also known as feature scaling) transforms the features into a space where all the
data for a particular feature has a mean of 0 and standard deviation of 1. Without this transformation, the
algorithm may have a hard time training when a particular feature is numerically much larger than other
features [Barzilay et al., 2019a]
82
pervised linear regression problem explores the space of possible hypotheses and finds the
hypothesis, 𝜃 ∈ R24×1, that minimizes a particular cost function, 𝐽 (i.e. 𝜃* = argmin 𝐽(𝜃))
𝑡ℎ𝑒𝑡𝑎*
[Barzilay et al., 2019d]. The cost function used here was squared loss: 𝐽(𝜃) = 1(𝜃𝑇𝑥 − 𝑦)
2
[Barzilay et al., 2019b], where 𝜃 ∈ R24×1 is the hypothesis class, 𝑥 ∈ R24×1 is the feature
vector, and 𝑦 ∈ R1×1 is the actual stiffness value of the modulated joint; 𝜃𝑇𝑥 ∈ R1×1 is the
algorithm’s prediction based on the hypothesis. The weights, 𝜃 ∈ R24×1, that minimized
squared error between the algorithm’s stiffness estimate and the actual stiffness can help
determine which motion characteristics are important in estimating stiffness.
The model was trained using gradient descent with a step size of 0.1, which was found
to be optimal via manual tuning. To train the model, algorithm 2 was used.
Algorithm 2 Supervised learning Algorithm
1: Find 𝜃𝑏𝑒𝑠𝑡 for some rando∑︁m fixed regularizer, 𝜆, using the training data, 𝐷𝑡𝑟𝑎𝑖𝑛.(𝜃𝑇𝑥− 𝑦)2
𝜃𝑏𝑒𝑠𝑡 = argmin
1 + 𝜆‖𝜃‖2
𝐷𝑡𝑟𝑎𝑖𝑛
𝜃 2
(𝑥,𝑦)∈𝐷𝑡𝑟𝑎𝑖𝑛
2: Find the best regularization, 𝜆*, parameter using 𝜃𝑏𝑒𝑠𝑡, and the the cross
validation data, 𝐷𝑣𝑎𝑙∑︁. 𝑇 2
𝜆* = argmin 1
(𝜃𝑏𝑒𝑠𝑡𝑥− 𝑦)
𝐷𝑣𝑎𝑙
𝜆 2
(𝑥,𝑦)∈𝐷𝑣𝑎𝑙
3: Find the optimal hypothesis, 𝜃*, using the optimal regularizer, 𝜆*, and the
the training data, 𝐷𝑡𝑟𝑎∑︁𝑖𝑛.
* (𝜃
𝑇𝑥− 𝑦)2
𝜃 = argmin 1 + 𝜆*‖𝜃‖2
𝐷𝑡𝑟𝑎𝑖𝑛
𝜃 2
(𝑥,𝑦)∈𝐷𝑡𝑟𝑎𝑖𝑛
4: Compute the test∑︁error using 𝜃* and the the test data, 𝐷𝑡𝑒𝑠𝑡.*𝑇(𝜃 𝑥− 𝑦)2
𝐸𝑟𝑟𝑜𝑟 = 1
𝐷𝑡𝑒𝑠𝑡 2
(𝑥,𝑦)∈𝐷𝑡𝑒𝑠𝑡
5: return: 𝜃* and 𝐸𝑟𝑟𝑜𝑟
Algorithm 2 was run on all 6 data sets individually. All programming and training was
done in MATLAB 2017b using a computer with a single Intel i7 core processor.
4.2.3 Analysis
After the algorithm was run on the data sets, two performance metrics were computed:
the root-mean-square (RMS) error and the coefficient of determination (𝑅2). Computing
the coefficient of determination allows for the results presented here to be compared to the
83
results presented by Huber et al. and in Chapter 3. Furthermore, to determine whether the
algorithm performed better using joint data or endpoint data, paired-sample t-tests were
conducted on the RMS error and the coefficient of determination, 𝑅2.
The values in 𝜃* ∈ R24×1 determine which motion characteristics were most important in
making a stiffness prediction. In accordance with Chapter 3, it was aimed to determine if a
greater emphasis on path information as opposed to trajectory information leads to better
stiffness estimates. To do so, for each of the 6 trained hypotheses, 𝜃*, the average magnitudes
of the position-related weights, 𝜃* *𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛, velocity-related weights, 𝜃𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦, and acceleration-
* *
related weights, 𝜃*𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛, were computed to produce the ratios:
𝜃𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 and 𝜃𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛* * .𝜃𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦 𝜃𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛
These ratios were then plotted with respect to the test RMS error. Finally, the coefficients of
determination between these ratios and the test RMS errors were computed, and hypothesis
testing was conducted to determine if this correlation is statistically significant.
All statistical analysis was conducted using MATLAB 2017b.
4.3 Results
Figure 4-1 shows the root-mean-square error of the trained linear classifier on each of the data
sets. The results were further divided by the test, validation, and training data. Specifically,
the RMS errors of the test data in the Cartesian experiments were 4.95, 4.93, and 2.46
Nm/rad in Experiments 1, 2, and 3 respectively, while the RMS errors of the test data in
the joint-space experiments were 8.01, 7.36, and 6.07 Nm/rad. A paired-sample t-test on
the RMS error showed that experiments using joint data performed statistically worse than
the experiments using Cartesian data (𝑝 < 0.001, 𝛼 = 0.05).
The bottom plots in Figure 4-1 show the optimal weights of each feature produced by the
algorithm. The magnitudes of these weights can be interpreted as the level of importance
of each feature in determining stiffness. Using the Cartesian data, the average magnitude of
the position related features, 𝜃*𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛, was 0.25, 0.41, and 0.25 for Experiments 1, 2, and 3,
respectively; the average magnitude of the velocity related features, 𝜃*𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦, was 0.28, 0.18,
and 0.05; and the average magnitude of the acceleration related features, 𝜃*𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛, was 0.06,
0.14, and 0.06. On the other hand, using the joint space experiment, the average magnitude
84
of the position related features, 𝜃*𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛, was 0.33, 0.48, and 0.30 for Experiments 1, 2, and
3, respectively; and the average magnitude of the velocity related features, 𝜃*𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦, was 0.13,
0.50, and 0.21; the average magnitude of the acceleration related features, 𝜃*𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛, was
0.19, 0.31, and 0.10. The algorithm learned the best by using the Experiment 3 Cartesian
Data set, in which it found position related features to be relatively more important than
the velocity and acceleration related features.
Figure 4-2 compares the ratio of the average magnitudes of the position-related and
*
velocity-related coefficients, 𝜃𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛* , and the ratio of the average magnitudes of the position-𝜃𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦
*
related and acceleration-related coefficients, 𝜃𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛* , to the test RMS error of that data𝜃𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛
set. Furthermore, a line of best fit and its computed coefficient of determination are shown.
There, it is seen that as this ratio decreases, error increases. Hypothesis testing on the 𝑅2
*
value of the 𝜃𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛* plot shows that this relation was not statistically significantly different𝜃𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦
*
from zero (𝑝 = 0.088,𝛼 = 0.05). However, hypothesis testing on the 𝑅2 value of the 𝜃𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛
𝜃*𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛
plot showed that this relation was statistically significant from zero (𝑝 = 0.044,𝛼 = 0.05).
These results demonstrate that algorithms that emphasize position-related features more
than velocity- and acceleration-related features perform better. However, this conclusion
may have been influenced by the small sample size.
Figure 4-3 shows the learning curves of each of the data sets, by plotting the RMS Error
as a function of each training iteration. In all six data sets, the error started to decrease
less rapidly around the 200𝑡ℎ iteration. However, in the joint space data sets, the test
error and cross validation error seemed to not converge as well as they did in the Cartesian
experiments. A pair-wise t-test on the magnitude of the difference between the last (1000𝑡ℎ
iteration) cross validation and the last (1000𝑡ℎ iteration) test values showed no statistical
difference in convergence between the classifiers trained using Cartesian data and joint data.
Figure 4-4 shows the linear classifier’s stiffness predictions versus the simulated stiffness
after being trained on each data set. Additionally, the associated coefficient of determination,
𝑅2, which measures the model’s goodness of fit, is shown. A paired-sample t-test on the 𝑅2
values showed that experiments using joint data performed significantly worse than the
experiments using Cartesian data (𝑝 < 0.001, 𝛼 = 0.05). Again, this conclusion may have
been influenced by the small sample size. This figure can be contrasted to Figure 3-5 of
85
Chapter 3, where human subjects reported predicting stiffness values using path information
in joint space.
86
87
Figure 4-1: Top: The root-mean-square error found from the trained algorithm. Bottom: The optimal weights produced
by the learning algorithm. Left values are position (pos) related, middle values are velocity (vel) related, and right values are
acceleration related (accel). Abbreviations X and Y stand for coordinate values of the endpoint in Cartesian space. Abbreviations
E and S stand for elbow and shoulder, respectively, and abbreviations Min, Max, Mean, and Std stand for the minimum,
maximum, mean, and standard deviation.
88
Figure 4-2: For all 6 trained algorithms. Left: The ratio of the average magnitude of the position-related weights, 𝜃*𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛,
divided by the average magnitude of the velocity-related weights, 𝜃*𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦, is plotted with respect to the test RMS error. Right:
The ratio of the average magnitude of the position-related weights, 𝜃*𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛, divided by the average magnitude of the acceleration-
related weights, 𝜃*𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛, is plotted with respect to the test RMS error. Both: Additionally, a line of best fit is produced, and
its associated coefficient of determination, 𝑅2, is displayed.
89
Figure 4-3: The learning curves of the trained data sets. In Experiment 1, the shoulder stiffness was held constant at 1 Nm/rad,
while the elbow stiffness varied from 1 to 50 Nm/rad. In Experiment 2, the elbow stiffness was held constant at 1 Nm/rad,
while the shoulder stiffness varied from 1 to 50 Nm/rad. In Experiment 3, both the shoulder and elbow stiffnesses varied
simultaneously from 1 to 50 Nm/rad. Top: the algorithm trained with Cartesian data. Bottom: The algorithm trained with
Joint Data.
90
Figure 4-4: A plot of the actual vs the training algorithm predicted stiffness values. The red line shows a linear fit to the data.
The associated coefficient of determination, 𝑅2, of that fit, is listed in each plot title.
4.4 Discussion
The first discussion point stems from the root-mean-square error plots in Figure 4-1. In the
Cartesian case, these values range from 2 to 6 Nm/rad, or 4 to 12% of the maximum simulated
stiffness. Considering that the actual scale ranges from 1 to 50 Nm/rad, one may conclude
that the algorithm is performing well. However, in the joint space case, these values range
from 6 to 10 Nm/rad, or 12 to 20% of the maximum simulated stiffness. Considering the scale
of the problem, these errors are unsatisfactory. Furthermore, this difference was considered
statistically significant (𝑝 < 0.001,𝛼 = 0.05). Since the algorithm trained better using
Cartesian data as opposed to joint data, one may conclude that endpoint Cartesian space may
contain more information about stiffness than does joint space. This observation contradicts
what human subjects reported in Chapter 3. There, subjects reported a higher interest in
the joint motion as opposed to the endpoint motion. In Chapter 3, it was mentioned that
the subjects’ self-reports may be biased, and that subjects may be unable to accurately
articulate their strategy used in estimating stiffness. The results here justify wariness in
interpreting the self-reported results in Chapter 3, and in fact, reinforce the need for further
exploration into whether humans use joint or endpoint motion to estimate joint stiffness.
One reason the algorithm trained better using the endpoint information manifests itself
in Figure 4-4, where the algorithm’s predictions for a simulated stiffness value are given.
There, it can be seen that when trained on the joint data, some of the stiffness predictions
were incorrect by a significant margin. For instance, in Experiment 1, there is an outlier,
where the machine estimated a stiffness of ∼100 Nm/rad for a simulated stiffness of ∼1
Nm/rad. These outliers highly skew the performance of the linear classifier.
In all cases, the test, validation, and training errors were similar. This suggests that these
errors were structural; regardless of how the data set was randomly divided, the classifier
performed similarly. Thus, it may be that a linear hypothesis class cannot accurately learn
the correlation between these features and the stiffness values because the relation is not
linear. To combat this problem, one might suggest using a polynomial feature mapping to
transform the original data into a higher dimensional space. However, a polynomial feature
mapping on the feature vector ∈ R24×1 used here would greatly increase the number of
91
features in the data set, which is likely to increase structural error. Another approach one
could use to appropriately address the structural error shown here, is to use a neural network
with ReLu5 activation functions in the hidden layers. Such a network can find non-linearities
in the data without using a more complicated data set [Barzilay et al., 2019c].
In all six data sets, the error starts to decrease less rapidly around the 200𝑡ℎ itera-
tion. However, in the joint space experiments of Figure 4-3, it may appear that the test
error and cross validation error do not converge as well as they do in the Cartesian exper-
iments. However, a paired-sample t-test showed there was no statistical difference between
the convergence of the two (𝑝 < 0.001,𝛼 = 0.05). Thus, one cannot conclude that the joint
experiments may be suffering from high variance6. This indicates that increasing the amount
of training will not improve performance, rather one should increase the complexity of the
neural network.
Figure 4-4 can be directly compared to the results of the human subject experiments
in the Huber et al. study [Huber et al., 2019, Huber et al., 2017], and in Chapter 3. The
best model produced here has an 𝑅2 value of 0.97, while the best subject in Figure 3-5 has
an 𝑅2 value of 0.90. The worst model produced here has an 𝑅2 value of 0.54, while the
worst subject in Figure 3-5 has an 𝑅2 value of 0.09. Furthermore, the machine’s average
coefficient of determination was 0.80, while the humans’ was 0.54. The machine’s worst
case is the humans’ average case, showing promising results for using machine learning to
estimate stiffness based on motion.
One of the questions this Chapter aimed to address was what motion parameters are im-
portant to determine these stiffness ratings. The answer to that question is found in Figures
4-1 and 4-2. It can be seen that Experiment 3, in Cartesian space, had the best performance.
The root-mean-square-error was about 2.5 Nm/rad. The average magnitude of that exper-
iment’s weights associated with the position values was 0.25, while the average magnitude
of the weights associated with the velocity and acceleration values were 0.05 and 0.06, re-
5Also known as the ramp function, the ReLu activation function outputs only the positive components
of the input (𝑓(𝑥) = max(0, 𝑥)). It is often used in Neural Networks to find non-linearities
6High variance is an error that arises when an algorithm does not receive enough data (or the data is not
helpful) to construct a good hypothesis class. A performance gap between the training error and validation
error is a sign of high variance, because it is likely that the algorithm is modeling random noise in the training
data, rather than the intended outputs. [Barzilay et al., 2019d].
92
spectively. The worst performance is seen in the Experiment 2 joint-space data. There, the
average magnitudes of the weights associated with position, velocity, and acceleration were
0.48, 0.50, and 0.31, respectively. In Figure 4-2, it can be seen that the hypothesis class
that valued position features more than velocity features performed better, whereas high
interest in velocity information led to sub-par results. This relation is also shown by the
coefficients of determination in Figure 4-2. Furthermore, this observation is consistent with
the conclusion drawn in Chapter 3: path information dominates trajectory information in
the perception of stiffness. Moreover, these results can also lead one to believe that velocity
information may serve as a distractor when estimating stiffness based on motion.
Presented here is a supervised linear regression algorithm trained on simulated arm mo-
tion data to estimate stiffness. In some cases, the trained linear classifier was able to perform
this task very successfully. The goal of this exercise was to move towards a new minimally
encumbering way of estimating joint stiffness, specifically in the context of wire-harnessing.
To generalize the simulated results presented here to human monitoring, the controller used
to produce the simulated movement must be congruent to the controller used to produce
actual human movement. The controller used to produce the movement here superimposed
joint-space and hand-space impedance, which has been found to be a competent description
of upper limb reaching movements [Hogan, 2017]. To extend this work to monitoring wire-
harness installation, there needs to be further exploration of the controller used to produce
arm movements during wire-harness installation. Furthermore, to apply this to the human
hand, more research needs to be conducted to understand the controller used to produce
finger movements in the hand.
4.5 Conclusion
Here, a supervised learning algorithm was used to train a machine to estimate simulated
arm stiffness based on the position, velocity, and acceleration information. When trained on
data in Cartesian endpoint space, the algorithm performed significantly better than when
it was trained on data in joint space. That observation contradicts what human subjects
self-reported in Chapter 3; there, subjects reported a higher interest in joint motion than
93
endpoint motion. Depending on the data set it was trained on, the algorithm had a root-
mean-square error that ranged from 2 to 10 Nm/rad (or 4 to 20% of the maximum simulated
joint stiffness). The linear classifiers that performed better had higher emphasis on position
values as opposed to velocity and acceleration, which reinforces the conclusion expressed in
Chapter 3: path information dominates trajectory information in the visual perception of
stiffness. In summary, presented here is a method that uses supervised linear regression to
train a machine to estimate joint stiffness using purely kinematic data. While some of the
results need to be further explored, the fact that the machine was able to do this task is a step
in the direction of using a machine to noninvasively estimate the stiffness of a wire-harness
installation worker’s hands and arms.
94
Chapter 5
Analysis of MuJoCo’s Contact Force
Measurements
5.1 Introduction
As previously mentioned, the goal of this thesis was to noninvasively measure a skilled wire-
harness worker’s motion, force, and impedance during wire-harness installation. Specifically,
this thesis aimed to provide a basis for solving equation 1.1 (𝐹 = 𝑍{𝑋0 − 𝑋}) using only
the measurement of 𝑋. Chapter 2 explored the possibility of using knowledge of existing
synergies to estimate 𝑋0 from 𝑋 in the human hand. Chapters 3 and 4 explored how
one could estimate stiffness, one important aspect of impedance, using solely kinematic
information. However, even with estimates of the impedance and zero-force trajectory, the
challenge still exists in calculating contact force in a high degree of freedom system. MuJoCo
may provide a possible solution to this problem.
MuJoCo is a physics-based simulation engine that specializes in multi-joint dynamics in
contact [Todorov et al., 2012]. This software has the capability of vastly reducing the amount
of extensive computation needed to explore how humans are capable of manipulating a highly
complex object, such as a wire harness. However, before using this tool, one must understand
and validate its capabilities.
This chapter aimed to validate and further inform the author of MuJoCo’s ability to
accurately estimate contact force for its use in estimating the forces that occur during wire-
95
harness installation. To achieve this, three different manipulators coming into contact, at
their endpoints, with a fixed, frictionless, rigid-body box were simulated. The manipulators
were either composed of one link, two links, or three links, and mimicked aspects of the human
index finger’s inertial, dynamic, and kinematic properties. In simulation, MuJoCo was used
to measure the manipulator’s joint kinematics, joint torques, and contact forces. With these
measurements, an expected joint torque was compared to a measured joint torque. It was
found that as the number of links in the manipulator was increased, so did error. Moreover,
as velocity of the manipulator increased, so did error. These values were skewed by large
initial impulses in contact force, when the link and box came into contact. Nonetheless, the
root-mean-square (RMS) error was always less than 3 Nm, 7% of the maximum applied joint
torque. So, it was concluded that MuJoCo’s ability to estimate contact force is acceptable.
5.2 Methods
In this chapter, MuJoCo’s abilities to measure contact force were analyzed. Primarily, the
following work serves as a check of the user’s understanding of the software tool and its
capabilities. To do so, a single “finger” moving in a planar-motion into contact with a
rigid-box was simulated. Force and torque measurements were taken from MuJoCo and
compared to what was expected from a simulation of a three-link manipulator in MATLAB.
A description of the kinematics is shown in Figure 5-1.
Prior to producing any simulations, care was taken to model the human index finger in
these simulations, as best as one could. The simulated finger was modeled as 3 rigid rods
(the proximal phalange, the middle phalange, and the distal phalange) and 3 joints (the
metacarpal [MCP], the proximal interphalangeal [PIP], and the distal interphalangeal [DIP]
joints). The radii and lengths of the phalanges were modeled according to the data described
in Clauser et al. [Clauser et al., 1988]. These values are found in Table 5.1. The fingers were
given a density of 1.16 g/cm3, which has been found to be the average density of the human
hand [Zatsiorsky, 2002]. The stiffness and damping of the finger were inspired by the works
of Jindrich et al. and Park et al. [Jindrich et al., 2004, Park et al., 2014], and were given
values of 50 Nm/rad and 1 Nm-s/rad, respectively. Furthermore, the finger was commanded
96
Figure 5-1: A three-link manipulator moving in planar motion. Joint angles, 𝜃, are measured
in relative coordinates. When comparing this three-link manipulator to a human finger,
joint 1 is the metacarpal joint, joint 2 is the proximal interphalangeal joint, and joint 3 is
the distal interphalangeal joint. Furthermore, link 1 is the proximal phalange, link 2 is the
middle phalange, and link 3 is the distal phalange. Link parameters are described in Table
5.1.
to follow a minimum-jerk trajectory (a trajectory found to accurately describe planar upper
limb motion [Flash and Hogan, 1985]) in joint space, starting and stopping at rest. This
implementation is shown in Equations 5.1, 5.2, and 5.3. Lastly, the finger was simulated to
follow the synergies produced by Santello et al. [Santello et al., 1998]. Specifically, the DIP
joint follows a trajectory 1.2 times the MCP joint, and the PIP joint follows a trajectory
0.95 times the MCP joint. This minimum-jerk trajectory is shown in Figure 5-2.
𝑡 𝑡 𝑡
𝜃 = 𝜃𝑖 + 𝜃𝑓 (10( )
3 − 15( )4 + 6( )5) (5.1)
𝐷 𝐷 𝐷
𝑡 𝑡 𝑡
𝜃 = 𝜃𝑖 + 𝜃𝑓 (30( )
2 − 60( )3 + 30( )4) (5.2)
𝐷 𝐷 𝐷
𝜏 = 𝐾(𝜃0 − 𝜃) + 𝐵(𝜃0 − 𝜃) (5.3)
In Equations 5.1, 5.2, and 5.3, 𝜃, its derivative, and 𝜏 represent position, velocity and
torque in relative1 coordinates. The subscripts 𝑖 and 𝑓 stand for initial and final, respectively.
The stiffness and damping terms are represented by 𝐾 and 𝐵, respectively. Equation 5.3 is
analogous to 1.1. Specifically, Equation 5.3 is in joint space, and the zero-force trajectory
1The relative coordinates mentioned here are shown in Figure 5-1
97
Figure 5-2: A plot of the position (deg), velocity (deg/sec), and torque (Nm) of a MuJoCo
simulation of a three-link finger moving in planar motion with a minimum-jerk, zero-force
trajectory. Here, the zero-force trajectory starts at rest and stops at rest after five seconds.
Then, it continues to be held statically at that final position for an additional five seconds.
is denoted by 𝜃𝑑 and 𝜃𝑑, while the actual trajectory is denoted by 𝜃 and 𝜃. The impedance
mapping that relates motion to force in 1.1 is represented here by the stiffness and damping
terms, 𝐾 and 𝐵. Lastly, with these values, joint torque, 𝜏 , is computed, which is the
rotational analog of force.
Diameter (cm) Length (cm)
Proximal Phalange 2.18 6.08
Middle Phalange 2.00 2.26
Distal Phalange 1.83 5.74
Table 5.1: Values used to describe the finger phalanges used in simulation. These values
came from the study done by Clauser et al. [Clauser et al., 1988]
In MuJoCo, the simulation is constructed as though it was a mechanical system comprised
of kinematically connected rigid bodies. First, a (rigid-body) object is defined. Next, joints
are attached to it with a certain set of degrees of freedom. Then, to actuate these joints,
motors are attached to them. Finally, sensors can be added to measure force and kinematic
98
data. Even though all of these “objects” are defined differently, one can attach them to the
same point. Additionally, the software can implement these components such that they do
not affect the overall dynamics of the system by adding a sensor or an actuator (i.e. the
sensors and actuators are massless). With all of these components, a manipulator can be
simulated, and force and kinematic measurements can be taken.
In the simulations presented here, the kinematics and torques felt at the joints and the
contact forces felt by the phalanges were of particular interest. Thus, the torque sensors
were defined to measure the torques felt at the joints. Specifically, they measured the 3-axis
torque in the joint. Due to the focus on planar motion, only the torque about the z-axis
was of interest. To measure contact force in MuJoCo, a touch sensor must be implemented.
This sensor is defined by the size of a particular “touch zone”. Here, the contact force sensor
was “attached” to each phalange, such that the touch zone encompassed it entirely. Touch
sensors measure the scalar normal force of all contact points that fall within the sensor
zone’s volume. In these simulations, it was important to exclude contact forces felt when the
phalanges collided with one another as they moved, as this is not what happens anatomically.
This was achieved, in MuJoCo, by simply excluding specific objects from being sensed by the
touch sensor. Finally, with these sensors implemented, the kinematic profiles of the joints,
their torques, and the force contact at the endpoint could be measured.
Given measurements of the kinematics, torques, and forces of a manipulator, Equation
5.4 was fully defined.
𝜇 = 𝐽(𝛼)𝑇 × 𝑓 (5.4)
Here, 𝜇 represents the torque vector in global2 coordinates. 𝛼 represents the joint displace-
ment in global coordinates. 𝑓 is the force vector in Cartesian coordinates. 𝐽 is the Jacobian,
which is found by taking the derivative of the forward kinematic equations. The superscript
𝑇 represents the transpose of the matrix. For a 3-link, planar manipulator, the forward
kinematic equations are shown in Equation 5.5, and the Jacobian is found in Equation 5.6.
𝑥 = 𝑙1 cos(𝛼1) + 𝑙2 cos(𝛼2) + 𝑙3 cos(𝛼3)
(5.5)
𝑦 = 𝑙1 sin(𝛼1) + 𝑙2 sin(𝛼2) + 𝑙3 sin(𝛼3)
2Here, global coordinates are measured with respect to a stationary (inertial) reference frame
99
⎡⎣ ⎤−𝑙1 sin (𝛼1) −𝑙2 sin (𝛼2) −𝑙3 sin (𝛼3)𝐽(𝛼) = ⎦ (5.6)
𝑙1 cos (𝛼1) 𝑙2 cos (𝛼2) 𝑙3 cos (𝛼3)
In Equations 5.5 and 5.6, the subscript 1 represents the first joint (MCP) and phalange
(proximal); the subscript 2 represents the second joint (PIP) and phalange (middle); and
the subscript 3 represents the third joint (DIP) and phalange (distal). Additionally, 𝑥 and
𝑦 denote the x and y positions in Cartesian coordinates, in accordance with Figure 5-1. It
is important to note that these equations are in global coordinates. However, the kinematic
and torque sensors, used in MuJoCo, are in relative coordinates. Thus, these equations
needed to be converted to relative coordinates, using the transformation matrix 𝑇 (equation
5.7).
⎢⎡ ⎤⎢⎢1 0 0⎣ ⎥⎥𝑇 = 1 1 0⎥⎦ (5.7)
1 1 1
Equation 5.4 is written in global coordinates. The transformation matrix 𝑇 relates the
relative joint coordinates to the global joint coordinates, 𝛼 = 𝑇×𝜃. Using this transformation
matrix, 𝑇 , Equation 5.4 can be rewritten in relative coordinates, as Equation 5.8.
⎡ ⎤ ⎡ ⎤ ⎡ 𝜏 = 𝑇 𝑇 × 𝐽𝑇 (𝜃)× 𝑓
⎢⎢⎢
⎤
𝜏1⎥⎥⎥ ⎢⎢⎢1 1 1⎥⎥ ⎢⎢ −𝑙
⎡ ⎤
⎥ ⎢ 1 sin 𝜃1 𝑙1 cos 𝜃1⎣ ⎦ ⎣ ⎦ ⎣ ⎥ 𝑓𝑥𝜏2 = 0 1 1 × − ⎥𝑙1 sin 𝜃1 + 𝜃 𝑙 ⎣ ⎦2 1 cos 𝜃1 + 𝜃2 ⎥⎦× (5.8)𝑓𝑦
𝜏3 0 0 1 −𝑙1 sin 𝜃1 + 𝜃2 + 𝜃3 𝑙1 cos 𝜃1 + 𝜃2 + 𝜃3
Equations 5.4 and 5.8 can be inverted so that force can be computed as a function of
the inverse of the Jacobian (𝑓 = 𝐽−𝑇𝜇). However, the Jacobian can only be inverted if it is
a square, non-singular matrix. In Equation 5.8, the Jacobian is clearly not square. When
the Jacobian is not square, it can be inverted using the pseudo-inverse, but this technique
finds a “best fit” solution, which is not always well-defined. It is a fact of mechanics that
calculating forces based on torques is not a well-posed problem. However, calculating torques
based on forces always results in a well-posed solution. Thus, MuJoCo’s computed forces and
100
kinematics were used to calculate the joint torques, and the calculated joint torque values
were compared to MuJoCo’s measured joint torque values.
In this study, there were three simulations. In the first simulation, only one joint (met-
carpal) and one link (proximal phalange) were simulated. In the second simulation, two
joints (MCP and PIP) and two links (distal and middle phalanges) were simulated. Finally,
in the third simulation, three joints and three links were simulated. In all three simulations,
the joints were commanded to follow a minimum-jerk trajectory and were subject to the
synergy profiles as described above. The most distal phalange had its motion opposed by a
frictionless box at some point during the simulation. In each simulation, the joint kinematics,
joint torques, and the phalange contact forces were measured. It is important to note that
the box was defined to have its frictionless surface lie along the x-axis. Thus, the computed
contact force was assumed to only be in the y-direction (i.e. the 𝑓𝑥 component in Equation
5.8 goes to zero). In both the two-link and three-link simulations, 𝜃𝑓 was chosen to be 30
degrees from the positive x-axis in Figure 5-1. When the end-effector came into contact
with the box and the zero-force trajectories of the more distal joints were still moving, the
end-effector slid along the box. This resulted in an undesired motion of the more proximal
joints. Since there is only one joint in the one-link case, 𝜃𝑓 was chosen to be 70 degrees from
the positive x-axis in Figure 5-1. In all the simulations, the initial position was 0 degrees (i.e.
along the x-axis in Figure 5-1). In all three cases, the velocity of the trajectory was varied.
This was done by changing the 𝐷 term in Equations 5.1 and 5.2. The duration, 𝐷, varied
between 30 seconds, 15 seconds, 5 seconds, and 2 seconds. Lastly, after the minimum-jerk
trajectory came to rest, the joints were commanded to hold the final position for 5 seconds.
This allowed for the exploration of measurements when the zero-force trajectory was sta-
tionary. The period where the manipulator was in contact with the box and the zero-force
trajectory was in motion, will be referred to as the dynamic case, while the period where the
zero-force trajectory was not in motion and the manipulator was in contact with the box,
will be known as the static case.
In summary, three simulations were run with four different durations each. With these
simulations, MuJoCo was used to measure position, velocity, torque, and contact force.
Then, the MuJoCo-given torques were compared to the torques calculated by Equation 5.8
101
in MATLAB.
5.3 Results
5.3.1 One-Link Simulations
Figure 5-3 shows the motion, force, and torque profiles of the MCP joint and the proxi-
mal interphalange in the one link MuJoCo simulation, where the minimum-jerk trajectory
duration was set to five seconds. Note that after 5 seconds, the zero-force trajectory stays
constant at the desired final position for an additional 5 seconds. The zero-force trajectory
is described by the solid red line, while the actual trajectory is described by the dashed
red line. A dashed black line at 2.3 seconds denotes the moment when the link comes into
contact with the box. Additionally, at this instance, both the contact force and the joint
torque start to increase. These results demonstrate what is qualitatively expected.
Quantitatively, the torque that MuJoCo produced was compared to a torque calculated
in MATLAB, based on the kinematic and force parameters given by MuJoCo (the blue line
in Figure 5-3), using Equation 5.8. This comparison is shown in the top plot of Figure
5-4 where MuJoCo’s measured joint torque is given by the solid line, and the joint torque
calculated in MATLAB is given by the dashed line. The MATLAB computed torque includes
an impulse, as the object initially comes into contact. The RMS error between the two is
shown in the bottom plot of Figure 5-4.
Figure 5-5 shows the average RMS error of the torques computed by MATLAB, and those
estimated by MuJoCo. In all four one-link simulations conducted, these values are separated
by the static (bottom) and dynamic (top) portions of the zero-force trajectory. The RMS
error increases as velocity is increased (minimum-jerk duration is decreased). However, these
values all have less than a 1 Nm error. In the static portion of the zero-force trajectory, RMS
error is less than 0.6 Nm.
102
Figure 5-3: A plot of the position (deg), velocity (deg/sec), torque (Nm), and force (N)
profiles of a single link commanded to follow a 5 second minimum-jerk trajectory followed
by 5 seconds of rest. The zero-force trajectory is described by the solid red line, while the
actual trajectory is described by the dashed red line. A dashed back line at 2.3 seconds
denotes the moment where the link comes into contact with the box.
103
Figure 5-4: Top: A plot of the MuJoCo computed torque and the MATLAB computed
torque. The torques computed here are for the simulation shown in Figure 5-3. Bottom: A
plot of the RMS error between the MuJoCo computed and MATLAB computed torques.
5.3.2 Two-Link Simulations
The kinematic profiles of the 5 second, two-link simulation are presented in Figures 5-6 and
5-7. Additionally, in the fourth plot of both figures, the x and y Cartesian reaction forces,
felt at the joint, and the contact forces, felt by the links, are shown. In Figure 5-6, a dashed
black line at 2.8 seconds denotes when the motion of the MCP joint is halted. Similarly, in
Figure 5-7, this is when contact force is felt by the proximal phalange, and the motion of
the PIP joint is opposed. At this moment in time, the PIP joint no longer perfectly tracks
its zero-force trajectory; however, it is still in motion.
Again, the torque that MuJoCo produced was compared to a torque calculated in MAT-
LAB, based on the kinematic and force parameters given by MuJoCo, using Equation 5.8.
This comparison is shown in the top plot of Figure 5-8, where MuJoCo’s measured joint
torques are given by the solid lines, and the joint torques calculated in MATLAB are given
by the dashed lines. The red line shows the torque of the MCP joint, and the green line
shows that of the PIP joint. A dashed black line indicates where contact first occurred. At
104
Figure 5-5: A plot of the RMS error as a function of the minimum-jerk trajectory average
velocity. As velocity increases, error increases. However, this error is still on the order of
less than 1 Nm in the dynamic case, and less than 0.6 Nm in the static case. Top: dynamic
case. Bottom: static case.
105
this point, MATLAB shows an initial impulse in the computed torque at both joints. In the
bottom plot of Figure 5-8, the RMS error between the MATLAB and MuJoCo computed
torques are shown. There is a spike in error right after the contact (shown by the black line).
However, this decreases significantly after initial contact.
Figure 5-9 shows the average RMS error between the torques computed by MATLAB
and those estimated by MuJoCo, of all four two-link simulations conducted. These values
are separated by the static (bottom) and dynamic (top) portions of the zero-force trajectory.
The RMS error increased as velocity was increased (minimum-jerk duration was decreased).
Similarly to the one-link case, the RMS error was small. Additionally, as velocity was
increased in the dynamic cases, error slightly increased. In the static case, error was less
than 0.04 Nm and independent of the average velocity of the movement, while in the dynamic
case, error was less than 1.2 Nm.
106
Figure 5-6: A plot of the MuJoCo produced position (deg), velocity (deg/sec), torque (Nm),
and force (N) profiles of the first joint (MCP) and link (proximal). The MCP joint was
commanded to follow a 5 second minimum-jerk trajectory followed by 5 seconds of rest. The
zero-force trajectory is described by the solid red line, while the actual trajectory is described
by the dashed red line. At 2.8 seconds the actual position deviates from the desired position.
Both the contact force of the proximal phalange, and the reaction forces felt by the MCP
joint are shown in the fourth plot.
107
Figure 5-7: A plot of the MuJoCo produced position (deg), velocity (deg/sec), torque (Nm),
and force (N) profiles of the second joint (PIP) and link (middle). The PIP joint was
commanded to follow a trajectory that is 0.95 times that of the first joint (Figure 5-6).
The zero-force trajectory is described by the solid green line, while the actual trajectory is
described by the dashed green line. Due to contact with the rigid box at 2.8 seconds, the
actual position deviates from the desired position. The contact force of the middle phalange,
and the reaction forces felt at the PIP joint are shown in the fourth plot.
108
Figure 5-8: Top: A plot of the MuJoCo computed torque and the MATLAB computed
torque of the two-link, 5 second simulation. MuJoCo’s estimated torque is given by the solid
line, and MATLAB’s computed torque is given by the dashed line. The red line shows the
torque of the MCP joint, and the green line shows the PIP joint. Initial contact is illustrated
by a vertical dashed black line at 2.8 seconds. Bottom: A plot of the RMS error between
the MuJoCo and MATLAB computed torques.
109
Figure 5-9: A plot of the mean RMS error as a function of the minimum-jerk trajectory
average velocity. The red line corresponds to the RMS error of the MCP joint, and the green
line corresponds to the RMS error in the PIP joint. Top: dynamic case. Bottom static case.
5.3.3 Three-Link Simulations
Motion profiles of the 5 second three-link simulation are shown in Figures 5-10, 5-11, and 5-12
respectively. A vertical dashed black line, at 2.7 seconds, denotes when the distal phalange
came into contact with the fixed box. Similar to the two-link case, once force contact was
felt, the motion of the MCP (Figure 5-10) was completely halted, while the PIP and DIP
(Figures 5-11 and 5-12) are both still able to move.
Again, the torque that MuJoCo estimated was compared to the torque calculated in
MATLAB, using Equation 5.8, where the kinematic and force parameters are given by Mu-
JoCo. This comparison is shown in the top plot of Figure 5-13, where MuJoCo’s estimated
contact force is given by the solid line, and the contact force calculated in MATLAB is given
by the dashed line. The red line shows the torque of the MCP joint; the green line shows the
PIP joint; and the blue line shows the DIP joint. Initial contact is indicated by a vertical
dashed black line at 2.7 seconds. At that point, MATLAB estimated a large initial contact
force. This correlates to large RMS errors after contact in the bottom plot of Figure 5-13.
110
Figure 5-10: A plot of the MuJoCo produced position (deg), velocity (deg/sec), torque
(Nm), and force (N) profiles of the first joint (MCP) and link (proximal). The MCP joint
was commanded to follow a 5 second minimum-jerk trajectory followed by 5 seconds of rest.
The zero-force trajectory is described by the solid red line, while the actual trajectory is
described by the dashed red line. At 2.7 seconds, the actual position deviates from the
desired position. In the fourth plot, the contact force of the proximal phalange and the
reaction forces felt by the MCP joint are shown.
Figure 5-14 shows the average dynamic and static RMS errors of the torques computed by
MATLAB and those estimated by MuJoCo, based on all four three-link simulations. The
RMS error increased as velocity was increased (minimum-jerk duration was decreased). In
the dynamic case, the RMS errors were less than 3 Nm, while in the static case, the RMS
errors were all less than 0.12 Nm.
111
Figure 5-11: A plot of the MuJoCo produced position (deg), velocity (deg/sec), torque
(Nm), and force (N) profiles of the second joint (PIP) and link (middle). The PIP joint
was commanded to follow a 5 second minimum-jerk trajectory followed by 5 seconds of rest.
The zero-force trajectory is described by the solid green line, while the actual trajectory
is described by the dashed green line. At 2.7 seconds the actual position deviates from the
desired position. In the fourth plot, the contact force of the middle phalange and the reaction
forces felt by the PIP joint are shown.
112
Figure 5-12: A plot of the MuJoCo produced position (deg), velocity (deg/sec), torque (Nm),
and force (N) of the third joint (DIP) and link (distal). The DIP joint was commanded to
follow a 5 second minimum-jerk trajectory followed by 5 seconds of rest. The zero-force
trajectory is described by the solid blue line, while the actual trajectory is described by the
dashed blue line. Initial contact is denoted by a vertical, dashed black line at 2.7 seconds.
In the fourth plot, the contact force of the distal phalange and the reaction forces felt by the
DIP joint are shown.
113
Figure 5-13: Top: A plot of the MuJoCo computed torque and the MATLAB computed
Torque. The torques computed here are for the simulation shown in Figures 5-10, 5-11, and
5-12. MuJoCo’s estimated contact force is given by the solid line, and the contact force
calculated in MATLAB is given by the dashed line. The red line shows the torque of the
MCP joint, the green line shows the PIP joint, and the blue line shows the DIP joint. Initial
contact is denoted by a vertical dashed black line at 2.8 seconds. Bottom: A plot of the
RMS error between the MuJoCo and MATLAB computed torques.
114
Figure 5-14: A plot of the RMS error as a function of the minimum-jerk trajectory average
velocity. As velocity increased, error also increased. The red line corresponds to the proximal
phalange, the green line corresponds to the middle phalange, and the blue line corresponds
to the distal phalange. Top: Dynamic case. Bottom: Static case.
115
5.4 Discussion
This chapter aimed to validate and further inform the author about MuJoCo’s ability to
accurately estimate contact force. Upon successfully doing so, MuJoCo could be used as a
tool to estimate contact forces of high degree-of-freedom bodies that manipulate complex
objects. Specifically, this thesis has taken interest in using MuJoCo to estimate the contact
forces used in wire-harness installation. With these goals in mind, three different manipula-
tors coming into contact, at their endpoints, with a fixed, frictionless, rigid-body box were
simulated. The manipulators were either composed of one, two, or three links, and were
given inertial and dynamic properties of a human index finger. Furthermore, each joint was
commanded to follow a minimum-jerk zero-force trajectory in joint space and was given a
stiffness of 50 Nm/rad and a damping of 1 Nm-s/rad. In simulation, MuJoCo was used to
measure the joint kinematics, torques, and contact forces of the finger. With these values,
Equation 5.8 was used to calculate the torque about the joints, based on the contact force
given by MuJoCo. Figures 5-5, 5-9, and 5-14 describe the main results. There, it can be seen
that the highest average RMS error between the Mujoco and MATLAB computed torques
was 3 Nm. This is only ∼7% of the maximum MuJoCo applied joint torque. Thus, it was
concluded that MuJoCo’s ability to estimate contact force is acceptable.
5.4.1 Force and Torque Impulse at Contact Onset
As the finger initially made contact with the box, there was an initial impulse in contact
force. This occurred in all the simulations shown in this chapter (Figures 5-3, 5-7, and 5-12).
This initial increase in contact force does make intuitive sense. It represents an instantaneous
change of momentum. In Figures 5-4, 5-8, and 5-13, the torque calculated by MATLAB has
an initial impulse as well. This was also expected as this calculation was based on the contact
forces given by MuJoCo. However, the joint torques produced by MuJoCo do not display
these impulses. MuJoCo specifically avoids these near infinite computations. In the case
of joint torques, MuJoCo requires that the user specifies actuator limits, and if those limits
are reached or exceeded, the simulation caps those actuator torques at the limit. However,
the results presented here do not show an impulse that reaches the specified joint torque
116
limit. This is likely due to MuJoCo’s numerical integrator, which aims to avoid such torque
and force impulses. The details of MuJoCo’s numerical integrator are not discussed here.
However, its intricacies are a topic that should be further explored.
5.4.2 Error in Static Zero-Force Trajectory
In all three simulations, the mean RMS error was less than 1 Nm, when the manipulator’s
zero-force trajectory was not in motion. This is less than 2% of the maximum torque applied.
In the static cases, one may conclude that MuJoCo accurately computes contact forces.
5.4.3 Error in Dynamic Zero-Force Trajectory
In Figures 5-5, 5-9, and 5-14, the RMS error increases with velocity. Figures 5-8 and 5-13
give insight as to why this is the case. As discussed above, in the top plots of Figures 5-4, 5-8,
and 5-13, there is an initial impulse in the MATLAB computed joint torques at the initial
point of contact. These impulses are large outliers that greatly skew the mean dynamic RMS
error for that simulation. Despite these errors, MuJoCo demonstrated acceptable accuracy
when computing contact force.
5.4.4 Next Steps
Moving forward, there is much room to further explore MuJoCo’s force computation capa-
bilities and try to better understand its strengths and weaknesses. One can do a sensitivity
analysis to understand which parameters MuJoCo is most sensitive to when calculating force.
Furthermore, the experiment presented in this paper was limited to planar motion and fric-
tionless contact. A deeper dive into how MuJoCo handles contact with friction could also be
informative, as the human hand is not frictionless. Lastly, replacing the rigid-body box with
a soft-body box might be informative to better understand how MuJoCo simulates non-rigid
objects such as a wire harness. Furthermore, when the finger comes in contact with a box
that has compliance, it is unlikely that there will be impulses in contact force as the two
objects initially collide. All in all, the proposed next steps promise further insight into how
MuJoCo simulates and estimates forces of multi-joint dynamics in contact with complex
117
objects. These insights will be important when MuJoCo is used to estimate contact forces
during wire-harness installation.
5.5 Conclusion
This chapter aimed to validate and further inform the author of MuJoCo’s ability to accu-
rately estimate contact force with a view to use it to estimate the forces that occur during
wire-harness installation. To achieve this, three different manipulators coming into contact,
at their endpoints, with a fixed, frictionless, rigid-body box were simulated. The manipula-
tors were either composed of one link, two links, or three links, and mimicked aspects of the
human index finger’s inertial, dynamic, and kinematic properties. In simulation, MuJoCo
was used to measure the manipulator’s joint kinematics, joint torques, and contact forces.
With these measurements, Equation 5.8 was used to calculate the torque about the joints,
based on the contact force given by MuJoCo. It was found that as the number of links in the
manipulator or the velocity were increased, the RMS error between the MATLAB-calculated
and MuJoCo-computed torques increased slightly as well. This occurred particularly when
the zero-force trajectory was moving. However, these values were skewed by large initial im-
pulses in contact force when the link and box came into contact. When the manipulator and
box were in contact with one another, and the zero-force trajectory was no longer in motion,
the RMS error was always less than 1 Nm. Thus, it was concluded that MuJoCo’s ability
to estimate contact force is sufficiently accurate for the application envisioned. Further ex-
ploration can be done to test its ability to simulate contact forces with non-rigid objects.
This will better inform whether or not MuJoCo can be used to simulate and estimate forces
during wire-harness installation.
118
Chapter 6
Conclusions and Future Work
6.1 Conclusions
This thesis was initially motivated by the paradox of human performance: despite infe-
rior ‘wetware’ and ‘hardware,’ humans outperform robots in tasks that require physical
interaction, especially physical interaction with non-rigid objects. This performance gap
is ever present in the manufacturing process known as wire-harnessing. Generally, on an
assembly line, robots can be used to quickly and accurately assemble a system. However,
wire-harnessing is done manually, creating a bottleneck in this process. To improve this
manufacturing operation, it was proposed to add a robot to help assist the human worker.
However, before adding a robot collaborator, it is important to first understand what ex-
actly the human is doing. Thus, this thesis focused on determining a method to explore how
humans do this task.
In tasks that require physical interaction, modulation of neuromuscular impedance is
important. However, to measure it is encumbering and can impede the worker. On the
other hand, today’s methods of kinematic monitoring are not invasive. Thus, this thesis
explored the feasibility of a method to measure the force and impedance of the wire-harness
worker’s hands and arms using solely motion information.
In Chapter 2, the idea of using a whole-hand, synergy-based Interaction Capture method
was explored. It was believed that the knowledge of hand synergies could help estimate
the hand impedance and zero-force trajectory. However, current knowledge of synergies
119
has not yet been extended to object manipulation. Thus, Chapter 2 focused on doing so
by comparing synergies observed in two experiments: in the first, subjects reached for and
grasped a tool or object commonly found in wire-harness installation, and in the second,
subjects manipulated those objects and tools to install a wire-harness on a mock electrical
cabinet. Upon comparing reach-and-grasp synergies to manipulation synergies, it was found
that only the first synergy was the same across experimental tasks.
Chapter 3 expanded on the work produced by Huber et. al [Huber et al., 2017, Huber et al., 2019],
which demonstrated that humans can correctly infer changes in limb stiffness from nontrivial
changes in multi-joint limb motion without force information or explicit knowledge of the
underlying limb controller. In particular, Chapter 3 explored what motion characteristics
subjects used to determine their stiffness estimates. The results suggested that path, not
trajectory, information is more important to subjects when estimating stiffness. Moreover,
subjects also reported a higher interest in joint information than endpoint information. This
exploration of how humans extract latent features of neuromotor control, such as stiffness,
from kinematics provides new insight into how humans interpret the motor actions of others.
Furthermore, it can help aid the development of a machine to do the same.
In Chapter 4, a supervised learning algorithm was used to train a machine to estimate
simulated arm stiffness based on position, velocity, and acceleration information. The algo-
rithm performed better when trained on Cartesian endpoint data than on joint space data.
That observation contradicts the findings in Chapter 3, where subjects self-reported a higher
interest in joint motion than endpoint motion. However, the better performing linear classi-
fiers had a higher emphasis on position values as opposed to velocity and acceleration, which
reinforces the conclusion expressed in Chapter 3: path information dominates trajectory
information in the visual perception of stiffness.
Given stiffness estimates from motion, using the methods presented in Chapters 2, 3,
and 4, the challenge of estimating contact force in a high degree-of-freedom space remains.
Chapter 5 aimed to validate and further inform the author of MuJoCo’s ability to accurately
estimate contact force, particularly for its use in estimating the forces that occur during
wire-harness installation. To achieve this, three different manipulators coming into contact,
at their endpoints, with a fixed, frictionless, rigid-body box were simulated. It was found
120
that as the number of links in the manipulator or velocity of the manipulator was increased,
MuJoCo’s error increased. Nonetheless, the average torque error was always less than 3 Nm,
which was about 7% of the maximum torque applied. Thus, it was concluded that MuJoCo’s
ability to estimate contact force is sufficiently accurate for its use envisioned here.
While the ability to monitor a human’s motion, force, and impedance during wire-harness
installation was not implemented, this thesis provides a basis to create a system to do so.
6.2 Future Work
6.2.1 Whole-Hand Interaction Capture
The exploration of synergies in Chapter 2 showed that only the first synergy was consis-
tent across both reach-and-grasp and object manipulation. Thus, existing knowledge of
higher-order, reach and grasp synergies would not provide fruitful insight of higher-order
manipulation synergies. However, the first synergy accounted for about 89% of the variance
in the hand postures in this wire-harnessing task. Furthermore, this synergy has been found
to be the most important one needed for stable grasp [Gabiccini et al., 2011]. Additionally,
this stable grasp could be achieved using different impedance values. This observation may
or may not extend to manipulation. To better understand this result, one could imple-
ment a soft-synergy model1 using MuJoCo to determine if stable object manipulation can
be achieved with only the lower-order synergies. Furthermore, it should be explored which
impedance values lead to stable object manipulation.
6.2.2 Visual Perception of Stiffness with Eye-Tracking
Chapter 3 showed that subjects more often reported having an interest in joint motion than
endpoint motion. However, Chapter 4 showed that training a machine using endpoint motion
data to estimate stiffness, returned more favorable results. That observation contradicts
what subjects reported in Chapter 3. It is possible that humans were unable to accurately
1The soft synergy model implements a hand in a dynamical equilibrium under two force fields. The first
force field is governed by the desired kinematic synergy, and the second force field is governed by the object
that repels the hand from penetrating it (Figure 1-4).
121
articulate what motion characteristics they used in estimating joint stiffness. To better
tease out this information, one could repeat the experiments presented in Chapter 3, using
eye-tracking to determine whether subjects looked at the joints or the endpoint.
6.2.3 Visual Perception of Stiffness Implementation
Chapter 4 suggests a few immediate next steps. First, it was seen that the performance
metrics used to measure the quality of the trained machine were greatly skewed by out-
liers. This was especially evident when the algorithm was trained using joint space data.
Deeper exploration into why the algorithm performed unsatisfactorily in predicting these
specific simulations may provide insight into how to improve the algorithm. Moreover, de-
termining where the machine performed badly may provide information about what motion
characteristics may inhibit humans from estimating stiffness based on motion.
Secondly, it was shown that in certain cases, the algorithm suffered from structural error.
Thus, a linear hypothesis class was unable to accurately predict stiffness when using motion
characteristics. This suggests that using a more complex hypothesis class, such as a neural
network with non-linear activation functions, might improve performance.
Additionally, to be able to reliably estimate human stiffness based on the simulation data,
it is imperative that the controller that produced the simulated movement is congruent
with how humans produce movement. The controller used to produce the movement in
Chapter 4 superimposed joint-space and hand-space impedance, which has been found to be a
competent description of the upper limb movements when reaching [Hogan, 2017]. To extend
this work to monitoring wire-harness installation, further work should explore the controller
used to produce arm movements in wire-harness installation. Moreover, to apply this to the
human hand, more research needs to be conducted to encompass the understanding of the
controller that was used to produce finger movements in the hand.
Furthermore, the current implementation estimated a single impedance value from the
kinematic characteristics of a single trajectory. However, in humans, impedance is inherently
time-varying. Thus, to monitor a human during a complex physical interaction task, the
machine must be able to estimate time-varying impedance. An algorithm to do this should
be considered and further pursued.
122
6.2.4 Further Exploration of MuJoCo
There is more room to further explore MuJoCo’s force estimation capabilities and to better
understand its strengths and weaknesses. One can do a sensitivity analysis to understand
which parameters MuJoCo is most sensitive to when calculating force. Furthermore, the
experiment presented in Chapter 5 was limited to planar motion and frictionless contact.
A deeper dive into how MuJoCo handles contact with friction could also be informative, as
the human hand is not frictionless. Additionally, expanding the simulation beyond planar
motion can create a better understanding of how MuJoCo handles gravity, a force that is
always at play during human object manipulation.
Most importantly, simulation of a non-rigid (compliant) object will be very be informative
for a better understanding of how MuJoCo can be used in estimating contact forces at play,
especially during wire-harness installation. Furthermore, this compliance can possibly reduce
the impulses in contact force seen as the finger initially collides with the rigid-body box. All
in all, the proposed next steps give further insight into how MuJoCo simulates and estimates
forces of multi-joint dynamic systems in contact with complex objects. Such insights will
be important when MuJoCo is to be used to estimate contact forces during wire-harness
installation.
123
124
Appendix A
Correlation Testing via Fisher
Transformation
The Pearson correlation coefficient, 𝑟, is not normally distributed when the population cor-
relation coefficient, 𝜌, is not zero. The common hypothesis tests, such as a student’s t-test,
assume that the data came from a normal distribution. Thus, to do hypothesis testing on
correlation coefficients, one must use the Fisher Transformation to convert 𝑟 into a value
that is normally distributed.
The theorem states: If the two variables that produce 𝑟 have a joint bivariate normal
distribution or come from a sufficiently large sample size, 𝑛, then the Fisher transforma-
tion 𝑟′ of the correlation coefficient r has a normal distribution 𝑁(𝜌, 𝑠𝑟′) where 𝑠 is, √ 1𝑟′ 𝑛−3
[Fisher et al., 1921, Fisher, 1915]. Thus, the Fisher transformation converts 𝑟 to the nor-
mally distributed1 𝑟′ using Equation A.1.
′ 1 1 + 𝑟𝑟 = 𝑙𝑛( ) (A.1)
2 1− 𝑟
This function has asymptotes at -1 and 1.
The confidence interval of a sample correlation coefficient in the Fisher domain, 𝑟′, is
1A normal distribution is completely determined by the parameters 𝜇 and 𝜎. 𝜇 is the mean of the normal
distribution and 𝜎 is the standard deviation [Whitlock and Schluter, 2009].
125
computed using Equation A.2,
𝐶𝐼 = 𝑟′ ± 𝑍𝑐𝑟𝑖𝑡 × 𝑠𝑟′ (A.2)
where 𝑍𝑐𝑟𝑖𝑡 is the boundary of acceptance and is determined by the range of the confidence
interval. For a 95% confidence interval 𝑍𝑐𝑟𝑖𝑡 = 1.96.
To convert this confidence interval from the Fisher domain to the original Pearson cor-
relation coefficient space, one must perform the inverse Fisher transformation,
exp(2𝑟′)− 1
𝑟 = ′
exp(2𝑟′
= tanh(𝑟 ) (A.3)
) + 1
126
Bibliography
[Banerjee et al., 2015] Banerjee, N., Long, X., Du, R., Polido, F., Feng, S., Atkeson, C. G.,
Gennert, M., and Padir, T. (2015). Human-supervised control of the ATLAS humanoid
robot for traversing doors. In 2015 IEEE-RAS 15th International Conference on Humanoid
Robots (Humanoids), pages 722–729. IEEE.
[Barsoum, 2016] Barsoum, E. (2016). Articulated Hand Pose Estimation Review.
[Barzilay et al., 2019a] Barzilay, R., Jaakkola, T., and Kaelbling, L. (2019a). Lecture notes
in 6.036 introduction to machine learning: Feature representation.
[Barzilay et al., 2019b] Barzilay, R., Jaakkola, T., and Kaelbling, L. (2019b). Lecture notes
in 6.036 introduction to machine learning: Gradient descent.
[Barzilay et al., 2019c] Barzilay, R., Jaakkola, T., and Kaelbling, L. (2019c). Lecture notes
in 6.036 introduction to machine learning: Neural networks.
[Barzilay et al., 2019d] Barzilay, R., Jaakkola, T., and Kaelbling, L. (2019d). Lecture notes
in 6.036 introduction to machine learning: Regression.
[Bennett et al., 1992] Bennett, D. J., Hollerbach, J. M., Xu, Y., and Hunter, I. W. (1992).
Time-varying stiffness of human elbow joint during cyclic voluntary movement. Experi-
mental Brain Research, 88(2):433–442.
[Bicchi et al., 2011] Bicchi, A., Gabiccini, M., and Santello, M. (2011). Modelling natural
and artificial hands with synergies. Philosophical Transactions of the Royal Society B:
Biological Sciences, 366(1581):3153–3161.
[Bidet-Ildei et al., 2006] Bidet-Ildei, C., Orliaguet, J.-P., Sokolov, A. N., and Pavlova, M.
(2006). Perception of elliptic biological motion. Perception, 35(8):1137–47.
[Blake and Shiffrar, 2007] Blake, R. and Shiffrar, M. (2007). Perception of Human Motion.
Annual Review of Psychology, 58(1):47–73.
[Cao et al., 2017a] Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. (2017a). Realtime Multi-
person 2D Pose Estimation Using Part Affinity Fields. In 2017 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pages 1302–1310. IEEE.
[Cao et al., 2017b] Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. (2017b). Realtime Multi-
person 2D Pose Estimation Using Part Affinity Fields. Cvpr, pages 1302–1310.
127
[Clauser et al., 1988] Clauser, C., Tebbetts, I., Bradtmiller, B., McConville, J., and Gordon,
C. C. (1988). Measurer’s Handbook: U.S. Army Anthropometric Survey, 1987-1988.
[Collins et al., 2019] Collins, J., Howard, D., and Leitner, J. (2019). Quantifying the reality
gap in robotic manipulation tasks. In Proceedings - IEEE International Conference on
Robotics and Automation, volume 2019-May, pages 6706–6712.
[Dayan et al., 2007] Dayan, E., Casile, A., Levit-Binnun, N., Giese, M. A., Hendler, T., and
Flash, T. (2007). Neural representations of kinematic laws of motion: Evidence for action-
perception coupling. Proceedings of the National Academy of Sciences of the United States
of America, 104(51):20582–20587.
[Diogo et al., 2012] Diogo, R., Richmond, B. G., and Wood, B. (2012). Evolution and ho-
mologies of primate and modern human hand and forearm muscles, with notes on thumb
movements and tool use. Journal of Human Evolution, 63(1):64–78.
[Erez et al., 2015] Erez, T., Tassa, Y., and Todorov, E. (2015). Simulation tools for model-
based robotics: Comparison of Bullet, Havok, MuJoCo, ODE and PhysX. In Proceedings
- IEEE International Conference on Robotics and Automation.
[Erol et al., 2007] Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., and Twombly, X. (2007).
Vision-based hand pose estimation: A review. Computer Vision and Image Understanding,
108(1-2):52–73.
[Fisher, 1915] Fisher, R. A. (1915). Frequency Distribution of the Values of the Correlation
Coefficient in Samples from an Indefinitely Large Population. Biometrika, 10(4):507.
[Fisher et al., 1921] Fisher, R. A. et al. (1921). 014: On the" probable error" of a coefficient
of correlation deduced from a small sample.
[Flash and Hogan, 1985] Flash, T. and Hogan, N. (1985). The coordination of arm move-
ments: an experimentally confirmed mathematical model. Journal of neuroscience,
5(7):1688–1703.
[Gabiccini et al., 2011] Gabiccini, M., Bicchi, A., Prattichizzo, D., and Malvezzi, M. (2011).
On the role of hand synergies in the optimal choice of grasping forces. Autonomous Robots,
31(2-3):235–252.
[Guarín and Kearney, 2017] Guarín, D. L. and Kearney, R. E. (2017). Estimation of Time-
Varying, Intrinsic and Reflex Dynamic Joint Stiffness during Movement. Application to
the Ankle Joint. Frontiers in Computational Neuroscience, 11:51.
[Herguedas et al., 2019] Herguedas, R., López-Nicolás, G., Aragüés, R., and Sagüés, C.
(2019). Survey on multi-robot manipulation of deformable objects. In IEEE Interna-
tional Conference on Emerging Technologies and Factory Automation, ETFA, volume
2019-Septe, pages 977–984. Institute of Electrical and Electronics Engineers Inc.
128
[Hermus et al., 2020] Hermus, J. R., Doeringer, J., Sternad, D., and Hogan, N. (2020). Sep-
arating Neural Influences from Peripheral Mechanics: The Speed-Curvature Relation in
Mechanically-Constrained Actions. Journal of Neurophysiology.
[Hogan, 1985a] Hogan, N. (1985a). Impedance control: An approach to manipulation: Part
I-theory. Journal of Dynamic Systems, Measurement and Control, Transactions of the
ASME, 107(1):1–7.
[Hogan, 1985b] Hogan, N. (1985b). Impedance control: An approach to manipulation: Part
II-implementation. Journal of Dynamic Systems, Measurement and Control, Transactions
of the ASME, 107(1):8–16.
[Hogan, 1985c] Hogan, N. (1985c). Impedance control: An approach to manipulation: Part
III-applications. Journal of Dynamic Systems, Measurement and Control, Transactions
of the ASME, 107(1):17–24.
[Hogan, 2017] Hogan, N. (2017). Physical Interaction via Dynamic Primitives. pages 269–
299. Springer, Cham.
[Hogan and Sternad, 2012] Hogan, N. and Sternad, D. (2012). Dynamic primitives of motor
behavior.
[Hopcroft et al., 1991] Hopcroft, J. E., Kearney, J. K., and Krafft, D. B. (1991). A Case
Study of Flexible Object Manipulation. The International Journal of Robotics Research,
10(1):41–50.
[Horak and Trinkle, 2019] Horak, P. C. and Trinkle, J. C. (2019). On the similarities and
differences among contact models in robot simulation. IEEE Robotics and Automation
Letters.
[Huber et al., 2017] Huber, M. E., Folinus, C., and Hogan, N. (2017). Visual perception
of limb stiffness. In 2017 IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS), pages 3049–3055. IEEE.
[Huber et al., 2019] Huber, M. E., Folinus, C., and Hogan, N. (2019). Visual perception of
joint stiffness from multijoint motion. Journal of neurophysiology, 122(1):51–59.
[Huh and Sejnowski, 2015] Huh, D. and Sejnowski, T. J. (2015). Spectrum of power laws for
curved hand movements. Proceedings of the National Academy of Sciences of the United
States of America, 112(29):E3950–E3958.
[Igo Krebs et al., 1998] Igo Krebs, H., Hogan, N., Aisen, M. L., and Volpe, B. T. (1998).
Robot-aided neurorehabilitation. IEEE Transactions on Rehabilitation Engineering,
6(1):75–87.
[Jindrich et al., 2004] Jindrich, D. L., Balakrishnan, A. D., and Dennerlein, J. T. (2004).
Finger joint impedance during tapping on a computer keyswitch. Journal of Biomechanics,
37(10):1589–1596.
129
[Kandel et al., 2000a] Kandel, E., Schwartz, J., Jessell, T., Siegelbaum, S., and Hudspeth,
A. (2000a). Principles of neural science. The McGraw-Hill Companies, fifth edition.
[Kandel et al., 2000b] Kandel, S., Orliaguet, J. P., and Viviani, P. (2000b). Perceptual
anticipation in handwriting: The role of implicit motor competence. Perception and
Psychophysics, 62(4):706–716.
[Knoedler et al., 2015] Knoedler, K., Dimitrov, V., Conn, D., Gennert, M. A., and Padir,
T. (2015). Towards supervisory control of humanoid robots for driving vehicles during
disaster response missions. In 2015 IEEE International Conference on Technologies for
Practical Robot Applications (TePRA), pages 1–6. IEEE.
[Krebs et al., 2002] Krebs, H. I., Volpe, B. T., Ferraro, M., Fasoli, S., Palazzolo, J., Rohrer,
B., Edelstein, L., and Hogan, N. (2002). Robot-aided neuro-rehabilitation: From evidence-
based to science-based rehabilitation.
[Kry et al., 2006] Kry, P. G., Pai, D. K., Kry, P. G., and Pai, D. K. (2006). Interaction
capture and synthesis. In ACM SIGGRAPH 2006 Papers on - SIGGRAPH ’06, volume 25,
page 872, New York, New York, USA. ACM Press.
[Lacquaniti et al., 1993] Lacquaniti, F., Carrozzo, M., and Borghese, N. A. (1993). Time-
varying mechanical behavior of multijointed arm in man. Journal of Neurophysiology,
69(5):1443–1464.
[Lacquaniti et al., 1982] Lacquaniti, F., Licata, F., and Soechting, J. F. (1982). The me-
chanical behavior of the human forearm in response to transient perturbations. Biological
Cybernetics, 44(1):35–46.
[Lee and Hogan, 2015] Lee, H. and Hogan, N. (2015). Time-varying ankle mechanical
impedance during human locomotion. IEEE Transactions on Neural Systems and Re-
habilitation Engineering, 23(5):755–764.
[Lee et al., 2016] Lee, H., Rouse, E. J., and Krebs, H. I. (2016). Summary of Human Ankle
Mechanical Impedance during Walking. IEEE Journal of Translational Engineering in
Health and Medicine, 4.
[Lee Rodgers and Alan Nice Wander, 1988] Lee Rodgers, J. and Alan Nice Wander, W.
(1988). Thirteen ways to look at the correlation coefficient. American Statistician,
42(1):59–66.
[Leo et al., 2016] Leo, A., Handjaras, G., Bianchi, M., Marino, H., Gabiccini, M., Guidi,
A., Scilingo, E. P., Pietrini, P., Bicchi, A., Santello, M., and Ricciardi, E. (2016). A
synergy-based hand control is encoded in human motor cortical areas. eLife, 5.
[Liu et al., 2018] Liu, Y., Xu, Y., and Li, S. B. (2018). 2-D Human Pose Estimation from
Images Based on Deep Learning: A Review. In Proceedings of 2018 2nd IEEE Advanced In-
formation Management, Communicates, Electronic and Automation Control Conference,
IMCEC 2018, pages 462–465. Institute of Electrical and Electronics Engineers Inc.
130
[Loula et al., 2005] Loula, F., Prasad, S., Harber, K., and Shiffrar, M. (2005). Recognizing
people from their movement. Journal of Experimental Psychology: Human Perception and
Performance, 31(1):210–220.
[Marzke, 1997] Marzke, M. W. (1997). Precision grips, hand morphology, and tools. Amer-
ican Journal of Physical Anthropology, 102(1):91–110.
[Mason et al., 2001] Mason, C. R., Gomez, J. E., and Ebner, T. J. (2001). Hand Synergies
During Reach-to-Grasp. Journal of Neurophysiology, 86(6):2896–2910.
[Mathis et al., 2018] Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N.,
Mathis, M. W., and Bethge, M. (2018). DeepLabCut: markerless pose estimation of
user-defined body parts with deep learning. Nature Neuroscience, 21(9):1281–1289.
[Mattar and Gribble, 2005] Mattar, A. A. and Gribble, P. L. (2005). Motor learning by
observing. Neuron, 46(1):153–160.
[Maurice et al., 2018] Maurice, P., Huber, M. E., Hogan, N., and Sternad, D. (2018).
Velocity-Curvature Patterns Limit Human-Robot Physical Interaction. IEEE Robotics
and Automation Letters, 3(1):249–256.
[Mussa-Ivaldi et al., 1985] Mussa-Ivaldi, F. A., Hogan, N., and Bizzi, E. (1985). Neural,
mechanical, and geometric factors subserving arm posture in humans. Journal of Neuro-
science, 5(10):2732–2743.
[Napier, 1956] Napier, J. R. (1956). THE PREHENSILE MOVEMENTS OF THE HUMAN
HAND. The Journal of Bone and Joint Surgery, 38(4):902–913.
[Nath et al., 2019] Nath, T., Mathis, A., Chen, A. C., Patel, A., Bethge, M., and Mathis,
M. W. (2019). Using DeepLabCut for 3D markerless pose estimation across species and
behaviors. Nature Protocols, 14(7):2152–2176.
[Park et al., 2014] Park, J., Pažin, N., Friedman, J., Zatsiorsky, V. M., and Latash, M. L.
(2014). Mechanical properties of the human hand digits: Age-related differences. Clinical
Biomechanics, 29(2):129–137.
[Pollick et al., 2001] Pollick, F. E., Paterson, H. M., Bruderlin, A., and Sanford, A. J. (2001).
Perceiving affect from arm movement. Cognition, 82(2).
[Rouse et al., 2014] Rouse, E. J., Hargrove, L. J., Perreault, E. J., and Kuiken, T. A. (2014).
Estimation of human ankle impedance during the stance phase of walking. IEEE Trans-
actions on Neural Systems and Rehabilitation Engineering, 22(4):870–878.
[Rouse et al., 2013] Rouse, E. J., Hargrove, L. J., Perreault, E. J., Peshkin, M. A., and
Kuiken, T. A. (2013). Development of a mechatronic platform and validation of methods
for estimating ankle stiffness during the stance phase of walking. Journal of Biomechanical
Engineering, 135(8).
131
[Sanchez et al., 2018] Sanchez, J., Corrales, J.-A., Bouzgarrou, B.-C., and Mezouar, Y.
(2018). Robotic manipulation and sensing of deformable objects in domestic and industrial
applications: a survey. The International Journal of Robotics Research, 37(7):688–716.
[Santello et al., 1998] Santello, M., Flanders, M., and Soechting, J. F. (1998). Postural hand
synergies for tool use. The Journal of neuroscience : the official journal of the Society for
Neuroscience, 18(23):10105–15.
[Santello et al., 2002] Santello, M., Flanders, M., and Soechting, J. F. (2002). Patterns
of hand motion during grasping and the influence of sensory guidance. The Journal of
neuroscience : the official journal of the Society for Neuroscience, 22(4):1426–35.
[Simon et al., 2017] Simon, T., Joo, H., Matthews, I., and Sheikh, Y. (2017). Hand keypoint
detection in single images using multiview bootstrapping. In Proceedings - 30th IEEE
Conference on Computer Vision and Pattern Recognition, CVPR 2017, volume 2017-
Janua, pages 4645–4653.
[Smith, 1988] Smith, L. (1988). A tutorial on Principal Components Analysis.
[Susman, 1994] Susman, R. L. (1994). Fossil evidence for early hominid tool use. Science,
265(5178):1570–1573.
[Todorov, 2014] Todorov, E. (2014). Convex and analytically-invertible dynamics with con-
tacts and constraints: Theory and implementation in MuJoCo. In 2014 IEEE Interna-
tional Conference on Robotics and Automation (ICRA), pages 6054–6061. IEEE.
[Todorov et al., 2012] Todorov, E., Erez, T., and Tassa, Y. (2012). MuJoCo: A physics
engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent
Robots and Systems, pages 5026–5033. IEEE.
[Van De Ruit et al., 2020] Van De Ruit, M., Cavallo, G., Lataire, J., Van Der Helm, F. C.,
Mugge, W., Van Wingerden, J. W., and Schouten, A. C. (2020). Revealing Time-Varying
Joint Impedance with Kernel-Based Regression and Nonparametric Decomposition. IEEE
Transactions on Control Systems Technology, 28(1):224–237.
[Viviani and Flash, 1995] Viviani, P. and Flash, T. (1995). Minimum-Jerk, Two-Thirds
Power Law, and Isochrony: Converging Approaches to Movement Planning. Journal
of Experimental Psychology: Human Perception and Performance, 21(1):32–53.
[Viviani and Stucchi, 1992] Viviani, P. and Stucchi, N. (1992). Biological Movements Look
Uniform: Evidence of Motor-Perceptual Interactions. Journal of Experimental Psychology:
Human Perception and Performance, 18(3):603–623.
[Walk and Homan, 1984] Walk, R. D. and Homan, C. P. (1984). Emotion and dance in
dynamic light displays. Bulletin of the Psychonomic Society, 22(5):437–440.
[Wei et al., 2016] Wei, S. E., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016). Con-
volutional pose machines. In Proceedings of the IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, volume 2016-Decem, pages 4724–4732.
132
[Weiss and Flanders, 2004] Weiss, E. J. and Flanders, M. (2004). Muscular and Postural
Synergies of the Human Hand. Journal of Neurophysiology, 92(1):523–535.
[Whitlock and Schluter, 2009] Whitlock, M. C. and Schluter, D. (2009). The analysis of
biological data. Number 574.015195 W5.
[Zago et al., 2018] Zago, M., Matic, A., Flash, T., Gomez-Marin, A., and Lacquaniti, F.
(2018). The speed-curvature power law of movements: a reappraisal. Experimental Brain
Research, 236(1):69–82.
[Zatsiorsky, 2002] Zatsiorsky, V. M. (2002). Kinetics of human motion. Human Kinetics.
133