The ability to track human operators' hand usage when working in production plants and factories is critically important for developing realistic digital factory simulators as well as manufacturing process control. We propose a proof-of-concept instrumented glove with only a few strain gage sensors and a microcontroller that continuously tracks and records the hand configuration during actual use. At the heart of our approach is a trainable system that can predict the fourteen joint angles in the hand using only a small set of strain sensors. First, ten strain gages are placed at various joints in the hand to optimize the sensor layout using the English letters in the American Sign Language (ASL) as a benchmark for assessment. Next, the best sensor configurations for three through ten strain gages are computed using a support vector machine (SVM) classifier. Following the layout optimization, our approach learns a mapping between the sensor readouts to the actual joint angles optically captured using a Leap Motion system. Five regression methods including linear, quadratic, and neural regression are then used to train the mapping between the strain gage data and the corresponding joint angles. The final proposed model involves four strain gages mapped to the fourteen joint angles using a two-layer feed-forward neural network (NN).
Skip Nav Destination
Article navigation
September 2019
Research-Article
High Degree of Freedom Hand Pose Tracking Using Limited Strain Sensing and Optical Training
Wentai Zhang,
Wentai Zhang
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Search for other works by this author on:
Jonelle Z. Yu,
Jonelle Z. Yu
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Search for other works by this author on:
Fangcheng Zhu,
Fangcheng Zhu
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Search for other works by this author on:
Yifang Zhu,
Yifang Zhu
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Search for other works by this author on:
Zhangsihao Yang,
Zhangsihao Yang
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Search for other works by this author on:
Nurcan Gecer Ulu,
Nurcan Gecer Ulu
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Search for other works by this author on:
Batuhan Arisoy,
Batuhan Arisoy
Vision Technologies and Solutions Group,
Siemens Corporate Technology,
755 College Road,
Princeton, NJ 08540
Siemens Corporate Technology,
755 College Road,
Princeton, NJ 08540
Search for other works by this author on:
Levent Burak Kara
Levent Burak Kara
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
e-mail: lkara@cmu.edu
Carnegie Mellon University,
Pittsburgh, PA 15213
e-mail: lkara@cmu.edu
1Corresponding author.
Search for other works by this author on:
Wentai Zhang
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Jonelle Z. Yu
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Fangcheng Zhu
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Yifang Zhu
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Zhangsihao Yang
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Nurcan Gecer Ulu
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
Carnegie Mellon University,
Pittsburgh, PA 15213
Batuhan Arisoy
Vision Technologies and Solutions Group,
Siemens Corporate Technology,
755 College Road,
Princeton, NJ 08540
Siemens Corporate Technology,
755 College Road,
Princeton, NJ 08540
Levent Burak Kara
Department of Mechanical Engineering,
Carnegie Mellon University,
Pittsburgh, PA 15213
e-mail: lkara@cmu.edu
Carnegie Mellon University,
Pittsburgh, PA 15213
e-mail: lkara@cmu.edu
1Corresponding author.
Manuscript received September 15, 2018; final manuscript received February 27, 2019; published online June 17, 2019. Assoc. Editor: Jitesh H. Panchal.
J. Comput. Inf. Sci. Eng. Sep 2019, 19(3): 031014 (7 pages)
Published Online: June 17, 2019
Article history
Received:
September 15, 2018
Revised:
February 27, 2019
Citation
Zhang, W., Yu, J. Z., Zhu, F., Zhu, Y., Yang, Z., Ulu, N. G., Arisoy, B., and Kara, L. B. (June 17, 2019). "High Degree of Freedom Hand Pose Tracking Using Limited Strain Sensing and Optical Training." ASME. J. Comput. Inf. Sci. Eng. September 2019; 19(3): 031014. https://doi.org/10.1115/1.4043757
Download citation file:
Get Email Alerts
Cited By
Haptic Interaction Methods for Freehand Contour Generation on a Refreshable Pin Display
J. Comput. Inf. Sci. Eng
LLM4CAD: Multimodal Large Language Models for Three-Dimensional Computer-Aided Design Generation
J. Comput. Inf. Sci. Eng (February 2025)
Integrating Graph Retrieval Augmented Generation with Large Language Models for Supplier Discovery
J. Comput. Inf. Sci. Eng
Related Articles
Development of a Strain Transferring Sensor Housing for a Lumbar Spinal Fusion Detection System
J. Med. Devices (June,2007)
Strain Transfer Analysis of Integrated Surface Acoustic Wave Sensors
J. Electron. Packag (December,2014)
A Multi-Sensor Approach for Rapid Digitization and Data Segmentation in Reverse Engineering
J. Manuf. Sci. Eng (November,2000)
Related Proceedings Papers
Related Chapters
Manipulability-Maximizing SMP Scheme
Robot Manipulator Redundancy Resolution
SYNNC: Symmetric Kernel Neural Network for Data Clustering
Intelligent Engineering Systems through Artificial Neural Networks Volume 18
Comparing Probabilistic Graphical Model Based and Gaussian Process Based Selections for Predicting the Temporal Observations
Intelligent Engineering Systems through Artificial Neural Networks, Volume 20