基于力跟踪的上肢康复机器人系统中视觉与触觉反馈融合技术研究

doi:10.3969/j.issn.1006-9771.2021.04.014

《中国康复理论与实践》 ›› 2021, Vol. 27 ›› Issue (4): 478-486.doi: 10.3969/j.issn.1006-9771.2021.04.014

基于力跟踪的上肢康复机器人系统中视觉与触觉反馈融合技术研究

王昱^1,^2,³,吴向东¹,施长城^2,³(),张佳楫^2,³,李娜^2,³,马冶浩^2,³,陶亮⁴,唐敏⁴,左国坤^2,³

^1.西南交通大学机械工程学院，四川成都市 610031
^2.中国科学院宁波材料与工程研究所，慈溪生物医学工程研究所，浙江宁波市 315300
^3.中国科学院宁波材料与工程研究所，医用植介入材料浙江省工程研究中心，浙江宁波市 315300
^4.宁波市康复医院神经康复科，浙江宁波市 315040

收稿日期:2020-08-21 修回日期:2020-11-19 出版日期:2021-04-25 发布日期:2021-04-20
通讯作者: 施长城 E-mail:changchengshi@nimte.ac.cn
作者简介:王昱(1994-)，男，汉族，山西忻州市人，硕士研究生，主要研究方向：人机交互技术|施长城(1981-)，男，汉族，重庆市人，博士，副研究员，硕士生导师，主要研究方向：多感觉反馈融合、生命体征监测与康复工程技术。
基金资助:
宁波市"科技创新2025"重大专项项目(2018B10073);浙江省重点研发计划项目(2019C03090);浙江省自然科学基金项目(LQ20F030003);中国博士科学基金项目(2019M662128)

Visual and Haptic Feedback Fusion Based on Force Tracking in Upper-limb Rehabilitation Robot System

Yu WANG^1,^2,³,Xiang-dong WU¹,Chang-cheng SHI^2,³(),Jia-ji ZHANG^2,³,Na LI^2,³,Ye-hao MA^2,³,Liang TAO⁴,Min TANG⁴,Guo-kun ZUO^2,³

^1.School of Mechanical Engineering, Southwest Jiaotong University, Chengdu, Sichuan 610031, China
^2.Cixi Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo, Zhejiang 315300, China
^3.Zhejiang Engineering Research Center for Biomedical Materials, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo, Zhejiang 315300, China
^4.Department of Neurorehabilitation, Ningbo Rehabilitation Hospital, Ningbo, Zhejiang 315040, China

Received:2020-08-21 Revised:2020-11-19 Published:2021-04-25 Online:2021-04-20
Contact: Chang-cheng SHI E-mail:changchengshi@nimte.ac.cn
Supported by:
Major Scientific and Technological Projects in Ningbo City(2018B10073);Key Research and Development Program of Zhejiang(2019C03090);Natural Science Foundation of Zhejiang Province(LQ20F030003);China Postdoctoral Science Foundation Grant(2019M662128)

摘要/Abstract

摘要： 目的

针对现有上肢康复训练系统提供视觉和触觉反馈无法关联的问题，本文以自主研发的末端牵引式上肢康复机器人为基础，研究基于力跟踪的视觉与触觉反馈融合技术及其对于上肢训练的效果。

方法

在虚拟环境中构建力学模型的基础上，本文设计两种与视觉反馈融合的触觉反馈，分别为两物体间靠近时的排斥力以及物体在介质平面移动时的摩擦力，进而采用基于力跟踪的机器人控制算法将虚拟环境中构建的力反馈通过操纵杆传递给用户。招募8例健康受试者，分别在有无反馈融合模式下进行对照训练。在训练过程中采集受试者与系统的实际和期望力反馈，以及三角肌前束、三角肌后束、肱二头肌和肱三头肌的表面肌电信号。计算有反馈融合模式下期望与实际力反馈之间的均方根误差，用于表征基于力跟踪的多感觉反馈融合效果。计算两种模式下的肌电积分值(iEMG)和单位时间内肌电幅值(EMG/T)以探究融合反馈技术对上肢运动训练的影响。

结果

在有反馈融合模式下，实际与期望交互力反馈的均方根误差为(0.757±0.171) N；肱二头肌、肱三头肌、三角肌前束和后束的iEMG均显著大于无反馈融合模式下(|t| > 7.965, P < 0.001)；前三块肌肉的EMG/T显著大于无反馈融合模式下(|t| > 6.363, P < 0.001)。

结论

设计的上肢康复机器人训练系统可以精确地将虚拟环境中构建的力反馈传递给用户，通过视觉与触觉融合增加机器人系统对于训练者外周神经功能的刺激，促使训练者付出更多的努力。基于力跟踪的视觉与触觉反馈融合技术的优势在于可以在虚拟环境中自由构建力学模型，力反馈模式不受空间位置的限制，且可以在同一位置叠加两种以上的力学模型，从而使力反馈效果与虚拟环境中的视觉反馈更加匹配，激发训练者的运动康复兴趣，增强人机交互体验感。

关键词: 上肢, 运动, 康复, 虚拟现实, 力反馈, 表面肌电, 多感觉融合反馈

Abstract: Objective

To solve the issue regarding a low correlation between visual and haptic feedback provided by the current upper-limb rehabilitation training system, this study was implemented based on the end-effector based upper-limb rehabilitation robot developed in the lab. A novel visual and haptic feedback fusion technology based on force tracking was investigated and its effect on upper-limb training was also studied.

Methods

Based on the force model constructed in a virtual environment, two types of haptic feedbacks correlated to the visual feedback were designed, including the repulsive force when two objects getting close and the friction force when the object moving above medium surfaces. The haptic feedback constructed in the virtual environment was delivered to the trainees by using force tracking based on robot controlling algorithm. Eight health subjects were recruited and trained with and without feedback fusion. In the training process, the actual and expected haptic feedbacks as well as the surface electromyography (EMG) signals from anterior deltoid, posterior deltoid, biceps, and triceps were collected. The root means square error (RMSE) between the actual and expected haptic feedback was calculated under the feedback fusion training mode to characterize the force tracking-based multi-sensory feedback fusion technology. The integrated EMG values (iEMG) and EMG amplitudes per unit time (EMG/T) under two training modes were measured to explore the effect of feedback fusion technology on the upper-limb motor training.

Results

Under feedback fusion training mode, the RMSE between actual and expected haptic feedback was (0.757±0.171) N. The values of iEMG from four muscles were significantly higher (|t| > 7.965, P < 0.001), and the values of EMG/T from the biceps, triceps and anterior deltoid were significantly larger under feedback fusion training mode than under the training mode without feedback fusion.

Conclusion

The proposed upper-limb rehabilitation robot training system could accurately transmit the haptic feedback constructed under the virtual environment to the trainees. This system could increase the stimulation to trainees' peripheral nervous function through visual and haptic feedback fusion as well as increase the trainees' training effort. The advantages of force tracking-based visual and haptic feedback fusion technology are to freely construct the force model under the virtual environment and the haptic feedback mode is not constrained by the spatial position. Moreover, two or more types of force models can be superimposed in the same spatial position by using this technology that could improve the matching effect between haptic feedback and visual feedback under a virtual environment. The trainees' motor rehabilitation interest could be stimulated and the experience feeling of human-robot interaction could also be enhanced.

Key words: upper-limb, motor, rehabilitation, virtual reality, haptic feedback, surface electromyography, multi-sensory feedback fusion

中图分类号:

R496

王昱,吴向东,施长城,张佳楫,李娜,马冶浩,陶亮,唐敏,左国坤. 基于力跟踪的上肢康复机器人系统中视觉与触觉反馈融合技术研究[J]. 《中国康复理论与实践》, 2021, 27(4): 478-486.

Yu WANG,Xiang-dong WU,Chang-cheng SHI,Jia-ji ZHANG,Na LI,Ye-hao MA,Liang TAO,Min TANG,Guo-kun ZUO. Visual and Haptic Feedback Fusion Based on Force Tracking in Upper-limb Rehabilitation Robot System[J]. 《Chinese Journal of Rehabilitation Theory and Practice》, 2021, 27(4): 478-486.

图/表 9

图1

图2

图3

图4

图5

图6

表1

表2

图7

参考文献 35

1	国家统计局. 中华人民共和国2019年国民经济和社会发展统计公报[J]. 中国统计, 2020(3): 8-22.
	National Bureau of Statistics. The People's Republic of China 2019 National Economic and Social Development Statistical Communique [J]. Chin Stat, 2020(3): 8-22.
2	中国脑卒中防治报告2019编写组. «中国脑卒中防治报告2019»概要[J]. 中国脑血管病杂志, 2020, 17(5): 272-281.
	China Stroke Prevention and Treatment Report2019Writing Group. Summary of "China Stroke Prevention and Treatment Report2019" [J]. Chin J Cerebrovasc Dis, 2020, 17(5): 272-281.
3	周慧琳,左国坤,施长城,等. 脑卒中患者上肢康复脑功能及其康复治疗技术[J]. 中国医学工程, 2017, 25(5): 28-33.
	Zhou H L, Zuo G K, Shi C C, et al. Brain function and rehabilita-tion therapies of upper limb rehabilitation in stroke patients [J]. Chin Med Eng, 2017, 25(5): 28-33.
4	王娜娜,路微波,吴毅,等上肢康复机器人对脑卒中患者上肢功能及日常生活能力影响的研究进展[J]. 中华物理医学与康复杂志, 2017, 39(9): 706-708.
	Wang N N, Lu W B, Wu Y, et al. Application of robot-assisted upper limb rehabilitation for stroke(review) [J]. Chin J Phys Med Rehabil, 2017, 39(9): 706-708.
5	李宇淇,曾庆,黄国志. 上肢康复机器人在脑卒中的临床应用进展[J]. 中国康复理论与实践, 2020, 26(3): 310-314.
	Li Y Q, Zeng Q, Huang G Z. Application of robot-assisted upper limb rehabilitation for stroke (review) [J]. Chin J Rehabil Theory Pract, 2020, 26(3): 310-314.
6	胡宇川,季林红. 从医学角度探讨偏瘫上肢康复训练机器人的设计[J]. 中国组织工程研究, 2004, 8(34): 7754-7756.
	Hu Y C, Ji L H. Designing the rehabilitation training robot for hemiplegic upper-limb from the view of the medical science [J]. Chin J Tissue Res, 2004, 8(34): 7754-7756.
7	李雅楠,左国坤,崔志琴,等. 虚拟现实技术在康复训练中的应用进展[J]. 中国康复医学杂志, 2017, 32(9): 1091-1094.
	Li Y N, Zuo G K, Cui Z Q, et al. Application progress of virtual reality technology in rehabilitation training [J]. Chin J Rehabil Med, 2017, 32(9): 1091-1094.
8	李娜,崔志琴,施长城,等. 实时视觉轨迹反馈对上肢运动学习的影响[J]. 中国康复理论与实践, 2019, 25(2): 217-223.
	Li N, Cui Z Q, Shi C C, et al. Effect of real-time visual trajectory feedback on upper limb movement learning [J]. Chin J Rehabil Theory Pract, 2019, 25(2): 217-223.
9	Sveistrup H. Motor rehabilitation using virtual reality [J]. J Neuroeng Rehabil, 2004, 1(1): 10-18.
10	林佳丽,贾杰. 脑卒中后感觉训练在上肢及手功能康复中的研究进展[J]. 中国康复医学杂志, 2020, 35(4): 488-492.
	Lin J L, Jia J. Research progress of sensory training in rehabilitation of upper limbs and hands after strok [J]. Chin J Rehabil Med, 2020, 35(4): 488-492.
11	Marchal-Crespo L, Rappo N, Riener R. The effectiveness of robotic training depends on motor task characteristics [J]. Exp Brain Res, 2017, 235(12): 3799-3816.
12	Wright Z A, Lazzaro E, Thielbar K O, et al. Robot training with vector fields based on stroke survivors' individual movement statistics [J]. IEEE Trans Neural Syst Rehabil Eng, 2018, 26(2): 307-323.
13	Majeed Y A, Awadalla S, Patton J L. Effects of robot viscous forces on arm movements in chronic stroke survivors: a randomized crossover study [J]. Res Square, 2020, 17(1): 156.
14	Lokesh R, Ranganathan R. Haptic assistance that restricts the use of redundant solutions is detrimental to motor learning [J]. IEEE Trans Neural Syst Rehabil Eng, 2020, 28(6): 1373-1380.
15	Di Diodato L M, Mraz R, Baker S N, et al. A haptic force feedback device for virtual reality-fMRI experiments [J]. IEE Trans Neural Syst Rehabil Eng, 2007, 15(4): 570-576.
16	Basalp E, Marchal-Crespo L, Rauter G, et al. Rowing simulator modulates water density to foster motor learning [J]. Front Robot AI, 2019, 6(24): 1-17.
17	Zitzewitz J V, Wolf P, Novakovic V, et al. Real-time rowing simulator with multimodal feedback [J]. Sports Tech, 2008, 1(6): 257-266.
18	Basalp E, Bachmann P, Gerig N, et al. Configurable 3D rowing model renders realistic forces on a simulator for indoor training [J]. Appl Sci, 2020, 10(3): 734
19	Keller U, Van Hedel H J A, Klamroth-Marganska V, et al. ChARMin: the first actuated exoskeleton robot for pediatric arm rehabilitation [J]. IEEE ASME Trans Mechatron, 2016, 21(5): 2201-2213.
20	李雅楠,左国坤,崔志琴,等. 上肢康复机器人建模与力反馈控制策略实验研究[J]. 制造业自动化, 2017, 39(5): 37-40.
	Li Y N, Zuo G K, Cui Z Q, et al. Upper-limb rehabilitation robot modeling and force feed back control research [J]. Manufacturing Automation, 2017, 39(5): 37-40.
21	李娜,崔志琴,左国坤,等. 基于末端牵引式康复机器人的定量化上肢运动功能评估研究[J]. 北京生物医学工程, 2018, 37(6): 581-588.
	Li N, Cui Z Q, Zuo G K, et al. Quantitative evaluation on upper limb motor function based on an end-effector rehabilitation robot [J]. Beijing Biomed Eng, 2018, 37(6): 581-588.
22	Duan J J, Gan Y H, Chen M, et al. Adaptive variable impedance control for dynamic contact force tracking in uncertain environment [J]. Rob Auton Syst, 2018, 102: 54-65.
23	王嘉津,左国坤,张佳楫,等. 腕功能康复机器人按需辅助控制策略研究[J]. 生物医学工程学杂志, 2020, 37(1): 129-135.
	Wang J J, Zuo G K, Zhang J J, et al. Research on assist-as-needed control strategy of wrist function-rehabilitation robot [J]. J Biomeical Eng, 2020, 37(1): 129-135.
24	Lahr G J G, Soares J V R, Garcia H B, et al. Understanding the implementation of impedance control in industrial robots [C]. Proceedings of the 13rd Latin America Robotics Symposium, 2016.
25	Al-Shuka H F N, Leonhardt S, Zhu W H, et al. Active impedance control of bioinspired motion robotic manipulators: an overview [J]. Appl Bionics Biomech, 2018, 2018: 8203054.
26	韩京清. 自抗扰控制技术[J]. 前沿科学, 2007(1): 25-32.
	Han J Q. Active disturbance rejection control technonlgy [J]. Frontier Science, 2007(1): 25-32.
27	韩京清. 从PID技术到"自抗扰控制"技术 [J]. 控制工程, 2002, 9(3): 13-18.
	Han J Q. From PID technology to ADRC technology [J]. Control Eng Chin, 2002, 9(3): 13-18.
28	马立,王敬萍,李丰甜,等. 精密定位系统的摩擦力建模与补 [J]. 光学精密工程, 2019, 27(1): 121-128.
	Ma L, Wang J P, Li F T, et al. Friction modeling and compensation of precision position system [J]. Optics Precision Eng, 2019, 27(1): 121-128.
29	Guidali M, Keller U, Klamroth-Marganska V, et al. Estimating the patient's contribution during robot-assisted therapy [J]. J Rehabil Res Dev, 2013, 50(3): 379-393.
30	Bisoffi A, Da Lio M, Teel A R, et al. Global asymptotic stability of a PID control system with coulomb friction [J]. IEEE Trans Automat Contr, 2018, 63(8): 2654-2661.
31	Kim S. Moment of inertia and friction torque coefficient identification in a servo drive system [J]. IEEE Trans Ind Electron, 2019, 66(1): 60-70.
32	丛爽, Alessandro D C. 两种补偿动态摩擦力的先进控制策略[J]. 自动化学报, 1998, 24(2): 236-240.
	Cong S, Alessandro D C. Two advanced control strategies for compensating dynamic friction [J]. Acta Automatica Sinica, 1998, 24(2): 236-240.
33	Phinyomark A, Phukpattaranont P, Limsakul C. Feature reduction and selection for EMG signal classification [J]. Expert Syst Appl, 2012, 39(8): 7420-7431.
34	Farina D, Jiang N, Rehbaum H, et al. The extraction of neural information from the surface EMG for the control of upper-limb prostheses: emerging avenues and challenges [J]. IEEE Trans Neural Syst Rehabil Eng, 2014, 22(4): 797-809.
35	何勇,施长城,左国坤,等. 训练轨迹对上肢肌肉协同的影响[J]. 北京生物医学工程, 2019, 38(5): 441-449.
	He Y, Shi C C, Zuo G K, et al. The effect of training trajectories on upper limb muscle synergy [J]. Beijing Biomed Eng, 2019, 38(5): 441-449.

肌肉	有反馈融合模式	无反馈融合模式	t值	P值
肱二头肌	2.196±2.315	1.513±1.160	-8.601	< 0.001
肱三头肌	2.352±2.119	1.848±1.565	-12.710	< 0.001
三角肌前束	4.845±2.780	3.227±1.746	-19.662	< 0.001
三角肌后束	3.784±4.044	3.133±2.911	-7.965	< 0.001

肌肉	有反馈融合模式	无反馈融合模式	t值	P值
肱二头肌	5.944±4.893	4.851±2.811	-6.363	< 0.001
肱三头肌	6.640±4.327	6.244±4.648	-7.710	< 0.001
三角肌前束	15.400±6.723	11.940±5.605	-19.607	< 0.001
三角肌后束	10.279±7.926	10.070±8.050	-1.891	0.059

[1]	罗丽华, 王雨生, 李剑锋, 董继革. 术后早期综合康复对儿童青少年肱骨髁上骨折伴尺神经损伤的效果[J]. 《中国康复理论与实践》, 2024, 30(1): 105-110.
[2]	王子豪, 李昕华, 蒋慧萍, 郭赛男, 梁秋曼, 史婷奇. 全膝关节置换术后短期膝关节功能及其影响因素[J]. 《中国康复理论与实践》, 2024, 30(1): 111-118.
[3]	周治宁, 周容, 肖燕文, 王博文, 吕娇娇, 刘宇. 多靶区经颅直流电刺激对健康成年人工作记忆-姿势控制双任务表现的影响[J]. 《中国康复理论与实践》, 2024, 30(1): 21-28.
[4]	林娜, 高菡璐, 卢惠苹, 陈燕清, 郑军凡, 陈述荣. 虚拟现实技术对脑卒中上肢功能影响的弥散张量成像研究[J]. 《中国康复理论与实践》, 2024, 30(1): 61-67.
[5]	陈珺雯, 陈谦, 陈程, 李淑月, 刘玲玲, 吴存书, 龚翔, 鲁俊, 许光旭. 改良八段锦身体活动对脑卒中患者心肺功能、运动功能和日常生活活动能力的效果[J]. 《中国康复理论与实践》, 2024, 30(1): 74-80.
[6]	魏梦力, 钟亚平, 周易文, 桂辉贤, 关烨明, 于婷婷. 单侧前交叉韧带重建患者步行双侧下肢肌肉协同模式差异[J]. 《中国康复理论与实践》, 2024, 30(1): 95-104.
[7]	史佳伟, 李凌宇, 杨浩杰, 王琴潞, 邹海欧. 预康复对全膝关节置换术后患者的有效性：系统综述的系统综述[J]. 《中国康复理论与实践》, 2023, 29(9): 1057-1064.
[8]	张冠聪, 黄秋晨, 顾蕊, 刘四海, 胡春英, 刘克敏. 不同神经肌肉训练方法对早期膝骨关节炎患者疼痛和运动功能效果的比较[J]. 《中国康复理论与实践》, 2023, 29(9): 1090-1097.
[9]	蔡华年, 费思先, 张忆晨, 孙青, 郭帅, 宋韬. 基于导纳控制的双边康复机器人运动辅助分析[J]. 《中国康复理论与实践》, 2023, 29(9): 1104-1109.
[10]	孙藤方, 任梦婷, 杨琳, 王耀霆, 王红雨, 闫兴洲. 高压氧治疗联合重复外周磁刺激干预脑卒中患者踝运动功能和平衡能力的效果[J]. 《中国康复理论与实践》, 2023, 29(8): 875-881.
[11]	马圣楠, 柯竟悦, 董洪铭, 李建萍, 张洪浩, 刘超, 沈双, 李古强. 核心稳定性训练干预前交叉韧带重建术后动态平衡及表面肌电的效果[J]. 《中国康复理论与实践》, 2023, 29(8): 882-889.
[12]	王亚楠, 刘西花. 脑卒中偏瘫患者主观和客观平衡功能测量的相关性及预测效能[J]. 《中国康复理论与实践》, 2023, 29(8): 890-895.
[13]	魏晓微, 杨剑, 魏春艳, 贺启令. 学校环境下适应性体育课程促进智力与发展性残疾儿童心理运动发展的系统综述[J]. 《中国康复理论与实践》, 2023, 29(8): 910-918.
[14]	王海云, 王寅, 周信杰, 何爱群. 基于“中枢-外周-中枢”理论的经颅直流电刺激结合针刺干预脑卒中患者中枢及上肢功能的效果[J]. 《中国康复理论与实践》, 2023, 29(8): 919-925.
[15]	陈怡婷, 王倩, 崔慎红, 李映彩, 张思鈺, 魏衍旭, 任慧, 冷军, 陈斌. 双侧序贯重复经颅磁刺激干预脑卒中患者上肢运动功能的效果[J]. 《中国康复理论与实践》, 2023, 29(8): 926-932.

基于力跟踪的上肢康复机器人系统中视觉与触觉反馈融合技术研究

Visual and Haptic Feedback Fusion Based on Force Tracking in Upper-limb Rehabilitation Robot System

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 35

相关文章 15

Metrics

本文评价

推荐阅读 0