Extracting blood-induced color changes on the face for non-contact heart rate estimation

Heart rate (HR) estimation is an essential component of health monitoring, and provides useful insights regarding the physiological and emotional state of humans. In the past decade or so, researchers have explored newer approaches for contactless HR estimation, primarily to overcome the discomfort or dermatitis associated with conventional methods that require physical contact. Non-contact HR estimation using cameras is an example of one such method. The method focuses on the blood volume pulse (BVP), that causes slight temporal changes in facial skin color captured in videos. By examining these color variations, it becomes possible to estimate HR. However, due to the small magnitude of these color changes, the accuracy of HR estimation is adversely affected by facial movements, ambient lighting variations, and noise.

To address these challenges, a team of researchers from Japan have now developed a novel method that leverages the temporal characteristics of the blood pulse. Importantly, it builds on the ability of the pulse to exhibit quasi-periodic behavior, which distinguishes it from noise artifacts. The study was led by Dr. Yoshihiro Maeda, Junior Associate Professor, from the Department of Electrical Engineering at the Tokyo University of Science and is published in Volume 11 of IEEE Access journal on 9 June 2023. Professor Takayuki Hamamoto and Kosuke Kurihara from the Tokyo University of Science and Associate Professor Daisuke Sugimura from the Department of Computer Science, Tsuda University, were also a part of this study.

The proposed method utilizes dynamic mode decomposition (DMD), a technique that analyzes spatio-temporal structures in multi-dimensional time-series signals. It also employs adaptive selection of the optimal spatio-temporal structure based on medical knowledge of HR frequencies. “Our method, unlike previous applications of DMD, effectively models and extracts the BVP signal by incorporating physics-informed DMD in a time-delay coordinate system, taking into account the nonlinearity and quasi-periodicity of the BVP dynamics,” explains Kosuke Kurihara, a Ph.D. student.

The proposed method relies solely on tracking time-series data from videos of a person’s face, eliminating the need for any attached detectors on the person’s body. In this method, the video time-series of the face, monitoring continuous changes, are converted into RGB time-series signals, which helps in extracting information of blood volume changes occurring beneath the skin. After effectively dealing with noise or misinformation that might creep into the data, the observed RGB signals are then converted to pulse wave information data.

Using the DMD method in a time-delayed coordinate system with conservative dynamics modeling, pulse waves containing major and accurate information can be extracted to estimate HR.

To demonstrate the efficacy of this method, the researchers used 67 facial videos from three publicly available datasets – namely TokyoTech Remote PPG dataset, MR-NIRP dataset, and UBFC-RPPG dataset. The results of this method were then compared with other non-contact HR estimation methods, including DistancePPG, SparsePPG, SAMC, Hierarchical, and MTTS-CAN.

Interestingly, the proposed method adaptively selects the dynamic mode that contains the most pulse wave components, based on the knowledge of the typical range of pulse wave components. As a result, the method showed a 36.5% improvement in estimation accuracy compared to conventional methods, especially in scenes with ambient light fluctuations.

“This achievement is expected to play a significant role as a fundamental technology for vital monitoring systems in the medical and fitness fields. The breakthrough contactless method holds great potential for non-contact heart rate estimation in various applications, such as remote health monitoring and physiological assessments,” concludes Dr. Maeda. The research findings provide new possibilities for enhancing healthcare technologies and improving overall patient comfort and well-being. Going ahead, further research will be needed to explore techniques that incorporate multispectral information, which can contribute to reducing noise and improving the accuracy of the method.

We wish Dr. Maeda and his team luck for their ongoing efforts towards addressing the remaining issues with this novel method.  

***

Reference                     

DOI: https://doi.org/10.1109/ACCESS.2023.3284465

About The Tokyo University of Science

Tokyo University of Science (TUS) is a well-known and respected university, and the largest science-specialized private research university in Japan, with four campuses in central Tokyo and its suburbs and in Hokkaido. Established in 1881, the university has continually contributed to Japan's development in science through inculcating the love for science in researchers, technicians, and educators.

With a mission of “Creating science and technology for the harmonious development of nature, human beings, and society", TUS has undertaken a wide range of research from basic to applied science. TUS has embraced a multidisciplinary approach to research and undertaken intensive study in some of today's most vital fields. TUS is a meritocracy where the best in science is recognized and nurtured. It is the only private university in Japan that has produced a Nobel Prize winner and the only private university in Asia to produce Nobel Prize winners within the natural sciences field.

Website: https://www.tus.ac.jp/en/mediarelations/

About Junior Associate Professor Dr. Yoshihiro Maeda from Tokyo University of Science

Dr. Yoshihiro Maeda is a Junior Associate Professor at the Department of Electrical Engineering, Tokyo University of Science. He received his B.E., M.E., and Ph.D. degrees in information engineering from the Nagoya Institute of Technology, Japan, in 2013, 2015, and 2019, respectively. He became an Assistant Professor with the Tokyo University of Science, Japan, in 2019. His research interests include image processing and multispectral sensing. He has more than 20 publications to his credit on topics like image filtering, and FIR filters.

About Professor Dr. Takayuki Hamamoto from Tokyo University of Science

Dr. Takayuki Hamamoto is a Professor at the Department of Electrical Engineering, Tokyo University of Science. He received his B.E. and M.E. degrees in electrical engineering from the Tokyo University of Science, Tokyo, Japan, in 1992 and 1994, respectively, and the Dr. (Eng.) degree in electrical engineering from The University of Tokyo, in 1997. His research interests include image processing, computer vision, and computational image sensors.

Funding Information

JSPS KAKENHI

Grant Number 23KJ1961 and 22K12080