A Multimodal Model of ECG and Heart Sound Signal by Considering Normal and Abnormal Heart

Prihatin Oktivasari - Politeknik Negeri Jakarta, Depok, 16425, Indonesia
Freddy Haryanto - Institut Teknologi Bandung, Jl. Ganesa 10, Bandung, 40132, Indonesia
- Suprijadi - Institut Teknologi Bandung, Jl. Ganesa 10, Bandung, 40132, Indonesia

Citation Format:

DOI: http://dx.doi.org/10.30630/joiv.6.4.1220


Analysis of the opening and closing of heart valves and the movement of blood flow in the heart are important in the domain of early detection of heart conditions. To build this correlation model, multimodal signals from electrocardiography and stethoscope are needed. Multimodal signaling was performed using primary data with the same sampling at 10 seconds by recording the PQRST heart signal in the lying position using electrocardiography and the heart sound in the sitting position using an electronic stethoscope. Experimental results showed that the number of R peaks is the same as the number of S1 sound peaks, and also the number of T peaks with the number of S2 sound peaks, so it can be concluded that there is a regular signal pattern relationship between S1-S2 and the RT wave, namely the relationship at the end of the first peak of the QRS wave. The cardiac signal due to ventricular depolarization (ventricular contraction), the appearance of an S1 heart sound, and the association of the end of the next peak of the T wave of cardiac signals indicate ventricular repolarization and the appearance of an S2 heart sound. This is consistent with the fact that electrical events in cardiac activity occur before mechanical events in normal heart conditions. Based on the study of HRV parameters, heart sound signals can be used to determine HRV parameters. The results show the same number of peaks in normal hearts, while in abnormal hearts, there are differences in results because abnormal heart conditions have an erratic rhythmic pattern.


Multimodal signal; heart sound; ECG signal; HRV.

Full Text:



L. G. Tereshchenko, “Electrocardiogram as a screening tool in the general population: A strategic review,” J. Electrocardiol., vol. 46, no. 6, pp. 553–556, Nov. 2013, doi: 10.1016/J.JELECTROCARD.2013.07.005.

P. Stanić Molcer, I. Kecskés, V. Delić, E. Domijan, and M. Domijan, “Examination of formant frequencies for further classification of heart murmurs,” SIISY 2010 - 8th IEEE Int. Symp. Intell. Syst. Informatics, pp. 575–578, 2010, doi: 10.1109/SISY.2010.5647147.

Zègre-Hemsey JK, Garvey JL, Carey MG. Cardiac Monitoring in the Emergency Department. Crit Care Nurs Clin North Am. 2016 Sep;28(3):331-45. doi: 10.1016/j.cnc.2016.04.009. Epub 2016 Jul 2. PMID: 27484661; PMCID: PMC5630152.

D. E. Becker, “Fundamentals of electrocardiography interpretation.,” Anesth. Prog., vol. 53, no. 2, pp. 53–64, 2006, doi: 10.2344/0003-3006(2006)53[53:FOEI]2.0.CO;2.

L. Saclova, A. Nemcova, R. Smisek, L. Smital, M. Vitek, and M. Ronzhina, “Reliable P wave detection in pathological ECG signals,” Sci. Rep., vol. 12, no. 1, pp. 1–14, 2022, doi: 10.1038/s41598-022-10656-4.

V. Nigam and R. Priemer, “Accessing heart dynamics to estimate durations of heart sounds,” Physiol. Meas., vol. 26, no. 6, pp. 1005–1018, 2005, doi: 10.1088/0967-3334/26/6/010.

A. Castro, A. Moukadem, S. Schmidt, A. Dieterlen, and M. T. Coimbra, “Analysis of the electromechanical activity of the heart from synchronized ECG and PCG signals of subjects under stress,” BIOSIGNALS 2015 - 8th Int. Conf. Bio-Inspired Syst. Signal Process. Proceedings; Part 8th Int. Jt. Conf. Biomed. Eng. Syst. Technol. BIOSTEC 2015, no. January, pp. 49–56, 2015, doi: 10.5220/0005202400490056.

N. F. Hikmah, A. Arifin, T. A. Sardjono, and E. A. Suprayitno, “A signal processing framework for multimodal cardiac analysis,” 2015 Int. Semin. Intell. Technol. Its Appl. ISITIA 2015 - Proceeding, pp. 125–130, Aug. 2015, doi: 10.1109/ISITIA.2015.7219966.

P. Bachtiger et al., “Point-of-care screening for heart failure with reduced ejection fraction using artificial intelligence during ECG-enabled stethoscope examination in London, UK: a prospective, observational, multicentre study,” Lancet Digit. Heal., vol. 4, no. 2, pp. e117–e125, 2022, doi: 10.1016/S2589-7500(21)00256-9.

S. Leng, R. S. Tan, K. T. C. Chai, C. Wang, D. Ghista, and L. Zhong, “The electronic stethoscope,” Biomed. Eng. Online, vol. 14, no. 1, pp. 1–37, 2015, doi: 10.1186/s12938-015-0056-y.

W. Phanphaisarn, A. Roeksabutr, P. Wardkein, J. Koseeyaporn, and P. Yupapin, “Heart detection and diagnosis based on ECG and EPCG relationships,” Med. Devices Evid. Res., vol. 4, no. 1, pp. 133–144, 2011, doi: 10.2147/MDER.S23324.

X. Bao, Y. Deng, N. Gall, and E. N. Kamavuako, “Analysis of ECG and PCG time delay around auscultation sites,” BIOSIGNALS 2020 - 13th Int. Conf. Bio-Inspired Syst. Signal Process. Proceedings; Part 13th Int. Jt. Conf. Biomed. Eng. Syst. Technol. BIOSTEC 2020, vol. 4, no. Biostec 2020, pp. 206–213, 2020, doi: 10.5220/0008942602060213.

F. Chakir, A. Jilbab, C. Nacir, and A. Hammouch, “Recognition of cardiac abnormalities from synchronized ECG and PCG signals,” Phys. Eng. Sci. Med. 2020 432, vol. 43, no. 2, pp. 673–677, Apr. 2020, doi: 10.1007/S13246-020-00875-2.

F. Safara, S. Doraisamy, A. Azman, A. Jantan, and A. R. Abdullah Ramaiah, “Multi-level basis selection of wavelet packet decomposition tree for heart sound classification,” Comput. Biol. Med., vol. 43, no. 10, pp. 1407–1414, 2013, doi: 10.1016/j.compbiomed.2013.06.016.

S. Kang, R. Doroshow, J. McConnaughey, A. Khandoker, and R. Shekhar, “Heart Sound Segmentation toward Automated Heart Murmur Classification in Pediatric Patents,” Proc. - 8th Int. Conf. Signal Process. Image Process. Pattern Recognition, SIP 2015, pp. 9–12, 2016, doi: 10.1109/SIP.2015.11.

W. Phanphaisarn, A. Roeksabutr, P. Wardkein, J. Koseeyaporn, and P. P. Yupapin, “MDER-23324-heart-defect-detection-and-diagnosis-based-on-ecg-and-epcg-r,” Med. Devices Evid. Res., pp. 4–133, 2011, doi: 10.2147/MDER.S23324.

V. Millette and N. Baddour, “Signal processing of heart signals for the quantification of non-deterministic events,” Biomed. Eng. Online, vol. 10, no. 1, p. 10, 2011, doi: 10.1186/1475-925X-10-10.

X. Cui et al., “On the Variability of Heart Rate Variability-Evidence from Prospective Study of Healthy Young College Students,” doi: 10.3390/e22111302.

A. Kardalinos, “The second heart sound,” Am. Heart J., vol. 64, no. 5, pp. 610–616, 1962, doi: 10.1016/0002-8703(62)90245-4.

K. Chauhan and R. K. Chauhan, “Image Processing for Automated Diagnosis of Cardiac Diseases,” Image Process. Autom. Diagnosis Card. Dis., pp. 1–223, 2021, doi: 10.1016/B978-0-323-85064-3.09990-8.

X. Bao, Y. Deng, N. Gall, and E. N. Kamavuako, “Analysis of ECG and PCG time delay around auscultation sites,” BIOSIGNALS 2020 - 13th Int. Conf. Bio-Inspired Syst. Signal Process. Proceedings; Part 13th Int. Jt. Conf. Biomed. Eng. Syst. Technol. BIOSTEC 2020, no. January, pp. 206–213, 2020, doi: 10.5220/0008942602060213.

P. M. Galen, R. A. Warner, and R. H. S. Selvester, “(19) United States (12),” vol. 1, no. 19, 2006.

H. & Choate and Stewart, “United States US 2004O26O188A1 (12) Patent Application Publication,” vol. 1, no. 10, 2004.

H. Posada-Quintero, “Electrodermal Activity: What it can Contribute to the Assessment of the Autonomic Nervous System,” Dr. Diss., no. December 2016, pp. 1–167, 2016.

A. M. Noor and M. F. Shadi, “The heart auscultation: From sound to graphical,” ARPN J. Eng. Appl. Sci., vol. 9, no. 10, pp. 1924–1929, 2014.

S. Raj and R. GV, “Second Heart Sound (S2),” Essentials Cardiovasc. Exam., pp. 48–48, 2016, doi: 10.5005/jp/books/12702_14.

M. V. Shervegar and G. V. Bhat, “Automatic segmentation of Phonocardiogram using the occurrence of the cardiac events,” Informatics Med. Unlocked, vol. 9, no. December 2016, pp. 6–10, 2017, doi: 10.1016/j.imu.2017.05.002.

S. M. E. A. Debbal, “Analysis of the four heart sounds statistical study and spectro-temporal characteristics,” J. Med. Eng. Technol., vol. 44, no. 7, pp. 396–410, 2020, doi: 10.1080/03091902.2020.1799095.

V. Kudriavtsev, V. Polyshchuk, and D. L. Roy, “Heart energy signature spectrogram for cardiovascular diagnosis,” Biomed. Eng. Online, vol. 6, pp. 1–22, 2007, doi: 10.1186/1475-925X-6-16.

E. Delgado-Trejos, A. F. Quiceno-Manrique, J. I. Godino-Llorente, M. Blanco-Velasco, and G. Castellanos-Dominguez, “Digital auscultation analysis for heart murmur detection,” Ann. Biomed. Eng., vol. 37, no. 2, pp. 337–353, 2009, doi: 10.1007/s10439-008-9611-z.

P. S. Vikhe, N. S. Nehe, and V. R. Thool, “Heart sound abnormality detection using short time fourier transform and continuous wavelet transform,” 2009 2nd Int. Conf. Emerg. Trends Eng. Technol. ICETET 2009, no. January, pp. 50–54, 2009, doi: 10.1109/ICETET.2009.112.

Y. L. Tseng, P. Y. Ko, and F. S. Jaw, “Detection of the third and fourth heart sounds using Hilbert-Huang transform,” Biomed. Eng. Online, vol. 11, no. 1, p. 8, 2012, doi: 10.1186/1475-925X-11-8.

C. Koning and A. Lock, “A systematic review and utilization study of digital stethoscopes for cardiopulmonary assessments,” J. Med. Res. Innov. offf, vol. 0, no. 2, pp. 1–11, 2021, doi: 10.25259/jmri_2_2021.

K. Khunti, “Accurate interpretation of the 12-lead ECG electrode placement: A systematic review,” Health Educ. J., vol. 73, no. 5, pp. 610–623, 2014, doi: 10.1177/0017896912472328.

K. H. Whitmer, “Assessment of Cardiovascular Function.” Iowa State University Digital Press, Feb. 2021.

C. Mishica, H. Kyröläinen, E. Hynynen, A. Nummela, H. C. Holmberg, and V. Linnamo, “Relationships between heart rate variability, sleep duration, cortisol and physical training in young athletes,” J. Sport. Sci. Med., vol. 20, no. 4, pp. 778–788, 2021, doi: 10.52082/jssm.2021.778.

M. B. Malarvili, I. Kamarulafizam, S. Hussain, and D. Helmi, “Heart sound segmentation algorithm based on instantaneous energy of electrocardiogram,” Comput. Cardiol., vol. 30, pp. 327–330, 2003, doi: 10.1109/cic.2003.1291157.


  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

JOIV : International Journal on Informatics Visualization
ISSN 2549-9610  (print) | 2549-9904 (online)
Organized by Department of Information Technology - Politeknik Negeri Padang, and Institute of Visual Informatics - UKM and Soft Computing and Data Mining Centre - UTHM
W : http://joiv.org
E : joiv@pnp.ac.id, hidra@pnp.ac.id, rahmat@pnp.ac.id

View JOIV Stats

Creative Commons License is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.