Multimodal emotion recognition using explainable ai for text, audio Telugu, Hindi, English, live video
Objective
The objective of this project is to develop a multimodal and Multilingual emotion recognition system that integrates text, Telugu, Hindi, English audio, and live video modalities to accurately identify emotions. By leveraging advanced machine learning and deep learning algorithms, including stacking models,Attention + CRNN, YOLOv8, and YOLOv12, the system processes textual data, audio signals, and live video streams to detect emotions such as happiness, anger, sadness, and surprise and more. The project aims to enhance emotion recognition performance across multiple channels, providing real-time insights through text-based sentiment analysis, speech emotion detection, and facial expression analysis. This technology has applications in mental health, human-computer interaction, and multimedia analysis.
Abstract
Emotion recognition from text, audio, and video is a complex yet important task for understanding human behavior and enhancing interaction systems. This project focuses on building a multimodal emotion recognition system using machine learning and explainable AI techniques. By integrating various data modalities text, audio (Telugu, Hindi, and English), and live video it aims to predict emotional states accurately. The system uses algorithms like Stacking (SVC, RF, LR, NB, KNN), CRNN, CNN, LSTM, and YOLO, with SHAP and Grad-CAM for explaining model decisions. The system allows for text-based emotion analysis, audio recognition from multiple languages, and live video emotion detection, providing users with detailed insights into their emotional states. With a user-friendly interface and a backend powered by Flask and SQLite, this solution demonstrates the power of combining diverse inputs and explainable AI to achieve high accuracy in emotion prediction.
Keywords:
Emotion Recognition, Text Analysis, Audio Emotion Recognition, Live Video, SHAP, Grad-CAM, Machine Learning, Explainable AI, Flask, YOLO.
NOTE: Without the concern of our team, please don't submit to the college. This Abstract varies based on student requirements.
Block Diagram
Specifications
HARDWARE REQUIREMENTS
Processor - I3/Intel Processor
Hard Disk - 160GB
Key Board - Standard Windows Keyboard
Mouse - Two or Three Button Mouse
Monitor - SVGA
RAM - 8GB
SOFTWARE REQUIREMENTS:
Operating System : Windows 7/8/10
Server side Script : HTML, CSS
Programming Language : Python
Libraries : Flask, Os, pandas, Scikit-learn, Numpy
IDE/Workbench : VsCode
Technology : Python 3.8+
Database : sqllite
Demo Video
Request for Video
Paper PublishingRequest Call Back
Would you like to receive a free callback now?
Choose the best time for callback:
Leave your message and we will contact you as soon as possible
6-2-85/B, Old Maternity Hospital Road, Thyagaraja Nagar, Tirupati, Andhra Pradesh – 517501
+91 9030333433
+91 9393939065
0877-2261612
Disclaimer - Takeoff Edu Group Projects are not associated or affiliated with IEEE in any way. The IEEE Projects mentioned here are mentioned in the context of student projects, whose ideas are derived from IEEE publications, not projects of or by IEEE.
