Harnessing Mixture of Experts for Abstractive Text Summarization

This project aims to develop an advanced text summarization system using a Mixture of Experts (MoE) approach combined with state-of-the-art transformer models. The system is designed to process and summarize long-form textual content such as movie summaries or articles from the CNN/Daily Mail dataset. To achieve high-quality and contextually rich summaries, the project leverages three powerful pre-trained models: PEGASUS, BART, and XLNet. Each of these models brings unique strengths to the summarization task—PEGASUS excels at generating abstractive summaries by pre-training on gap-sentence prediction n, BART is effective due to its bidirectional encoder and autoregressive decoder, and XLNet offers strong generalization by modeling language autoregressively while capturing bidirectional context. These models are integrated within a Mixture of Experts framework, where input data is intelligently routed through the most suitable model or combination of models. This dynamic selection mechanism enhances the model's ability to adapt to different writing styles, tones, and complexities found in diverse text sources. The use of the CNN/Daily Mail or movie summary datasets provides a comprehensive training ground, exposing the system to real-world textual variance and enabling it to learn the structure and semantics required for high-quality summarization. Overall, this project aims to build a robust and efficient summarization system that outperforms single-model baselines by harnessing the collective strengths of multiple transformer experts through MoE. Keywords: PEGASUS, BART, XLNet, CNN/Daily Mail.