Deep Factorization for Speech Signal