The Hermite matrix polynomials have been generalized in a number of ways and many of these generalizations have been shown to be important tools in applications. In this paper we introduce a new generalization of the Hermite matrix polynomials and present the recurrence relations and the expansions of these new generalized Hermite matrix polynomials. We also give new series expansions of the matrix functions exp(xB), sin(xB), cos(xB), sinh(xB) and cosh(xB) in terms of these generalized Hermite matrix polynomials and thus prove that many of the seemingly different generalizations of the Hermite matrix polynomials may be viewed as particular cases of the two-variable polynomials introduced here. The generalized Chebyshev and Legendre matrix polynomials have also been introduced in this paper in terms of these generalized Hermite matrix polynomials.