In this paper the definition of Hermite-Hermite matrix polynomials is introduced starting from the Hermite matrix polynomials. An explicit representation, a matrix recurrence relation for the Hermite-Hermite matrix polynomials is given and differential equations satisfied by them are presented. A new expansion of the matrix exponential for a wide class of matrices in terms of Hermite-Hermite matrix polynomials are proposed.