This study is devoted to understand the mechanisms of mineral fillers in asphalt at the mixture and mastic scales. To do so, the physical properties of asphalt mixtures and mastics were evaluated; in addition, a chemical investigation was presented on the mastic scale for the sake of providing insight behind the possible mechanisms between asphalt binder and mineral filler. Three mineral fillers namely hydrated lime (HL), limestone (LS), and cement bypass dust (CD) were incorporated into an asphalt binder at different filler-to-binder (F/B) ratios. For the mechanistic evaluation of the mixtures, the indirect tensile strength (ITS), moisture damage evaluation, resilient modulus, and static creep-recovery tests were carried out. The fracture energies also were quantified for all the studied mixtures. The results of penetration and softening point tests, as well as their pertinent penetration index (PI), were discussed to elucidate some physical properties of the mastics. The nuclear magnetic resonance (NMR) spectroscopy was performed on the mastics to obtain information on molecular structures of the formed composites. The findings of this study emphasize that the physical properties of mixtures and mastics are highly dependent on type and concentration of mineral filler. The 1H-NMR and 13C-NMR results proved that the inclusion of the studied fillers did not change the molecular structure of the base asphalt binder; hereby the physical mechanisms stand behind the attained properties. It is believed that the findings of this work would contribute to growing knowledge of the mechanisms of mineral fillers in asphalt mixtures.