Laboratory evaluation of elastomer- and plastomer-modified asphalt binders using different grades of asphalt binders and produced asphalt concrete mixes is the subject of this paper. The evaluated polymer modifiers in this study were an elastomer [commercially available styrene-butadiene-styrene (SBS) and a plastomer (functionally modified olefin commercially known as Eastman EE-2)], blended separately with two penetration-grade binders (60/70 and 80/100) at polymer/binder ratios of 2%, 4%, and 6% (by mass). The rheological properties of the polymer-modified binders (PMBs) were tested using a rotational viscometer, dynamic shear rheometer, and bending beam rheometer. The effect of the polymers on the rheological properties of the asphalt binders was investigated before and following standardized short- and long-term oxidative aging. Hot-mix asphalt mixes were prepared and evaluated in terms of the number of performance tests, which included indirect tensile strength, moisture susceptibility, resilient modulus, creep-recovery strain properties, and indirect tension fatigue. Analysis of the obtained PMBs indicated that the addition of the elastomer and plastomer polymers to petroleum asphalts was very useful in obtaining a number of desirable characteristics. The main indicators of such improvements are improved rutting resistance of the unaged and short-term aged binders, and the addition of higher percentages of the polymers resulted in an upward shift of the rutting resistance without impacting the fatigue properties of the binders. The addition of up to 6% of the polymers to the binders raised the performance grade (PG) of the PMBs by at least two grades from their base PG. For the softer binder (i.e., Pen. 80/100), 6% SBS pumped the PG of the binder three grades up. The introduction of varying amounts of elastomer and plastomer polymers can significantly influence the resultant mechanistic properties of mixtures.