Plastic waste is a growing concern today, as landfills fill up and pollution leads to global climatic problems. Some research has been conducted on using plastics in asphalt, but the effect that high-density polyethylene (HDPE) can have in the performance of the asphalt is not understood. This research examines the impact on the moisture damage, rutting, and cracking resistance of recycled HDPE plastics as a portion of aggregates in a dense graded asphalt mix. Two types of recycled HDPE were used—flakes and pellets. Mixes with 5% by volume of recycled HDPE were designed and compared with a control mix. The results indicate that the use of recycled HDPE allowed for a reduction of 0.5% binder requirement. Tensile strength ratio testing showed that, with no anti-stripping agent, the HDPE-modified mixes had acceptable moisture resistance. Both types of HDPE-modified mixes showcased superior rutting resistance based on the flow number parameter. However, the use of HDPE decreased the cracking resistance of the mixes measured with the Illinois Flexibility Index Test at intermediate temperatures. The dynamic modulus test indicated a decrease in the stiffness of the mixes at low temperatures and confirmed the increase of the stiffness at high operating temperatures. Overall, the results indicate that the use of HDPE as partial replacement of aggregates is feasible and future research could explore balancing the performance of the HDPE-modified mix.