Preplaced aggregate fibrous concrete (PAFC) is a revolutionary kind of concrete composite that is gaining popularity and attracting the interest of academics from across the world. PAFC is a uniquely designed concrete prepared by stacking and packing premixed fibers and coarse aggregate in a steel mold. The gaps between the fibers and aggregates are subsequently filled by injecting a cement grout with high flowability. This study investigates the impact performance of three different sizes of PAFC beams. Steel and polypropylene fibers were used in a 3% dosage to make three different beam sizes, measuring 550 × 150 × 150 mm, 400 × 100 × 100 mm, and 250 × 50 × 50 mm. According to ACI Committee 544, all beams were subjected to a drop weight flexural impact test. Compressive strength, impact energies at initial crack and failure, ductility index, and failure mode were evaluated. Additionally, analytical modeling was used to compute the failure impact energy for the fibrous beams. The results showed that the addition of fibers increased the capacity of the tested beams to absorb greater flexural impact energy. Compared to polypropylene fibers, steel fibers had better crack propagation and opening resistance because of their higher tensile strength and crimped and hooked end configuration. For all large-size beams, the analysis of the percentage increase in impact energy at the failure stages was found to be 5.3 to 14.6 times higher than the impact energy at cracking.