You can Google exactly what the military was after when the 5.56 was brought in. Off the top, it needed to be in .22 cal, penetrate a steel plate and steel helmet at 500 yards, be in a rifle weighing less than 10lbs (can't remember exactly what weight they wanted), and capable of supersonic speed at 500 yards out. The round was designed to work optimally in a 20" 1:12 twist barrel. With the M4, you begin to see a decrease in muzzle velocity and, therefore, a poor performance in the round due to the shorter barrel. The military has been after another round for a while now and has fielded some... the .458 SOCOM and 6.8 SPC come to mind. The thing is cost and logistics in equipping units with the parts and ammo to convert their current stock of M16/M4s. I have a friend still in and he says there are rumors of them testing the 6.8 SPCII.
I think the requirements for the round were from Gen Wyman after Korea, he wanted to punch a hole in a steel helmet at 500 yds with a supersonic round, fired from a platform that weighed < 6 lbs with 20rds loaded.
But the military had experienced the logistical problems and change in warfighting back in WWII. The biggest problems that they had was the lack of a universal infantry weapon. Infantry units carried Garands, M2 carbines, BARs, M3s, Thompsons...all with different parts and ammunition requirements - to say nothing of the variance with British and other Allied weapons. NATO standardized on a 7.62, so the US went to .30 carbines in Korea, but were consistently out-classed by the AK-47. So the decision was made to go to a universal automatic infantry rifle with a round that could outperform the .30 and still retain the logistic benefits. That's when Gen Wyman's request for an intermediate cartridge was reconsidered, and they rechamberd the AR-10 (which had lost the competition to be the M14 for mostly political reasons) from 7.62 to 5.56.
When Vietnam happened, troops couldn't carry enough 7.62 ammo to Pin Down/Blow Up the enemy with automatic fire from the M14, nor could they effectively concentrate fire due to the heavy weight of the M14 and the high recoil of the 7.62. That's when the DoD stopped making M14s and went to the AR-15/M16.