Did you see the video of an organized mob trying to force their way in a small set of doors and nearly crushing an officer? The intent was there... the premeditation was there... for at least a large segment of the mob. You don't come equipped with the weapons and supplies that these terrorists had unless you intend to use them.
The hangers on may have 'milled about the Capitol' but the violence preceding by some makes ANYONE who entered the building guilty by association. Anyone with a shred of morals and decency would not participate in this in any shape or form. Every extra person who participated in this effort added additional numbers leading the officers to be further outnumbered and unable to thwart the attack.
Here's the video of the incident if you care to watch it.
https://www.insider.com/video-pro-trump-mob-crush-officers-head-2021-1