Your comment on the outer steel tube is right. Although pierced by thousands of 'small' openings for light, the external steel skin was the only structural element maintaining stability against lateral forces (primarily wind - but in this tragic case, airplane impact). The central core was not able to support anything but vertical loads and was braced against buckling by the lateral restraint of the floor trusses.
Once the floor trusses failed (as a result of fire primarily) the central core (also weakened by fire) buckled and allowed the upper floors to collapse. Once in motion, there was little that could have been done to ever stop the whole lot coming down.