Amorphous solids are critical in the design and production of nanoscale devices, but under strong confinement these materials exhibit changes in their mechanical properties which are not well understood. Phenomenological models explain these properties by postulating an underlying defect structure in these materials but do not detail the microscopic properties of these defects. Using machine learning methods, we identify mesoscale defects that lead to shear banding in model polymer nanopillars well below the glass transition temperature as a function of pillar diameter. Our results show that the primary structural features responsible for shear banding on this scale are fluctuations in the diameter of the pillar. Surprisingly, these fluctuations are quite small compared to the diameter of the pillar, less than half of a particle diameter in size. At intermediate pillar diameters, we find that these fluctuations tend to concentrate along the minor axis of shear band planes. We also see the importance of mean “softness” as a classifier of shear banding grow as a function of pillar diameter. Softness is a new field that characterizes local structure and is highly correlated with particle-level dynamics such that softer particles are more likely to rearrange. This demonstrates that softness, a quantity that relates particle-level structure to dynamics on short time and length scales, can predict large time and length scale phenomena related to material failure.