Glasses have liquid-like structure, but solid-like properties. An important question concerns the relation between the macroscopic flow behavior and the microscopic structure. However, for atomic glasses, microscopic configurations are prohibitively difficult to visualize due to the small molecular length scales. Here, we use a colloidal glass to directly visualize and analyze particle configurations of quiescent and sheared colloidal glasses. We determine the free volumes of the particles, and relate this free volume distribution directly to the free energy of the glass. This allows us to obtain novel insight into the relation between rigidity/flow and changes in the amorphous structure. We identify a clear change in the free energy at the transition from homogenous to inhomogenous flow.