The concept of sustainable cities and communities is endorsed as one of the seventeen goals of sustainable development. Since buildings represent an essential element of the city, they play a primary role in achieving the social, economic, and environmental sustainability of cities. Previous studies have pointed to the lack of emphasis on the social aspect of buildings. Aiming to fill the gap, this research is focused on identifying barriers that hinder the adoption of social sustainability (SS) measures and practices in residential buildings, as a first step in overcoming these barriers and enabling faster achievement of SS goals. The initial platform of barriers was derived from a comprehensive review of the published literature, international reports, and green building rating systems. For the selection of critical barriers, the Delphi method was used with the participation of 60 international experts. Of 58 barriers initially identified from the literature and experts’ suggestions, 29 were selected as important and classified into five groups. Further, in each of these groups three barriers were singled out, the overcoming of which would facilitate and speed up buildings’ SS to the greatest extent. These results provide insight into barriers to SS for policy makers, developers, and planners, invite further studies on this topic, and provide a starting point for other researchers to identify the most relevant barriers in different contexts, i.e., countries and regions with their specific characteristics. This will further create the conditions for the elimination of barriers to SS by focusing on the most critical issues.