You can have a maximum of 6 people, starting from level 7 of the Chief's Residence, though not every upgrade yields you a new villager.