The Tribez never leave a good chief! Remember that and always take care of your people.Your workers can disappear only in case your happiness level has reduced. Note that number of your workers cannot be more than your happiness level. Try to build some decorations and social buildings for your people and they will return.