Fabric is produced in the Spinnery that you can build only on the Farmer's Bay island. You'll need food and silk for production, with silk available at the Silkworm farm. You can also buy fabric for gems from the Store any time you want.