The silkworms start to drop after completing the quest "Catching Silkworms".

 

Silkworms can be found in treasure chests or collected from Pineapple Bushes and Cherry and Mango Trees on Farmer's Bay, each yielding a different quantity with varying probability.