The silkworms start to drop after completing the quest for finding Silkworm fireflies.