Pour warm water (about 110°F) into a small bowl. Sprinkle yeast over the water, stir gently, and let sit for 5 to 10 minutes until foamy and bubbly. If it does not bubble, discard and start with fresh yeast.
In a stand mixer bowl, combine warm milk, pumpkin puree, granulated sugar, melted butter, and salt. Mix on low speed for 1 to 2 minutes until smooth, then pour in the foamy yeast mixture and stir until fully incorporated.
Switch to the dough hook. Add 3 cups of flour and mix on low for about 2 minutes until a shaggy dough forms. Gradually add the remaining ½ cup flour, 1 tablespoon at a time, until the dough pulls away from the bowl but is still slightly sticky. Knead on medium speed for 6 to 8 minutes until smooth and elastic.
Lightly grease a large bowl. Shape dough into a ball, place in the bowl, and flip once to coat. Cover with a towel or plastic wrap and let rise in a warm spot for about 1 hour, until doubled in size.
Stir together granulated sugar, cinnamon, and nutmeg in a small bowl. In another bowl, whisk melted butter and pumpkin puree until smooth. Line a 9x5-inch loaf pan with parchment paper, letting it hang over the sides.
Punch down the risen dough and turn it onto a lightly floured surface. Roll into a 12x20-inch rectangle, about ¼ inch thick, keeping the thickness even.
Brush the dough with half the pumpkin butter mixture and sprinkle evenly with half the cinnamon-sugar. Roll tightly into a log from one long edge. Cut into 12 equal 1-inch slices. Dip each slice into the remaining pumpkin butter, then the remaining cinnamon-sugar. Stack 3 to 4 slices together and place cut-side up in the prepared pan, standing upright like books.
Cover the pan loosely with a towel and let rise in a warm spot for 30 to 45 minutes, until puffy and nearly doubled. The slices should be touching and filling the pan nicely.
Preheat oven to 350°F. Bake on the middle rack for 35 to 40 minutes, until golden brown and the internal temperature reaches 190°F. Tent with foil after 25 minutes if browning too quickly. Cool in the pan for 10 minutes, then lift out using the parchment overhang and cool completely.
Whisk together powdered sugar, milk, and vanilla extract until smooth and pourable. Drizzle over the cooled bread. Pull apart the layers and enjoy.