Preheat the oven to 350°F and generously grease a loaf pan.
In a large bowl, mix all of the wet ingredients: the butter, white sugar, brown sugar, eggs, buttermilk, and pumpkin puree. Use a hand mixer or the paddle attachment on a stand mixer.
Add the dry ingredients (flour, pumpkin pie spice, baking soda, baking powder, and salt) and mix until just combined. Don't overmix.
Pour into the loaf pan and bake at 350°F for about 45 minutes, until a toothpick comes out clean.
For the frosting, beat the butter and cream cheese until smooth. Add the powdered sugar and vanilla and beat until fluffy.
Let the loaf cool completely, then frost.