Preheat oven to 350°F. Butter a 9-inch square cake pan.

Combine 2/3 cup whole wheat pastry flour, 2/3 cup brown sugar and 1 teaspoon pumpkin pie spice in a bowl. Using your fingers or a pastry blender, cut 5 tablespoons butter into dry ingredients until no separate pieces of butter remain and mixture is crumbly. Set aside 2/3 cup streusel, and then add slivered almonds to the remainder. Set both streusels aside.

In a bowl, whisk together all-purpose flour, remaining 1 cup whole wheat pastry flour, 1 1/2 teaspoons pumpkin pie spice, baking powder and salt.

Using an electric mixer, beat remaining 1 cup butter and granulated sugar and 3/4 cup brown sugar together until the mixture is tan and fluffy, about 3 minutes. Add 1 egg at a time, beating for about 1 minute after each addition. Blend in vanilla, and then pumpkin. Add flour mixture in three batches, mixing only until no white streaks remain.

Spoon half the batter into the prepared pan, and top with reserved 2/3 cup streusel without almond slivers. Cover with remaining cake batter, and top with streusel with almonds.

Bake for 40 to 50 minutes, or until a cake tester or toothpick comes out clean from the center.