In a medium bowl, combine butterscotch pudding and milk. Whisk until pudding is dissolved. Refrigerate for 10 minutes.

Meanwhile, use a food processor to crush Butterfinger candy bars into coarse crumbs. Measure out 1 1/2 cups of the crushed candy bars and set aside. Save the remaining crushed Butterfingers for topping the cake.

Using an electric mixer on high speed, beat the cream cheese until light and fluffy. Add the brown sugar and mix well.

Mix in the vanilla and the powdered sugar. Add the heavy cream and beat on medium speed for about 2 minutes.

Fold in the 1 1/2 cups crushed Butterfingers, followed by 1/2 of the frozen whipped topping.

Line the bottom of a 9×13 pan with chocolate graham crackers, cutting as needed to fit the pan.

Spread 1/3 of the cream cheese mixture over the crackers (use a little of the mixture on the bottom of the crackers to hold them in place, if needed).

Layer half of the pudding over the cream cheese mixture.

Add a second layer of graham crackers, followed by a second layer of the cream cheese mixture and the remaining pudding.

Top with a third layer of graham crackers and the remaining cream cheese mixture.

Place the cake in the refrigerator for at least two hours, or until ready to serve. Just before serving, top with the remaining frozen whipped topping and sprinkle with the remaining crushed Butterfingers.