In a large saucepan, add the milk over medium heat.
In a bowl, whisk together the sugar and egg yolks until well combined, about 2 minutes.
Add the sugar-egg yolk mixture, cornstarch and vanilla extract to the milk and whisk well to combine.
Cook until the mixture is bubbling, whisking occasionally so the bottom doesn’t burn.
Once the mixture is bubbling, cook for another 1 to 2 minutes, until the mixture thickens. Note: It will thicken up more as it cools.
Remove from the heat and stir in the butter and vanilla, mixing until the butter has melted.
Let the pudding cool for 30 to 60 minutes.
Add the heavy whipping cream, powdered sugar, vanilla and salt to a large mixing bowl.
Beat the cream with an electric mixer until stiff peaks form. You will know they are stiff peaks when you remove the mixer blade and the whipped cream forms a peak that stands up straight and does not curl.
In a medium to large trifle bowl, add 1½ cups of the pudding, spreading it evenly across the bottom of the bowl.
Layer with slices of one banana.
Top the banana slices with ½ of the pound cake chunks, followed by 2 cups of whipped cream, Nilla wafers, the remaining pudding, slices of one banana, the remaining pound cake chunks and the remaining whipped cream.
Sprinkle with crushed Nilla Wafers and additional sliced bananas.
Refrigerate until ready to serve.