Break the sponge into small pieces and put them in the bottom of a glass trifle bowl. Sprinkle over the sherry and raspberries and toss.
Make the custard. Gently heat half the double cream in a pan until hot (but not boiling). Meanwhile, in a bowl, whisk the egg yolks, sugar and cornflour into a smooth paste.
When the cream is hot, pour it over the egg mixture and whisk vigorously. Return to the pan and – whisking all the time – cook until thick.
If you are adding them, slice the bananas and put them on top of the raspberry sponge mix. Then pour the custard on top and leave it to cool.
Whip up the remaining cream and spread it on top. Chill for 3–6 hours before sprinkling the toasted almonds over the top.