Preheat an oven (with one of its racks positioned in the center) to 350°F.
On a large, rimmed baking sheet, place the squash. Drizzle with olive oil and toss well to coat, arranging the slices in a single layer. Transfer to the oven and roast for 20 minutes. Flip and continue cooking until the slices are tender and just barely beginning to brown at the edges, about 30 minutes more. Cool slightly, then peel off and discard the skin. Set aside the 3 most even and round slices from the neck of the squash (without a hole in the center). Transfer the remaining slices to a food mill or food processor and puree the rest of the flesh. Measure 2 cups (17½ ounces) and set aside to cool completely; refrigerate or freeze and reserve the rest for another use.
Meanwhile, make the spiced syrup: To a small pot, add the sugar, rum, vanilla, allspice, and cinnamon. Bring to a boil, stirring to dissolve the sugar, then remove from heat and set aside.
Make the graham cracker crumble: To a large bowl, add the graham crackers and ginger and use the end of a rolling pin or a pestle to pulverize the crackers to a fine powder. Add the melted butter and toss gently until the crumbs are completely moistened then set aside.
Make the pumpkin mousse: To a medium bowl, add the gelatin sheets and enough cold water to submerge, turning the sheets over a few times to evenly moisten and prevent them from sticking together. Set aside to “bloom” until completely soft and saturated, about 5 minutes.
In a large bowl, whisk together the squash purée, maple syrup, cinnamon, nutmeg, clove, and curry powder and set aside.
In a small bowl, whisk together the yolks and sugar and set aside.
To a small pot over medium heat, add ½ cup heavy cream and bring just to a simmer. Remove from heat, then remove the gelatin sheets from their soaking liquid, squeezing out as much excess water as possible. Add the gelatin to the warm cream, and stir well to completely dissolve. Whisk the cream-gelatin mixture into the yolk-sugar mixture until the sugar is completely dissolved. Strain the liquid over the squash mixture, discarding any solids. Whisk well to combine then set aside.
In the bowl of a stand mixer fitted with a whip attachment, beat the remaining ¼ cup heavy cream to medium-stiff peaks. Gently fold the whipped cream into the pumpkin mixture to lighten until homogenous.
Assemble the charlotte: Press 2 large sheets of plastic wrap along the inside of a 2-quart charlotte mold to line it with at least 4 inches of overhang from the edges. Brush the flat side of the lady fingers generously with the spiced rum syrup, then arrange them standing along the outside of the mold, flat side in. (Fit as many as you can without overlapping.) Scoop ⅓ of the mousse into the mold and use the back of a spoon or a spatula to spread it into an even layer. Sprinkle half of the crumble evenly over the mousse, then add another ⅓ of the mousse, smoothing again into an even layer. Add the reserved squash slices, squeezing and trimming it as needed to cover the mousse in a single, tight-fitting layer. Add a layer of the remaining crumble, followed by the remaining mousse. Cover the final mousse layer with the remaining lady fingers. Rap the mold a few times gently on the table to settle the ingredients, then use the overhanging plastic wrap to cover. Refrigerate for at least 12 or up to 24 hours to allow the gelatin to set.
When you are ready to serve the charlotte, peel back the top layer of plastic wrap, place a flat, medium, inverted plate over the top of the charlotte, flip, and use the loose plastic to ease the cake out of its mold and onto the plate (it should release easily). Remove and discard the plastic wrap. Use a lean, warm knife to cut the charlotte into wedges and serve chilled, with warm or room temperature caramel sauce on the side.