Heat oven to 400°. Line a baking sheet with parchment paper and set aside. Place 3⁄4 cup sugar on a separate parchment paper-lined baking sheet and bake until beginning to color at edges, 6–7 minutes; reduce heat to 225°. Meanwhile in the bowl of a stand mixer fitted with a whisk, beat egg whites on medium until foamy, about 5 minutes. Increase speed to medium-high and gradually add hot sugar to egg whites until combined; add zest and vanilla and beat until meringue is cool and glossy, about 5 minutes more. Spread meringue onto prepared baking sheet into a 3⁄4-inch-thick rectangle; bake until crisp and slightly golden, about 2 1⁄2 hours. Cool, then break into 1 1⁄2-inch pieces.
Cook cream and rosemary in a 2-qt. saucepan over medium-low, stirring, until slightly thickened, about 30 minutes. Strain cream through a fine sieve into a bowl; reserve 1 cup cream. Cover and chill remaining cream. Combine remaining sugar, the flour, and salt in saucepan; whisk in egg yolks. While whisking, gradually stir in reserved 1 cup cream; cook over medium, stirring constantly, until thickened, 3–4 minutes. Transfer pastry cream to a bowl and press plastic wrap directly onto surface; chill for 1 hour.
Beat remaining cream in a bowl until stiff peaks form; fold in pastry cream, then three-quarters each meringue pieces and mandarin segments. Transfer to a serving platter and top with remaining meringue pieces and mandarin segments.