Preheat a large, nonstick skillet over medium-high heat with 2 turns of the pan of EVOO, about 2 tablespoons. Add in the sliced andouille and cook until crisp and golden brown. Remove to a paper towel-lined plate to drain excess fat.
Generously butter a 9x13" casserole dish. Toss the andouille with the torn bread and sliced peaches, and place the mixture into the casserole dish.
In a large mixing bowl, combine the eggs, milk, half-and-half, vanilla, nutmeg and salt, and whisk until blended. Pour over the bread and sausage mixture, making sure it's covered evenly with the milk-egg mixture. Cover with foil and place in the refrigerator overnight.
The next day, preheat oven to 375°F. Bake for 40 minutes, until puffed and lightly golden.