The outgoing executive producer says he's still not sure how he feels about leaving