Support Planning
Microsoft
Support Planning
Multiple Locations, United States
Save
Overview
With more than 45,000 employees and partners worldwide, the Customer Experience and Success (CE&S) organization is on a mission to empower customers to accelerate business value through differentiated customer experiences that leverage Microsoft’s products and services, ignited by our people and culture. We drive cross-company alignment and execution, ensuring that we consistently exceed customers’ expectations in every interaction, whether in-product, digital, or human-centered. CE&S is responsible for all up services across the company, including consulting, customer success, and support across Microsoft’s portfolio of solutions and products. Join CE&S and help us accelerate AI transformation for our customers and the world.
Within CE&S, the Customer Service & Support (CSS) organization builds trust and confidence for every person and organization through delivering a seamless support experience. In CSS, we are powered by Microsoft’s AI technology to help consumers, businesses, partners, and more, resolve their issues quickly and securely, helping prevent future problems from occurring and achieving more from their Microsoft investment.
We are seeking a data-driven, business-savvy, and analytically minded Sr Support Planner for AI Agent Quality Evaluation to join our team. This pivotal role ensures that our AI agents, AI assistants, and AI models - particularly those focused on understanding human intent, language, and conversation—consistently deliver high-quality, high-performance customer interactions.
This work will include ‘voice of the customer’ with a primary goal being world-class customer experiences. To be successful, you will work across and with a number of stakeholders – leveraging content, best practices, but also influencing to help drive change. The role includes a full set of status reporting via RoB (rhythm of the business) activities. This opportunity enables and promotes career growth as a recognized expert in support & service planning.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Qualifications
Required Qualifications:
- Bachelor's Degree in Computer Science, Information Technology, Business, Social Science, Marketing, Communications, or related field AND 2+ years of technical/customer services, technical/customer support, business process operations, product development/management, product marketing, business or product strategy, technical consulting, or related experience
- OR 5+ years of technical/customer services, technical/customer support, business process operations, product development/management, product marketing, business or product strategy, technical consulting, or related experience.
- Analytical skills with hands-on experience in AI model evaluation, natural language processing (NLP), or conversational AI systems
- Understanding of AI model quality and business performance metrics
- Knowledge of linguistic structures and conversational flow dynamics
- Experience designing and managing evaluation pipelines for AI systems
Additional or Prefferred Qualifications:
- 5+ years of call center experience
- Sharp eyes for language quality and context nuance
- Comfort working across cross-functional teams (AI, product, UX, data)
- Curiosity about how language and AI intersect in real-world scenarios
- Passion for raising the bar on AI-Human experience
Support Planning IC4 - The typical base pay range for this role across the U.S. is USD $85,100 - $169,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $112,000 - $185,300 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
Microsoft will accept applications for the role until July 22, 2025.
#CES #CSS
Responsibilities
- Evaluate AI-generated responses (text, voice, or chat) for linguistic accuracy, tone, coherence, and contextual appropriateness
- Design and apply quality scoring rubricsto assess AI agent performance across diverse use cases and interaction types
- Analyze structured and unstructured data to uncover patterns, inconsistencies, and opportunities for improvement
- Collaborate with data scientists and engineers to recommend tuning strategies, threshold adjustments, and prompt optimizations
- Develop and lead human-in-the-loop evaluationsto train and refine AI models and drive measurable business impact
- Support A/B testing and experimentation to assess the impact of AI model changes on user experience and key performance indicators (KPIs)
- Generate performance reports and dashboards to monitor quality trends, accuracy, and deflection rates
- Ensure adherence to ethical, inclusive, and responsible AI practices—particularly in language generation and conversational design
- Monitors key performance indicators (KPIs) for assigned features or services for issues and makes recommendations to improve the customer experience.
- Develops and/or reviews new support scenarios (e.g., features, customer entry points, capabilities) prior to or during in-life releases to identify moderately complex customer experience scenarios that may drive support incidents; documents the end-to-end customer experience to identify and communicate risks, and identifies approaches to remove them with relevant teams. Builds plans in response to new support offerings or improvement opportunities, and identifies approaches to remove them with relevant teams.
- Gathers feedback on existing feature/program/product scenarios to identify blockers and applies existing approaches to remove them with relevant teams; identify ways to increase efficiency and/or return on investment (ROI) Return of Investment.
- Identifies broad dependencies and risks in specific products/programs that they are responsible for relative to other products/programs makes recommendations to mitigate them.
- Analyzes the experience and the voice of the customer and delivery team at a product line level to identify new opportunities to drive customer loyalty and return on investment; raises these opportunities to stakeholders for sponsorship and buy in and reviews experiences of other business units, and identifies ways to incorporate into their strategies to drive continuous improvement and growth.
- Leverages status reports, data, and other communication methods, using storytelling to influence stakeholders to communicate issues and improvement opportunities. Highlights risks and issues through existing data and reporting.
- Manages moderately complex stakeholder relationships across product group, engineering, and support ability with a clear rhythm of business (ROB), and active communication, to drive customer and support experience alignment.
Other:
- Embody our culture and values