UX Metrics Reset + AI Metrics

3 live sessions | One hour each | Three hours total

What are your metrics teaching your product to do?

A product optimized for engagement can demand more attention. A service optimized for cost reduction can transfer work to customers. An AI agent optimized for completion can finish tasks that still need extensive checking.

The dashboard might show progress. But understanding the customer journey requires deeper contextual evidence.

UX and Service Design have an important role here: questioning what success means, who benefits and who carries the costs.

This Masterclass revisits familiar UX KPI metrics, then extends that thinking to LLM voice, chatbots and agents. We’ll examine long-term customer value, AI safety and what productivity means. With AI, we can bring magical savings of time, boost efficiency and power-charge customer productivity. While at the same time we can case people to stop and interpret, babysit a runaway bot or switch to another channel out of frustration or tarnished trust.

Overview: UX Metrics Reset + AI Metrics

Review what counts as value. Build measures that support better decisions.

Across three interactive sessions, you’ll examine how metrics shape design choices, develop measures for an AI experience and assess value across the complete workflow.

We’ll connect user research, behavioral evidence, service performance and commercial outcomes. You’ll learn how to make a credible impact case while keeping uncertainty, accessibility and safety visible.

Three live, interactive sessions with Frank Spillers. One hour each. Three hours total.

Each session combines short teaching, examples, guided practice and discussion. Everything needed for the exercises is introduced live. No pre-watching or homework is required.

Why attend?

Your team may already track conversion, satisfaction, retention or task completion. Those measures need journey-level context before they can support a claim about customer value.

A customer’s lifetime value describes their financial value to an organization. Long-term value to the customer concerns what that person continues receiving. A useful measurement approach examines both.

We’ll discuss tensions between monetization, usefulness and customer choice, using Apple as one case for examining these questions.

AI adds further challenges: A quick answer can contain errors, or a completed task can exceed the user’s instructions. Sometimes faster workflow can demand more human supervision.

AI and UX ROI expert Frank Spillers will help you investigate these trade-offs and decide what evidence should guide investment, redesign or further testing.

Who this is for

• UX and service design practitioners reviewing success measures or demonstrating design impact.
• Researchers and design leaders connecting human outcomes with operational and commercial evidence.
• Product and AI teams assessing conversational experiences, agent behavior and productivity claims.

Yes: You can apply a real work challenge or use a supplied case.

What you’ll get

Build a Metrics Reset Case Sheet across the three sessions. It will contain:

• A review of one existing metric: What it rewards, what it misses and which outcome matters.
• An AI measurement plan: Defined measures, evidence sources, user groups and a baseline.
• A complete-workflow value estimate: Human effort, system costs, quality requirements and uncertain assumptions.
• A short decision brief: What the evidence supports, what needs investigation and one practical next action.

A taste of the measurement problem

An illustrative AI support scenario:

An AI service handles more conversations and reduces staff handovers. It responds quickly and the dashboard reports improved efficiency.

But: Customers still repeat information, check answers and return when their problems remain unresolved. Staff spend time repairing cases that appeared complete. Has the service improved? What would establish that?

In this 3-part Masterclass, we’ll examine how verified resolution, repeat contact, customer effort and staff recovery work change the interpretation. We’ll also consider when a successful human handover represents a good outcome.

Agenda

Session 1: Reset what counts as success

Examine the outcomes your metrics encourage.

We’ll explore how engagement, conversion, satisfaction and retention can support useful decisions…and where they need additional evidence. You’ll connect one user outcome with an organizational result. Then identify how an existing measure could misrepresent progress.

Topics include:

• Engagement, compulsive use and people’s control over their attention.
• Immediate monetization and long-term customer value.
• Retention, customer choice and reasons for staying.
• Accessibility, wellbeing and outcomes beyond revenue.
• Measures that improve while the underlying problem continues.

Live practice: Review a success claim and propose a stronger way to assess it.

You leave with: A metric review, a clearer outcome and an evidence question.

Session 2: Measure AI interaction and agent behaviour

Define success for systems that converse, generate and act.

We’ll introduce measurement questions across three AI experience types: LLM Chatbots, Agents and AI Voice experiences. You’ll explore how interaction quality, reliability and safety fit together. We’ll consider when clarification, refusal or escalation represents appropriate behaviour.

We’ll also examine whose experience disappears inside an average. Results may differ across tasks, languages, accents, disabilities and levels of experience.

Live practice: Choose one AI case and define a focused measurement plan, including evidence sources and thresholds for review.

You leave with: Measures for your selected case, a practical baseline and clear criteria for acceptable performance.

Session 3: Test productivity and build a credible value case

Account for the work behind the output.

AI productivity claims need to include briefing, supervision, verification, correction and recovery. You’ll assess three separate dimensions:

• Human effort: Time actively spent doing or overseeing work.
• Delivery time: Elapsed time until an acceptable result.
• Total cost: Labor, tools, integration and ongoing operation.

We’ll compare results against a baseline with consistent quality requirements. You’ll distinguish freed capacity from realised savings and consider whether work has moved to customers or other teams.We’ll then build a transparent value estimate using low, expected and high scenarios. Safety and accessibility requirements will remain explicit alongside financial results.

Live practice: Review an AI productivity claim, account for hidden effort and prepare a short decision brief.

You leave with: A value estimate, visible assumptions and one evidence action for the next 30 days.

How the interactive Masterclass works

• Learn it live: Short explanations introduce each method before you apply it.
• Compare interpretations: Discuss what the numbers suggest and what further evidence could change.
• Develop one case: Carry your chosen challenge through the three sessions.
• Choose how you contribute: Write, discuss, observe or sketch. You can pass on an activity.

Who you’ll learn from

Frank Spillers is the founder of UX Inner Circle and Experience Dynamics. He brings more than 25 years of UX and Service Design experience across government, enterprise and inclusive design.

Frank will help connect measurement choices with user evidence, service outcomes and decisions teams can act on.

Common questions

Do I need to attend the Lightning Lesson first?

No. The Masterclass introduces the foundations needed for the exercises. The Rethinking Metrics for UX and Service Design Lightning Lesson (FREE) provides an optional introduction to the topic.

Do I need analytics or coding experience?

No coding or advanced statistics is required. We’ll focus on defining measures, interpreting evidence and making proportionate claims.

Do I need a live AI product?

You can use your own project or a supplied scenario. Examples cover voice, chatbots and agents. Your detailed work will focus on one selected case.

Does this apply outside commercial products?

Yes. Public services, internal tools, healthcare and charities can apply the approach to resolved needs, access, service quality, safety and avoidable repeat work.

Will we calculate ROI?

Yes. We’ll build a simple scenario-based estimate with explicit costs, benefits and assumptions. We’ll also identify important outcomes that need their own measures.

Is there preparation or homework?

No required preparation or homework. You can bring an existing metric or AI value claim if you have one.

What if I miss a session?

Sessions are recorded. Each live session begins with a short recap. Attending all three provides the full sequence of practice, feedback and discussion.

Join to review the measures shaping your work and develop stronger evidence of value.