Log in

No account? Create an account

Previous Entry | Next Entry

I work in an office full of programmers, UX folks and data crunchers. They're primarily geeks with a fair number of fans, though that dries up rapidly the higher up the managerial food chain you go. One of the cool thing about an office of geeks and nerds is that we don't have subscriptions to McLean's or Business World or Money Magazine; no, we have a subscription to Wired. Gotta love it.

Yesterday at lunch, I was paging through an older issue when I found this cool article: Why Quants Don’t Know Everything.

For me it was especially interesting, since I work with surveys and analyze data all the time. Plus, the focus of everything in our call center's is an agent's stats. AHT, OSAT and the dreaded NPS (DUH DUH DUUUUUUUH!) Here's the snippet that really caught my attention:

    ... The most common problem is that all these new systems—metrics, algo­rithms, automated decision making processes—result in humans gaming the system in rational but often unpredictable ways. Sociologist Donald T. Campbell noted this dynamic back in the ’70s, when he articulated what’s come to be known as Campbell’s law: “The more any quantitative social indicator is used for social decision-making,” he wrote, “the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

    On a managerial level, once the quants come into an industry and disrupt it, they often don’t know when to stop. They tend not to have decades of institutional knowledge about the field in which they have found themselves. And once they’re empowered, quants tend to create systems that favor something pretty close to cheating. As soon as managers pick a numerical metric as a way to measure whether they’re achieving their desired outcome, everybody starts maximizing that metric rather than doing the rest of their job—just as Campbell’s law predicts.

I've always thought it was nuts that so much focus is on NPS. It's such a brutal way to measure success, but it's the industry standard and all the cool kids are doing it, so it's what we do. I spent many years in different call centres as a phone monkey, and I always hated that I could ace the customer calls and be a MVP CSAT ROCKSTAR but still not qualify for bonuses and be in danger of being put on a 'correction plan' because my AHT was always too high. I always though, dammit! I'm doing a fantastic job overall - why is the sky falling if I'm a few seconds too long? And thus, I would let my CSAT go to hell to try and make better time; it was really a no win situation.

Anyway, I laughed when I read this article. I was discussing it with one of the guys in the lunch room and said, "Dude, it like the Heisenberg's Principle!" because, really, it is - in this case, the act of observing and tracking a specific stat, declaring it the Mecca of all stats, changes the behaviour of the people who now bend over backwards to hit and exceed at that stat, and corrupt the experience the stat was based on in the first place.

Man, if this doesn't sign, seal and deliver on the fact that I AM A NERD, I am not sure what does.



( 4 comments — Leave a comment )
Oct. 16th, 2014 04:07 am (UTC)
***Pets you on your little geeky head***
Oct. 16th, 2014 04:23 am (UTC)
Ooh, talk nerdy to me...
Oct. 16th, 2014 05:34 pm (UTC)
What you subsidize, you get more of
If the company is subsidizing the wrong behaviours, of course things will go off the rails. To what extent are the right behaviours being subsidized? Do people who figure out how to game the system and help correct it get praised and rewarded? It's quantified whack a mole.
Oct. 16th, 2014 07:16 pm (UTC)
Re: What you subsidize, you get more of
Subsidizing is the right way to put it. Bonuses, shift bids, rewards if all kinds are tied into stats, the numero uno of all stats being the Lofty NPS - even though it's something that agents rarely have control over. But because it is valued so highly and they are judged on it, they will bend over backwards to find ways to manipulate customers into scoring them higher - because otherwise, they suffer the consequences. And thus, the usefulness of the score is devalued.

It was so bad that, with one of our big clients, we changed the survey methodology to 30 minute delay outbound calls. The rep doesn't offer the survey; the IVR calls you 30 minutes later. Sure, the survey volume dropped - a lot of people hang up on IVR calls (me included) but the scores from the ones who completed it plummeted about 20 points across the board for OSAT and NPS when it was no longer being pitched by an agent.

And of course, the agents are penalized for that drop. *rolls eyes* You just can't win.
( 4 comments — Leave a comment )


Geek by Shaddyr

Latest Month

November 2018



Powered by LiveJournal.com
Designed by Tiffany Chow