When educated people like doctors, lawyers, managers, etc., ask for "interpretability," what they mean is, "what are the 'key variables' we can control that will help produce the outcomes we want?"

Simpler statistical models, and particularly linear models, provide easy-to-grasp "answers" to that question -- for example, "to get more of y, we need more of x1 and x2, and less of x3."

An entire generation of leaders and professionals has been trained to think this way by statistics professors and teachers who have been rigorously trained to think that way too. (For background, I highly recommend Leo Breiman's now-classic paper, "Statistical Modeling: The Two Cultures:" https://projecteuclid.org/download/pdf_1/euclid.ss/100921372... )

Alas, there's no going back. Simpler models do not work for complicated tasks like, say, predicting the next best move in a game of Go, for which there are no 'key variables' in the traditional sense.

It goes further. Folks who care strongly about the interpretability factor generally only care because it gives them authority and political control -- they definitely don't actually believe it's a superior, nor even cost-effective, way of obtaining prediction accuracy.

Robin Hanson summarized this well in his article "Who about forecast accuracy?" [0]. Here's a nice quote:

> The benefits of creating and monitoring forecast accuracy might be lower than we expect if the function and role of forecasting is less important than we think, relative to the many functions and roles served by our pundits, academics, and managers.

> Consider first the many possible functions and roles of media pundits. Media consumers can be educated and entertained by clever, witty, but accessible commentary, and can coordinate to signal that they are smart and well-read by quoting and discussing the words of the same few focal pundits. Also, impressive pundits with prestigious credentials and clear “philosophical” positions can let readers and viewers gain by affiliation with such impressiveness, credentials, and positions. Being easier to understand and classify helps “hedgehogs” to serve many of these functions.

> Second, consider the many functions and roles of academics. Academics are primarily selected and rewarded for their impressive mastery and application of difficult academic tools and methods. Students, patrons, and media contacts can gain by affiliation with credentialed academic impressiveness. In forecasts, academic are rewarded much more for showing mastery of impressive tools than for accuracy.

> Finally, consider next the many functions and roles of managers, both public and private. By being personally impressive, and by being identified with attractive philosophical positions, leaders can inspire people to work for and affiliate with their organizations. Such support can be threatened by clear tracking of leader forecasts, if that questions leader impressiveness.

FWIW, having worked for a while in quant finance, I can testify that this precisely nails it. Even when it would be cheap, easy, obviously worthwhile, and likely impactful, projects to switch from politically-controlled, simplistic regressions couched in the buzzwords of academic finance and instead embrace principled statistical computing, proper model fitting hygiene, machine learning, etc., such improvements will never be considered. There's too much political gatekeeper rent-seeking at the higher levels of the management who oversee the investment process. Sure, they'll claim plausible deniability by saying something is "a black box" or making a fallacious comparison with an algorithmic trading firm that lost money, but the real reason is to protect their fiefdom, over which they are the lord and master because of their credentialed authority in some "interpretable" buzzword.

[0] < http://www.cato-unbound.org/2011/07/13/robin-hanson/who-care... >

> they definitely don't actually believe it's a superior, nor even cost-effective, way of obtaining prediction accuracy.

Prediction accuracy is not the only goal. There are only a handful of times in my life where I've seen a machine learning task where the training data was a perfect match for the business objective. The training objective is a proxy for the actual objective, where the gap between the two includes a lot of systematic bias. Interpretability is necessary to make sure the model isn't merely exploiting that gap. (And also to motivate further improvements.)

I don't think what you are saying is incompatible with my comment. Asking a question such as, "what is it about the data that is causing the model to be fitted like this?" is very healthy and important, but is rarely what anyone means by 'interpretability.'

Instead, 'interpretability' is usually taken to mean something more akin to: which readily human-understandable cognitive structures (out of the set of such structures that I am the authortity on) is the model directly (and easily) mappable to? By mapping onto those structures, I can now weave a story about the model within the domain in which I am the established authority. If such mappings don't exist, or are very difficult to compute, then my authority is threatened.

Having said all of this, the most egregious abuses of the nefarious side of 'interpretability' that I have seen have all come in quant finance, which is a situation in which the optimization criteria are literally the identically same thing as the actual objective.

Given that when prediction accuracy is paramount to the definition of success you still see politics trumping science, I have very little confidence that when you move to domains where the objective criteria of the models differs from the real-life objective in non-trivial ways, then somehow 'interpretability' will be used appropriately rather than overwhelmingly just as a political tool.

> Asking a question such as, "what is it about the data that is causing the model to be fitted like this?" is very healthy and important, but is rarely what anyone means by 'interpretability.'

What I see people mean by interpretability is one step back. It's literally just "what is the model actually doing?" in few enough concepts that meat machines can hold them in memory. Sometimes accompanied by a second question "is the thing it is doing dumb or smart?"

In my field (search) there are large classes of things that "work" but are dumb or annoying, or just not the product we want to provide, and we have to make product decisions to avoid them. So in that context figuring these things out is really important.

>Having said all of this, the most egregious abuses of the nefarious side of 'interpretability' that I have seen have all come in quant finance, which is a situation in which the optimization criteria are literally the identically same thing as the actual objective.

You know more about this than me, but it seems like there might be big differences between models in how they respond to unquantifiable downside risk of rare events. How do you deal with that without interpretability? How do you examine the assumptions the model is making that you may not want it to make?

With respect, I think there's a fundamental error in this reply. You're making a dogmatic statement about what interpretability is taken to mean, which assumes of course that the vast body of work, both in academia, journalism, and business meetings, holds a consistent view. The point of this paper, however, in part, is to point out that this view is mistaken.

You are incorrect in your claim. The paper does make a good point that under the umbrella term "interpretability" there are varied and conflicting understandings of what, specifically, is meant.

That fact has nearly zero relevance to what I'm saying.

When I say that people look for mappings from the model to a space of cognitive structures over which they have authority, it could be (and observed in practice always is) a smattering of all the different things talked about in the paper -- authority over which diagnostics represent 'trust,' authority over which visualizations are adequately informative, authority over explaining 'how' an algorithm works, and so on.

Though all of these are discordant takes on the specifics of interpretability, as discussed in the paper, they still are unified in the sense that the true intention in appealing to any of them is to map things into a domain where someone wields the authority. It's not about actually caring about trusted diagnostics, or caring about quality visualizations, or caring about communicating the intuition behind a model. Those are just the plausible deniability excuses proffered to try to win the battle over which faction's authority gets center stage.

I don't require that anyone have a cross-domain consistent view of the specific definition of interpretability in order for my claims to be a viable explanation, and so it is perfectly compatible with the paper. The paper looks at the on-the-ground specifics used in the arguments over interpretability, meanwhile politics explains the behind-the-scenes motivations of any of those specific choices.

The Moral Mazes chapter Dexterity with Symbols is a good place if you want to read on the evidence about how this is carried out in management bureaucracy. It has actually been formally studied in some interesting ways.

I have good reason to believe the paper's author agrees with this argument.

I can think of another application of your/Hanson's idea. It always seemed to me, was that for all the handwringing over bad comment field culture or low quality content, site owners would rather have that than give up a finger of control.

Rather that try to define quality and think about the processes which could bring it about, they have a "I know it when I see it" attitude to quality. They value their personal discretion to promote, bury, editorialize and censor far higher than actually being a high-quality truly open forum.

I would think quantitative finance is the one area where models are checked, because there's real money on the line. If you're a sell side strategist it might not matter, but if your bonus is on the line, you'll keep an open mind.

But that's the problem. Bonuses are awarded based on who can argue politically. It's similar to creating Dutch books. Managers will work hard to set a up a situation such that, no matter what the actual performance outcome really is, they have ready-made, plausible-seeming arguments for why it wasn't their fault and why they did what was asked and it was impossible to avoid the particular outcome. Then they push the burden of blame for a failed outcome somewhere down the hierarchy. It's not a perfect science, and yes, sometimes managers do get dinged badly for failing to be objective enough, but most often even what that does happen the incentive scheme doesn't cause them to become more open-minded with regard to modeling techniques, rather more cynical and aggressive with political techniques.

I thought the exact same thing going into finance. If real money is on the line, surely they care about which techniques are most rigorously justified in a given model context, right?

Absolutely not. The incentive schemes from clients (who often don't actually fire bad investment managers when they should, and who also engage in chasing returns momentum despite the long and, by now, boring and uncontroversial history of that not paying off) don't actually punish inefficiency. It's a big political mess.

Really the best you can hope for is that they'll hire you for marketing purposes. Hey look, we brought in a shiny new expert in deep learning -- we're cutting edge, we swear! They won't actually let you do any real work with deep learning, of course. It will all be Excel jockey bullshit on factor models in which you'll do obviously fallacious things like directly compare the t-stats of two different model fits as a means for model selection. Maybe you'll write an ineffectual white paper on something slightly more advanced from time to time. But the big reason you're hired is to look good on paper and smoke cigars and drink brandy with the right person who wants you as a political darling in order to win arguments from authority about how you definitely should not migrate away from Excel/VBA.

This is not hyperbole, sadly.

Interesting. After watching the show Billions, and reading up on how much money hedge fund managers make on fees (seems totally ridiculous), I wonder how common is illegal insider trading for hedge funds? No matter how good your model is, you won't beat someone with information your model doesn't have.

Read up on SAC (Point72). Quite a bit of insider trading.

It isn't just about having great models. 51% of the world can't beat the median return. Everyone has great models. It's figuring out where the models are wrong that matter. (Example 1: Most Mortgage models assumed that housing values in all US markets couldn't go negative at the same time) Sometimes that's by qualitative insights. But that's very very hard. And sometimes it's by having someone give some info that they shouldn't.

I work for a large-ish bank. Sadly can confirm everything this guy says is completely accurate.

I'm trying to get out...

These are strong arguments. Unfortunately, while sometime interpretability addresses a fundamental (not frivolous concern), it's also true that often interpretability is sometimes contrived to serve some irrational bias or political motive.

I think this is expressed in the paper's discussion section:

"We should be careful when giving up predictive power, that the desire for transparency is justified and isn’t simply a concession to institutional biases against new methods."

For hedge funds, doesn't it still ultimately come to the bottom line? My experience with traders and hedge fund quantis is ultimately they get paid a % on revenue or gains - for junior people it's more political, but the more senior they get the more it's correlated.

This doesn't count sell side marketing work, or slow money (insurance, mutual fund) buy side work where people get paid for Assets Under Management versus a % of the gains.

Not really. Some hedge funds might be somewhat better, but there are many kinds of hedge funds. A large number of them function in almost precisely the same manner as a traditional asset manager, and are compensated and incentivized in the same way. As a non-manager employee at these firms, your bonus is still usually expressed as a percentage of your base salary with the caveat that it is awarded in a discretionary way that is based on firm performance, but is not given by any explicit formula connected to profits or losses.

Some firms also don't even bother to give any details whatsoever, and the job offer will simply say there is a discretionary bonus, no percentage, no description at all. I actually turned down a hedge fund job because of this. I told them that in order for me to feel comfortable accepting an entirely discretionary bonus, with no baseline or agreed upon way of relating it to base salary or profits and losses, I would need a much higher base salary, and they weren't willing to negotiate about it.

The number of firms, even among extremely quant-heavy hedge funds, who award bonuses in a manner that is not overwhelmingly political is exceedingly tiny.

What this means is that the same political incentives affect even most hedge funds, and so they care far less about the mathematical rigor of what they are doing than about how to sell political stories about it. If "interpretability" sells political stories, then that's what they'll do.

Where are the hedge funds using sophisticated statistics? Maybe Renaissance. Maybe. Where else? Certainly not DE Shaw. Certainly not PDT. Certainly not G-Research. Certainly not Coatue. And on and on. When you interview at these places, and see how the sausage is made, it is eye-opening and alarming to understand just how little their business utilizes or cares about mathematics, statistical rigor, and often not even proper software design. They are just more of the same kinds of shoddy software shops cranking out ad hoc code for rapidly varying political whims, but with far better branding.

All this discussion makes me think about that bet Warren Buffet issued some years back about dumb index vs managed hedge fund. Political authority is an interesting slant to all this. Makes me also think about Thomas Sowells "varieties of nothing", ie putting on a show to make it look like someone is doing something when their net contribution is near nil. http://www.npr.org/2016/03/10/469897691/armed-with-an-index-...

Maybe the prevalent mental model regarding models is wrong? The real elephant in the room is the idea that "all models" are similar, and therefore occupy the same space regarding interpretability, forecasting, and other uses. Models are designed to solve specific problems, usually within specific scopes. Confusion, misinterpretation, and error occur when consumers of models don't understand (or simply ignore) the limitations of these models.

Whenever I sit down to build a new model, or work with a new-to-me model, I start with questions like:

- What questions was this model built to answer? Be careful when asking other questions, and know the limitations of those answers.

- If it's been in use for a while, how effective has it been? How is that efficacy measured? A highly accurate decomposition of historical behavior may involve a lot of post-fact knowledge that has no predictive power, so be careful interpreting cross-use results.

- What are the ranges and distributions of the input variables? What are their boundary conditions? This is not always applicable to all variables, especially unstructured data, but when it is applicable, it is usually straightforward.

- What are the ranges of the model outputs? Do they have any boundary conditions? Do boundary effects impact the situations in which the model can be applied? For example, when working with a binned variable, often bins 0 and n include less homogenous data than the rest of the bins.

- How is model accuracy calculated? This is usually far less objective than model constructors are willing to admit to. How does your error vary along different dimensions? Which dimensional error do you look at more closely, and why?

In my experience, the best way to reason about models is to work with a lot of different models, and to be honest about their flaws. That learning generally happens more efficiently, and more broadly, in the real world than in the classroom. With the recent rise in popularity of applied models, we have lots of inexperience modelers out there, so these growing pains are to be expected.

It seems like you're talking about human-created models (e.g. a manually-constructed decision tree), whereas the paper's primary concern is machine-created models (e.g. the weights derived mechanically by a neural network).

So the problem would be: how do you get the "modelers" (i.e. the machines) to be "honest about the flaws" of the models that they generated?