How to solve problems: The basic schema

In this post I want to layout a general schema for how I approach problem solving in different contexts. Most of what I lay out is my synthesis of underlying principles that have governed my own thinking about various issues,as well as a ton of material that I have read.

My aim will be to continually refine and update this post as I apply this to tease out my conclusions in problem solving for specific questions.

Again, not all of this is original and revolutionary- much of it has been ground covered by thinkers who have most influenced me such as Karl Popper, Friedrich Von Hayek, Charles Darwin etc.

Broadly speaking there are two types of problems- "simple" and "complex". I don't mean "easy" and "difficult", several things (such as the theory of relativity) are difficult but "simple" in that you need to invoke very few principles to explain a vast number of phenomena.

"Complex" problems have one or many of the following characteristics:

1. Several agents: Unlike simple problems, which involve only one or two agents, complex problems involve the interaction of thousands of agents. I use the term "agent" and not "person" to make this applicable to problem solving situations where there are entities that are not necessarily a human being but act as if they have purpose or agency (to use a technical philosophy term- any entity where a teleological explanation is the best way to describe and understand it). Corporations would be one example, genes would be another. An aspect to note is that these agents largely optimize for their interest- that is, they are "selfish". This does not mean there is no altruism. just that the primary purpose of each agent is usually self-interest.

2. Information having three properties- being big, local and transient: Unlike simple problems, the information needed to solve complex problems is massive. It is also local, that is the information is distributed among several entities, and there is no efficient way to engineer gathering and storing the data centrally. Thirdly the data is transient, in that it is continually refreshed- so even if you somehow managed to warehouse it centrally you would need to keep refreshing it so from an engineering standpoint it would not be computable realistically.

Here is the simple idea: Simple problems can be solved efficiently by an "intelligent designer", complex problems need to be solved by "evolutionary systems".

Why?

Simple problems have only a few agents, and information that can be centrally gathered and processed and is not local or transient. In such cases an intelligent designer can process such information and solve the problem.

For complex problems no single entity can solve the problem since they would not even be able to gather the information required to do so, and even if they did, they would not be able to realistically impose their will on multiple agents.

So such problems require a problem solving system- not a single designer.

What would be the characteristics of such a system?

1. Experimentation or hypothesis generation: When there is no single place to access data to solve a problem you need tentative conjectures at a solution. Hence the problem solving system should allow experimentation so that several attempts can be made at solving the problem. This is also important since whatever solution is found is also transient (since the information itself is transient). so a process of continual experimentation is required

2. Failure and feedback or error elimination: The second step of the problem solving is that experiments should be allowed to fail, since such failure conveys information. When a tentative conjecture is wrong, it should be allowed to quickly fail, so that agents can infer that the conjecture is wrong and retry with a different experiments. Error elimination leads naturally to proliferation of "good solutions". Proliferation can be accelerated with active efforts (this is the difference between natural selection and artificial selection)

The system that has these two characteristics is an evolutionary system. I am using evolutionary in the broad philosophical sense and not just in the biological sense. Any system that has experimentation or hypothesis generation by multiple agents, in environments that are rich with massive, local and transient information, and allow for error elimination and feedback is an evolutionary system.

So, whenever you need to solve a problem, ask yourself, if it is simple or complex. If it is complex, set up an evolutionary system i.e. setup a a system that allows experimentation, that has a feedback mechanism and which will ruthlessly eliminate errors.

Let me take a simplified example (simplified in terms of making a direct argument without much nuance for reasons of clarity of explanation) to illustrate my point on problem solving for complex problems:

Example #1: Problem statement: How should India's GDP growth rate be boosted?

This is clearly a complex problem. It involves literally a billion producers and consumers. The information needed is the consumer needs of billions of people in billions of markets (e.g., what kind of toothpaste do they want, how much will they spend on mobile phones, on steel, on movies etc.) - the information is massive, local (what people in Bhakti Park spend on movies is different from what people in Bandra will spend) and transient.

What is the evolutionary system needed: The free market

The market needs to be free to allow experimentation. A hyper regulated market, where entrepreneurs cannot experiment with different business models to serve customer needs will be unable to solve each piece of this problem. A free market will allow local experiments and innovations such as dabbawalas to serve hot food to consumers in Mumbai at work. 

The market should also allow failure. Bankruptcy laws that allow firms to go out of business painlessly will convey information to all other firms about what works and what doesn't.

No single person knows what is needed to boost India's GDP. the beauty of evolutionary systems is that no one person or entity needs to solve the problem. The evolutionary system harnesses the problem solving ability of a billion people, each one working on a minor piece of the problem, forever experimenting, failing and learning , and that's how the full puzzle is cracked.

A footnote: Nothing of what i have said is original. But it is worth repeating, for one reason.

Some people support free markets since "they work". They point to the example of the US vs. the USSR etc. here, the free market is portrayed as a necessary evil- I have heard several people say that while Communism is a moral ideal, it doesn't work since there is no incentive for people to work. In the view of such people, a free market is the solution to a human failing- that is selfishness. Not only do I think this is naive empiricism since there may be several other factors that may govern the success or failure of a nation's economy, it also misses the point- of taking only one aspect of what makes the economy an evolutionary system- namely the multiple (and selfish) agents. But this totally ignores the fact that even if people were not selfish, the information required to solve the problem would simply not be available to a central planning authority (such as the Planning commission)

Some support free markets on the grounds of political philosophy. Most libertarians would be of this worldview- that economic freedom is a basic human right. I am sympathetic to this point of view, and even share it.

But there is a deeper reason also for my support of free markets. My economics is determined not just by empirical evidence, or my political philosophy (libertarianism) but my my understanding of epistemology (the study of knowledge in philosophy). Not only do free markets work and uphold basic human freedoms, they are the only way to deal with the nature of knowledge in a complex economic world- where information is massive, local and transient. In a world of Martians of homogeneous, unchanging and few consumer preferences, central planning may be ok- but not so with the basic nature of human beings and the world we inhabit.

In my subsequent blogposts I explore this notion of evolutionary systems to address more nuanced problems than the one on free markets.

An Indian in Indochina: A view from the "tributary"

For the next 6 months I will be based at Hanoi, Vietnam. I  will be capturing some of my experiences and reflections here in an occasional blogseries titled "An Indian in Indochina"

All modern nation-states require to construct a sense of identity and shared consciousness (of course, in a deeper sense, isn't all identity constructed?). Myth making, a historical revisionism and revivalism of a glorious past etc. are common ways that   elites use to construct such a sense of national identity among the larger population. 

In much of East Asia, the construction of national identity has also been animated by an impulse to resist the overwhelming and homogenizing influence of the historical hegemon of the region- China. 

China looms large in the consciousness of the Vietnamese sense of identity. Nationalist myths, not only in modern Vietnam, but in their history over thousand years, involve stories of men and women standing up to Chinese imperialism. (read this fascinating story of a sisterhood of women generals defeating the Chinese http://en.wikipedia.org/wiki/Trung_Sisters )

The Vietnamese self-perception is of a hardy and proud people, spiritedly resisting the rapacious and brutal Chinese forces, relentlessly seeking to pummel them into submission. 

Much of this is based on reality. In his deeply perceptive book- "When China Rules the World",  Martin Jacques wrote about how China is not a conventional nation-state, but has a character of a "tributary state". A tributary state is an imperial power, which perceives itself to be a rightful hegemon, but which is content to allow local rulers to wield administrative power as long as they acknowledge its supremacy. In ancient China, the neighbors- Korea, Japan, Vietnam etc. used to explicitly acknowledge this by paying tribute to the Chinese emperor. 

In many ways, this attitude seems to shape the conduct of Chinese foreign policy towards its neighbors even today. Indeed, Chinese accounts of the latest maritime disputes is bewilderment about the pesky tributaries not willingly accepting their status as inferior appendages of China, and having the temerity to stand-up to china instead. 

The fascinating thing is that though the national identity in East Asian countries is constructed in opposition to China, there is an implicit and un-self conscious acknowledgement of the centrality of China to defining their sense of self. 

Look no further than nomenclature for this. The Chinese name for China- "Zhongguo" - immodestly means "Center of the world". What is fascinating is that even the Japanese name for China- "Chuka Minkoku" also means "Center of the world" ! It's almost like the Japanese acknowledge the centrality of China. Similarly the Vietnamese also define themselves in relation to China- indeed the name Vietnam itself means "the south Viet" or "the land of Viets in the south", thus defining themselves in relation to their larger and more powerful neighbor. 

It is this curious dualism that is fascinating- the influence of China on culture and identity is undeniable, but its "soft power" is outweighed by its malign and arrogant "hard power". 

The most pithy quote to illustrate the modern Vietnamese identity toward China: In a conversation with a Vietnamese colleague I asked him if there was any residual anti-American feeling due to the Vietnam war. He replied "Now, we see the US as a potential ally. After all, we fought the US only for 20 years, and the French for only 200 years, but we have been resisting China for over 2000 years"!

Why Ben Bernanke is like "Baasha"/ Rajinikanth

The Federal Reserve's move to undertake QE3 is one of the biggest intellectual and policy shifts in the field of economics.

In this post I try to explain the nature of the policy and how it will work, by using an analogy from a scene from South Indian superstar- Rajinikanth's most famous movie- "Baasha". Watch the total kvlt scene below (unfortunately could not get a clip with subtitles, but like several Rajini scenes you can figure out what's going on even if you don't speak the language)

So here's what happens in the scene: The girl (Rajinikanth's sister) goes to the owner of a medical college to seek admission. He then lewdly propositions her; she then goes crying to Rajinikanth, who joins her for the next meeting with the college owner.

When the guy still doesn't yield, Rajinikanth asks him if he can have a word with him alone. That's when he reveals himself to be "Baasha"- a renowned mafia don. Then you see the other guy abjectly surrendering, saying that Rajinikanth could not just have one seat in the college, but the entire college if he chooses to!

Notice that Rajinikanth does not actually have to do anything- he doesn't have to actually beat up the guy or even explicitly threaten him.  In fact once he knows that it is "Baasha", the rational reaction is to give him whatever he wants without putting up a fight, since resistance would be a costly exercise in futility. Hell, it's Rajinikanth, who could win a fight with him!

Ben Bernanke's strategy to solve the unemployment crisis (based on a revolutionary new policy framework in economics called "market monetarism")  is based on a similar idea.

In a recession of the kind that we see in the US now, the essential problem is a shortfall of demand- that people are not spending. This becomes a self-reinforcing cycle. As an example, if I own a cement factory, I will not produce much during a recession since people are not constructing homes; because I am not producing cement I don't employ too many people in my factory- leading to unemployment; this unemployment means people do not have enough money and hence will not construct homes, which means I will not produce cement, hence the cycle continues.

Hence we get stuck in a recession because people in the market have low expectations of demand. When people act as if there will be low demand, it becomes a self-fulfilling prophecy.

So what can we do to change people future expectations of demand?

Well the only entity that can do so is the Federal Reserve- an entity which is nearly as powerful as Rajinikanth!

Here's how it works.

Step 1: The Federal Reserve explicitly lays out its goal. Here the goal is to grow the economy by a certain rate to ensure that unemployment reduces (just as Rajinikanth makes it clear that he really really wants that medical seat)

Step 2: The Federal Reserve then hints at what it will do if the goal is not reached. In Rajinikanth's case the hint was that he would beat up the guy, in the Fed's case it is that they would 'print money' to buy mortgage backed securities to lower long term interest rates to encourage spending

Remember the Federal Reserve has unlimited power to print money, just as Rajinikanth has unlimited power to beat people up. Resistance is futile, you might as well expect that the superpower will achieve what he/it wants and behave accordingly.

Then the superpower doesn't even have to carry on it's threat! Rajinikanth does not actually have to beat up the guy...the Federal Reserve did not buy a single mortgage backed security, yet stock prices still went up!

The reason is that the cement manufacturer now knows that he may as well accept the fact that we are going to have higher demand. This is because if the Federal Reserve does not get the demand it wants in the economy, it is going to print money and buy cement itself! (albeit indirectly). The Fed can print enough money to buy over the whole world- so it's a credible promise/threat that it will increase demand

Given that there is going to cement demand in the country it's the rational thing for me to produce cement; which means I will employ people in my factory; which means they will have money to spend, which means they will construct homes, which means that I need to produce cement. The hitherto negative spiral is converted to a positive one. The market acts as if the monetary infusion has already taken place, and then the monetary infusion is hardly required! (hence the name 'market monetarism')

Never take on Rajinikanth. Never take on the Fed. Assume that they will get what they want, and change your behavior accordingly, resistance is futile.

Why I find this appealing than fiscal stimulus (i.e. the Government spending to increase demand in the economy), apart from grounds of economic ideology (which is that it involves Government interference in the economy which as a libertarian I am opposed to) , is it's beautiful aesthetic symmetry. Recessions are caused due to low expectations, hence what you need to do is to change expectations. Fiscal stimulus means you have to actually interfere, a 'market monetary' stimulus involves just credibly promising/threatening to interfere, that's enough to change expectations.

Market monetarism started as an idea proposed by a few intrepid bloggers- Scott Sumner, David Beckworth, Nick Rowe etc. and has now become Fed policy. We are witnessing one of the most consequential shifts in how we conceptualise the roles of institutions such as the Fed in the economy. These ideas require to be understood and popularised in the wider public so that we have more public discourse about these policies.

Intellectual climates and ideology

Scott Sumner put up a very thought provoking post recently- " What if Matt Yglesias had been born in 1955?"

The basic argument of the post is that ideology is largely shaped by the intellectual climate that one is exposed to. If that was all that the post asserted, that would hardly be earth-shattering insight. But the intriguing part was the transmission mechanism- how does intellectual climate shape ideology.

It is not a simplistic story of affiliating with the ideology that happens to be more prevalent when one is young.

Here's a paragraph from the post that shows it:

Liberals (in the American sense) seemed so illogical during the 1970s, so naive.  I suppose my younger readers are now shaking their heads in disbelief:  “Everyone knows the conservatives are the stupid party, and always have been.”  They’d point to global warming denial, creationism, irrational fears of hyperinflation, or claims that “the military isn’t that expensive,” and hence a good way to employ the youth of America.

Yes, I know all that, but everyone is shaped by what was happening when they were young.  I recall:

1.  Progressive hostility to the hard truths of cost/benefit analysis.

2.  Hysteria over chemicals in food, even though science didn’t back them up.

3.  Denial of the role of money in inflation and support for crack-pot solutions such as wage price controls (Yes, I know about Nixon, I’m talking intellectuals now.)

4.  Denial of the disincentive effects of welfare programs.

5.  Soft on communism.  By that I don’t mean pro-communist, I mean anti-anti-communist.  When I was young calling someone an “anti-communist” was basically an insult in liberal company.  If you called Mao or Castro a brutal tyrant you were viewed as an embarrassment; as something of a something of a McCarthyite.  Chilean economic policies were viewed as evil.  Now the Chilean socialists have adopted them.

6.  Denial that punishment deters crime.

And I could go on and on.  It seemed to me that liberals weren’t willing to engage in clear, hard-headed, logical thinking.

 Building on Scott Sumner's example, here is my explanation of the transmission mechanism is as follows:

1) Due to various factors (exogenous to this explanation) there is a particular ideology that is more anchored in logic and analytical thinking. For example in the 1950s, conservatism was the intellectual ideology. In the current decade, liberal/ progressive ideologues (in the US) are much more intellectually rigorous than most of their influential conservative counterparts
 2)  Based on your own propensity towards intellectual/ analytical / logical thinking, you would then affiliate with the group that best mirrors that. Hence, what you affiliate with is not an ideology, but people who are equally smart. The ideology is an incidental by product. This makes perfect anthropological sense too, quality of thinking is a signal for potential power (this is called "expert power" in behavioral sciences); and people tend to affiliate with others who show the capacity for such power
3) Then, do to in-group vs out-group reasons, you would double down and defend "your side" even when the intellectual climate changes. Witness the vast mass of Republican economists reduced to the level of having to defend the various lunacies of the party's current anti-intellectual idiot core (which Krugman calls having to play for "Team Republican")

I like this explanation since it makes sense anthropologically, and since it explains why the concentration of "intellectuals" shifts so much over time. It is not the ideology per se (one can be an intellectual conservative or liberal or libertarian). It is actually more about how we go about social and intellectual affiliations. This explains why an intellectual Scott Sumner growing up in the 1950s ends up as a conservative, while a Matt Yglesias ends up as a liberal

What would be interesting to investigate is how the intellectual climate itself shifts- what causes a massive concentration of intellectuals in one ideological group or the other

I can relate this to my own life. I was growing up in the 1990s in newly liberalizing India. The wages of a sterile socialism had hit us hard with the balance of payments crisis and the overall impending macroeconomic collapse of India. The rapid and visible change that I witnessed in the wake of economic liberalization, accompanied by the logically sound and rigorous arguments made for economic freedom (as opposed to the ridiculously trite and vacuous rhetoric of the left (" companies should work on a no profit, no loss basis"; "manual labor should be respected, so every worker in a company should be paid the same amount- from the CEO to a sweeper"!- believe me these were actual arguments at the time!)

A lot of my libertarianism was probably shaped by that

Why Obamacare's contraception mandate offends me...as a finance professional

The latest controversy associated with the Patient Protection and Affordable Care Act (aka "Obamacare") is the mandatory requirement for employers to provide health insurance for employees that includes coverage for contraception (for example, the birth control pill)

Predictably, there has been outrage in the US, with social conservatives raving about Obama's "War on religion" and even non-religious libertarians being unhappy with the imposition of a state imposed mandate upon people to undertake an activity that conflicts with their values.

But even if you're not outraged, on grounds of either religion or economic and civil liberties, the coverage of contraception by the insurance mandate does not make any sense from the standpoint of conceptual finance

Chicago economist, John Cochrane, explains it well:

" Critics are missing the larger point. Why should the Department of Health and Human Services (HHS) decree that any of us must pay for "insurance" that covers contraceptives?

I put "insurance" in quotes for a reason. Insurance is supposed to mean a contract, by which a company pays for large, unanticipated expenses in return for a premium: expenses like your house burning down, your car getting stolen or a big medical bill.

Insurance is a bad idea for small, regular and predictable expenses. There are good reasons that your car insurance company doesn't add $100 per year to your premium and then cover oil changes, and that your health insurance doesn't charge $50 more per year and cover toothpaste. You'd have to fill out mountains of paperwork, the oil-change and toothpaste markets would become much less competitive, and you'd end up spending more" 

Read the full article here

As a person who works in financial services, a pet peeve of mine is the use of a financial product for a purpose that is totally divorced from its raison d'etre

The use of insurance for a regular, predictable and small expense is just wrong, wrong, wrong!

As I have outlined elsewhere in this blog, the 2 main purposes of finance is to help deal with intertemporal tradeoffs and "states of the world". Savings, investments and loans are the most common device for intertemporal tradeoffs (if I don't need money today, but need it tomorrow- for example to retire to the Bahamas, I will invest or same the money; if I need it today (for example for a house or a car), but cannot fund it from immediate incomes or savings, I will take a loan)

Insurance is the financial product which finds use in managing different "states of the world". It is a means to manage risk; to deal with uncertainty. Death; accident or personal injury; loss of assets; large unanticipated expenses (such as an operation or chemotherapy) are good candidates

Birth control pills clearly fail this test. They entail a small, regular and entirely predictable expense as Cochrane says. Hence it should be funded purely through current incomes and savings.

So called "insurance" is not just unnecessary, it is counterproductive. All insurance involves paperwork (since to claim the money from the insurance company you need to show invoices) Hence there is a waste of time, money etc. in claims processing which will just push up insurance costs

As a libertarian, I am opposed to Government mandated health insurance in any case. But whether you are a social conservative or liberal; libertarian or socialist; the Obama contraception mandate should be opposed on grounds of its violation of the concepts of finance