In this post I want to layout a general schema for how I approach problem solving in different contexts. Most of what I lay out is my synthesis of underlying principles that have governed my own thinking about various issues,as well as a ton of material that I have read.
My aim will be to continually refine and update this post as I apply this to tease out my conclusions in problem solving for specific questions.
Again, not all of this is original and revolutionary- much of it has been ground covered by thinkers who have most influenced me such as Karl Popper, Friedrich Von Hayek, Charles Darwin etc.
Broadly speaking there are two types of problems- "simple" and "complex". I don't mean "easy" and "difficult", several things (such as the theory of relativity) are difficult but "simple" in that you need to invoke very few principles to explain a vast number of phenomena.
"Complex" problems have one or many of the following characteristics:
1. Several agents: Unlike simple problems, which involve only one or two agents, complex problems involve the interaction of thousands of agents. I use the term "agent" and not "person" to make this applicable to problem solving situations where there are entities that are not necessarily a human being but act as if they have purpose or agency (to use a technical philosophy term- any entity where a teleological explanation is the best way to describe and understand it). Corporations would be one example, genes would be another. An aspect to note is that these agents largely optimize for their interest- that is, they are "selfish". This does not mean there is no altruism. just that the primary purpose of each agent is usually self-interest.
2. Information having three properties- being big, local and transient: Unlike simple problems, the information needed to solve complex problems is massive. It is also local, that is the information is distributed among several entities, and there is no efficient way to engineer gathering and storing the data centrally. Thirdly the data is transient, in that it is continually refreshed- so even if you somehow managed to warehouse it centrally you would need to keep refreshing it so from an engineering standpoint it would not be computable realistically.
Here is the simple idea: Simple problems can be solved efficiently by an "intelligent designer", complex problems need to be solved by "evolutionary systems".
Why?
Simple problems have only a few agents, and information that can be centrally gathered and processed and is not local or transient. In such cases an intelligent designer can process such information and solve the problem.
For complex problems no single entity can solve the problem since they would not even be able to gather the information required to do so, and even if they did, they would not be able to realistically impose their will on multiple agents.
So such problems require a problem solving system- not a single designer.
What would be the characteristics of such a system?
1. Experimentation or hypothesis generation: When there is no single place to access data to solve a problem you need tentative conjectures at a solution. Hence the problem solving system should allow experimentation so that several attempts can be made at solving the problem. This is also important since whatever solution is found is also transient (since the information itself is transient). so a process of continual experimentation is required
2. Failure and feedback or error elimination: The second step of the problem solving is that experiments should be allowed to fail, since such failure conveys information. When a tentative conjecture is wrong, it should be allowed to quickly fail, so that agents can infer that the conjecture is wrong and retry with a different experiments. Error elimination leads naturally to proliferation of "good solutions". Proliferation can be accelerated with active efforts (this is the difference between natural selection and artificial selection)
The system that has these two characteristics is an evolutionary system. I am using evolutionary in the broad philosophical sense and not just in the biological sense. Any system that has experimentation or hypothesis generation by multiple agents, in environments that are rich with massive, local and transient information, and allow for error elimination and feedback is an evolutionary system.
So, whenever you need to solve a problem, ask yourself, if it is simple or complex. If it is complex, set up an evolutionary system i.e. setup a a system that allows experimentation, that has a feedback mechanism and which will ruthlessly eliminate errors.
Let me take a simplified example (simplified in terms of making a direct argument without much nuance for reasons of clarity of explanation) to illustrate my point on problem solving for complex problems:
Example #1: Problem statement: How should India's GDP growth rate be boosted?
This is clearly a complex problem. It involves literally a billion producers and consumers. The information needed is the consumer needs of billions of people in billions of markets (e.g., what kind of toothpaste do they want, how much will they spend on mobile phones, on steel, on movies etc.) - the information is massive, local (what people in Bhakti Park spend on movies is different from what people in Bandra will spend) and transient.
What is the evolutionary system needed: The free market
The market needs to be free to allow experimentation. A hyper regulated market, where entrepreneurs cannot experiment with different business models to serve customer needs will be unable to solve each piece of this problem. A free market will allow local experiments and innovations such as dabbawalas to serve hot food to consumers in Mumbai at work.
The market should also allow failure. Bankruptcy laws that allow firms to go out of business painlessly will convey information to all other firms about what works and what doesn't.
No single person knows what is needed to boost India's GDP. the beauty of evolutionary systems is that no one person or entity needs to solve the problem. The evolutionary system harnesses the problem solving ability of a billion people, each one working on a minor piece of the problem, forever experimenting, failing and learning , and that's how the full puzzle is cracked.
A footnote: Nothing of what i have said is original. But it is worth repeating, for one reason.
Some people support free markets since "they work". They point to the example of the US vs. the USSR etc. here, the free market is portrayed as a necessary evil- I have heard several people say that while Communism is a moral ideal, it doesn't work since there is no incentive for people to work. In the view of such people, a free market is the solution to a human failing- that is selfishness. Not only do I think this is naive empiricism since there may be several other factors that may govern the success or failure of a nation's economy, it also misses the point- of taking only one aspect of what makes the economy an evolutionary system- namely the multiple (and selfish) agents. But this totally ignores the fact that even if people were not selfish, the information required to solve the problem would simply not be available to a central planning authority (such as the Planning commission)
Some support free markets on the grounds of political philosophy. Most libertarians would be of this worldview- that economic freedom is a basic human right. I am sympathetic to this point of view, and even share it.
But there is a deeper reason also for my support of free markets. My economics is determined not just by empirical evidence, or my political philosophy (libertarianism) but my my understanding of epistemology (the study of knowledge in philosophy). Not only do free markets work and uphold basic human freedoms, they are the only way to deal with the nature of knowledge in a complex economic world- where information is massive, local and transient. In a world of Martians of homogeneous, unchanging and few consumer preferences, central planning may be ok- but not so with the basic nature of human beings and the world we inhabit.
In my subsequent blogposts I explore this notion of evolutionary systems to address more nuanced problems than the one on free markets.
My aim will be to continually refine and update this post as I apply this to tease out my conclusions in problem solving for specific questions.
Again, not all of this is original and revolutionary- much of it has been ground covered by thinkers who have most influenced me such as Karl Popper, Friedrich Von Hayek, Charles Darwin etc.
Broadly speaking there are two types of problems- "simple" and "complex". I don't mean "easy" and "difficult", several things (such as the theory of relativity) are difficult but "simple" in that you need to invoke very few principles to explain a vast number of phenomena.
"Complex" problems have one or many of the following characteristics:
1. Several agents: Unlike simple problems, which involve only one or two agents, complex problems involve the interaction of thousands of agents. I use the term "agent" and not "person" to make this applicable to problem solving situations where there are entities that are not necessarily a human being but act as if they have purpose or agency (to use a technical philosophy term- any entity where a teleological explanation is the best way to describe and understand it). Corporations would be one example, genes would be another. An aspect to note is that these agents largely optimize for their interest- that is, they are "selfish". This does not mean there is no altruism. just that the primary purpose of each agent is usually self-interest.
2. Information having three properties- being big, local and transient: Unlike simple problems, the information needed to solve complex problems is massive. It is also local, that is the information is distributed among several entities, and there is no efficient way to engineer gathering and storing the data centrally. Thirdly the data is transient, in that it is continually refreshed- so even if you somehow managed to warehouse it centrally you would need to keep refreshing it so from an engineering standpoint it would not be computable realistically.
Here is the simple idea: Simple problems can be solved efficiently by an "intelligent designer", complex problems need to be solved by "evolutionary systems".
Why?
Simple problems have only a few agents, and information that can be centrally gathered and processed and is not local or transient. In such cases an intelligent designer can process such information and solve the problem.
For complex problems no single entity can solve the problem since they would not even be able to gather the information required to do so, and even if they did, they would not be able to realistically impose their will on multiple agents.
So such problems require a problem solving system- not a single designer.
What would be the characteristics of such a system?
1. Experimentation or hypothesis generation: When there is no single place to access data to solve a problem you need tentative conjectures at a solution. Hence the problem solving system should allow experimentation so that several attempts can be made at solving the problem. This is also important since whatever solution is found is also transient (since the information itself is transient). so a process of continual experimentation is required
2. Failure and feedback or error elimination: The second step of the problem solving is that experiments should be allowed to fail, since such failure conveys information. When a tentative conjecture is wrong, it should be allowed to quickly fail, so that agents can infer that the conjecture is wrong and retry with a different experiments. Error elimination leads naturally to proliferation of "good solutions". Proliferation can be accelerated with active efforts (this is the difference between natural selection and artificial selection)
The system that has these two characteristics is an evolutionary system. I am using evolutionary in the broad philosophical sense and not just in the biological sense. Any system that has experimentation or hypothesis generation by multiple agents, in environments that are rich with massive, local and transient information, and allow for error elimination and feedback is an evolutionary system.
So, whenever you need to solve a problem, ask yourself, if it is simple or complex. If it is complex, set up an evolutionary system i.e. setup a a system that allows experimentation, that has a feedback mechanism and which will ruthlessly eliminate errors.
Let me take a simplified example (simplified in terms of making a direct argument without much nuance for reasons of clarity of explanation) to illustrate my point on problem solving for complex problems:
Example #1: Problem statement: How should India's GDP growth rate be boosted?
This is clearly a complex problem. It involves literally a billion producers and consumers. The information needed is the consumer needs of billions of people in billions of markets (e.g., what kind of toothpaste do they want, how much will they spend on mobile phones, on steel, on movies etc.) - the information is massive, local (what people in Bhakti Park spend on movies is different from what people in Bandra will spend) and transient.
What is the evolutionary system needed: The free market
The market needs to be free to allow experimentation. A hyper regulated market, where entrepreneurs cannot experiment with different business models to serve customer needs will be unable to solve each piece of this problem. A free market will allow local experiments and innovations such as dabbawalas to serve hot food to consumers in Mumbai at work.
The market should also allow failure. Bankruptcy laws that allow firms to go out of business painlessly will convey information to all other firms about what works and what doesn't.
No single person knows what is needed to boost India's GDP. the beauty of evolutionary systems is that no one person or entity needs to solve the problem. The evolutionary system harnesses the problem solving ability of a billion people, each one working on a minor piece of the problem, forever experimenting, failing and learning , and that's how the full puzzle is cracked.
A footnote: Nothing of what i have said is original. But it is worth repeating, for one reason.
Some people support free markets since "they work". They point to the example of the US vs. the USSR etc. here, the free market is portrayed as a necessary evil- I have heard several people say that while Communism is a moral ideal, it doesn't work since there is no incentive for people to work. In the view of such people, a free market is the solution to a human failing- that is selfishness. Not only do I think this is naive empiricism since there may be several other factors that may govern the success or failure of a nation's economy, it also misses the point- of taking only one aspect of what makes the economy an evolutionary system- namely the multiple (and selfish) agents. But this totally ignores the fact that even if people were not selfish, the information required to solve the problem would simply not be available to a central planning authority (such as the Planning commission)
Some support free markets on the grounds of political philosophy. Most libertarians would be of this worldview- that economic freedom is a basic human right. I am sympathetic to this point of view, and even share it.
But there is a deeper reason also for my support of free markets. My economics is determined not just by empirical evidence, or my political philosophy (libertarianism) but my my understanding of epistemology (the study of knowledge in philosophy). Not only do free markets work and uphold basic human freedoms, they are the only way to deal with the nature of knowledge in a complex economic world- where information is massive, local and transient. In a world of Martians of homogeneous, unchanging and few consumer preferences, central planning may be ok- but not so with the basic nature of human beings and the world we inhabit.
In my subsequent blogposts I explore this notion of evolutionary systems to address more nuanced problems than the one on free markets.
No comments:
Post a Comment