In observing the behavior of any group of social animals, one thing that stands out is, a pyramid shaped social hierarchy will eventuall form. This is so common that we take it for granted. Of course there's always a hierachy - why would't there be? When things are common, we stop asking why they are there, but wouldn't it be nice to find an explanation?
The big reason to try to explain common behavior is that there must be some value in it. If a person wants to accomplish something, such as build the next big app, they invariably must form a group to achieve their goal. Whether they succeed or not has a lot do with how they manage to create that group - whether it's successful recruitment, appropriate delegation, or common characterics amongs its members.
Why social hierachies exist then is an important question, and today we try to answer from the perspective of data science.
In data science, dimensional complexity means how many varaibles are we considering when we train our model. Let's say we are training a model to predict Igor's favorite car. We have four in this sample cars:
+-----------+-------------+---------------+
| make | year | model |
+-----------+-------------+---------------+
| Ford | 1998 | Taurus |
| Toyota | 1997 | Camry |
| Honda | 2000 | Accord |
| Tesla | 2019 | Model S |
+-----------+-------------+---------------+
Igor chose the Honda Accord from this sample, and we can use that information on other samples. Every time we add a row to the data set, we increase the work we have to do linearly. When we add another column, it's a different story. Let's say we add color.
+-----------+-------------+---------------+----------------+
| make | year | model | color |
+-----------+-------------+---------------+----------------+
| Ford | 1998 | Taurus | Red |
| Toyota | 1997 | Camry | Blue |
| Honda | 2000 | Accord | Yelow |
| Tesla | 2019 | Model S | Green |
+-----------+-------------+---------------+----------------+
To try to predict by color, our model has to consider the year, model, and make as well. Yellow on a honda may do it for him, but not on a Tesla. Adding another dimension increases our computational costs exponentially.
Let's say you have a long commute to work. Today, there's something wrong with your car. You have to shift to first gear for your breaks to work. Not a problem you say. If you concentrate really hard, you can do it. Let's also say you have to have to turn your wheel twice as much to turn your car left as you did before. If you concentrate more than you ever have, and you also get lucky, you may make it. Add a screaming, hungry kid to the back seat, and may you rest in peace.
Our ability to deal with dimensional complexity is pretty bad. When work applications ask for 3 years of experience for a junior dev role, what this says is that we are so bad at dealing with new variables, the only way around it is experience - practice, day and in and day out,for years in dealing with particilar variables in order not to screw everything up.
But, driving a car is a lone task, and does not require a group of people. Let's say you are building a house. You hired Chau to wire electricity, Mary to lay the bricks, and Pendleton to do the plumbing. If Pendleton starts too soon, Mary won't be able to lay the bricks. If Mary lays the bricks incorrectly, Chau can not do his work. Each person is a dimension you have to worry about. Every time you add a person, you increase the dimensional complexity of building a house.
How do people who build skyscrapers manage to succeed? If you have 5 brick layers, and 3 electricians, you hire someone to manage all the brick layers, and someone to manage the electricians, your dimensional complexityS remains 3, though you've increased your team by 8. Your team also looks like a pyramid - large at the base, but small at the top.
Dealing with complexity does not stop at building houses - in fact, it is one thing all tasks have in common. Some work has to get done. Some orchestration of the workers must take place, and that work in itself must be done by workers. This naturally forms a pyramid structure. How many 'levels' there are in a pyramid, vs nodes at each level is determined our brains' capacity to handle dimensional complexity.
What can be surmised from this perspective on group behavior?
If Rodrigo the plumber wants to join Igor's house buildling company, where is he going to plug into the pyramid? Going to Igor, Igor says my hands are full. Going to Igor's plumbing manager, Ben, he says I got enough to worry about. Going to one of the plumbers, Pendleton, is tough because Rodrigo and Pendleton are competitors for the same position. For Pendleton to be promoted to manager, there must be too many plumbers for Pendlton's manager to handle.
New members of a group are added to the bottom, and whoever was there earliest has the advantage, regardless of skill. If Ridrigo is a better plumbing manager than Ben the incombent, Igor still has to deal with the complexity of a new employee he doesn't know. It's not enough that Rodrigo is better, he will likely have to be added to the bottom regardless. From an evolutinary standpoint, this helps to explain mortality.
From the perspective of evolution, incombent advantage is a serious problem. In two groups of species A, temporarily divided by geography, of group Aa has longer lived individuals than Ab, this would pose a risk for Aa if the two populations ever met. New generations having to compete against older, established members will slow down adaptation.
If longer arms are selected for by their environment, yet older generations are longer lived, the advantage of longer arms may be realized at a lower rate. In other words, incombency being a greater advantage than even better physical traits means those traits are not selected for. If those populations ever converged, the shorter lived Ab group may outcompete Aa.
This explains a 'sunset mechanism' apparent in all species. Elephants eventually stop regrowing teeth, wood peckers get dizzy, and humans get cancer. That a short lifespan would be an evolutinary advantage seems counter intuitive, however taking into consideration how much it helps newer generations, the value is clear.
Some countries are more successful than other countries, and one of the traits that successful countries have is democracy. At its heart, democracy has the pattern of replacing incombent leaders. A population with short lived governing class would have a similar advantage as a population with a shorter average life span. This principle also applies to geographical regions.
Looking at Africa, a place where modern science agrees humans came from, one would wonder why it is in such bad shape, considering places like Europe and America are newer. One could say Africa 'was there first'.
If you are an African thousands of years ago, and you land in Europe, having taken a great risk to get away from an enemy tribe, you have a dilemma on your hands. First, you are with a small group of people in a strange land. It's cold, you don't know what you can or can't eat, and you're hungry.
As you leave your boat in search of food, everywhere you look, there's something missing: no other people around to compete for resources. With no set hierarchy in place, if a European came up with a new invention, he doesn't have to worry about upsetting some other, older, European. The slate is clean, as they say.
The success of America then is explained likewise. It is newer to Europe, and was able to outcompete it, because eventually Europe developed an incombent class. The tragic extension of this is that America will eventually also give birth to a class of people who use their position to prevent competition from replacing them. Luckily, there's always Mars.