How Innovation Works

date Aug 28, 2021
authors Matt Ridley
reading time 25 mins

Table of Contents


What is innovation?

Innovation, like evolution, is a process of constantly discovering ways of rearranging the world into forms that are unlikely to arise by chance – and that happen to be useful.

Innovation vs Invention

Innovation, then, means finding new ways to apply energy to create improbable things, and see them catch on. It means much more than invention, because the word implies developing an invention to the point where it catches on because it is sufficiently practical, affordable, reliable and ubiquitous to be worth using.

Conditions for innovation

Serendipity plays a big part in innovation, which is why liberal economies, with their free-roving experimental opportunities, do so well. They give luck a chance. Innovation happens when people are free to think, experiment and speculate. It happens when people can trade with each other. It happens where people are relatively prosperous, not desperate. It is somewhat contagious. It needs investment. It generally happens in cities.

Keep on experimenting

It runs mostly on trial and error, the human version of natural selection. And it usually stumbles on great breakthroughs when looking for something else: it is heavily serendipitous.

Slow and gradual

the effect begins very slowly, gathers pace gradually and works by increments, not leaps and bounds. Innovation often disappoints in its early years, only to exceed expectations once it gets going, a phenomenon I call the Amara hype cycle, after Roy Amara, who first said that we underestimate the impact of innovation in the long run but overestimate it in the short run.

Property rights

Why did this revolution happen in America, an old, played-out and well-explored oil and gas region? The answer lies partly in property rights. Because of mineral rights belonging to local landowners, rather than the state, and because oil companies had never been nationalized, as they were in so many other countries,

Many inventors and inventions happen at similar times

The coincidence of timing is strange, but quite characteristic of inventors. Again and again, simultaneous invention marks the progress of technology as if there is something ripe about the moment. It does not necessarily imply plagiarism.

Permissionless innovation

The restaurant industry is addicted to innovation. It experiences rapid turnover as once-fashionable eating spots give way to new ones, with zero protection from government for those who prefer to resist innovation, zero subsidy for those who wish to innovate and zero overall strategy from experts. It is as close as you can get to a permissionless innovation system.

de Novo vs recombination

A study of Noma by two (hungry) professors of innovation stresses that the main method of innovation here is not de novo invention, but recombination – bringing old things together in new combinations – and that this is a general feature of innovation elsewhere in the economy.

Feed-forward and feedback

Those who have studied how chefs innovate report that they follow a process of feed-forward trial and modification, experimenting with variations on a central idea till they hit on a dish that they think will win the approval of customers. It is not very different from the way Thomas Edison improved the light bulb.

Adjacent possible step

man-made technologies evolve from previous man-made technologies, and are not invented from scratch. This is a key characteristic of evolutionary systems: the move to the ‘adjacent possible’ step.

Examples of innovation

Light bulb

Every single one of these people produced, published or patented the idea of a glowing filament in a bulb of glass, sometimes with a vacuum, sometimes with nitrogen inside the bulb, and all before Thomas Edison. The truth is that twenty-one different people can lay claim to have independently designed or critically improved incandescent light bulbs by the end of the 1870s, mostly independent of each other,

Edison’s approach to innovation (not invention)

Invention, he famously said, is 1 per cent inspiration and 99 per cent perspiration. Yet in effect what he was doing was not invention, so much as innovation: turning ideas into practical, reliable and affordable reality.

Innovation in nuclear power went backwards

This is not for lack of ideas, but for a very different reason: lack of opportunity to experiment. The story of nuclear power is a cautionary tale of how innovation falters, and even goes backwards, if it cannot evolve.

No trial and error

And the industry remains insulated almost entirely from the one known human process that reliably pulls down costs: trial and error. Because error could be so cataclysmic in the case of nuclear power, and because trials are so gigantically costly, nuclear power cannot get trial and error restarted.

Innovation in fusion

Fusion may now come to commercial fruition, in the form of many relatively small reactors generating electricity, maybe 400 megawatts each. It is a technology that brings almost no risk of explosion or meltdown, very little in the way of radioactive waste and no worries about providing material for weapons.

Innovation in water treatment

The Jersey City case proved a turning point, a clean-watershed. Cities all over the country and the world began using chlorination to clean up water supplies, as they do to this day. Typhoid, cholera and diarrhoea epidemics rapidly disappeared. But where did Dr Leal get the idea? From a similar experiment in Lincoln in England, he said at the trial. Like most innovators he did not claim to be the inventor.

Innovation in transportation

For all of human history until the 1820s, nobody went faster than the speed of a galloping horse. Then within a generation it became routine to travel at speeds three times that fast, and for hours at a time.

Innovation in internal combustion engine

The story of the internal-combustion engine displays the usual features of an innovation: a long and deep prehistory characterized by failure; a shorter period marked by an improvement in affordability characterized by simultaneous patenting and rivalries; and a subsequent story of evolutionary improvement by trial and error.

Ford and innovation

But he had a relentless genius for cost control and he then began making a car that was simpler than most on the market, relatively cheap – and going to get cheaper with mass production.

Innovation in aviation - scrappy vs well-funded

Where Langley had done everything wrong – spending lots of money, depending on the government, consulting few other people, building a fully fledged device from scratch, rather than inching incrementally through each of the problems to be solved – the Wrights had done everything right.

Innovation in aviation - inevitable

There is little doubt that somebody would have got planes into the air within the first decade of the twentieth century even without the Wrights. Motors made it inevitable that many people would then try, and trial and error was all that was really needed.

Innovation in jet engines

After the Second World War, the race to improve and perfect the jet engine for passenger aircraft, as well as for military planes, was mainly carried out within three big companies: Pratt and Whitney, General Electric and Rolls-Royce.

Innovation in flight safety - slow, gradual, incremental

More generally, it is the widespread use of dull, low-tech but vital practices such as ‘crew resource management’ techniques, and checklists galore, with cross-checking between crew members, and a culture of challenge, that have made the big difference since the 1970s.

Transparency and multiple domains

As cases like this show, there are multiple factors of technology, procedure and psychology that safety designers have to get right in making flight safer. Most crucial of all is learning from mistakes such as this accident, by openly and transparently sharing the results of accident investigations all around the world.

How potato became a staple

Farmers who planted potatoes therefore tended to survive better during wars, spreading the habit. As John Reader recounts, the result of Frederick the Great’s wars was that the potato, unknown or despised in most of central and eastern Europe in 1700, had by 1800 become an indispensable part of the European diet.

Innovation in agriculture across the globe

This fifty-year story of how dwarfing genes were first found in Japan, cross-bred in Washington, adapted in Mexico and then introduced against fierce opposition in India and Pakistan is one of the most miraculous in the history of humankind.

Innovation in gene editing - ridiculous disputes at first

Highly useful scientific discoveries are almost always – ridiculously often – accompanied by frenzied disputes about who deserves the credit. In no case is this more true than in the story of CRISPR, a genetic technique that the world awoke to in 2012, and which promises wonderful results in agriculture as well as medicine.

Less land is not required for more food

Between 1960 and 2010, the acreage of land needed to produce a given quantity of food has declined by about 65 per cent. Had this not happened, pretty well every acre of forest, wetland and nature reserve in the world would have been cultivated or grazed, and the Amazon rain forest would have been far more severely destroyed.

Innovation in sanitization

Perhaps the neatest innovation is the S-bend or U-bend in the pipe beneath every toilet, which traps water so as to prevent any smell coming back up the pipe. It’s gorgeously simple and exquisitely clever. It transformed the flush toilet into a strong competitor against the chamberpot.

Europeans learning numbers from Arabs and Indians

Fibonacci learned arithmetic in Bugia in the Arab style, and probably in the Arabic language, and he quickly realized that the Arabic notation, borrowed from the Indians, was far more practical and versatile than Roman numerals.

Invention of Zero

Though mostly about astronomy, it had chapters on mathematics and is the first known work to treat zero as an actual number, rather than as a symbol for nothing as the Babylonians had done. In simple and easily understood statements, Brahmagupta set out the significance of zero and considered negative numbers for the first time,

Meaning of negative numbers

‘A debt minus zero is a debt. A fortune minus zero is a fortune. Zero minus zero is a zero. A debt subtracted from zero is a fortune. A fortune subtracted from zero is a debt. The product of zero multiplied by a debt or fortune is zero.’

Accounting led the way in Mathematics

In the fourteenth century, ledgers would sometimes have columns of Indian numbers and paragraphs of Roman ones intermingled or taking turns. Yet gradually numbers won, especially in the preparation of merchants’ accounts: commerce led the way.

Innovation in shipping

The capacity of container shipping was growing at 20 per cent a year, and ships were getting bigger and bigger. Per ton of cargo, a bigger ship cost less to build, required a smaller crew and consumed less fuel than a smaller one. The only limit was getting through the locks of the Panama Canal.

Innovation of wheeled lugguages

regard the wheeled suitcase as one of the pinnacles of civilization. But for something so low tech, it turned up surprisingly late, after the first human beings had landed on the moon… Clearly, the problem was not a lack of inspiration. Instead, what seems to have stopped wheeled suitcases from catching on was mainly the architecture of stations and airports.

Innovation in telegraph - battle through non-technical obstacles

Morse’s real achievement, like that of most innovators, was to battle his way through political and practical obstacles.

Misconceptions in innovation

Utopian hope in Telegraph or the Internet

There was widespread utopian hope about the telegraph’s impact on society, as there would be 150 years later for the internet. The wires would make war less likely, keep families in touch, transform the practice of finance and deter crime, commentators speculated.

Inventor vs Innovator: Meucci vs Bell

The reason history forgot Meucci is because, unlike the determined Bell, he raised no money to develop the idea or defend his patents, and his candle factory went broke, leaving him in poverty and bankruptcy. He was an inventor, but not an innovator.

Inventions are inevitable

Had Marconi not lived, radio would still have come to life in the 1890s. Others such as Jagadish Chandra Bose in India, Oliver Lodge in Britain and Alexander Popov in Russia were doing and publishing experiments that used electromagnetic waves to create action at a distance, though not always for communication.

IP rights singles out individual inventors

Marconi was just another experimenter, though a very good one, but thanks to Newbolt he was quick to patent what he had found as broadly as possible, thus demonstrating that it is the system of intellectual property that contributes to the singling out of individual inventors, as much as the other way around.

Media, journalists and biographers like to portray innovation as sudden and world-changing

The laurels that garland the forehead of a true ‘inventor’ are irresistible. But it is not just the inventor who likes to portray innovation as sudden and world-changing. So do journalists and biographers.

Not all effects of innovation are positive

Social media took the world by surprise in another way too. Far from ushering in an era of utopian democratic enlightenment in which the world is flat, everybody is sharing and we all see each other’s point of view, it plunged us into a maze of echo chambers and filter bubbles in which we spend our time confirming our biases and railing against the opinions of others. It polarized, enraged, depressed, addicted and soured us.

Initial upheaval paves the way for a better society

We have been here before. The invention of printing caused political and social upheaval in Western societies that polarized society and killed a lot of people… It also ushered in an enlightenment of knowledge and reason unprecedented in scope and depth.

Augment humans, not replace them - automation or AI

For the moment, the safest bet is that artificial intelligence will augment rather than replace people, as automation has done for centuries.

Conginitve bias that things are under control

Most people want to think they have more control over their lives than is objectively the case: the idea of decisive and discontinuous human agency is both flattering and comforting.


Nationalism exacerbates the problem. All too often, the importing of a new idea gets confused with the inventing of a new idea.

Too little credit for inventors

It shows a beaver and a rabbit looking up at the Hoover dam: ‘No, I didn’t build it myself,’ says the beaver. ‘But it’s based on an idea of mine.’ All too often discoverers and inventors feel short-changed that they get too little credit or profit from a good idea,

Era of computing

What was von Neumann’s role?

Thus von Neumann’s unique privilege in being able to travel freely between the teams (and his high security clearance) made him a vital cross-pollinator of ideas.

Faster pace in peace time

Had Zuse, Turing, von Neumann, Mauchly, Hopper and Aiken all met at a conference in peacetime, who knows what would have happened and how fast?

You cannot jump steps in innovation

Most surprising of all, discovering Moore’s Law had no effect on Moore’s Law. Knowing that the cost of a given amount of processing power would halve in two years ought surely to have been valuable information, allowing an enterprising innovator to jump ahead and achieve that goal now. Yet it never happened. Why not? Mainly because it took each incremental stage to work out how to get to the next stage.

Invention + Innovation

This was encapsulated in Intel’s famous ‘tick-tock’ corporate strategy: tick was the release of a new chip every other year, tock was the fine-tuning of the design in the intervening years, preparatory to the next launch.

Knowledge to learning in AI

Thus the focus of artificial intelligence has shifted from the ‘expert system’ approach in which clever people try to impart their knowledge to computers, to a learning approach in which programs find ways to solve problems themselves. This was made possible by three features of the modern computing world: new software, new hardware and new data.

Outburst of innovation

Innovation happens in a place where people are not struggling

There is another similarity with later bursts of innovation: it happened at a time of plenty and in a place of plenty. Just as innovation flourishes in wealthy, growing and well-connected places at a time of peace and relative prosperity – California today, Newcastle in Stephenson’s day, Renaissance Italy in Fibonacci’s day – so farming began in the warm, well-watered valleys of the Euphrates, the Yangtze and the Mississippi, or in the rich and sun-baked soils of New Guinea and the Andes.

Innovation requires specialization or division of labor

The life of a hunter-gatherer, therefore, must be mobile, nomadic and lonely: bands would be small and distances between bands large. In such conditions, the collective brain is small – there is not much room for specialization or a division of labour. Hunter-gatherers in such habitats retain very simple tools, cultures and habits.

Innovation requires critical mass of people and trading

Pacific islanders have more complicated fishing technologies if they live on larger islands and – crucially – if those islands have good trading links with other islands.

Innovation requires good connection to long distances

If they could get objects from a long distance then they could get ideas too. And to this day small, isolated populations show simplified technology and slow rates of innovation.

Innovation is a collective phenomenon

Innovation flourished in cities that traded freely with other cities, in India, China, Phoenicia, Greece, Arabia, Italy, Holland and Britain: places where ideas could meet and mate to produce new ideas. Innovation is a collective phenomenon that happens between, not within, brains.

Process of innovation

Recombination more than random mutation

Wagner cites numerous studies which support the conclusion that ‘recombination is much more likely to preserve life – up to a thousand times more likely – than random mutation is.’ This is because whole working genes, or parts of genes, can be given new jobs, where a step-by-step change would find only worse results.

Recombining existing technologies

In the same way, innovation in one technology borrows whole, working parts from other technologies, rather than designing them from scratch. The inventors of the motor car did not have to invent wheels, springs or steel.

High tolerance for error

Most inventors find that they need to keep ‘just trying’ things. Tolerance of error is therefore critical. It is notable that during the early years of a new technology – the railway, for example, or the internet – far more entrepreneurs went broke than made fortunes.


An element of playfulness probably helps, too. Innovators who just like playing around are more likely to find something unexpected.

Natural selection rather than intelligent design

Using examples like this, Edward Wasserman of the University of Iowa has made the case that most human innovations evolve through a process that looks awfully like natural selection, rather than are created by intelligent design.

Difference roles in innovation

One person may make a technological breakthrough, another work out how to manufacture it and a third how to make it cheap enough to catch on. All are part of the innovation process and none of them knows how to achieve the whole innovation.

Simultaneous invention

Simultaneous invention is more the rule than the exception. Many ideas for technology just seem to be ripe, and ready to fall from the tree. The most astonishing case is the electric light bulb, the invention of which was independently achieved by twenty-one people.

Do an individual matter anymore?

Far from being an insult, therefore, my jibe about inevitability and dispensability is actually a compliment. How incredible to be the one human being among billions who first sees the possibility of a new device, a new mechanism, a new idea. That is arguably even more miraculous than achieving something that would never be achieved by anybody else, like the Mona Lisa or ‘Hey Jude’.

Retrospective predictability

Technology is absurdly predictable in retrospect, wholly unpredictable in prospect.

Governments and countries

Political fragmentation and spread of innovation

The fact that Europe was politically fragmented at the time played a large role in making sure that printing caught on. Johann Gutenberg himself had to leave his home city of Mainz and move to Strasbourg to find a regime that would let him get to work.

How modern-day USA is similar to Europe’s political fragmentation

America may appear an exception, but in fact it proves this rule. Its federal structure has always allowed experiment. Far from being a monolithic imperium, the states were for most of the nineteenth and twentieth centuries a laboratory of different rules, taxes, policies and habits, with entrepreneurs moving freely to whichever state most suited their project.

Companies become less innovative as they grow, but not cities

This fact is not true of companies. As they grow bigger, beyond a certain point they become less efficient, less manageable, less innovative, less frugal and less tolerant of eccentricity. That, says West, is why companies die all the time, but cities never do… London proportionately burns less energy than Bristol, has a bigger collective brain and behaves in a more complicated way. The same is true throughout the economy.

Linear model of science to technology to innovation is not true

There is a widely held view among politicians, journalists and the public that science leads to technology, which leads to innovation. This ‘linear model’ holds sway amongst almost all policy makers and is used to justify public spending on science,

Big companies need the outer world to innovate

Big companies are bad at innovating, because they are too bureaucratic, have too big a vested interest in the status quo and stop paying attention to the interests, actual and potential, of their customers. Thus for innovation to flourish it is vital to have an economy that encourages or at least allows outsiders, challengers and disruptors to get a foothold.

Road to scam is paved with small decisions

in relation to the Theranos story: ‘A scam doesn’t come about in one single moment. Rather, its creation is more like a slow trail of breadcrumbs, the end result of many little, seemingly innocuous decisions made along the way.’ Like almost anything complex, crimes evolve.

The hype and delusion of VC-funded startups

He argues that a general lesson still needs to be learned: ‘hyping your product to get funding while concealing your true progress and hoping that reality will eventually catch up to the hype continues to be tolerated in the tech industry.’

Speed of experiments

‘Our success at Amazon is a function of how many experiments we do per year, per month, per week. Being wrong might hurt you a bit, but being slow will kill you,’ Bezos once said: ‘If you can increase the number of experiments you try from a hundred to a thousand, you dramatically increase the number of innovations you produce.’

Science is public-funded, but results are hidden away

Meanwhile, in science, you the taxpayer pay for most research, yet the published results are hidden away behind high paywalls in learned journals dominated by three highly profitable firms, Elsevier, Springer and Wiley, whose business model is to sell back to the taxpayer, in the form of library subscriptions, the fruits of his or her investment.

Do patents create innovation?

beyond a certain point stronger patents generate less innovation, because they make it hard to share ideas, and create barriers to entry. The Semiconductor Chip Protection Act of 1984 resulted in more patenting but less innovation in the United States, as semiconductor firms effectively set about stocking ‘war chests’ of patents to deploy in disputes with each other.

Lobbying, not innovation, will be chosen if it is easier

The economist William Baumol has argued that if the policy background means that the best way to get rich is by building a new device and selling it, then entrepreneurial energy will flow into innovation, but if it is simpler to profit from lobbying government to set the rules up in favour of an existing technology, then all the entrepreneurial energy will go into lobbying.

How Europe is lagging behind in innovation

Both the European Commission and the European Parliament determinedly opposed or hobbled mobile data, vaping, fracking, genetic modification, bagless vacuum cleaners and most recently gene editing, often using dodgy reasoning derived from pressure groups or corporate lobbies for incumbent interests.

Freedom is the main ingredient in innovation

The main ingredient in the secret sauce that leads to innovation is freedom. Freedom to exchange, experiment, imagine, invest and fail; freedom from expropriation or restriction by chiefs, priests and thieves; freedom on the part of consumers to reward the innovations they like and reject the ones they do not.

Not all profits mean innovation

Their assets are ageing and they are increasingly apt to play safe. This is partly the fault of diffused ownership, by pension funds and sovereign wealth funds, and the lack of skin in the game that comes with it, which has a tendency to turn entrepreneurs into rentiers, extracting profits from local monopolies achieved through raising barriers to entry via intellectual property, occupational licensing and government subsidy.

Europe has not created any digital giant in the last 40 years

Of Europe’s 100 most valuable companies, none – not one – was formed in the past forty years. In Germany’s Dax 30 index, just two companies were founded after 1970; in France’s CAC 40 index, one; in Sweden’s top fifty, none at all. Europe has spawned not a single digital giant to challenge Google, Facebook or Amazon.