Why are over optimized systems so brittle?

I’ve been working in the Pacific for the last 20 years designing and building information systems. One of the key issues that keeps cropping up is the problem of focusing too much on designing and building systems that are highly efficient. There seems to be a general lack of understanding that focusing too much on efficiency creates problems.

The basic rule of thumb is that generally speaking the more optimised a system becomes, the more brittle it becomes. This is because as a system that is over optimized doesn’t have any redundancy options available when the environment changes.

You see this pattern playout consistently across:

Ecological systems – when apex predators who are over optimized for their environment go extinct due to changes in their environment

Stock market systems – when automatic trading algorithms go haywire due to some unexpected event and have run away feedback loops that results in flash crash scenarios

Software Systems – that are over optimized for a specific use case that become redundant when user requirements change

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.