Is Complexity an Illusion

Getting your Trinity Audio player ready…

With DeepSeek.

Okay, so I need to summarize this paper titled “Is Complexity an Illusion?” by Michael Timothy Bennett. Let me start by reading through the content provided. The paper seems to discuss the relationship between simplicity, complexity, and generalization in the context of artificial general intelligence (AGI), physics, and biology.

First, the abstract mentions that simplicity is often considered key to general intelligence because simpler models generalize better. However, the author argues that the correlation between simplicity and generalization depends on interpretation. Previous work showed that generalization comes from “weak” constraints rather than simple forms, and experiments showed that weak constraints improved generalization by 110-500%. The current paper aims to show that all constraints can have equally simple forms if forms are spatially extended, leading to a correlation between simplicity and generalization when weak constraints are made simple. The conclusion is that complexity doesn’t causally affect generalization but appears to due to confounding factors.

The introduction discusses complexity as a subjective measure and mentions Ockham’s Razor, Solomonoff Induction, and AIXI. It questions why simplicity correlates with objective performance if complexity is about form, not function. Key questions revolve around whether complexity is an artifact of abstraction and why sample efficiency and simplicity are correlated.

The formalism section defines environments, abstraction layers, tasks, policies, and proxies (weakness and simplicity). The learning and inference definitions explain how policies are evaluated based on their ability to generalize. The arguments and results section posits that complexity is subjective and an illusion caused by abstraction layers. Proposition 1 states that without abstraction, complexity can be minimized without affecting sample efficiency. Proposition 2 suggests that with finite vocabularies, weakness can confound simplicity and sample efficiency. The paper argues that abstraction layers are goal-directed (like natural selection favoring adaptability), leading weak constraints to take simple forms for efficiency.

The references include works on entropy, Kolmogorov complexity, AIXI, Solomonoff Induction, and related topics.

Now, to summarize this effectively, I need to capture the main thesis, the formalism presented, the key propositions, and the conclusion. The core idea is that complexity isn’t inherently linked to generalization; instead, it’s the “weakness” of constraints (their generality) that matters. The perceived correlation between simplicity and generalization arises because, in practical scenarios with finite vocabularies and abstraction layers, weak constraints are represented as simple forms. Thus, complexity is an illusion created by the abstraction process rather than a fundamental property affecting performance.

I should make sure to mention the shift from form (complexity) to function (weakness), the role of abstraction layers, and the implications for AGI and natural systems like biology. Also, highlight the experimental results showing that weak constraints outperform simple ones, and how the paper reconciles this with the observed correlation by introducing confounding through finite vocabularies and goal-directed abstraction.

Summary of “Is Complexity an Illusion?” by Michael Timothy Bennett

Thesis: Complexity is an illusion arising from abstraction layers, not a causal factor in generalization. The correlation between simplicity and generalization is confounded by the interplay of “weak” constraints (generality) and finite vocabularies in goal-directed systems.

Key Contributions:

1. Simplicity vs. Weakness:
   • Simplicity (form) and generalization (function) are distinct. Previous work showed that weak constraints (policies with larger extensions, allowing broader applicability) are necessary and sufficient for sample-efficient generalization, outperforming simple policies by 110–500% in experiments.
   • Complexity, often conflated with generalization, is subjective and tied to form rather than function.
2. Subjectivity of Complexity:
   • Without abstraction (i.e., a vocabulary of all possible declarative programs), all behaviors have equal complexity. Complexity becomes meaningful only when filtered through an abstraction layer (e.g., human-designed systems or biological cognition).
3. Confounding in Finite Vocabularies:
   • In spatially extended systems with finite vocabularies, weak constraints (which maximize generalization) often adopt simple forms due to practical efficiency. This creates an illusory correlation between simplicity and generalization.
   • Example: Natural selection favors adaptable organisms, which use weak policies. To minimize search space and computational cost, these policies are encoded simply within finite abstraction layers.
4. Goal-Directed Abstraction:
   • Abstraction layers (e.g., computer architectures, biological systems) evolve to balance versatility and tractability. Weak constraints are prioritized for adaptability, but their simplicity emerges from finite representation limits, not inherent causal efficacy.

Implications:

• AGI Research: Challenges the primacy of simplicity/complexity metrics (e.g., Solomonoff Induction) in favor of weakness-driven frameworks.
• Biology and Physics: Explains how natural selection and physical systems “illusionistically” link simplicity to functionality through constrained representations.
• Philosophy: Reinforces that complexity is observer-dependent, shaped by the interaction between objective environments and subjective abstraction layers.

Conclusion: Complexity does not intrinsically drive generalization but appears influential due to confounding factors in practical systems. The efficiency of goal-directed processes (e.g., evolution, machine learning) necessitates that weak, generalizable constraints adopt simple forms, creating the illusion of a causal link between simplicity and performance.