My exploration of Uri Alon’s book, An Introduction to Systems Biology, has led me down a fascinating path.
The book introduces the key concept of network motifs, an idea which Alon passionately advocates for. My first encounter with network motifs was in the Zoom In paper (https://distill.pub/2020/circuits/zoom-in/), which led me to Alon’s book. A network motif is defined as a recurring pattern within a network that appears significantly more often than would be expected by chance. According to Alon, these motifs or patterns in evolved networks don’t occur randomly, but rather serve a purpose in enhancing the fitness of these networks, echoing the survival of the fittest principle.
Previously, my understanding of complex networks was limited to macroscopic tools like power laws, akin to the focus of learning curves in the deep learning training processes. I viewed the system as an indivisible whole and focused on its statistical properties. Network motifs, however, offer a different perspective by providing insights at a microscopic level, making them appear like individual parts of an engineered machine.
Alon uses transcription networks as an example to illustrate the concept of network motifs. While it’s beneficial for me to learn some specifics of transcription networks, the methodology can be equally applicable to studying other complex networks. It would be interesting to see the exploration of motifs in social networks or neural networks, for example.
Another crucial point Alon’s book explores is the robustness often exhibited by biological systems. Living in a noisy environment, our survival is largely dependent on this robustness. We all know that, but that was probably all I knew before. This book offers an explanation of the origins of robustness or, seen from another angle, the principles of designing robust systems. For example, if the failure rate of a task is 1/n, repeating it twice reduces the failure rate to 1/n^2, a principle known as proofreading. Our cells perform this proofreading all the time. Knowledge of these simple design principles could be quite beneficial for engineers. The book also provides several other such principles for creating robust systems.
Towards the end, the book discusses a method to generate modularity in evolution. As a graduate student studying evolutionary algorithms, I often complained about the lack of understandable solutions and the absence of modularity generated by these algorithms. It was as if they were pieces of clutter that worked by accident. Alon’s book, however, demonstrates a way to create modularity in the solutions. By alternating the fitness function between two targets sharing common components, modularity emerges. This implies that an evolutionary algorithm can create modularity if there’s a set of targets that share basic building blocks (much like a diverse training dataset in deep learning). The reason we previously obtained unmodular solutions was due to optimization for a single target, resulting in overfitting (how can I miss that!). From this perspective, large language models with extensive training datasets should demonstrate modularity, with motifs hidden and waiting to be discovered.
In addition to target alteration, the book mentions another method to create modularity: assigning costs to long-distance edges (connections between distant nodes). This approach was demonstrated in the Seeing is Believing paper (https://arxiv.org/pdf/2305.08746.pdf), resulting in sparser networks that are easier to understand.
To sum up, Alon’s book offers valuable lessons from biology that I believe could be relevant in the realm of AI, particularly in the emerging field of mechanistic interpretability. I anticipate delving further into the biological sphere in my future studies.
Any feedback? We can discuss it under this Tweet.