2.3.10. Greece

Laws of Thermodynamics

• Zeroth Law: If two systems are in thermal equilibrium with a third, they are in thermal equilibrium with each other.
• First Law: Energy is conserved; it can only change forms.
  • The total energy of the universe remains constant in any process.
  • For thermodynamic cycles, the net heat supplied equals the net work done.
  • Represented mathematically as: ΔU = Q - W
• Second Law: The entropy of an isolated system not at equilibrium tends to increase over time, reaching a maximum at equilibrium.
• Third Law: As temperature approaches absolute zero, the entropy of a system approaches a minimum or zero value.

Topology of Deep Learning

• Deep learning excels in image recognition and natural language processing but lacks theoretical understanding of its internal mechanisms.
• The "black box" nature of deep learning limits its further advancement.
• Topology offers a perspective to understand deep learning by analyzing network architecture, loss functions, and learned representations.
• Topology studies properties invariant under continuous deformations.
• Topological Data Analysis (TDA) infers topological features from data, providing a tool to understand data structure.

Basic Concepts of Topology

• Topological Space: A set X with a topology T, where T is a collection of subsets of X meeting specific axioms.
  • Axioms include: the empty set and X are in T; arbitrary unions of sets in T are in T; finite intersections of sets in T are in T.
  • Open sets are the sets in T
• Metric Space: A set X with a distance function d: X × X → ℝ satisfying specific axioms.
  • Axioms include: d(x, y) ≥ 0, and d(x, y) = 0 if and only if x = y; d(x, y) = d(y, x); d(x, z) ≤ d(x, y) + d(y, z).
  • An open ball with radius r centered at x is defined as B(x, r) = {y ∈ X | d(x, y) < r}.
• Simplicial Complex: A set of points, line segments, triangles, and their n-dimensional counterparts (simplices).
  • A k-simplex σ is the convex hull of k + 1 affinely independent points (v₀, v₁, ..., vₖ), denoted as σ = [v₀, v₁, ..., vₖ].
  • A face of a simplex is the convex hull of a subset of its vertices.
  • A simplicial complex K includes every face of its simplices, and the intersection of any two simplices is either empty or a face of both.
• Vietoris-Rips Complex Vrϵ(X): This complex helps construct simplicial complexes from points:
  • Parameterized by ε.
  • Each data point is a 0-simplex.
  • A k-simplex [v₀, v₁, ..., vₖ] is in Vrϵ(X) iff d(vi, vj) ≤ ε for all 0 ≤ i, j ≤ k.
• Alpha Complex: A subcomplex of the Delaunay triangulation.
  • Contains simplices whose circumradius is at most α (a parameter).
• Homology: Detects topological features, defining k-chains as formal sums of k-simplices, forming a vector space Ck.
  • The boundary operator ∂k: Cₖ → Cₖ₋₁ maps a k-simplex to its boundary.
  • The k-th homology group Hk is Zk / Bk, where Zₖ are k-cycles and Bₖ are k-boundaries.
  • The k-th Betti number βₖ is the rank of Hk, representing the number of k-dimensional holes.
    • β₀: Number of connected components.
    • β₁: Number of loops.
    • β₂: Number of voids.
• Persistent Homology: Tracks topological features over a range of scales in a filtration of simplicial complexes.
  • Features are born when they appear and die when they disappear.
  • Persistence measures the difference between death and birth times, indicating significance.
  • Visualized via persistence diagrams, plotting birth time vs. death time for each feature.

Analyzing Network Architecture and Loss Function

• Network architecture in deep learning can be seen as a topological space, impacting model behavior.
• Complexity can be measured via neurons, layers, connections, and topology.
• Analyzing the network architecture with persistent homology relates to performance
• Topological loss functions can encourage the learning of representations with certain topological properties.
• Deep neural networks with simple loss landscapes generalize better.
• Topological optimization methods to avoid local minima or saddle points.

Understanding Representations Learned by Deep Learning

• Representations learned by the deep learning models capture the structure of data.
• They can be analyzed as points in high-dimensional space.
• The persistent homology of representations relates to performance

Designing new deep learning models

• The topological properties of the data can be incorporated into the deep learning models.
• Topological autoencoders learn representations that preserve the topological structure of the data
• Topological graph neural networks designed for learning graph representations.