Evolution of strategies in iterated games: a deep multi-agent reinforcement learning approach with hybrid genetic and CMA-ES optimization

The emergence of cooperation among self-interested agents in social dilemmas remains a central unsolved problem in evolutionary game theory, multi-agent systems, and behavioral science. While memory-one reciprocal strategies such as Tit-for-Tat have been extensively studied, the evolutionary properties of memory-two bilateral reciprocity strategies—which condition on a two-round bilateral history over a 216 = 65, 536-element strategy space—remain comparatively undercharacterised. This study presents a unified agent-based computational framework integrating a Q-learning multi-agent reinforcement learning (MARL) engine with Moran, Wright-Fisher, and replicator dynamics selection mechanisms to systematically investigate the evolutionary fate of memory-two (M2) bilateral reciprocity strategies across eight controlled simulation experiments. Under canonical Prisoner's Dilemma conditions (T = 5, R = 3, P = 1, S = 0, N = 100, μ = 0.05, β = 2.0, ε = 0.02), the simulation converged to a high-cooperation quasi-equilibrium with mean cooperation rate ρC = 0.814 ± 0.029. The M2 strategy class collectively captured 61.0% of the evolutionary equilibrium population, with all three M2 strategies achieving fixation probability Pfix = 1.000 when invading an AllD-resident population—a 10-fold enrichment over the neutral expectation. Cooperation was robust to behavioral noise up to ε≈0.15, beyond which a noise-induced phase transition collapsed cooperative equilibria. Selection pressure, mutation rate, game class, and population size were systematically varied, confirming that M2 dominance is robust across all tested conditions. These results establish memory-two bilateral reciprocity as the dominant evolutionary strategy class in noisy iterated social dilemmas and provide a rigorous computational characterization of the conditions under which it emerges and persists.