The research behind the Brain.
The cognitive architecture, affect dynamics, and memory retrieval this system implements are structurally inspired by a specific, citable body of work in cognitive science, affective computing, and information retrieval. Voice and vision draw on a shorter, more supporting literature further down this page.
These are mechanisms this architecture draws structural inspiration from — not benchmarks it claims to beat. Citing HippoRAG doesn't mean this system outperforms HippoRAG's own reported numbers, and citing Anderson's ACT-R doesn't mean this reimplements ACT-R in full. This project's own measured numbers live at /benchmarks, independently captured against this codebase and labeled with their provenance — not derived from, or compared against, any paper below. Every link on this page was checked to actually resolve to the paper it claims to be before publishing.
Affective Computing & Appraisal
Instead of coarse categorical emotion labels or a stateless prompt prefix, this system represents affect as a continuous point in a 3-dimensional Pleasure-Arousal-Dominance space, updated each turn by an appraisal step, and layers a tonic + phasic endocrine simulation on top that actually changes LLM sampling parameters.
1. Phasic burst decay: phasic(t) = peak · e^(−λt), where λ = ln(2) / half-life. Cortisol's half-life is 4500s and dopamine's is 90s — deliberately asymmetric, because a fright should linger far longer than a reward's glow.
2. Tonic + phasic split: the tonic terms are pure functions of current valence/arousal and so are perfectly anti-correlated by construction; only the phasic channels let the agent be stressed and rewarded at once.
3. Sampling modulation: cortisol narrows LLM temperature, dopamine widens top-p, fatigue shortens the token ceiling — see the endocrine docs for the exact formulas and the live simulator on the playground.
Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament
Current PsychologyThe 3-dimensional PAD affect space the agent's continuous mood state is represented in, instead of discrete emotion labels.
What are emotions? And how can they be measured?
Social Science InformationComponential appraisal framing behind mapping affect to observable vocal/behavioral parameters.
Affective Computing
MIT PressFoundational text for treating emotion as a computational, measurable signal rather than a UI flourish.
EMA: A process model of appraisal dynamics
Cognitive Systems ResearchProcess model for how appraisal of an event updates ongoing affect over time, underlying the appraisal -> PAD update step.
Affective computing with primary and secondary emotions in a virtual human
Autonomous Agents and Multi-Agent SystemsLayered primary/secondary emotion architecture informing the tonic (slow) vs. phasic (fast burst) split in the endocrine model.
IEMOCAP: Interactive emotional dyadic motion capture database
Language Resources and EvaluationReference corpus design for dyadic emotional interaction, relevant background for the affect-labeling pipeline.
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
IEEE International Conference on Automatic Face and Gesture Recognition (FG)Multimodal affective-corpus methodology background for continuous valence/arousal annotation.
ACT-R Memory & Hybrid Vector-Graph Retrieval
Memory retention doesn't decay linearly, nor stay indefinitely static. Recall scoring combines a single-term approximation of Anderson's ACT-R base-level activation with a Personalized-PageRank graph boost over a Neo4j semantic network and an affect-gated similarity term.
Where hours_since_lastis time elapsed since the memory's last recollection, d ≈ 0.5 is the decay rate, importance is a stored per-memory weight, and dist_emois emotional distance between the memory's affect and the agent's current affect. Try the exact formula live, including massed-vs-spaced recall, on the playground.
An integrated theory of the mind
Psychological ReviewThe ACT-R base-level activation formula -- ln(recall_count) minus a recency decay term, plus context terms -- is the direct source of the memory-scoring math this system ports almost verbatim.
HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
Advances in Neural Information Processing Systems (NeurIPS 2024)Neurobiologically-inspired long-term memory via graph indexing -- the direct structural precedent for the Neo4j + Personalized PageRank retrieval boost.
Cognitive Architectures for Language Agents
Transactions on Machine Learning Research (TMLR)Framework for grounding an LLM-driven agent in classical cognitive-architecture concepts (working/long-term memory, action selection).
From Local to Global: A Graph RAG Approach to Query-Focused Summarization
Microsoft Research / arXiv preprintGraph-structured retrieval-augmented generation, background for combining a semantic vector store with a relational graph.
Retrieval-Augmented Generation for knowledge-intensive NLP tasks
Advances in Neural Information Processing Systems (NeurIPS)The RAG pattern this system's memory retrieval is a specialized, affect-gated instance of.
Unsupervised dense information retrieval with contrastive learning
Transactions on Machine Learning Research (TMLR)Contrastive dense-retrieval training background for the embedding model used in vector memory search.
BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models
Advances in Neural Information Processing Systems (NeurIPS)Standard reference for evaluating retrieval quality across domains -- methodology context, not a benchmark this project reports against.
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
arXiv preprintMulti-granularity embedding design background relevant to the dynamic-resolution truncation approach used at high cognitive load.
Turn-Taking & Interaction Latency
Natural human conversation runs on turn-transition gaps around 200ms. Sequential ASR → LLM → TTS pipelines routinely blow past that; the speculative pre-generation and Voice-Activity-Projection-style prediction below are the literature this project's turn-taking reflex draws from.
Turn-taking in Conversational Systems and Human-Robot Interaction: A Review
Computer Speech & LanguageSurvey of turn-taking latency norms in human conversation and human-robot interaction; the ~200ms reference figure for natural turn-gaps.
Applying General Turn-taking Models to Conversational Human-Robot Interaction
ACM/IEEE International Conference on Human-Robot Interaction (HRI)Applying general (non-robot-specific) turn-taking models to embodied human-robot interaction, the setting this project targets.
Voice Activity Projection: Self-supervised Learning of Turn-taking Events
Proceedings of InterspeechThe Voice Activity Projection concept behind predicting an upcoming turn boundary before the user finishes speaking.
TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialogue
Proceedings of InterspeechTransformer-based turn-taking prediction, an early precedent for combining linguistic and acoustic signals for turn boundaries.
Multilingual Turn-taking Prediction Using Voice Activity Projection
Proceedings of LREC-COLINGExtends VAP-style turn prediction across languages -- relevant background for a locally-run, language-agnostic pipeline.
Smooth turn-taking by a robot using an online continuous model to generate turn-taking cues
Proceedings of ICMIContinuous (not just binary) turn-taking cue generation for a robot -- direct precedent for the idle-time proactive-scheduling behavior.
A Finite-State Turn-Taking Model for Spoken Dialog Systems
Proceedings of NAACL-HLTEarly finite-state turn-taking model -- the older endpoint-detection approach the VAP-style methods above were built to improve on.
Theory of Mind May Have Spontaneously Emerged in Large Language Models
arXiv preprintBaseline evidence (and its limits) for LLM Theory-of-Mind performance, the comparison point for this system's separate, deterministic ToM concept-tracker.
Edge Middleware & Local Multi-Agent Inference
Agents in this system are separate processes coordinated over NATS JetStream rather than function calls, with latency-critical voice and STT paths written in Rust behind a Python control plane, so it can run entirely on local hardware.
Exploring the performance of ROS2
Proceedings of the International Conference on Embedded Software (EMSOFT)Latency characterization of a DDS-based robotics middleware -- the comparison point for choosing a lighter publish-subscribe layer instead.
A Study on Modern Messaging Systems - Kafka, RabbitMQ and NATS Streaming
arXiv preprintComparative messaging-system survey informing the choice of NATS JetStream as the inter-agent signal bus.
Towards Understanding the Runtime Performance of Rust
Proceedings of ASEEmpirical basis for using Rust for the latency-critical voice and STT binaries, with a Python control plane atop.
Characterizing the Performance of Accelerated Jetson Edge Devices for Training Deep Learning Models
Proceedings of the ACM on Measurement and Analysis of Computing Systems (POMACS) / ACM SIGMETRICSEdge-hardware performance characterization relevant to running the full mesh on constrained local devices.
Profiling Apple Silicon Performance for ML Training
arXiv preprintApple Silicon ML performance profiling, directly relevant to running this system's inference locally on macOS hardware.
Robust speech recognition via large-scale weak supervision
Proceedings of ICMLThe Whisper speech-recognition model whose local variants (whisper.cpp) back this system's STT paths.
The Llama 3 Herd of Models
arXiv preprintOne of the open-weight local LLM families this system runs against via Ollama.
Lifespan Development & Neuromorphic Memory
A companion relationship that matters is one that deepens over a real timeline, not one that resets each session. Complementary Learning Systems theory — fast episodic memory consolidating into slower, structured long-term memory — is the direct model behind this system's sleep-cycle consolidation pass.
The hippocampal indexing theory
Behavioral NeuroscienceThe indexing-theory account of the hippocampus as a pointer structure over cortical memory traces -- the biological metaphor behind graph-indexed episodic memory.
Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory
Psychological ReviewComplementary Learning Systems theory -- fast episodic vs. slow consolidated memory -- the direct model for the sleep-cycle consolidation pass.
AriGraph: Learning and Planning in Graph-Based Episodic Memory
arXiv preprintGraph-based episodic memory for planning agents, a contemporary parallel to this system's Neo4j episodic graph.
Episodic Memories Generation and Evaluation Benchmark for Large Language Models
arXiv preprintBenchmark methodology for episodic-memory evaluation -- context for how this project's own Recall@K figures should and shouldn't be compared to published numbers.
The Life Cycle Completed (Extended Version)
W. W. Norton & CompanyPsychosocial development-stage theory, background for framing a companion relationship as something that deepens over a real timeline rather than resetting each session.
Where this sits next to commercial humanoid platforms
Humanoid platforms like Figure, Tesla Optimus, Unitree's G1, Engineered Arts' Ameca, and Kyoto's ERICA are named here only as industry context for the conversational-AI space this project sits in — their public specs and demos are vendor/lab claims, not peer-reviewed publications, and this project makes no benchmark comparison against them here.
This matters for scope, too: physical robotics body-hardware actuation is explicitly not implementedin this codebase — it's realized only via fail-closed external dispatcher stubs. This project is a cognitive/affect/memory architecture (the Brain) with a voice pipeline and appraisal-only vision input; it does not drive a physical robot body today.