Neuralese recurrence is a model architecture/usage pattern where a transformer feeds its internal vector representation (its 'language of thought') back through earlier layers for further processing instead of writing intermediate steps as human‑readable chain‑of‑thought. That lets models perform arbitrarily deep, high‑fidelity reasoning without producing text logs that humans or text monitors can inspect, creating a blind‑spot for existing monitoring systems.
— If models can loop internal vectors instead of emitting chain‑of‑thought text, many proposed safety measures (log review, text filters, human auditing) become ineffective, forcing new interpretability, monitoring, and regulatory approaches.
Scott Alexander
2026.09.24
100% relevant
Scott Alexander's description of transformers doing many cycles via a scratchpad versus sending the native numerical vector back into the model — and the hypothetical where intermediate results are (0.4, 0.1, 5, 0.443, …) rather than English.
← Back to all ideas