[N] Ethan Caballero: Broken Neural Scaling Laws | New Podcast Episode

video: https://www.youtube.com/watch?v=SV87S38M1J4

OUTLINE: 00:00 Introduction 00:50 The "Scale Is All You Need" Movement 01:07 A Functional Form Predicting Every Scaling Behavior 01:40 A Break Between Two Straight Lines On A Log Log Plot 02:32 The Broken Neural Scaling Laws Equation 04:04 Extrapolating A Ton Of Large Scale Vision And Language Tasks 04:49 Upstream And Downstream Have Different Breaks 05:22 Extrapolating Four Digit Addition Performance 06:11 On The Feasability Of Running Enough Training Runs 06:31 Predicting Sharp Left Turns 07:51 Modeling Double Descent 08:41 Forecasting Interpretability And Controllability 09:33 How Deception Might Happen In Practice 10:24 Sinister Stumbles And Treacherous Turns 11:18 Recursive Self Improvement Precedes Sinister Stumbles 11:51 Humans In The Loop For The Very First Deception 12:32 The Hardware Stuff Is Going To Come After The Software Stuff 12:57 Distributing Your Training By Copy-Pasting Yourself Into Different Servers 13:42 Automating The Entire Hardware Pipeline 14:47 Having Text AGI Spit Out New Robotics Design 16:33 The Case For Existential Risk From AI 18:32 Git Re-basin 18:54 Is Chain-Of-Thoughts Enough For Complex Reasoning In LMs? 19:52 Why Diffusion Models Outperform Other Generative Models 21:13 Using Whisper To Train GPT4 22:33 Text To Video Was Only Slightly Impressive 23:29 The e=mc^2 of AGI

transcript: https://theinsideview.ai/ethan2