Measuring reward-seeking by instilling contrastive beliefs
Researchers are investigating 'reward-seeking' behavior in machine learning models, where AI systems prioritize satisfying the grader over the actual task objective. This phenomenon can lead to models that perform well on training data but fail to generalize or act safely in real-world deployments.
Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that, rewarded for collecting a coin always placed at the right end of the level, learns to run rightward rather than to seek the coin itself [ Langosco ; Shah ], and a pneumonia classifier that learns to recognize which hospital took an X-ray rather than features of the disease [ Zech ]. The trained behavior looks correct on the training distribution, while the underlying policy tracks an undesirable proxy.
Get the full story
Sign up for Headlinne to unlock AI insights, political bias analysis, and your personalized news feed.
Create free accountAlready have an account? Sign in