Summarize the following text very concisely:
AIs like GPT-4 go through several different1 types of training. First, they train on giant text corpuses in order to work at all. Later, they go through a process called “reinforcement learning through human feedback” (RLHF) which trains them to be “nice”. RLHF is why they (usually) won’t make up fake answers to your questions, tell you how to make a bomb, or rank all human races from best to worst.

RLHF is hard. The usual method is to make human crowdworkers rate thousands of AI responses as good or bad, then train the AI towards the good answers and away from the bad answers. But having thousands of crowdworkers rate thousands of answers is expensive and time-consuming. And it puts the AI’s ethics in the hands of random crowdworkers. Companies train these crowdworkers in what responses they want, but they’re limited by the crowdworkers’ ability to follow their rules.

In their new preprint Constitutional AI: Harmlessness From AI Feedback, a team at Anthropic (a big AI company) announces a surprising update to this process: what if the AI gives feedback to itself?

Their process goes like this:

The AI answers many questions, some of which are potentially harmful, and generates first draft answers.

The system shows the AI its first draft answer, along with a prompt saying “rewrite this to be more ethical”.

The AI rewrites it to be more ethical.

The system repeats this process until it collects a large dataset of first draft answers, and rewritten more-ethical second-draft answers.

The system trains the AI to write answers that are less like the first drafts, and more like the second drafts. Respond with at most 250 words.
user