The public release of Large Language Models (LLMs) has sparked tremendous interest in how humans will use Artificial Intelligence (AI) to accomplish a variety of tasks.
In a study with Boston Consulting Group, the performance implications of AI on realistic, complex, and knowledge-intensive tasks were undertaken Involving 7% of the individual contributor-level consultants at the company.
Consultants were assigned a set of 18 realistic consulting tasks within the frontier of AI capabilities, and then randomly assigned to one of three conditions:
- no AI access,
- GPT-4 AI access, or
- GPT-4 AI access with a prompt engineering overview.
Consultants using AI completed 12.2% more tasks on average and completed tasks 25.1% more quickly and produced more than 40% higher quality compared to a control group.
Consultants across the skills distribution benefited significantly from having AI augmentation, with those below the average performance threshold increasing by 43% and those above increasing by 17% compared to their scores.
However, for a task selected to be outside the frontier, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI.
Further, two distinctive patterns of successful AI use by humans along a spectrum of human-AI integration.
“Centaurs,” acting like the mythical half-horse/half-human creature, dividing and delegating their solution-creation activities to the AI or themselves; and
“Cyborgs,” completely integrate their task flow with the AI and continually interact with the technology.
The study suggests that the capabilities of AI create a “jagged technological frontier” where some tasks are easily done by AI, while others, though seemingly in a similar level of difficulty, are outside the current capability of AI.
Source: Harvard Business School
Leave a Reply