Where’s AI up to, where’s AI headed?

Air Street Capital recently released its sixth annual report on the state of AI. In a nutshell, the report highlights the impressive progress of AI in consumer-facing applications. OpenAI’s ChatGPT-4 remains a leading large language model (LLM) across various tests, surpassing its predecessor, GPT-3.5, by a significant margin. GPT-4 exhibits improved accuracy and safety, making it less likely to generate inappropriate content.

The report discusses the pursuit of creating a single multimodal language model capable of handling text, images, sound, and more. While GPT-4 is a step in this direction, it faces challenges in visual tasks, with other AI models closely matching its capabilities.

Two key factors contributing to GPT-4’s success are

• its massive training data and

• the use of Reinforcement Learning from Human Feedback (RLHF).

However, these factors present challenges, including the need for significant human involvement and potential limitations in data availability.

The report addresses concerns about the availability of quality training data and the potential solution of using synthetic data or training on the same dataset multiple times. It also highlights the challenge of “context length” in AI, as longer contexts may lead to performance issues.

The competition in AI is influenced by the high development costs, but there are emerging techniques like fine-tuning smaller models on larger models’ outputs, “less is more” approaches, and self-improvement through training on own outputs. Openness and opacity in AI development are also discussed, with both GPT-4 and open-source models making strides in the field.

The report mentions the development of specialized AI applications by the open-source community and the potential of small language models (SLMs) trained on curated datasets to rival larger models. Crystal-ball gazing in the report includes predictions for AI in 2024, such as AI-generated songs hitting the music charts and self-improving AI agents achieving significant milestones.

Source: Lexology