Measuring Progress In the direction of AGI: A Cognitive Framework

To grasp AI capabilities throughout these cognitive skills, we suggest a three-stage analysis protocol that benchmarks system efficiency in relation to human capabilities:

1. Consider AI techniques throughout a broad suite of cognitive duties overlaying every capability, utilizing held-out take a look at units to stop information contamination
2. Acquire human baselines for a similar duties from a demographically consultant pattern of adults
3. Map every AI system’s efficiency relative to the distribution of human efficiency in every capability

Going from idea to follow

Defining these cognitive skills is a vital first step, however we’d like greater than a framework to measure progress. To place this idea into follow, we’re launching a brand new Kaggle hackathon — “Measuring progress towards AGI: Cognitive skills”. The hackathon encourages the neighborhood to design evaluations for 5 cognitive skills the place the analysis hole is the biggest: studying, metacognition, consideration, government features and social cognition.

Members can use Kaggle’s newly launched Neighborhood Benchmarks platform to construct and take a look at their evaluations in opposition to a lineup of frontier fashions.

We’re providing a complete prize pool of $200,000: $10,000 awards for the highest two submissions in every of the 5 tracks, and $25,000 grand prizes for the 4 best possible total submissions. Submissions are open March 17 via April 16, and we’ll announce the outcomes June 1. Head over to the Kaggle web site to begin constructing.