To grasp AI capabilities throughout these cognitive skills, we suggest a three-stage analysis protocol that benchmarks system efficiency in relation to human capabilities:
- Consider AI techniques throughout a broad suite of cognitive duties overlaying every capability, utilizing held-out take a look at units to stop information contamination
- Acquire human baselines for a similar duties from a demographically consultant pattern of adults
- Map every AI system’s efficiency relative to the distribution of human efficiency in every capability
Going from idea to follow
Defining these cognitive skills is a vital first step, however we’d like greater than a framework to measure progress. To place this idea into follow, we’re launching a brand new Kaggle hackathon — “Measuring progress towards AGI: Cognitive skills”. The hackathon encourages the neighborhood to design evaluations for 5 cognitive skills the place the analysis hole is the biggest: studying, metacognition, consideration, government features and social cognition.
Members can use Kaggle’s newly launched Neighborhood Benchmarks platform to construct and take a look at their evaluations in opposition to a lineup of frontier fashions.
We’re providing a complete prize pool of $200,000: $10,000 awards for the highest two submissions in every of the 5 tracks, and $25,000 grand prizes for the 4 best possible total submissions. Submissions are open March 17 via April 16, and we’ll announce the outcomes June 1. Head over to the Kaggle web site to begin constructing.
