Science of Software - Computing Education Research

SOS-CER Auto-Grading

Automated Grading with the Canary Framework

Canary Framework Overview

The Canary Framework is a set of open source software engineering tools and supplemental resources that support automated grading. Details about how the Canary Framework is used at NC State Computer Science in our SIGCSE 2018 paper.

Students push their code submissions to [GitHub]( The continuous integration system, Jenkins, pulls their code and evaluates it as appropriate for the class. We run the Jenkins master and executors on the Virtual Computing Lab (VCL).

Setup Instructions


We have created (or are creating) tooling to support the Canary Framework automated grading system and other teaching related tasks. Information about each tool are set of tools are below:

  • GitHubManager: tool that creates GitHub repositories, adds students to teams, and updates repository permissions from push (write) to pull (read) when a project is closed.
  • JenkinsJobResources: tools to create and close Jenkins jobs.
  • BuildResources: scripts and build configuration files to evaluate student submissions.
  • JenkinsWebScraper: Selenium program that scrapes results from the most recent build on Jenkins for use when grading.
  • BuildDataCollector: Forthcoming tooling that supports the collection of student build information into a database for data analysis and grading.
  • FeedbackFileGenerator: generates a PDF feedback file with information about a student’s grade on an assignment from a grade spreadsheet.