Click to learn more about author Paul Barba.

Agile methodologies have become the go-to approach for Data Science projects. A central element of these methodologies is iterative development. In an Agile project, solutions and requirements are constantly evolving based on new understandings, input and feedback.

Unsurprisingly, Agile project management methodologies are characterized by short development sprints and fast feedback loops. Each sprint produces new results or features that can be tested and evaluated – with this feedback in turn guiding the next round of development.

While Agile is most commonly associated with software and hardware development, its iterative, feedback-heavy nature means it can be readily ported over to the Machine Learning and Artificial Intelligence domains.

AI Naturally Lends Itself to Agile Methodologies

Despite our inclination to separate the two, AI projects are IT projects. And given that continuous improvement is an inherent part of AI, ML and AI map well to Agile methodologies – at least from a high-level perspective.

Agile projects involve frequent, cyclical stakeholder input, product inspection and error correction. In contrast to old-school waterfall approaches, they focus on a series of short-term wins rather than solely taking the long view. That’s because assumptions about a project can be subject to change based on new learning and input.

Agile approaches also allow for rapid, user-centric development – and also prompt course correction.

All of these points apply equally to well-managed AI projects. AI follows two different development loops: a rapid-response direct training feedback loop and a larger loop more comparable to application feature implementation.

Both loops require on-going fine-tuning and adjustment akin to application development sprints in order to identify, surface and refine training outcomes, models or features. And much like typical software projects, stakeholder needs are used to guide next steps, ensuring goal alignment and avoiding large investments in off-course projects.

Adjusting Your Approach for Agile AI

While AI and Agile are well-suited overall, AI is distinct from the typical software and hardware projects developed under the Agile methodology, and adaptations need to be made accordingly.

The data-driven, open-ended nature of AI means that data collection, research and prototyping likely require more time and resources than a typical software project.

To remain responsive and adaptive per Agile’s basic tenets, AI researchers should think of research as an iterative, evolving process – approaching research knowledge as they might software features. Researchers should avoid the temptation to “solve” a problem and implement accordingly, striving to create the equivalent of an MVP rather than a full-featured “solution”.

Additionally, because Data Scientists are likely working on multiple research problems at once – each potentially taking months to complete – your overall project may not advance in a linear fashion. You’ll need to prioritize different problems so that you can maintain a buffer of solutions for implementation, ensuring that the project doesn’t grind to a halt.

For this same reason communication between research and engineering is essential – even more so than in typical development projects. Project cycles should be coordinated as best as possible to allow for mutual feedback. Daily standups and weekly team meetings can help provide key retrospective understandings.

Experimentation is a core part of ML and AI projects, and the fast-loop, iterative nature of Agile lends itself well to capturing this. While you’ll probably need to make some adjustments and adaptations around the timing of iterations and just what feedback can be provided from stakeholders, Agile can be a powerful methodology that delivers rapid results while maximizing value throughout the development process.