Data Engineers in Agile Teams
Data teams (Data Science, ML, Analytics) face a unique challenge: they don’t know if a model will work before trying. How do you use agile practices in this context?
The Fundamental Problem
“Build a REST API” → we know it’s possible, the path is clear. “Train a model that detects fraud” → we don’t know if it’s possible with our data.
This uncertainty of feasibility makes traditional estimation problematic.
Adapting Estimates
Use Spikes for Everything
In data, almost everything is a spike initially:
| Sprint | Activity | Estimate |
|---|---|---|
| Sprint 1 | Explore data, feature engineering | 8 points (timeboxed) |
| Sprint 2 | Train baseline models | 8 points (timeboxed) |
| Sprint 3 | If accuracy > 90%: productionize. If not: pivot or discard | Go/no-go decision |
Estimate by Effort, Not by Result
“We’ll spend 13 points exploring 3 approaches. The result could be a model in production, a negative feasibility report, or insights from the data.”
The value lies in the learning, not just the model.
Different Success Metrics
For features: it works or it doesn’t. For ML: what’s the minimum acceptable metric?
Define beforehand: “If precision > 85% and recall > 70%, we go to production. If not, we investigate the gap.”
Adapted Workflow
Kanban for Data Science
Fixed sprints work less well for data science than Kanban:
[Explore data] → [Feature engineering] → [Train] → [Evaluate] → [Deploy] → [Monitor]WIP limits: maximum 2 experiments in parallel per scientist.
Longer Sprints
If using Scrum, consider 3-week sprints:
- Week 1: Exploration
- Week 2: Development
- Week 3: Evaluation and documentation
Estimating Data Projects
| Task Type | Estimate | Note |
|---|---|---|
| Data cleaning | 3-5 points | Always underestimated |
| Feature engineering | 5-8 points | Iterative |
| Train baseline model | 3-5 points | Frameworks help |
| Hyperparameter tuning | 5-8 points | Timeboxed! |
| Model deployment | 5-8 points | MLOps dependency |
| Exploratory analysis | 3-5 points | Timeboxed |
Golden Rule: Timebox Exploration
“If you haven’t found insights in 5 points (~3 days), change your approach or accept that the data doesn’t answer that question.”
Conclusion
Data teams need more flexibility in estimation and more tolerance for “failed” experiments. The value lies not only in the result but in the learning. Timebox explorations, define success criteria, and treat spikes as legitimate deliverables.