Picture this. Monday morning standup. Your junior data scientist raises their hand with that look. You know the one.

"So… I was trying to set up MLflow for our new project, and I can't figure out how to give the interns read-only access to our experiments."

"Just use the authentication feature," you say confidently. "MLflow has auth now."

"Wait, which MLflow instance?" they ask. "The one for Project Alpha or the Research sandbox?"

You pause. "The one on the newer version. I think that's… Project Alpha?"

"Actually, I think that's still on 2.8. It's the customer success environment that we updated to 2.11 with auth enabled."

Four hours later, you're both staring at documentation, environment variables, and a half-configured setup that somehow made things worse. You're now maintaining three different MLflow environments, each on different versions, each with different capabilities. Experiments are scattered across instances. Someone accidentally logged Project Alpha's production runs to the Research sandbox last week. No one has a complete picture of what's happening where.

The interns can either see everything or nothing. There's no middle ground.

Welcome to the reality of open-source MLflow at scale where features that mature teams need feel like afterthoughts, and growing teams discover that what worked for 5 people breaks at 50. At this point, you might as well dedicate one of your data scientists to be the full-time MLflow fixer.

First, Let's Be Fair to MLflow

Before I dive into the pain points, let's acknowledge what MLflow gets right. It democratized ML experiment tracking. It's open-source, supports every major framework, and has an incredible community. For research projects and small teams with strong DevOps capabilities, who can manage orchestration, set up data caching, optimize resource usage, handle infrastructure scaling, and maintain multiple environments, it can work beautifully.

But this post is about what happens when you scale beyond that sweet spot of a single team/project. When you need mature team features. When "good enough" isn't good enough anymore.

The Authentication Reality: More Complex Than You Think

The real authentication challenge isn't about local instances. It's about what happens when your team grows. When you have 8 data scientists across 3 projects, suddenly everyone sharing one MLflow instance becomes chaos. So you need proper access controls.

Here's the catch: authentication only works on remote tracking servers. Now you're facing a choice:

• Set up and maintain your own tracking server infrastructure
• Or pay for a managed service (AWS SageMaker charges $300-500/month just for MLflow hosting)

Either way, you're now dealing with:

• Configuring environment variables for secret keys and ensuring they don't accidentally leak
• Restarting all services with authentication flags enabled (meaning downtime for your team)
• Managing a separate auth database
• Hoping your team properly handles credentials without saving them in notebooks or scripts

And after all that setup? You get four permission levels: READ, EDIT, MANAGE, and NO_PERMISSIONS. These work at the experiment and model level, which is something, but without project isolation or team boundaries? Here's what the nightmare looks like:

• Every user sees every experiment name (even if they can't access it)
• No way to separate "Team A's production models" from "Team B's research experiments"
• Managing permissions for 50 users across 200 experiments becomes a full-time job
• New team members need individual permissions set for every relevant experiment
• One misconfigured permission and sensitive data is exposed or critical experiments are locked

Multiple teams I've worked with discovered this the hard way. After investing significant time in MLflow auth setup, they realized they'd need custom wrappers just for basic requirements like "data scientists can see all experiments but only edit their own." Their workaround? An elaborate spreadsheet tracking who should access what, updated manually every sprint.

The Versioning Wild West: A Design Choice with Consequences

Let's talk about model versioning. In MLflow, registering models in the Model Registry is optional, a design choice for flexibility. And flexibility is great… until it isn't.

Last month, I watched a team try to track down which model was actually in production. Simple question, right?

Forty-five minutes later, they're still searching. Why? Because MLflow's Model Registry is optional, and when something's optional in a fast-moving team, it becomes "that thing we'll do later."

Their MLflow instance had:

• 500+ experiments scattered across folders
• Maybe 50 properly registered models
• No consistent naming conventions
• Multiple "final_model_v2_actually_final" variations

Even when models are registered, finding them isn't straightforward. As one ML engineer told me recently: "MLflow worked okay for tracking metrics, but the UI wasn't the easiest to navigate. That was one of the reasons we started looking for something more streamlined."

The Model Registry exists and works well when used consistently, but without enforcement, it relies entirely on team discipline. When production breaks at 2 AM, discipline isn't what saves you. Process is.

The Collaboration Desert

Perhaps the most frustrating limitation is how MLflow treats machine learning like a solo sport. No built-in commenting. No approval workflows. No team collaboration features. This isn't an oversight. It's a design decision to keep MLflow focused on tracking rather than workflow management.

But here's what that means in practice. A team lead needs to review models before production. Their "review process"?

1. Data scientist posts in Slack: "Please review experiment mlflow_123456"
2. Team lead opens MLflow, searches for the experiment
3. Reviews metrics in isolation with no context
4. Responds in Slack with approval
5. Someone manually tracks this in a spreadsheet

In 2025, we're using Slack and spreadsheets to manage model deployments. Let that sink in.

The Real Cost of "Free"

Yes, MLflow is open source. Yes, it's "free." But here's the actual invoice:

Setup complexity: That "basic" auth requires infrastructure changes, remote servers, and careful configuration. It's basic in name only.

Maintenance overhead: Every missing feature spawns workarounds. Cleanup scripts, access tracking spreadsheets, custom notification systems. And someone maintains all of this.

Scale friction: What works for 5 people breaks at 50. Manual processes and tribal knowledge don't scale.

Security gaps: Even with auth enabled, you're missing audit trails, project isolation, and granular permissions that mature teams need.

Logging chaos: No unified logging approach means "tracked" experiments that can't be compared. One person logs "accuracy," another logs "acc", rendering reproducibility impossible.

The "free" tool ends up costing senior engineer salaries to bandage its limitations.

The Scale Wall Is Real

MLflow is like that starter apartment you loved in college; perfect when you're young and scrappy, but eventually, you need more than a futon and milk crates.

If your team is:

• Writing auth wrappers and permission systems
• Managing model versions in spreadsheets
• Afraid to clean up experiments
• Building collaboration workflows outside MLflow
• Spending more time on MLflow maintenance than ML development

Then you've hit the scale wall.

When Building Makes Sense (And When It Doesn't)

Many teams successfully build around MLflow's limitations. If you have:

• Strong DevOps/platform capabilities
• Time to invest in tooling
• Relatively stable team size
• Great documentation culture

Then MLflow + custom tooling might work perfectly.

But if you're:

• Scaling rapidly
• Short on platform engineering resources
• Needing enterprise features yesterday
• Spending more time on MLflow maintenance than ML development

Then it's time to consider alternatives.

What Modern ML Infrastructure Looks Like

Let's paint a picture of what teams actually need:

• Project-based isolation: Teams work in their sandbox without seeing everyone's experiments while still being able to share resources they want to share with specific colleagues or teams.
• Granular permissions: Not just basic read/write/delete permissions, but workflow-specific rights (who can promote to production?).
• Automatic versioning: Every run, every model, every dataset tracked without human intervention. No forgotten registrations, no inconsistent naming.
• Audit everything: Who changed what, when, and why, with full history.
• Smart resource optimization: Intelligent caching and reuse system, saving both time and money.
• Native collaboration: Comments, reviews, and approvals where the work happens.
• Zero-setup auth: Security by default, not as an afterthought.

This isn't fantasy. Modern ML platforms build these features in from day one. They understand that ML is a team sport played at enterprise scale.

How Valohai Makes MLflow's Problems Disappear

Yes, it's a Valohai blog. But hear me out because the contrast is almost comical.

Security by default: Every Valohai project is isolated. Teams can't accidentally delete each other's work because they can't even see it. RBAC isn't an afterthought. It's built into the foundation. And from a user's perspective? It's dead simple. No handling keys, no environment variables, no auth database management. Just log in and work.

Automatic versioning: Every single run is versioned. Not "if you remember to register it." Not "if you follow the naming convention." Every. Single. Run. Your interns can't accidentally ship an untracked model because untracked models don't exist.

Smart storage management: Valohai automatically manages artifact lifecycles. Set retention policies once, and old experiments clean themselves up. That storage monthly bill? Gone. Your artifacts are stored in your own cloud buckets with intelligent lifecycle rules.

Built-in collaboration: Comment directly on experiments. Set up approval workflows without leaving the platform. No more Slack archaeology to figure out why a model was approved. It's all there, in context, where the work happens.

Zero-setup mature features: Project isolation, audit logs, and granular permissions work out of the box. Not after two weeks of configuration. Not with custom scripts. Just… working.

The best part? Valohai runs in your infrastructure. Your object storage, your security rules, and your compliance requirements are all respected. It's not about giving up control; it's about gaining capabilities without the maintenance nightmare.

The Valohai Difference: MLflow's Missing Features, Built In

Here's what happens when you switch from MLflow to Valohai based on real examples from teams who made the jump:

Week 1: "Wait, I don't need to set up authentication?" Every project is secure by default. No remote servers, no environment variables, no prayer circles. Just create a project and invite your team.

Week 2: "The interns can see their experiments but not production?" Project isolation means teams work in their own spaces. No accidental deletions. No security theater. Just logical boundaries that make sense.

Month 2: "We haven't had a single 'which model is in production?' meeting" Every run is versioned. Every deployment is tracked. Full lineage from data to production. The spreadsheet tracking system? Retired with honors.

Month 3: "Our approval process is… actually on the platform?" No more Slack archaeology. Reviews, comments, and approvals happen where the work lives. Your future self will thank you when debugging production issues.

Time for an Honest Conversation

I'm not here to bash MLflow as it democratized ML tracking, and that's huge. But there's a difference between "can work" and "works well at scale."

The teams thriving with Valohai aren't special. They just decided that fighting infrastructure wasn't the best use of their ML talent. They wanted:

• Security without the setup nightmare
• Versioning that just happens
• Storage costs that make sense
• Collaboration where work happens
• Infrastructure that scales with them, not against them

If you're fighting MLflow's limitations more than using its features, let's talk.

I'd love to hear what creative workarounds you've built. Sometimes the best therapy is knowing you're not alone in the infrastructure struggle. And when you're ready to see what ML infrastructure looks like when it's actually built for teams? Let me show you Valohai in action.

Start with a technical dry-run where we'll take one of your actual projects and run it on Valohai during a two-hour screen share. See the difference in real-time. If that looks promising, we can move to a two-week team trial. No massive migrations. No philosophical commitments. Just your workflows running on infrastructure that respects both your security needs and your sanity.

Because your data scientists shouldn't need a PhD in platform engineering and spreadsheet management just to ship models safely.