Valohai Blog

DLOps, deep learning operations, is an evolution of MLOps, looking to answer the unique operational challenges that deep learning sets. A skeptic may look at it as unnecessarily muddying the waters with a new buzzword.

Support for Spark has been one of the most requested features as Spark has become almost ubiquitous for data scientists and engineers working with structured data. We’ve heard the calls and Valohai now supports Spark natively.

One of the unique aspects of Valohai is that despite being a proprietary platform it can run in fully private, even airgapped, environments. Why is this important? Machine learning often revolves around data that is sensitive and thus data security is a fundamental requirement.

The AIIA blueprint is an excellent starting resource for teams looking to implement their stack for machine learning development. The initiative draws inspiration from other popularized tech stacks.

As a data scientist or a machine learning engineer, you have probably heard about Kubeflow and MLflow. They are often compared against each other despite being quite different.

Datum is a version-controlled file inside the Valohai platform. Every datum is immutable by design. We have introduced three new improvements for more flexibility over datums.

Putting together a suitable dataset for training a model can be one of the biggest challenges. Data augmentation is an approach where you start with an existing dataset and expand it to have more variety.

Let’s start by defining interpretability in the context of machine learning and AI. In simple terms, it means how easily a human can interpret how the model arrived at a decision.

Summer is here and hopefully, for most of us, it means time to decompress. But if you are like me and learning is relaxing, podcasts are a great way to enjoy the summer weather while learning.

This article shows an example of a pipeline that integrates Valohai and Superb AI to train a computer vision model using pre-trained weights and transfer learning. For the model, we are using YOLOv3, which is built for real-time object detection.

It's time for an update on what's been happening under the hood of the Valohai platform. We'd like to highlight three major features we've added in the past two months: Support for Kubernetes and Spot instances and the Valohai Python utility library.

Why a Machine Learning model is not a product if there is no MLOps. Our approach to implementing training as a reproducible process, and how this process intertwines with our CI/CD pipeline.

It's becoming more important to think about the competencies of a team rather than expecting every individual to be an expert at everything related to machine learning.

Computer vision is one of the most disruptive technologies of the recent decade. To develop computer vision systems requires massive, upfront investments. Or it used to, before Superb met Valohai.

How can MLOps make consultant-client relationships more productive? Starting with machine learning is a massive, strategic undertaking, and many are turning to consultancies and contractors to take the first steps with AI.

March 16th, we held a webinar to follow up on our MLOps eBook. Together with our co-authors, we wanted to tackle the goal we set for MLOps in the eBook: “The goal of MLOps is to reduce technical friction to get the model from an idea into production in the shortest possible time to market with as little risk as possible.”

In the past few months, we've rolled out three new features that highlight end-to-end automation on our platform: Deployment nodes in pipelines, Pipeline scheduler & Model monitoring.

If you are involved with production machine learning in any way, understanding MLOps is essential. For people with software development experience, the easiest way to understand MLOps is to draw a parallel between it and DevOps.

Valohai MLOps platform provided the infrastructure for the Black-Box Optimization Challenge for the NeurIPS 2020 conference. The competition was organized together with Twitter, Facebook, SigOpt, ChaLearn, and 4paradigm.

The bus factor is a common term in software engineering describing the risk of a key contributor disappearing unexpectedly from a project – because they get hit by a bus. In machine learning the bus factor is magnified significantly.

Should a machine learning model be retrained each time new observations are available (or otherwise very frequently)? The answer is “it depends”, but this article looks at two components to consider: the use case and the costs.

Buying an MLOps platform is tricky and for that reason we’ve introduced a model where teams can sign up for a two-week proof-of-concept project to test out our platform with their environment and projects.

As you start incorporating machine learning models into your end-user applications, the question comes up: “When is the model good enough to deploy?” There simply is no single right answer.

Modern tooling and shared work methods (CI/CD, version control, microservices) have enabled companies to scale their throughput in software development exponentially. A machine learning pipeline brings similar scale to machine learning.

To make it easier to consider what tools your organization could use to adopt MLOps, we’ve made a simple template that breaks down a machine learning workflow into components.

Machine learning and artificial intelligence allow businesses to gain new insights and improve their business processes. However, they expose companies to additional risks because humans do not explicitly program the algorithms. Let's look at some of these risks and how data scientists and compliance officers can help mitigate them.

Using the MLOps platform allows you to manage everything about machine learning in production, where each new update doesn’t feel like an entirely new project and easily dovetails to the last.

To make an analogy to a more traditional industry, machine learning is shipping goods while MLOps is containerization. And much like containerization of global shipping, MLOps is equal parts process and infrastructure.

Imagine this; you are working on a notebook that takes ages to run, and it bogs down your computer, so it's even hard to multitask. We've seen this countless times, which is why we are introducing Minihai.

For a long time, most machine learning initiatives have been stuck in a persistent state of proofs-of-concept. However, in the past year, we’ve seen a rapid acceleration of machine learning models getting real-world use. Consequently, machine learning engineers are increasingly sought after – nearly catching up to data scientists in posted jobs.

Valohai, the MLOps platform company, is collaborating with Twitter and Facebook to launch a competition for the annual The Neural Information Processing Systems (NeurIPS) conference to advance the optimization of machine learning models towards more accurate AI solutions. The goal is to find better optimization algorithms for machine learning.

Levity decided to adopt Valohai to manage their machine learning infrastructure & model serving through Kubernetes instead of hiring an MLOps engineer.

Most software development teams have adopted continuous integration and delivery (CI/CD) to iterate faster. However, a machine learning model depends not only on the code but also the data and hyperparameters. Releasing a new machine learning model in production is more complex than traditional software development.

Grid search and random search are the most well-known in hyperparameter tuning. They are also both first-class citizens inside the Valohai platform. You define your search space, hit go, and Valohai will start all your machines. It does a search over the designated area of parameters you’ve defined. It is all automatic and doesn’t make you launch or shut down machines by hand. Also, you don't accidentally leave machines running costing you money. But we’ve been missing one central way for hyperparameter tuning, Bayesian optimization. Not anymore!

Finding the right subreddit to submit your post can be tricky, especially for people new to Reddit. There are thousands of active subreddits with overlapping content. If it is no easy task for a human, I didn’t expect it to be easier for a machine. Currently, redditors can ask for suitable subreddits in a special subreddit: r/findareddit.

A lot of companies and teams are going fully remote for the first time due to the Coronavirus. We at Valohai are big believers in remote work. Having practiced with a distributed team for a good 4 years we would like to share some of our thoughts on remote work in Machine Learning. A lot of major pain points we have seen revolve around tooling.

In this blog post we will explore how you can use DVC for your data version control and how you can automate your data version control with and without DVC inside the Valohai platform.

It’s a running joke among developers that the cloud is just a word for somebody else’s computer. But the fact remains, that by leveraging the cloud you can reap benefits that you couldn’t achieve with your on-premises server farm.

Machine learning (ML) platforms take many forms and usually solve only one or a few parts of the ML problem space. So how do you make sense of the different platforms that all call themselves ML platforms?

When doing machine learning in production, the choice of the model is just one of the many important criteria. Equally important are the definition of the problem, gathering high-quality data and the architecture of the machine learning pipeline.

Selko.io builds solutions for multi-disciplinary project teams working in large companies. These teams work according to project documents that usually have several hundreds of pages. Finding the relevant sections for each team member is a real burden in the project-based working environment.

This article is the story of us at Selko.io, productionizing our machine learning workflows. We'll describe Selko's route from starting the company to developing our first ML models. We'll also walk through how we built a fully working machine learning solution combining our UI, backend, and orchestration layer for machine learning tasks. And of course, how we went from a homegrown ML orchestration platform to Valohai. To give you some context, let's first dive into the history of the company.

One of the key challenges for a Data Science team is the search for an accurately labelled dataset for solving the given problem. While it is easy to build a basic model that is reasonably accurate for a demo to the business, going beyond it towards a production worthy solution needs gold standard ground truth data.

Apache Airflow is a popular platform to create, schedule and monitor workflows in Python. It has more than 15k stars on Github and it’s used by data engineers at companies like Twitter, Airbnb and Spotify.

After looking at a lot of Java/JVM based NLP libraries listed on Awesome AI/ML/DL I decided to pick the Apache OpenNLP library. One of the reasons comes from the fact another developer (who had a look at it previously) recommended it. Besides, it’s an Apache project, they have been great supporters of F/OSS Java projects for the last two decades or so. It also goes without saying that Apache OpenNLP is backed by the Apache 2.0 license.

Only the companies that invest into machine learning today will exist 10 years from now. The ones that look to the sidelines will be eaten by their competition.

Continuous Integration (CI) in software development is the process of testing that a change in one place doesn’t break something else. Continuous Delivery (CD), on the other hand, is an extension to CI where every change in the code is also deployed. Both are and have been core parts in the advancements of Extreme Programming, i.e. rapid small-batch development. This, on its hand, has been the main contributor to advancements in rapid software development.

Valohai is the enterprise-grade machine learning platform for data scientists that build custom models by hand. In addition to writing code with classic IDEs like PyCharm or VSCode, we also have native support for data scientists preferring to use Jupyter notebooks.

We are all aware of Machine Learning tools and cloud services that work via the browser and give us an interface we can use to perform our day-to-day data analysis, model training, and evaluation, and other tasks to various degrees of efficiencies.

They say data is the new gold. But without a data catalog, your data is just scattered around like random nuggets of gold in a desert full of rocks, pebbles and sand. Data catalogs help you keep track of the data you have but also, in the case of machine learning models, what data has affected which model. Data brings meaning to machine learning because unlike software, machine learning models are 90% data and 10% code.

One of the more exciting things we have under development (or, should we say, in the pipeline) right now is our Pipeline system. Since our mission is to enable CI/CD style development for AI and machine learning, there's a logical next step up from just (well, "just" might be the understatement of the year here) running your code in a repeatable manner with Valohai.

One of the hottest areas of application for deep learning is undoubtedly self-driving cars. We’ll go through the problem space, discuss its intricacies and build a self-driving solution utilizing the Unity game engine, training a neural network on top of the Valohai platform. Regardless of the technologies used, you’ll get an understanding of the basics as well as the code to tweak for yourself.

We all understand the importance of reproducibility of machine learning experiments. And we all understand that the basis for reproducibility is tracking every experiment, either manually in a spreadsheet or automatically through a platform such as Valohai. What you can’t track what you’ve done it’s impossible to remember what you did last week, not to mention last year. This complexity is further multiplied with every new team member that joins your company.

Some time ago I came across this life-cycle management tool (or cloud service) called Valohai and I was quite impressed by its user-interface and simplicity of design and layout. Previous to that I had written a simple pipeline using GNU Parallel, JavaScript, Python and Bash - and another one purely using GNU Parallel, and Bash.

Software patents raised a lot of hairs twenty years ago, mainly because while governments are slow to react to change, software evolves rapidly, and patents thus live on for too long in comparison to hardware. Let’s in this blog post take a look at how AI patents are similar and different from software patents and what challenges can be seen in AI patenting.

Production-grade machine-learning algorithms never come out perfect on the first try. They require the same approach to iteration and testing as any other software project. But validating machine-learning algorithms is particularly hard—harder than writing simple unit or integration tests. And iterating on machine-learning algorithms gets harder as the team contributing to it grows.

All over the world, patents are known as the best way to protect inventions. They provide inventors with a period of up to 20 years to use an exclusive, monopoly-like position in the commercial exploitation of their creations. It is the key for getting returns on the investments they made during the research and development of their new technological solutions.

I see the quote “AI is the new electricity” thrown around in about every other blog post. I think there is truth in it, but I also think most people don’t go to the bottom of what it really means for their business. Let’s first define what we mean by AI: in this context, I’m referring to new advances in machine learning and deep learning.

Nyris is developing a high-performance visual search engine that understands the content of an image. The visual search engine works as an easy-to-use API that companies can use to inject visual search as a part of their solution.

Valohai is a deep learning platform that helps you execute on-demand experiments in the cloud with full version control. Jupyter Notebook is a popular IDE for the data scientist. It is especially suited for early data exploration and prototyping.

Pointlessly staring at live logs and waiting for a miracle to happen is a huge time sink for data scientists everywhere. Instead, one should strive for an asynchronous workflow. In this article, we define asynchronous workflows, figure out some of the obstacles and finally guide you to a next article to look at a real-life example in action in Jupyter Notebooks.

Valohai executions can be triggered directly from the CLI and let you roll up your sleeves and fine-tune your options a bit more hands-on than our web-based UI. In part one, I showed you how to install and get started with Valohai’s command-line interface (CLI). Now, it’s time to take a deeper dive and power up with features that’ll take your daily productivity to new heights.

Ville Tuulos, machine learning infrastructure architect, was the first to publicly dissect Netflix’s Machine Learning infrastructure at QCon in November 2018 in San Francisco. If you haven’t seen the talk yet, read the summary of his talk here! All the pictures used here, are from Ville's presentation.

In our series of machine learning infrastructure blog posts, we recently featured Uber’s Michelangelo. Today we’re happy to be interviewing Ville Tuulos from Netflix. Ville is a machine learning infrastructure architect at Netflix’s Los Gatos, CA office. 

As new Valohai users get acquainted with the platform, many fall in love our web-based UI - and for good reason. Its responsive, intuitive and gets the job done with just a few clicks. But don’t be fooled into thinking that’s the end of the interface conversation. We know it takes different [key]strokes for different folks, so Valohai also includes a command-line interface (CLI) and the REST API.

Last week we had the pleasure of joining our partner SwiftStack at our joint booth at the NVIDIA GTC 2019 conference in San Jose. GTC touts itself as the premier AI conference and it sure was.

One of the core design paradigms of Valohai is technology agnosticism. Building on top of the file system and in our case Docker means that we support running very different kinds of applications, scripts, languages and frameworks on top of Valohai. This means most systems are Valohai-ready because of these common abstractions. The same is true for TensorBoard as well.

In this blog post we’ll look at which parts a machine learning platform consists of and compare building your own infrastructure from scratch to buying a ready-made service that does everything for you.

Running a local notebook is great for early data exploration and model tinkering, there’s no doubt about it. But eventually you’ll outgrow it and want to scale up and train the model in the cloud with easy parallel executions, full version control and robust deployment. (Letting you reproduce your experiments and share them with team members at any time.)

SwiftStack and Valohai, in joint partnership, announce the world’s first peta-scale ML solution that covers everything from computation to data management in a multi-cloud environment. The solution provides a global namespace removing silos and enabling universal access to all your data in all your machine learning use-cases. It has built-in support for Azure, Google Cloud, AWS and SwiftStack.

We may live in the era of “Big Data,” and yet the access to it is somewhat restricted; especially, when we talk about high-quality data. This blogpost will address the question of acquiring data for your Machine Learning projects from the perspective of EU and US copyright laws.

In this third part, we will move our Q-learning approach from a Q-table to a deep neural net.

When we founded Valohai two years ago, we were lucky to make friends with team leads for Uber’s Michelangelo machine learning platform. Michelangelo has been an inspiration in building Valohai for the other 99.999…% of companies that aren’t Uber but still need to speed up their machine learning through automation.

By now you’ve surely heard about Kubeflow, the machine learning platform based out of Google. Kubeflow basically connects TensorFlow’s ML model building with Kubernetes’ scalable infrastructure (thus the name Kube and Flow) so that you can concentrate on building your predictive model logic, without having to worry about the underlying infrastructure. At least in theory.

In this second part takes these examples, turns them into Python code and trains them in the cloud, using the Valohai deep learning management platform.

There’s only one way to grow your deep learning team effectively: by adding new people to it! (We were just as shocked as you are by this revelation!)

This is the first part of a tutorial series about reinforcement learning. We will start with some theory and then move on to more practical things in the next part. During this series, you will not only learn how to train your model, but also what is the best workflow for training it in the cloud with full version control using the Valohai deep learning management platform.

This tutorial will demonstrate how to take a single cell in a local Jupyter Notebook and run it in the cloud, using the Valohai platform and its command-line client (CLI).

Since the rise of the deep learning revolution, springboarded by the Krizhevsky et al. 2012 ImageNet victory, people have thought that data, processing power and data scientists were the three key ingredients to building AI solutions. The companies with the largest datasets, the most GPUs to train neural networks on, and the smartest data scientists were going to dominate forever.

Valohai now supports random search for hyperparameter optimization (which we call the Tasks feature), which has been proven in the aptly named paper Random search for hyper-parameter optimization to be an efficient way to find “neighborhoods” of likely-to-be-optimal hyperparameter values, which can then be iterated further to find the really good values.

Watch a recording of the webinar on version control in machine learning that was held on 22th of November 2018. During the webinar we discussed about the topics below and answered multiple questions addressed by the attendees.

PocketFlow is an open-source framework from Tencent to automatically compress and optimize deep learning models. Especially edge devices such as mobile phones or IoT devices can be very limited on computing resources so sacrificing a bit of model performance for a much smaller memory footprint and lower computational requirements is a smart tradeoff.

Microsoft's Cognitive Toolkit or CNTK is an open source framework for building Deep Learning models. This relatively new framework has been gaining traction so we decided to make sure Valohai supports it well. One of the benefits over competing frameworks has been CNTK’s ground up support for multi-node, multi-GPU training, something that for instance TensorFlow has been struggling to tackle well. If you are doing work on really large datasets, you should maybe give it a try.

Synthetic data is artificially created information rather than recorded from real-world events. A simple example would be generating a user profile for John Doe rather than using an actual user profile. This way you can theoretically generate vast amounts of training data for deep learning models and with infinite possibilities.

You might have heard that every individual subject to automated decision making by machine learning models has a right to an explanation of the result. I bet you feel drops of sweat forming on your forehead when you receive an inquiry from a manager saying that he needs details about how a certain decision was made. If thinking about this scenario gives you chills, you are in the right place. Read further and learn how to tackle the transparency issue.

When meeting with teams that are working with machine learning today, there is one point above everything else that I try to teach. It is the importance of storing and versioning of machine learning experiments and especially how many things there actually are that need to be stored.

You know what really grinds my gears? When I have a deep learning model that I want to train and I have to SSH into my AWS instance, install all the drivers and libraries, run my code and then forget to shut down my machine! Once, I ended up forgetting one up over the weekend that cost my employer over $10 000!!!

Recreating experiments inside Valohai could be a whole lot easier and we’ve heard your cries!

All of us have seen those fear mongering headlines about how artificial intelligence is going to steal our jobs and how we should be very careful with biased AI algorithms. Bias means that the algorithm favors certain groups of people or otherwise guides decisions towards an unfair outcome. Bias can mean giving a raise only to white male employees, increasing criminal risk factors of certain ethnic groups and filling your news feed only with topics and point of views that you are currently consuming – instead of giving a broad, balanced view of the world and educating you.

Valohai and Microsoft cross lightsabers in the battle for artificial intelligence, through Microsoft’s global ScaleUp Program.

Just lately we’ve been playing around with IBM PowerAI in order to ensure our customers can leverage it in large-scale on-premise training. PowerAI in itself is IBM’s solution for deep learning consisting of software and hardware to help you quickly train deep learning models. Today we’re happy to announce that Valohai fully supports PowerAI and our customers can start using it!

Valohai is turning 2 years old in three weeks. The paperwork was done on October 16th, 2016. It’s been a thrilling ride so I’ll take this chance to write a few words about why we really started this company.

Whitesnake cover bands of the 2020s. Although both might be sporting the same hobo beards, Data Scientists are getting their work done with just sticks and stones as their tools while us Software Engineers have every tool in the universe.

Developing a machine learning model for a new project starts with certain common groundwork and exploration, to understand your data and figure out the approaches to try. A popular choice for this groundwork is Jupyter, an environment where you write Python code interactively.

Reproducibility and replicability are cornerstones of the scientific method. Every so often there’s a sensationalized news article about a new scientific study with astounding results (for instance, we’re looking forward to seeing what’s hot at ICML 2018.

If machine learning is a team sport, like I so frequently hear, machine learning platforms must be the playing fields. And to up your machine learning game, you must have the proper environments to do it.

With the promise of relieving strain on the transport network in maritime cities using Artificial Intelligence and autonomous driving technology, Finnish software powerhouse Reaktor set to build a solution for future waterways.

After spending two days at the AI Summit fair in London and having several conversations with people from different business backgrounds, I wanted to clarify why machine learning infrastructure is one of the biggest things to concentrate on when building production level machine learning models.

Today’s machine learning teams consist of people with different skill sets. There are a bunch of different roles that are needed, but today I am going to talk about the two key roles that I get asked about the most: machine learning researcher / data scientist vs. machine learning engineer.

Smart recommendation in apps and websites is not an additional feature that differentiates top industries from others. Most users take for granted that they will be suggested products that they like. Collaborative filtering has been widely used to predict the interests of a user by collecting preference and tastes information from many users. It is often combined with content-based filtering, especially for tackling the cold-start problem.

In the age of technology, conventional methods are being automated, and computers are taking over. Similarly, for energy distribution, smart grids are replacing traditional energy distribution grids which allow efficient distribution and demand-side management.

Jacques Marais used machine learning to scan Africa’s elephant population from aerial infrared and color images taken from a plane. The built models were trained first in 2015 with local GPU hardware in three weeks. When the models were retrained in 2017 with the Valohai platform the work was completed in three days while the detection accuracy increased from 56% up to 67% and the overdetection rate dropped dramatically.

Valohai, a machine learning (ML) platform-as-a-service company, has raised $1.8M in funding to help international companies accelerate machine learning development and scale their model deployment. The round was led by Nordic seed stage investment company Superhero Capital, with participation from Reaktor Ventures and Business Finland, the Finnish Funding Agency for Innovation.

We at Valohai are building machine learning platform-as-a-service. Underneath this mouthful of a buzzword we are actually trying to solve a real world problem I've seen being tackled over and over in dozens of big and small organizations applying or researching machine learning.