For the last months, we’ve been hard at work bringing the Core Engine to market as a commercial product. Our vision was and remains the commoditization of reproducible Machine Learning. Every pipeline for every model should forever be reproducible and have a deployable model at its end. All the invaluable lessons we’ve learned in this time, from customers, users, friends, and strangers, are now culminating in the next step towards our goal: We will make the Core Engine available under an Open Source License.

The upcoming days will be dedicated to providing an extensible, open, and smartly abstracted version of the codebase we’ve been using so far to power our commercial product.

Vision and reality

Above the realities of any project and startup stands a bigger vision. From our early days, we’ve always shared the belief that whatever we build should be available with as little barrier to adoption as possible - and Open Source is just the next stage of achieving that goal.

Next to that lofty vision is our vision for the Core Engine as a solution to a problem: making reproducible Machine Learning a reality. We are bridging the gap between your ML workflows and the complexity of ML tooling - in a unified tech-stack and without unnecessary steep learning curves.

We will have your back from sourcing data to trained models - and their deployment. Let me walk you through a preview of what that’ll look like.

Concepts

Artifact Stores

Every pipeline step, from sourcing data to training your model, will persist it’s output artifacts in a so-called artifact store. By default, the artifact store is a local directory inside your project. To allow for collaboration within teams and distribution of workloads on processing backends like Google Dataflow, we also support Google Cloud Storage buckets as artifact store.

NOTE: Support for AWS S3 and Azure Blob Storage is already on the roadmap.

Backends

Not every pipeline is suitable for local environments. Production workloads often require scheduling and distribution. That’s why we provide a few native integrations, e.g to processing backends like Google Dataflow or training backends like Google Cloud Platform. Custom backends can easily be added through extensible interfaces.

Tracking

Tracking is baked into the very fabric of the Core Engine. Inputs and outputs for every pipeline step are automatically tracked (default: local), and lineage can be established easily for all processed data up until serving time.

Comparability

An essential feature of the Core Engine is the guaranteed comparability of pipelines within a project. Automated tracking and a unified artifact store allow for the comparability of pipelines across processing backends.

Pipeline layout

Sourcing data

Core Engine pipelines start at sourcing data. For faster processing down the (pipe-)line, all data will be converted into TFRecords. To avoid unnecessary recomputation all processing is automatically versioned and cached. Results are conveniently stored immutably in a configurable artifact store. Sourcing data will automatically be executed on the configured backend (default: local).

Custom code and arbitrary pipeline steps

We are fully aware, workflows are very heterogeneous. The extensible interface design of the Core Engine allows you to easily extend a pipeline step, or build your own pipeline components. We’re exposing some convenient functionality, but you can dive deeper into the rabbit hole and eve run arbitrary TFX components, too.

NOTE: TFX support can be subject to change in future releases.

Splitting

We’re shipping some batteries included making your life easier. From simple ratios to complex categorical or other hybrid split scenarios, you don’t have to reinvent the wheel (because we did). And where we fall short, you can use the extensible interface to run your own splits.

Preprocessing

Similar to splitting, we’re shipping a few batteries included, but provide an extensible interface to add your own code. As an added convenience, the Core Engine has native support for tf.transform and will embed the resulting transformation into the serving graph of your TensorFlow model.

Training

At launch, the Core Engine comes with native support for Keras models. For convenience, passing a model_fn is enough, but this functionality can be extended, similar to all the other pipeline steps before. Additionally, training can be offloaded to special training backends like Google AI Platform.

NOTE: Support for other libraries is already on the roadmap.

Evaluation

Every pipeline automatically provides you with pre-generated evaluation notebooks. There are a few tools we’ve found to be exceptionally useful, so we baked them in for you. Namely, we’re shipping these Notebooks with Tensorboard, TensorFlow Model Analysis (TFMA) as well as TensorFlow Data Validation (TFDV). You have, of course, access to all raw results to run your own evaluation.

Serving

Our value chain ends at serving. The heterogeneity of serving environments and use-cases guided our hand, and we’re shipping support for two of our favorite approaches: Seldon Core and Google AI Platform. Configuring Serving for your pipelines is optional, and can be added after running pipelines. For complex, custom scenarios we’re exposing an extensible interface for you to integrate the Core Engine in your existing workflows with limited overhead.

Timeline

Going from closed-source and commercial to open-source is an involved process, and we’re working hard at making it happen. As of writing this, we’re looking at an anticipated release in early December 2020. If you have any questions or would like to talk to us about your special use-case feel free to reach out at support@maiot.io.

To receive updates and to be the first to know about our final release make sure to subscribe to our newsletter!

* indicates required

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.


If you want to keep in touch with the latest blog posts, please subscribe to our RSS Feed

More from us:

Never miss out on our blogposts

Subscribe to our newsletter.

* indicates required

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

Get started with the Core Engine - for free!