MLOps on Google Cloud: Building an ML Workflow That Runs Itself

What the ML workflow looks like on Google Cloud

A typical MLOps workflow on Google Cloud runs: data in BigQuery or Cloud Storage, features computed and served in a consistent way, training orchestrated via Vertex AI Pipelines, models versioned in the Vertex AI Model Registry, deployment to Vertex AI endpoints or batch prediction, and drift monitored via Vertex AI Model Monitoring.

The value of naming the entire chain is that it makes the gaps obvious. Most teams have training and they have deployment. Very few have versioned data, very few have a registry that actually reflects reality, and even fewer have monitoring in place that would actually pick up a silent regression.

• Data layer. BigQuery for structured and tabular data, Cloud Storage for images, audio and documents. Keeping your training data queryable in the warehouse pays off again and again.
• Feature consistency. The same computation for training and serving, whether via a managed feature store or a shared transformation library. That's the structural fix for training-serving skew.
• Orchestration. Vertex AI Pipelines runs containerised steps with defined inputs and outputs, caches steps that don't change and records the lineage of every run.
• Registry. Versioned models with their evaluation metrics and lineage, so what's actually deployed and why is a matter of record rather than recollection.
• Serving. Online endpoints for low-latency inference, batch prediction for scheduled scoring. Getting this one right has a significant cost impact.
• Monitoring. Drift and skew detection on deployed models, configured explicitly against the training baseline.

Making training a pipeline, not a notebook

That one change is what delivers most of the benefit that teams are after when they talk about MLOps. A notebook that produces a good model is a research result. A pipeline that produces that model in a repeatable fashion is an operational capability.

1. Break it down into steps. Data extraction, validation, transformation, training, evaluation and registration as separate containerised components with declared inputs and outputs.
2. Parameterise everything. Date ranges, hyperparameters and data versions as pipeline parameters, so that a rerun on new data is just a parameter change rather than an edit.
3. Validate the data as a step. Schema, distribution and volume checks that fail the pipeline loudly rather than quietly training on corrupted input.
4. Evaluate against the incumbent. The candidate must be compared to the currently deployed model on the agreed criteria, and the run must fail if it doesn't clear the bar.
5. Register rather than deploy. A successful run registers the candidate; promotion to production stays a separate, deliberate step.
6. Schedule and trigger. On a cadence, on new data arriving or on a drift alert — whichever matches the rate at which your problem actually moves.

The caching is worth taking advantage of. If only the training step changes, then the extraction and transformation steps can be reused, which dramatically shortens the iteration loop during development.

Choosing between the training options

A practical sequence would be to use BigQuery ML or AutoML to establish an honest baseline very quickly, then go to custom training only if a custom model is going to beat that baseline by a meaningful amount. A lot of teams spend a quarter building something that in the end doesn't beat a baseline they never actually measured.

Serving: online or batch

This decision has a much bigger cost impact than most teams realise, and it's typically made by default rather than as a considered decision.

• Batch prediction for scheduled scoring. If you're consuming your predictions on a cadence (nightly churn scores, weekly demand forecasts), batch is vastly cheaper, as there's no endpoint sitting around idle between runs.
• Online endpoints for request-time inference. Required when you need to make a prediction as part of a user interaction. Make sure the machine type is right-sized and that autoscaling is configured deliberately.
• Watch idle endpoint cost. A provisioned endpoint which is serving only occasional traffic is a recurring charge for capacity that you're not using — one of the most common avoidable costs on the platform.
• Consider precomputation. Where the input space is bounded, scoring in batch and serving from a lookup table is faster and cheaper than live inference.

Monitoring and retraining

Model Monitoring picks up two separate problems, and both of these require configuration against a baseline derived from the training data.

• Training-serving skew. Production inputs are different from training inputs from the very beginning — usually a pipeline defect, and the cheapest to catch straight after deployment.
• Prediction drift. Inputs drift away from the training distribution over time as the world changes. That's the expected long-term failure mode.
• Attribution drift. Which features are driving the predictions has changed, which can sometimes flag a problem before the overall distributions have visibly shifted.
• Set thresholds you will act on. An alert that nobody ever responds to teaches people to ignore the channel. You need to decide in advance whether a drift alert results in an investigation, retraining or a rollback.
• Close the loop with outcomes. Where ground truth comes in later, feed it back to measure the realised accuracy. Drift detection is the early warning; realised accuracy is the verdict.

Retraining triggers should be chosen according to how fast the problem is actually moving. If the change is gradual, a scheduled trigger is appropriate. If it's unpredictable, you want a drift trigger. If new labelled data comes in in batches, you want a data-volume trigger.

How SoftmaxAI builds on Google Cloud

We're a Google Cloud Partner and we build these systems the same way we build everything else. We have a written, client-approved specification prior to any code being written. We have an agent-accelerated build led by senior engineers, and every single decision that matters is owned by a human.

• Right-sized to your model count. We build the maturity level that your workload justifies. Full continuous delivery for two stable models is cost without benefit.
• Reproducibility as a requirement. Versioned data, tracked runs and a registry that shows what is actually deployed, so any model can be rebuilt and audited.
• Monitoring configured, not assumed. Drift baselines, alert thresholds and the agreed response are all part of the delivery, not something that comes in at a later stage.
• Across the Google Cloud stack. Including BigQuery data analytics, Vertex AI and Gemini, and migration and modernisation where the foundation needs work first.

Background reading: MLOps versus DevOps for what makes ML operations different, and the ML and AI services on GCP for the wider platform.