Examples
End-to-end ML pipeline examples covering training, inference, recommendation systems, and model packaging. Each example is a complete, working workflow you can run locally or on a Michelangelo cluster.
All examples live in python/examples/.
By Use Case
Training & Fine-tuning
| Example | Description | Runtime | Difficulty |
|---|---|---|---|
| California Housing (XGBoost) | Full pipeline — feature prep, Spark preprocessing, distributed XGBoost training, and pusher step that exports model + eval report to storage and registry. | Ray + Spark | Beginner |
| BERT Text Classification | Fine-tune BERT for linguistic acceptability classification on the CoLA benchmark (GLUE). Uses HuggingFace Transformers with distributed Ray training. | Ray | Intermediate |
| GPT Fine-tuning with LoRA | Parameter-efficient fine-tuning using LoRA (1.29% trainable params) on the Stanford Alpaca instruction-following dataset. Includes perplexity and generation quality evaluation. | Ray | Advanced |
| Nomic AI Embedding Training | Train a long-context Nomic BERT model (2048 tokens) on WikiText using PyTorch Lightning with distributed Ray execution. | Ray | Intermediate |
| MovieLens Collaborative Filtering | Neural Collaborative Filtering on MovieLens-100k. Minimal smoke test for the LightningTrainer SDK — trains on CPU with a single Ray Train worker. | Ray | Beginner |
Recommendation Systems
| Example | Description | Runtime | Difficulty |
|---|---|---|---|
| Amazon Books Recommendation | Dual-encoder recommendation system using Qwen-based architecture for Amazon Books. Demonstrates Chronon feature engineering on Spark and distributed Ray training. | Ray + Spark | Advanced |
Inference
| Example | Description | Runtime | Difficulty |
|---|---|---|---|
| HuggingFace Batch Inference | Batch inference with two backends: HuggingFace Transformers (CPU/GPU) and vLLM (optimized GPU with tensor parallelism). Configurable sampling parameters (temperature, top-p, max tokens). | Ray | Intermediate |
Model Packaging
| Example | Description | Runtime | Difficulty |
|---|---|---|---|
| Custom Model Packaging | Package a custom model with CustomTritonPackager for Triton Inference Server. Demonstrates the Model interface (save/load/predict) with raw and deployable packaging modes. | — | Advanced |
| Custom PyTorch Model Packaging | Package a PyTorch model (TorchLinearModel) with CustomTritonPackager. Uses numpy arrays for I/O (as required by the Model interface) with internal torch conversion. | — | Advanced |
Running Examples
Most examples follow the same pattern:
cd python
poetry install --extras "trainer example"
PYTHONPATH=. poetry run python examples/<example_dir>/<script>.py
For remote execution on a Michelangelo cluster, append remote-run:
PYTHONPATH=. poetry run python examples/pipelines/california_housing_xgb/california_housing_xgb.py remote-run \
--project ma-examples \
--image ghcr.io/michelangelo-ai/examples:main
See each example's README for specific prerequisites and run instructions.
What's Next?
- New to Michelangelo? Start with California Housing (XGBoost) — it covers the full pipeline end-to-end
- Want to understand the framework? Read Getting Started with Pipelines for a guided walkthrough
- Ready to deploy? See Deploy a Model after training