Skip to main content

Examples

End-to-end ML pipeline examples covering training, inference, recommendation systems, and model packaging. Each example is a complete, working workflow you can run locally or on a Michelangelo AI cluster.

Most examples live in python/examples/. Heavier examples with their own dependency footprint — currently the California Housing pipelines (XGBoost + Spark, and PyTorch Lightning) — live in the separate michelangelo-examples repo instead, to keep this core repo's dependencies lean.

By Use Case​

Training & Fine-tuning​

ExampleDescriptionRuntimeDifficulty
California Housing (XGBoost)Full pipeline — feature prep, Spark preprocessing, distributed XGBoost training, and pusher step that exports model + eval report to storage and registry. Lives in michelangelo-examples.Ray + SparkBeginner
California Housing (PyTorch Lightning)Same California Housing use case as the XGBoost variant, using tabular_trainer's train_tabular() for distributed Ray Train instead. Also has a local-only runner for a quick, sandbox-free trial. Lives in michelangelo-examples.Ray + SparkBeginner
BERT Text ClassificationFine-tune BERT for linguistic acceptability classification on the CoLA benchmark (GLUE). Uses HuggingFace Transformers with distributed Ray training.RayIntermediate
GPT Fine-tuning with LoRAParameter-efficient fine-tuning using LoRA (1.29% trainable params) on the Stanford Alpaca instruction-following dataset. Includes perplexity and generation quality evaluation.RayAdvanced
Nomic AI Embedding TrainingTrain a long-context Nomic BERT model (2048 tokens) on WikiText using PyTorch Lightning with distributed Ray execution.RayIntermediate
MovieLens Collaborative FilteringNeural Collaborative Filtering on MovieLens-100k. Minimal smoke test for the LightningTrainer SDK — trains on CPU with a single Ray Train worker.RayBeginner

Recommendation Systems​

ExampleDescriptionRuntimeDifficulty
Amazon Books RecommendationDual-encoder recommendation system using Qwen-based architecture for Amazon Books. Demonstrates Chronon feature engineering on Spark and distributed Ray training.Ray + SparkAdvanced

Inference​

ExampleDescriptionRuntimeDifficulty
HuggingFace Batch InferenceBatch inference with two backends: HuggingFace Transformers (CPU/GPU) and vLLM (optimized GPU with tensor parallelism). Configurable sampling parameters (temperature, top-p, max tokens).RayIntermediate

Model Packaging​

ExampleDescriptionRuntimeDifficulty
Custom Model PackagingPackage a custom model with CustomTritonPackager for Triton Inference Server. Demonstrates the Model interface (save/load/predict) with raw and deployable packaging modes.—Advanced
Custom PyTorch Model PackagingPackage a PyTorch model (TorchLinearModel) with CustomTritonPackager. Uses numpy arrays for I/O (as required by the Model interface) with internal torch conversion.—Advanced

Running Examples​

Most examples follow the same pattern:

cd python
poetry install --extras "trainer example"
PYTHONPATH=. poetry run python examples/<example_dir>/<script>.py

For remote execution on a Michelangelo AI cluster, append remote-run:

# From the michelangelo-examples repo (https://github.com/michelangelo-ai/michelangelo-examples):
pip install "michelangelo-examples[california-housing]"
python -m michelangelo_examples.california_housing.pipelines.xgb_train.pipeline \
  remote-run \
  --image ghcr.io/michelangelo-ai/michelangelo-examples:california-housing \
  --storage-url s3://michelangelo/workflows \
  --environ AWS_ENDPOINT_URL=http://minio:9091 \
  --environ AWS_ACCESS_KEY_ID=minioadmin \
  --environ AWS_SECRET_ACCESS_KEY=minioadmin \
  --environ REGISTRY_ENDPOINT=michelangelo-apiserver:15566 \
  --yes

See each example's README for specific prerequisites and run instructions.

What's Next?​