Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
EXO Infrastructure as Code
- Comprehensive overview of EXO deployment architectures: single-node, multi-node, and RDMA clusters.
- Automating dependency installation (Xcode, uv, Node.js, Rust) via configuration management tools.
- Leveraging Nix flakes to create reproducible EXO builds and consistent developer environments.
- Developing Ansible playbooks or shell scripts for unattended cluster provisioning.
Reproducible Builds and CI Integration
- Pin dependencies and construct the dashboard within CI pipelines.
- Execute EXO smoke tests using GitHub Actions or GitLab CI runners.
- Create golden images and snapshot-based rollback workflows for macOS and Linux virtual machines.
- Version custom model cards alongside application code to maintain traceability.
Cluster Discovery and Networking Automation
- Configure mDNS and static DNS to ensure reliable libp2p node discovery.
- Automate network profile creation and Thunderbolt bridge management on macOS systems.
- Utilize custom namespaces (EXO_LIBP2P_NAMESPACE) to logically separate development, staging, and production clusters.
- Implement firewall rules and network segmentation for secure multi-tenant environments.
Storage and Model Lifecycle Management
- Formulate strategies for EXO_MODELS_DIRS and EXO_MODELS_READ_ONLY_DIRS configurations.
- Mount NFS or SAN shares as read-only model repositories to facilitate rapid provisioning.
- Apply garbage collection policies for stale caches and define retention rules for versioned weights.
- Automate pre-downloads of models and conduct health checks prior to rolling updates.
Monitoring and Alerting
- Forward EXO logs to centralized logging platforms such as ELK, Loki, or Splunk.
- Construct Grafana dashboards utilizing output from EXO_TRACING_ENABLED.
- Set up alerts for cluster membership changes, Out-of-Memory (OOM) events, and inference latency spikes.
- Correlate macmon hardware telemetry with model performance regressions to identify root causes.
Update, Rollback, and Disaster Recovery
- Stage EXO binary updates on a canary node before executing fleet-wide rollouts.
- Perform model-level rollbacks by switching between quantized versions without requiring re-downloads.
- Backup and restore cluster states, custom namespaces, and cached weights to ensure data integrity.
- Document recovery runbooks for scenarios involving complete cluster rebuilds.
Security Hardening and Compliance
- Enforce TLS at the reverse proxy layer (nginx, traefik) for the dashboard and API interfaces.
- Implement API rate limiting and IP whitelisting mechanisms for EXO endpoints.
- Isolate clusters using VLANs and enforce zero-trust network policies.
- Audit access logs and maintain an accurate inventory of deployed models and their versions.
Requirements
- Proficiency in DevOps practices such as CI/CD, Infrastructure as Code (IaC), and container orchestration.
- Familiarity with system administration and package management on macOS or Linux platforms.
- Solid understanding of networking principles, DNS, and storage concepts.
Audience
- DevOps engineers
- Infrastructure architects
- Site Reliability Engineers (SREs) managing on-premise AI workloads
21 Hours
Testimonials (2)
Craig was extremely involved in the training, always making sure we are paying attention, adapted the examples to our day-to-day activities and always provided an answer when asked, even if the information was not added in the presentation.
Ecaterina Ioana Nicoale - BOOKING HOLDINGS ROMANIA SRL
Course - DevOps Foundation®
High level of commitment and knowledge of the trainer
