1. 2024: Crisis for the Team and the Community​

In 2024, while I was busy growing alongside the launch of Project Widearth, my company (MAXST) ran into difficulties and I ended up leaving the team. I had wanted to grow more within that team, but after careful consideration of various circumstances, I decided to move on. I was separated from colleagues I had worked with for years and faced the big challenge of finding a new team. Still, I made the most of a rare period of rest — taking care of my health, keeping a calm mind, and focusing on applications and interviews.

• Career: Left MAXST in October 2024 / (MLOps/DevOps Engineer)
• Project: Launched digital twin platform Widearth / (MLOps/DevOps Lead)
• Certifications: Earned CKA, CKAD, CAPA, GitHub Foundations
• Challenges: Job searching after resignation · Company in turmoil, country in turmoil

Then in December, a state of emergency martial law was declared in South Korea. My sense is that this triggered a wave of hiring freezes and I received many rejection notices as a result. My own skills were also lacking, but I believe the political climate had a significant impact as well. I kept studying to fill the gaps. I also did everything I could to protect the community. Going through something I never imagined would happen in my lifetime — something that seemed impossible in the 21st century, something only found in history books — made me viscerally feel how difficult it is to live without a stable country and economy.

Fortunately, the people I care about and my family gave me the strength to keep going without giving up on finding a new job. Until the very last day of December, I gave my best to everything within my reach. I did what I could in the streets, and at home I spent my time polishing my resume rather than reflecting. That was how I closed out the year I left my first workplace. Even so, with the support of family and friends, I held on well and kept hoping for good news.

2024 was the hardest year of my life. It was also the year I received the most help from others. I am deeply grateful to everyone who stood by us. Thank you all for your perseverance. 🤝

2. 2025: A New Team and Growth​

2025 was a year of recovery for me. Remarkably, many things stabilized quickly. I'm grateful to be able to look back on those days like this.

• Career: Joined Hyundai AutoEver Dev Environment Platform Team in February 2025 / (AI/MLOps Engineer)
• Project: Codebeamer AI Assistant PoC / (AI Architect)
• Certifications: Earned KCNA, KCSA / (Targeting CKS in H1 2026)
• Challenges: Adapting and delivering results after my first job change · Hands-on work and tech lead for Generative AI projects · Planning to expand Generative AI initiatives going forward

2.1. New Team, New Colleagues​

It wasn't until early February, when I received the offer and made the decision to join, that I could finally breathe again. I'm grateful that my experience and potential were held in high regard.

In February 2025, I joined the Hyundai AutoEver Dev Environment Platform Team. It's the ICT arm of the Hyundai Motor Group, and I was delighted to return to the automotive field that I had been deeply immersed in during my graduate studies. Our team supports the software development environment for automotive technology — something I had experienced firsthand during my time as a researcher.

Being part of a large corporation means you can make bold new attempts on top of a stable foundation. However, as projects mature, a strong lock-in effect tends to develop, tying individuals to specific tasks. To keep growing, I need to take on more challenges than I did at a startup, while also improving stability.

I met great colleagues. My previous workplace had many people with similar career stages and ages, but the new team has colleagues across all experience levels, which allowed me to share and absorb a wide range of experiences. Everyone was relaxed and highly skilled. With their help, I was able to adapt quickly. I drank a lot of coffee. ☕️

The Dev Environment Platform Team is an organization within the Hyundai Motor Group that supplies collaboration tools for automotive software development:

• Commercial Application Lifecycle Management (ALM) tools like Codebeamer for integrated management of requirements, issues, and testing,
• Popular commercial SW collaboration tools such as GitLab, Jira, and Confluence,
• BitWinHub, an in-house automation tool for integrating SW development platforms (internally developed)

These form the toolchain provided to the organization.

I was now able to work at the intersection of the automotive field I had been deeply involved with since graduate school and the DevOps domain focused on supporting people and platforms. Experiences that had been evaluated in pieces during other interviews all came together here as a cohesive set of career assets. Being recognized in that way only made me want to contribute more to the new company.

Over the past 10 months, I've been able to learn the culture of a group company where thousands of people work together, share my experience, and contribute to the team in my own way. Being a manufacturing-based company with a long history, it had a different scale and set of pros and cons compared to an agile startup. I also got to hear a lot about the pride and struggles of my teammates. I felt genuinely proud that people listened to the ideas and insights I brought to the table.

2.2. New Role, New Project​

I joined as a mid-level engineer. Based on my experience in DevOps and Generative AI, I was assigned the role of maintaining and improving the team's platform. After a brief adjustment period and through a process of role refinement, my responsibilities became more focused on Generative AI projects. My existing DevOps skills also proved useful since we were aiming for live services. (I later received feedback that it was great having someone who could wear multiple hats.)

I joined a part of over 10 members, within a group of 4. The first project I was assigned to was "Driving the PoC for an AI assistant to support Codebeamer." As a group of four, we worked together on the following:

• Reviewing the target platform and Generative AI technologies
• Developing AI agents and designing the UI
• Data preprocessing and building a knowledge-base training pipeline
• Code review, CI/CD, and monitoring setup

As the AI Architect, I worked alongside team members who had limited Generative AI project experience, sharing my knowledge and growing together. Along the way, we successfully overcame both project management and technical challenges. After about 6 months of hands-on work, we conducted an internal beta test and wrapped up the PoC successfully. We built up the planning capabilities and technical stack needed to drive AI projects going into 2026.

2.3. The Pressure of a First Job Change​

There were colleagues who advised me from the very start not to try too hard, and I knew I was the type who could easily fall into impostor syndrome. But before I knew it, I was caught in an unsustainable routine. I was waking up early, studying on the bus on my way to work, proposing and driving various initiatives, communicating to be a good colleague, asking for feedback, and coming home to find more things to do.

This is a fine routine for a growing professional, but I took it too far. After pushing through the first half of 2025 this way, I did manage to make a good first impression and deliver consistent results. I received positive feedback at year-end as well, but my body had taken a toll over that short period. As project involvement and work requests will only increase, pacing myself is absolutely essential.

Lately, I've been trying to leverage Generative AI for work automation — doing more in the same amount of time. Come in early, leave early.

2.4. Regrets and the Path Forward​

Through last year's AI project, I was able to implement a successful GenAI service development cycle in a short period of time. Of course, there were a few things I wish had gone better.

• ⏩ Speed Is Everything: AWS 'innovate faster'
• 📝 Validate with Results: I'll secure technical validation through a portfolio
• 🤝 Collaboration: I'll boldly build disposable tools for collaboration when needed

I believe that addressing these areas in my personal capabilities will lead to excellent project execution and teamwork:

1. Execution was fast, but I was obsessed with perfection.
   • Issue: Initial implementation was completed quickly, but I stuck to incrementally improving the existing framework rather than pivoting to improve results.
   • Solution: Build flexible plans and iterate quickly based on data.
   • Examples: Rebuilding the knowledge base, changing the tech stack, etc.
   • Going forward: Drawing from AWS's 'innovate faster' principle to pursue planning and implementation that enables rapid iteration.
2. I lacked the process of persuading through results.
   • Issue: When technical decisions beyond the original plan were needed, I had no working demo to show teammates. Not having prepared these in advance meant decision-making consumed a lot of time.
   • Solution: When exploring AI technologies like RAG, MCP, and A2A, prepare examples in advance.
   • Examples: A working technical portfolio deployed on internal cloud
   • Going forward: Build and continuously update a technical portfolio. It can serve as a tool for decision-making and also as onboarding material.
3. I could have done better on UX for collaboration.
   • Issue: I wrote solid code for data processing and evaluation, but didn't build a UI to share it and gather feedback. This created a structure that relied solely on result reports and made it hard to share results in real time. Even small updates required rebuilding the source code.
   • Solution: AI makes it easy to build disposable apps. FE/BE/DB can be designed for fast collaboration and decision-making.
   • Examples: Knowledge-base search performance evaluation UI, expert evaluation UI, AI demo tools
   • Going forward: I'll boldly build disposable tools whenever collaboration calls for it. I expect productivity to increase 3–4x compared to before.

3. Closing: Here's How I Plan to Keep Growing​

I plan to keep doing what I did last year (2025). That means deepening my automotive SW domain knowledge in the new team and becoming more seasoned in hands-on work and collaboration. I also plan to further spread Generative AI within the team and company, applying it to work to expand productivity without limits.

• 💪 Grind in the Work: I want to build expertise in Generative AI and LLMOps platforms within the automotive SW development domain, and further develop my practical and collaborative skills.
• 🦾 Leveraging Generative AI: I'll spread knowledge of Generative AI usage to the team and company, applying it to work to expand productivity without limits.
• 🤝 Collaboration: Every day, I'll strive to be a colleague who is great to work with.
• 🏥 Health Management: I'll make sure to exercise consistently so I can keep working for the long haul.

Because I always dream of being a staff engineer who works hands-on and solves technical problems directly, I will keep learning and engaging with others to maintain my practical skills. I'll gratefully take on the opportunities that come my way, share good opportunities with others, and grow together. No one works alone. Every day, I'll strive to be a colleague who is great to work with. And to do that, I need to take care of my health too. This year (2026), I'll commit to exercising consistently.

Thank you for reading.

The Kubernetes and Cloud Native Security Associate (KCSA) certification validates foundational knowledge of security practices and principles for Kubernetes and the broader cloud native ecosystem. Created by the Cloud Native Computing Foundation (CNCF) in collaboration with The Linux Foundation, KCSA is designed for individuals who want to demonstrate their understanding of cloud native security concepts and best practices.

The KCSA exam is a proctored, online test that consists of 60 four-choice multiple-choice questions to be completed within 90 minutes. All questions are theoretical and there are no hands-on tasks. A passing score of 75% is required, and the certification is valid for 2 years. The exam covers security aspects across the cloud native landscape, including:

• Kubernetes Security Fundamentals: Core security concepts, pod security, network policies, and RBAC
• Container Security: Container image security, scanning, and runtime security
• Cloud Native Security Tools: Security tools and practices in the CNCF ecosystem
• Security Best Practices: Secure configuration, secrets management, and compliance
• Threat Detection and Response: Monitoring, logging, and incident response in cloud native environments

Unlike hands-on security certifications like CKS (Certified Kubernetes Security Specialist), KCSA focuses on theoretical knowledge and understanding of security concepts, making it an excellent entry point for those new to cloud native security.

2. My Experience​

My KCSA certification 2025. Issued on October 27, 2025. Expires on October 27, 2027. You can verify my certification here.

I took the KCSA exam on October 27, 2025, and passed it successfully. Coming from hands-on experience with Kubernetes security through my CKA and CKAD certifications, I found the KCSA exam to be an excellent way to validate and systematize my theoretical understanding of cloud native security practices.

The exam format consists entirely of four-choice multiple-choice questions with no hands-on components:

• All 60 questions are theoretical multiple-choice questions (no command-line tasks)
• Focus on security concepts, principles, and best practices rather than practical implementation
• Covers security aspects across the entire cloud native ecosystem, not just Kubernetes

The questions tested knowledge across various security domains, including Kubernetes security policies, container security, secrets management, network security, compliance, and security tools in the CNCF ecosystem. While my practical experience with Kubernetes was helpful, the exam required a comprehensive understanding of security principles and how they apply to cloud native environments.

To aid my preparation, I enrolled in the Kubernetes and Cloud Native Security Associate (KCSA) course on KodeKloud, taught by Mumshad Mannambeth and Nimesha Jinarajadasa. I bought basic plan on KodeKloud to access this course (Pro plan is not necessary). This comprehensive 6.78 hour course covered all exam domains including cloud native security overview, Kubernetes cluster component security, security fundamentals, threat models, platform security, and compliance frameworks. The course's hands-on labs, interactive content, and regular updates with the latest security information were particularly valuable. While KodeKloud provides hands-on lab environments for practice, the actual KCSA exam consists entirely of four-choice multiple-choice questions with no hands-on components. The course's interactive quizzes and mock exams helped reinforce security concepts and prepare for the actual exam format. The structured learning path and continuous content updates made it possible to pass the exam smoothly.

3. Tips for Exam Preparation​

Based on my experience, here are some important tips for those preparing for the KCSA exam:

Essential Resources Before the Exam:

• Official CNCF Curriculum: Review the official KCSA curriculum to understand the exact exam domains and their weightings
• CNCF Security Landscape: Familiarize yourself with security-related projects in the CNCF Landscape - understand security tools, their purposes, and how they fit into the cloud native security ecosystem. The landscape provides a comprehensive view of security projects categorized by their maturity levels (Graduated, Incubating, Sandbox)
• CNCF Blog: Regularly review the CNCF Blog for the latest security updates, best practices, and announcements about new security projects. The blog often covers security-related topics that may appear on the exam
• Recent CNCF Security Additions: Pay special attention to security projects that joined the CNCF ecosystem in the 2-3 months before your exam date. These newer security tools can appear on the exam, so staying current is crucial
• Kubernetes Security Documentation: Review official Kubernetes security documentation, especially on pod security, network policies, RBAC, and secrets management

Study Strategy:

• If you're already familiar with Kubernetes security (e.g., from CKS or security-focused work), the preparation can be completed more efficiently
• Focus on understanding security "why" and "what" rather than just the "how" - KCSA tests conceptual security knowledge through four-choice multiple-choice questions only
• Practice with mock exams to get comfortable with the multiple-choice format (remember: no hands-on tasks on the actual exam)
• Review security best practices and common vulnerabilities in cloud native environments
• Study CNCF security projects and their roles in the security ecosystem

Important Reminder: The KCSA exam covers security aspects across the entire cloud native landscape, not just Kubernetes. Make sure to allocate study time for container security, secrets management, compliance, threat detection, and security tools beyond your Kubernetes knowledge.

4. Summary​

The KCSA certification serves as an excellent foundation for understanding cloud native security. For those with hands-on Kubernetes experience (like CKA/CKAD/CKS holders), it provides an opportunity to validate and expand theoretical security knowledge. For beginners, it offers a structured learning path into cloud native security practices.

Key characteristics of KCSA:

• Entry-level security certification: Accessible to those new to cloud native security
• Broad security coverage: Covers security aspects across the entire cloud native ecosystem, not just Kubernetes
• Theoretical focus: Tests understanding of security concepts and principles
• Complementary: Pairs well with hands-on certifications like CKS or practical security experience

The KCSA certification validates a solid foundation in cloud native security technologies and serves as a stepping stone for more advanced security certifications or as a way to systematize existing security knowledge in the cloud native context.

Related wiki pages​

• Kubernetes
• Containers
• CNCF
• Linux Foundation

Related article​

• KCNA: Kubernetes and Cloud Native Associate

The Kubernetes and Cloud Native Associate (KCNA) certification validates foundational knowledge of Kubernetes and the broader cloud native ecosystem. Created by the Cloud Native Computing Foundation (CNCF) in collaboration with The Linux Foundation, KCNA is designed for individuals who want to demonstrate their understanding of cloud native technologies and Kubernetes fundamentals.

The KCNA exam is a proctored, online test that consists of 60 four-choice multiple-choice questions to be completed within 90 minutes. All questions are theoretical and there are no hands-on tasks. A passing score of 75% is required, and the certification is valid for 2 years. The exam covers:

• Kubernetes Fundamentals: 46% - Core concepts, architecture, and basic operations
• Container Orchestration: 22% - Container concepts, orchestration principles, and container runtimes
• Cloud Native Architecture: 16% - Cloud native principles, patterns, and design
• Cloud Native Observability: 8% - Monitoring, logging, and observability tools
• Cloud Native Application Delivery: 8% - CI/CD, GitOps, and deployment strategies

Unlike hands-on certifications like CKA or CKAD, KCNA focuses on theoretical knowledge and understanding of cloud native concepts, making it an excellent entry point for those new to the cloud native ecosystem.

2. My Experience​

My KCNA certification 2025. Issued on October 22, 2025. Expires on October 22, 2028. You can verify my certification here.

I took the KCNA exam on October 22, 2025, and passed it successfully. Coming from hands-on experience with Kubernetes through my CKA and CKAD certifications, I found the KCNA exam to be a good way to validate and systematize my theoretical understanding of the cloud native ecosystem.

The exam format consists entirely of four-choice multiple-choice questions with no hands-on components:

• All 60 questions are theoretical multiple-choice questions (no command-line tasks)
• Focus on concepts and principles rather than practical implementation
• Covers broader cloud native topics beyond just Kubernetes

The questions tested knowledge across the entire cloud native landscape, including CNCF projects, cloud native architectures, observability tools, and application delivery practices. While my practical experience with Kubernetes was helpful, the exam required a broader understanding of the ecosystem.

To aid my preparation, I enrolled in the Kubernetes and Cloud-Native Associate (KCNA) course on KodeKloud, taught by Mumshad Mannambeth. I bougth basic plan on KodeKloud to access this course (Pro plan is not necessary). This comprehensive 9 hour course covered all exam domains including cloud-native architecture, container orchestration, Kubernetes fundamentals, application delivery, and observability. While KodeKloud provides hands-on lab environments for practice, the actual KCNA exam consists entirely of four-choice multiple-choice questions with no hands-on components. The course's interactive quizzes and mock exams were particularly valuable for reinforcing concepts and preparing for the actual exam format. If you are familar with Kubernetes, It actually takes 1 week to complete this course. The structured learning path helped me systematically build knowledge across the entire cloud native ecosystem.

3. Tips for Exam Preparation​

Based on my experience, here are some important tips for those preparing for the KCNA exam:

Essential Resources Before the Exam:

• Official CNCF Curriculum: Review the official KCNA curriculum to understand the exact exam domains and their weightings
• CNCF Landscape: Familiarize yourself with the CNCF Landscape - understand the categorization of projects (Graduated, Incubating, Sandbox) and their purposes. The exam may include questions about recently added projects
• Recent CNCF Additions: Pay special attention to projects that joined the CNCF ecosystem in the 2-3 months before your exam date. These newer projects can appear on the exam, so staying current is crucial

Study Strategy:

• If you're already familiar with Kubernetes (e.g., from CKA/CKAD), the KodeKloud course can be completed in about 1 week
• Focus on understanding the "why" and "what" rather than just the "how" - KCNA tests conceptual knowledge through four-choice multiple-choice questions only
• Practice with mock exams to get comfortable with the multiple-choice format (remember: no hands-on tasks on the actual exam)
• Review CNCF project documentation to understand each project's role in the cloud native ecosystem

Important Reminder: The KCNA exam covers the entire cloud native landscape, not just Kubernetes. Make sure to allocate study time for observability tools, CI/CD practices, GitOps, and cloud native architecture patterns beyond your Kubernetes knowledge.

4. Summary​

The KCNA certification serves as an excellent foundation for understanding the cloud native ecosystem. For those with hands-on Kubernetes experience (like CKA/CKAD holders), it provides an opportunity to validate and expand theoretical knowledge. For beginners, it offers a structured learning path into cloud native technologies.

Key characteristics of KCNA:

• Entry-level certification: Accessible to those new to cloud native technologies
• Broad coverage: Covers the entire cloud native ecosystem, not just Kubernetes
• Theoretical focus: Tests understanding of concepts and principles
• Complementary: Pairs well with hands-on certifications like CKA or CKAD

The KCNA certification validates a solid foundation in cloud native technologies and serves as a stepping stone for more advanced certifications or as a way to systematize existing knowledge.

I shared my certification on LinkedIn with this post.

Related wiki pages​

• Kubernetes
• Containers
• CNCF
• Linux Foundation

Related article​

• KCSA: Kubernetes and Cloud Native Security Associate

The AI/MLOps Engineer position at Hyundai AutoEver focuses on developing enterprise-scale AI platforms and ML infrastructure for the automotive industry. Key responsibilities include:

• Building enterprise AI platforms and LLM+RAG systems for automotive domain knowledge
• Developing ML pipeline infrastructure for software-defined vehicles
• Supporting autonomous driving development with MLOps practices
• Implementing cloud-native solutions for scalable AI services

2. My Experience and Capabilities​

My background uniquely positions me for this role, standing at the intersection of three core capabilities:

• Automotive Domain Knowledge: 8+ years of R&D experience in computer vision and ADAS (Advanced Driver Assistance Systems).
• Infrastructure & DevOps Expertise: Designing and operating cost-effective systems based on hybrid Kubernetes clusters.
• GenAI & MLOps Capabilities: Hands-on experience in building LLM+RAG systems, ML pipelines, and automation solutions.

This convergence of domain expertise, robust infrastructure engineering, and modern AI capabilities forms a cohesive foundation for contributing to the innovation of software-defined vehicles (SDVs).

• Technical Stack:
  • MLOps: Kubeflow, Katib, Training Operator, JupyterHub
  • LLMOps: RAG, Vector DBs, Ollama
  • Infrastructure: Kubernetes, AWS EKS, Terraform, Argo Projects
  • Computer Vision: SLAM, SfM, Supervised/Unsupervised Learning, Object Detection
  • Soft Skills: Attitude to be a good co-worker ☺️

3. Share to the World​

See my LinkedIn post for more details: New Role: AI/MLOps Engineer at Hyundai AutoEver

GitHub Foundations certification validates core skills in GitHub's basic structure and usage, from fundamental Git operations to enterprise collaboration features like CodeSpaces. This certification covers essential GitHub knowledge that every developer should possess in 2024, including:

• Basic Git and GitHub usage
• Advanced GitHub features: Issues, Pull Requests, Projects, Team Management
• Latest GitHub products: CodeSpaces, Copilot
• Enterprise features and administration

The exam consists of 75 multiple-choice questions to be completed within 120 minutes. 60 questions are scored while 15 are unscored experimental items. The certification is valid for 3 years upon passing.

2. My Experience​

My GitHub Foundations certification 2024. Issued on November 19, 2024. Expires on November 18, 2027. You can verify my certification here.

I prepared for the certification using two main resources:

1. Microsoft Learn - GitHub Foundations (Free)

   • Comprehensive hands-on labs
   • Always up-to-date with latest GitHub features
   • Practical exercises through GitHub Skills platform

2. LinkedIn Learning - Prepare for the GitHub Foundations Certification (Paid)

   • Video-based learning
   • Good for review and consolidation
   • Covers enterprise features thoroughly

Additionally, there's an excellent free comprehensive video guide from freeCodeCamp that covers both exam content and practical tips:

This video provides a thorough overview of the exam format and includes detailed explanations of key concepts. It's particularly helpful for understanding what to expect during the exam.

The exam tests both basic and advanced GitHub knowledge. While most questions were straightforward for experienced GitHub users, some enterprise-related questions were challenging. The exam interface provided features like:

• Question flagging for review
• Multiple-choice and multiple-select questions
• Keyboard shortcuts for efficient navigation

I found the enterprise and team management sections particularly challenging, as these features are difficult to practice without enterprise access. However, the hands-on experience from Microsoft Learn's labs proved invaluable for the practical aspects of the certification.

3. Summary​

The GitHub Foundations certification is an excellent starting point for developers looking to validate their GitHub skills. While it may seem basic for experienced users, the certification covers important aspects of modern GitHub features that many developers might have overlooked, such as GitHub Copilot, CodeSpaces, and enterprise collaboration tools.

For those preparing for this certification, I recommend:

• Taking advantage of the free Microsoft Learn resources
• Practicing with GitHub Skills hands-on labs
• Focusing on enterprise features and team management concepts
• Watching for certification discount opportunities (sometimes available at 50-100% off)

While I don't plan to pursue other GitHub certifications immediately, I'm keeping an eye on upcoming GitHub AI certifications, particularly around GitHub Copilot and GitHub Models. For now, I'll focus on applying these validated skills in real-world projects and helping others in their GitHub journey.

I hope this post helps those considering the GitHub Foundations certification. Remember, effective version control and collaboration skills are fundamental to modern software development - whether you pursue this certification or not, mastering these skills is essential for any developer's career growth.

• Title: Development of AR & Digital Twin Platform Widearth
• Website: https://widearth.world
• Period: Jan 2024 - Oct 2024 (10 months)
• Role: Lead ML/Infra ~ MLOps/DevOps + ML Backend + SRE [Contribution 75%]
  • DevOps & SRE
    • IaC, GitOps, CI/CD Pipelines, Monitoring, Logging, Notifications, Multi-Deployment Scenarios, Emergency Calls
  • Hybrid Clusters
    • AWS EKS + Bare Metal Kubernetes, API Gateway Pattern, Dynamic Instance Management, GPU Cost Optimization
  • ML Workloads
    • ML APIs, ML Pipelines, Data lake, Dockerizing, Model CI/CD
• Results: Successful service launch ~ Small Team, All Features Implemented, Improved Availability, Cost Savings
  • Service Launch
    • 15 people total, 8 developers, 1 infrastructure manager participated in planning and development to contribute to the launch and operation of the platform
  • High-Efficiency ML
    • Real-time execution of ML pipelines in the on-premises infrastructure of the hybrid cluster. Produced more than 300 space maps in the production environment, and reduced monthly cloud costs by 70%, saving approximately KRW 15 million per month compared to the previous service
  • High-Availability Infrastructure
    • Hybrid cluster and disaster response to implement a service with 96% annual availability and a downtime of 14 days
• Skills
  • AWS EKS
  • Kubespray
  • Python/FastAPI
  • Argo Workflows
  • Argo CD
  • Bitbucket Pipelines
  • Karpenter

Details​

We configured a hybrid architecture for the digital twin platform.

Please refer to the omitted/simplified parts due to internal security regulations.

Widearth's operation flow and user behavior patterns

Widearth is a B2B2C business that uses digital twin technology to create space maps for the following two users. It is a platform business that provides services to general consumers after passing through the main customers.

• Description of users
  • User A (Digital twin content provider)
    • User A can create a space map (2) by using the workspace (1) they own.
    • The created space map is stored in the cloud and User A can share it through the content studio (3). The shared target is User B (digital twin content consumer).
    • User A can also modify the content (4), add content, and process personal information for privacy.
    • The candidate group of User A is 'a subject that makes a profit by sharing space through communication', and examples are as follows:
      • Real Estate / Travel / Art Museum / Restaurant / Heavy Industry / Virtual Space / Disabled-Friendly Facilities
      • Their plans are 'to meet the needs of customers when they are not on site'.
        • Before visiting the site / After visiting the site / When it is not possible to visit the site / When the site has changed
  • User B (Digital twin content consumer)
    • User B is the subject that consumes the space map service provided by User A.
    • The candidate group of User B is 'a subject that makes a profit by consuming space through communication', and examples are as follows:
      • Potential Real Estate Consumer / Potential Traveler / Potential Art Museum Visitor / Potential Restaurant Customer / Potential Employee / Virtual Space User / Disabled Person And Helper/Protector
      • They are customers who can get information about the space and use the service from User A by paying for it.

Widearth's architecture corresponding to the public cloud area. The area where users interact

• The area where users interact, mainly the public cloud area of the platform.
• It is configured with an MSA (microservice architecture or mini-service architecture) that separates FE/BE/pipeline. Since the databases of each service are not strictly divided, it is appropriate to call it a 'mini-service architecture'.
• It can operate in single/multi-cluster and all servers are designed to be horizontally scalable according to usage/usage environment.
• The pipeline is explained in detail in the hybrid architecture below.

Widearth's architecture corresponding to the hybrid cloud (pipeline). The area where users interact

• The area where the pipeline is operating, using both on-premises and public cloud.
• The pipeline is configured with the same specifications on both the on-premises and public cloud. To do this, a 'file storage' and an 'object storage' are configured in each cloud.
• The database for logging and metadata for the back office is configured only in the public cloud for availability and management.
• The pipeline is allocated to the available area of the on-premises data center first. If the on-premises does not respond or all areas are allocated, it is allocated to the public cloud second and the pipeline is activated.
  • If there is an outage/disaster in the on-premises data center,
  • If the request exceeds the available capacity of the on-premises data center.
• The public cloud is allocated to the cluster that services FE/BE with the minimum resource allocation for the pipeline. If the pipeline is allocated to the public cloud, the node is allocated and activated.

Through this hybrid architecture, we can operate a platform with high performance, high efficiency, and high availability.

1. About the Certification​

What is the Argo Project?​

The Argo Project is a suite of tools that combine with Kubernetes clusters to make them behave like a single DevOps platform — and it's getting more attention as Kubernetes adoption grows. It consists of four components:

• Argo Workflows: An orchestration engine for general-purpose workflows, including ML and CI/CD pipelines
• Argo CD: A declarative continuous-deployment tool (the most well-known of the four)
• Argo Rollouts: A deployment-strategy manager that supports advanced update strategies
• Argo Events: An event-driven automation tool that reacts to code changes and other triggers

At first glance it looks like a bundle of CI/CD tools, but each component is actually designed to work generically alongside Kubernetes. You can use Argo Workflows to build data/ML pipelines instead of CI pipelines, and Argo CD to deploy Kubernetes resources — or even Argo CD itself — instead of a web server.

What is the CAPA Certification?​

The CAPA (Certified Argo Project Associate) exam tests your understanding and hands-on knowledge of all four Argo apps (Workflows, CD, Events, Rollouts). A few key facts:

• It's an internationally recognized technical certification administered by the CNCF and the Linux Foundation.
• It's an entry-to-intermediate Associate-level exam — there's currently no higher-level Argo certification.
• It's a live, proctored exam taken through the PSI Secure Browser environment.
• You have 90 minutes to answer 60 multiple-choice questions, and you need 75/100 to pass. You can take it up to twice.
• It's valid for two years from the date you pass, and you'll know whether you passed within about 10 minutes of submitting.

If you've taken Kubernetes certifications like CKA, CKAD, or CKS, this exam environment will feel familiar.

2. My Experience​

My CAPA certification 2024. Issued on September 14, 2024, expires on September 15, 2026. You can verify it here.

Why did I pursue it?​

1. I'd been actively using the Argo Project at work for two years and wanted to take stock of where I stood.

• Used Argo Workflows as the engine for data pipelines and CI/CD pipelines
• Used Argo CD to deploy web applications and Kubernetes resources
• Adopted Argo Rollouts to manage multiple deployment strategies as code
• Used Argo Events for event sourcing in custom CI/CD setups

2. Adoption of the Argo Project is growing in Korea, and I wanted to keep up with the industry trend.

Companies ranging from large IT firms to unicorn AI startups — Karrot (당근), Toss, Devsisters, Sionic AI, and more — are increasingly adopting the Argo Project. I wanted to properly absorb a trending technology, and go beyond simply documenting my experience by securing a verifiable, internationally recognized credential.

Did it actually help?​

It helped a lot. I used Argo Workflows and Argo CD almost every day, but I'd never organized that knowledge systematically. Preparing for this exam gave me a chance to consolidate what I knew and fill in the gaps.

Before passing the exam, I was able to firm up my understanding of the design intent and use cases for all four Argo apps, and discovered hidden features in Argo Rollouts and Argo Events — the two I'd used the least. I tried out every Argo Rollouts deployment strategy (canary, blue/green, delayed rolling, custom updates, and more) and a variety of Argo Events triggers (Kubernetes resource triggers, S3 triggers, custom webhook triggers, and more). I'd only ever used the features I needed day-to-day — this was my first real chance to put to use everything the contributors had built.

After passing the exam, I started reading the official docs much more carefully, and kept finding features I didn't know about even in tools I used every day, like Argo Workflows and Argo CD — which surfaced a lot of things I now want to refactor. A few examples:

• Argo Workflows SSO guest accounts
  • "Can a new hire monitor workflows without being able to delete resources?" → Yes — combine SSO with RBAC to keep initial permissions to a minimum. (This overlaps nicely with skills tested in CKA.)
• Argo CD Sync Windows
  • "Can we turn off auto-sync during our Thursday-early-morning maintenance window?" → Yes — add a deny window for Thursdays in syncWindows, and syncing will be paused during that time.
• Argo CD ApplicationSet Git Generators
  • "Can we sync and control the incidental resources that pop up during operations?" → Yes — ApplicationSet lets you sync derived Kubernetes resources with subdirectories of a git repository.

In the end, this reaffirmed something true of every open-source tool: the real skill is reading the official docs well and putting them to use.

How did I prepare?​

Since this is such a new exam, study material is scarce. There's plenty of Korean-language material for Argo Workflows and Argo CD, but for Argo Rollouts and Argo Events I had to rely almost entirely on the official documentation.

There are barely any third-party courses, but there is an official one: DevOps and Workflow Management with Argo (LFS256). Going through it start to finish should cover the exam scope well. For some reason I bought just the exam without the course — I think I assumed that, since I was already using Argo at work, I could pass without much extra studying. That assumption cost me quite a bit of suffering. For your own sanity, I'd recommend buying the bundle when there's a discount.

I ended up self-studying with the official documentation and the GitHub source code. The exam scope is laid out in the CAPA Curriculum PDF, but since it was hard to predict exactly what would show up, I studied every item in it. My first pass was a quick read-through of the docs; the second pass was implementing the parts I thought were likely to appear. Since this is a multiple-choice exam rather than a hands-on one, it's not open-book and you can't verify things by running them — so it really pays off to experiment beforehand. For experimentation, I used Kubernetes simulators like docker-desktop, kind, microk8s, and orbstack — orbstack was my go-to on macOS for spinning up clusters quickly.

How does the exam work?​

Buy the exam from the exam page (cheaper if you can find an event coupon), then book a date on the Training & Certification dashboard. On exam day, go through the readiness checklist shown on the same dashboard (see the screenshot below), then click Take Exam to begin.

The CAPA exam readiness checklist on the Linux Foundation Training & Certification dashboard.

A few things worth knowing about how it's run:

• It's a multiple-choice exam with 4 options per question. There are surprisingly many obviously-wrong options, which makes the exam feel easier than expected — but there's almost always one option that looks deceptively close to the right answer, so read the English explanations carefully.
  • The difficulty felt comparable to CKAD. Roughly speaking, I'd rank related certifications as KCNA < CKAD = CAPA < CKA < CKS.
  • To score 75, you just need to be comfortable with the core features of all four Argo apps. To score 100, you'll also need to know the names and usage of rarely-used detail features, and be able to tell apart answers that look almost — but not quite — correct.
• You take it through the PSI Secure Browser — familiar territory if you've taken CKA, CKAD, or CKS. Do double-check supported OS versions ahead of time, though: I was running a macOS developer beta and had to roll it back at the last minute before my exam.
• It's not open-book. You can't access the official docs or a CLI, so there's no looking things up or verifying commands on the spot.
• You can review all 60 questions again after finishing, and there's a review toggle to flag the ones you're unsure about.
• You can request a short break (up to about 15 minutes) to use the restroom mid-exam, but the clock appeared to keep running during the break — so it's best to go beforehand, or be quick about it (I'd love to hear if others experienced this differently).

What's the exam content like?​

Per the exam's terms of service, I won't describe specific questions.

• A few questions show you Argo Workflows YAML and ask you to determine the execution order — watch out, because "this can't be executed" is sometimes one of the options.
• There are more GUI-related questions than I expected, in addition to CLI ones — it's worth browsing the GUI of the latest Argo Workflows and Argo CD versions before your exam.
• Some questions test whether you know if a setting belongs at spec.a.b, spec.b.a, or spec.c in the YAML.
• Overall, the difficulty isn't too high — if you've used these tools in production, you should be able to pass comfortably.

A couple more tips that helped me:

• Flag any question you're unsure about with review and move on — you can sometimes pick up hints (e.g., correct YAML syntax) from the options in other questions.
• Since it's multiple-choice, eliminating the options you're sure are wrong improves your odds even when you have to guess.

3. Summary​

I received my pass notification about a minute after clicking submit. I'm used to Kubernetes exams that take 24 hours to grade, so this caught me off guard — but it also meant the joy of passing arrived that much sooner 😊 This exam left me wanting to use the Argo Project even more, and even better, going forward.

• If a higher-level, hands-on certification is ever introduced, I imagine the difficulty would increase significantly — and I'd love to take that one too.
• The subheading structure of this post was inspired by this tech blog post (Korean). Thanks for the great write-up.
• Thanks as well to our team's lead, who first introduced me to the Argo Project, and to the colleague who studied alongside me.
• If you have any questions about the CAPA certification, feel free to leave a comment any time.

The Argo Project is becoming increasingly important in the Kubernetes ecosystem, and I think CAPA is a meaningful way to validate your expertise with these tools. I hope this post helps those of you considering the certification. Show our adorable octopus mascot some love 🐙

Photo by Maxim Hopman on Unsplash

1. Overview​

Our team has built an on-premise MLOps cluster utilizing the latest open-source projects, including Kubeflow, JupyterHub, and Data Pipelines. With limited GPU resources, the architecture is designed to reduce underutilized GPU time and optimize the cost of running ML workloads. Additionally, a single training job can be accelerated by utilizing multiple GPUs in parallel, allowing it to leverage all GPUs in the cluster beyond the resources of a single GPU server.

With on-premise resources, our MLOps cluster provides the following advantages:

• Resource Utilization: The cluster can efficiently utilize all available GPU resources in the cluster, reducing underutilized GPU time. 24/7 GPU usage has increased GPU utilization by three times againt the previous environment.
• AutoML: The cluster supports AutoML workflows, enabling automated model selection, hyperparameter tuning. The cluster can run more than 800 experiments for two weeks. The results were used for commercialization.
• Parallel Training: A single training job can be accelerated by utilizing multiple GPUs in parallel, allowing it to leverage all GPUs in the cluster.
• Remote Notebooks: The cluster provides researchers with on-demand Jupyter notebooks to distribute resources for ML research. Researchers can easily access the cluster and run experiments without worrying about resource constraints.

The following diagram shows the architecture of the on-premise cluster for MLOps:

Architecture Diagram of the On-Premise Kubernetes Setup for MLOps; Users interact with Kubeflow for managing and automating machine learning workflows. Kubeflow/Katib utilizes the Argo Workflows as a workflow engine for hyperparameter tuning and Kubeflow/Training Operator for managing training jobs. Argo Workflows orchestrates these machine learning pipelines. The MySQL cluster stores workflow metadata and results, while the Minio cluster handles artifact storage. The cluter contains GPU servers providing computational power for training models. This setup ensures efficient data handling and workflow management in an on-premise Kubernetes environment.

2. Problem Statement​

AI researchers in our organization were facing challenges with limited GPU resources. Each researcher was allocated a single GPU server, which led to underutilized GPU time when the server was idle and insufficient resources for large-scale experiments.

To address these challenges, we proposed integrating the GPU servers into the cluster and implementing an MLOps platform based on open-source projects. Our goals were followings:

• Resource Sharing: Allow researchers to share GPU resources in the cluster and leverage all available GPUs for their experiments.
• User-Friendly Environments: Provide researchers with an easy-to-use environment for developing and running ML experiments.
• AutoML Support: Implement AutoML workflows to automate model selection, hyperparameter tuning, and model evaluation.
• Parallel Training: Enable parallel training of models by utilizing multiple GPUs in the cluster.
• Remote Notebooks: Provide researchers with on-demand Jupyter notebooks to distribute resources for ML research.

The project started with a Proof of Concept (PoC) to demonstrate the benefits of the MLOps platform and its impact on AI research.

3. Methods​

To achieve the architecture described above, we implemented the following steps:

3.1. Setup On-Premise Cluster​

We already had an on-premise cluster with GPU servers. We integrated the GPU servers into the cluster and configured the cluster to support MLOps workflows. The cluster deployed the following core components:

• Argo Workflows: General purpose workflow controller for Kubernetes to run ML pipelines. Using Argo Workflows, we could provide researchers with a flexible and scalable workflow system without extra setup.
• GPU Operator: Operator to manage GPU resources in the cluster and enable researchers to utilize all available GPUs for their experiments.
• Kubeflow: Machine learning toolkit for Kubernetes that provides a platform for running, training, and deploying ML models. We used Kubeflow to manage trains and experiments.

We chose these open source project to build license-free environments and easily customizable environments for researchers and devops engineers.

3.2. Implement Data Pipelines with Argo Workflows​

• No, Kubeflow this time: Kubeflow is a machine learning toolkit for Kubernetes. It provides a platform for running, training, and deploying ML models. We used Kubeflow to manage trains and experiments. But, in this case, we used Argo Workflows to manage data pipelines. The reason is that Argo Workflows is more flexible and easier to use for data pipelines. There are some facts of Kubeflow for data pipelines:

  • Instant Static Graph: Kubeflow Pipeline shows a built-in static graph of the pipeline. It provides a visual representation of the pipeline so that researcher can easily understand how the pipeline works.

  • Requiring Pre-Build: Kubeflow requires pre-build data pipelines, which can be time-consuming and inflexible. Jobs of Kubeflow Pipeline require pre-build images for training and inference. So it is not easy to modify the pipeline rapidly. CI/CD pipelines might be helpful, but it still requires extra effort. This fact was a key disadvantage deciding to use Argo Workflows.

• Again, Argo Workflows: Argo Workflows is general purpose workflow controller for cloud native environments. With GPU Operator, the workflow can provide any graph structure of data pipelines. Generally, Argo Workflows are defined in YAML files, but we used Hera, an Argo Workflows SDK that allows us to define workflows in Python scripts. This approach made it easier to create and modify workflows for ML researchers. There are some facts of Argo Workflows for data pipelines:

  • No Pre-Build: Argo Workflows does not require pre-build images for data pipelines. Researchers can define the pipeline in Python scripts and run it directly. This flexibility allows researchers to modify the pipeline easily and run experiments quickly.

    In training stage, you can use pre-built images and add some script to traing models. Or you can use scratch CUDA images and setup the environment in the workflow with Hera SDK. In inference stage, you can choose same way as training stage.

  • No built-in graph. Everything are DIY: Argo Workflows does not provide a built-in graph of the pipeline. Researchers need to define the graph structure in the Python script. This flexibility allows researchers to create custom pipelines tailored to their specific needs. This can be both pro and con. If you need a visualized graph, you can use tools like Grafana or TensorBoard to monitor the pipeline.

  Simply, if you have Argo Workflows in your cluster, you already have a data pipeline system via Argo Workflows and Hera SDK.

3.3. Setup Katib AutoML in Kubeflow​

Katib AutoML provides hyperparameter tuning for ML models and visualized parallel coordinates graph for researchers. We used Katib to automate the process of tuning hyperparameters and selecting the best model for the given dataset. Katib supports external engines like Argo Workflows, so we integrated Katib with Argo Workflows to manage the AutoML process.

The core function of Katib is continuous hyperparameter tuning. Researchers can define the search space of hyperparameters and the objective metric to optimize. Katib will run multiple experiments in parallel, tuning the hyperparameters based on the objective metric. Researchers can monitor the progress of the experiments and select the best model based on the results. And GPU resources can be used 24/7 for AutoML experiments. We ran 800+ experiments for two weeks and used the results for commercialization.

3.4. Setup Training Operator in Kubeflow​

We chose Kubeflow Training Operator to manage distributed training environments. The Training Operator provides a high-level abstraction for training jobs, allowing researchers to utilize all available GPUs in the cluster for parallel training. But in v1.6, the version we used, the Training Operator does not support external engine like Argo Workflows. So we used pure Kubeflow Training Operator to manage distributed training environments.

The Training Operator allows researchers to define training jobs with custom resources, such as the number of GPUs, memory, and CPU cores. We configured a certain training job with 8 GPUs from 2 GPU servers, and the job was distributed across the GPUs for parallel training. This approach allowed researchers to accelerate the training process and utilize all available GPU resources in the cluster without setting up additional infrastructure.

3.5. Deploy JupyterHub for Researchers​

We chose JupyterLab in the first place, but we decided to use JupyterHub for researchers to access the cluster. JupyterHub provides current status of the cluster and researchers can figure how many resources are available in the cluster.

We setup the pre-built notebook images for researchers to use. The images contain the necessary libraries and tools for ML research, such as TensorFlow, PyTorch, and scikit-learn. Researchers can easily access the data lakes with provisioned NFS storage and run experiments without worrying about resource constraints.

4. Summary​

In this project, we built an on-premise MLOps cluster with the latest open-source projects, including Kubeflow, Argo Workflows, and JupyterHub. The cluster provides researchers with a flexible and scalable environment for developing and running ML experiments. With limited GPU resources, the cluster can efficiently utilize all available GPUs in the cluster, reducing underutilized GPU time and optimizing the cost of running ML workloads.

5. Future Works​

In the future, we are planning to enhnace user experience. ML researchers asked two convinent functions:

• Monitoring: Researchers want to monitor the progress of their experiments and visualize the results in real-time. We are planning to integrate Grafana to provide monitoring dashboards for researchers.
• SSH Access for Notebooks: Researchers also want to access Jupiter notebooks via SSH. The custom spawner for JupyterHub will be implemented to provide SSH access for researchers. or we will chatbot generates stack of notebooks directly.

6. Used Skills​

Highlighted skills used in this project:

• Argo Workflows
• Kubeflow
• Kubeflow/Katib
• Kubeflow/Training Operator
• JupyterHub
• Grafana

The Certified Kubernetes Application Developer (CKAD) program was created by the Cloud Native Computing Foundation (CNCF) in collaboration with The Linux Foundation to help develop the Kubernetes ecosystem. The CKAD program allows individuals to demonstrate their competence in designing, building, and deploying cloud-native applications for Kubernetes. Commonly, developers are mainly responsible for making workloads in Kubernetes.

Similar to the CKA exam, the CKAD exam is a hands-on, proctored, performance-based test that requires solving multiple issues from a command line running Kubernetes. The exam focuses on the skills required to be a Kubernetes Application Developer in industry-ready work environments.

2. My Experience​

My CKAD certification 2024. Issued on June 1, 2024. Expires on May 31, 2026. You can verify my certification here.

I took the CKAD exam on May 31, 2024, and passed it successfully. The exam was challenging, and I achieved a score that met the passing requirements. I prepared for the exam by studying the official Kubernetes documentation, taking online courses, and practicing with Kubernetes clusters in KodeKloud.

3. Share to the World​

The Certified Kubernetes Administrator (CKA) program was created by the Cloud Native Computing Foundation (CNCF) in collaboration with The Linux Foundation to help develop the Kubernetes ecosystem. As one of the highest velocity open-source projects, Kubernetes use is exploding. The CKA program allows individuals to demonstrate their competence in a hands-on, command-line environment.

The CKA exam is a hands-on, proctored, performance-based test that requires solving multiple issues from a command line running Kubernetes. The exam focuses on the skills required to be a Kubernetes Administrator in industry-ready work environments.

2. My Experience​

My CKA certification 2024. Issued on March 16, 2024. Expires on March 15, 2027. You can verify my certification here.

I took the CKA exam on March 15, 2024, and passed it successfully. The exam was challenging, but I was able to pass it on my first attempt. I prepared for the exam by studying the official Kubernetes documentation, taking online courses, and practicing with Kubernetes clusters.

To aid my preparation, I enrolled in the Certified Kubernetes Administrator (CKA) with Practice Tests course on Udemy, which helped me understand the core concepts of Kubernetes. I also utilized the KodeKloud platform to practice with Kubernetes clusters and complete hands-on labs. Below is my course completion certificate from KodeKloud:

You can verify this course completion certificate here.

3. Share to the World​

My team and I built a hybrid Kubernetes cluster that combines on-premise and public cloud resources. This architecture helps us overcome the limitations of on-premise resources and optimize the cost of public cloud usage.

Our application consists of two parts: a web application and machine learning (ML) pipelines. The web application is hosted on the public cloud, while the ML workload runs on the on-premise GPU servers. The hybrid cluster architecture provides the following benefits:

• Reliability: Hosting the web applications and databases on the public cloud helps us achieve high availability and reliability 24/7.
• Cost Optimization: By using the on-premise GPU servers for ML workloads, we reduced the cost of public cloud resources by over 50%.
• Scalability: When the on-premise cluster runs out of resources, we can easily scale out to the public cloud. Even if the on-premise cluster is down, the public cloud extends backup nodes to ensure high availability for ML workloads.

The following diagram shows the architecture of the hybrid cluster:

Architecture Diagram of the Hybrid Cluster for DevOps; Users interact with a web application hosted on a public cloud platform like AWS EKS, while the ML pipelines execute on an on-premise cluster. We utilize Argo Workflows as the workflow controller. A metric collector monitors the liveness and availability of the on-premise cluster and triggers scale-out of backup pipelines in the public cloud when necessary. The metrics server is implemented using FastAPI.

2. Problem Statement​

Initially, we deployed a public AWS EKS cluster to support our web application and ML pipeline. However, running ML workloads on the public cloud proved to be cost-ineffective due to the high expense of GPU instances (e.g., g4dn) on AWS EC2, resulting in unacceptable monthly costs amounting to thousands of dollars.

To mitigate these costs, we transitioned to on-premise clusters equipped with GPU servers for our ML pipelines. To facilitate communication between the public cloud and our on-premise clusters, we needed to expose endpoints on the on-premise cluster.

A metric collector in the public cloud was set up to monitor the liveness and availability of the on-premise cluster. In the event of the on-premise cluster going down or lagging, the metric collector would automatically scale out public cloud resources to maintain high availability of the ML pipelines.

3. Methods​

To achieve the architecture described above, we implemented the following steps:

3.1. Public Cluster: AWS EKS​

We deployed a public AWS EKS cluster to host our web application and database. The EKS cluster deployed the following components:

• Web Application: A web application that serves as the front-end and CMS for users.
• Database: PostgreSQL and MySQL databases that store user data and application logs.
• Metric Collector: A metric collector that monitors the liveness and pipeline availability of the on-premise and public clusters.
• Backup ML Pipelines: Backup pipelines that are automatically scaled out in the event the on-premise cluster is not available.

3.2. On-Premise Cluster: Bare-Metal Kubernetes​

We deployed on-premise bare-metal Kubernetes clusters with GPU servers. The on-premise cluster deployed the following components:

• ML Pipelines: Machine learning pipelines that process data and train models. The pipelines run on GPU servers to accelerate the training process. We used Argo Workflows as a pipeline controller.
• Metrics Server: A metrics server that exposes the liveness and availability of the on-premise cluster to the public cloud. We used FastAPI and Prometheus SDK to implement the metrics server.

3.3. Scheduling ML Pipelines​

If the on-premise cluster is down or lagging, the metric collector in the public cloud will automatically scale out the backup ML pipelines to ensure high availability of the ML workloads. The metric collector will also scale in the backup pipelines when the on-premise cluster is back online.

4. Summary​

This is a case study of a loosely-coupled hybrid cluster that combines on-premise and public cloud resources. The architecture provides high availability, cost optimization, and scalability for our application.

We reduced the cost of public cloud resources by over 50% by running ML workloads on on-premise GPU servers. The public cloud provides high availability and reliability for our web application and database.

The metric collector alerts us when the on-premise cluster is unavailable and automatically scales out the backup pipelines to ensure high availability of the ML workloads. This allows us to adjust the scale of the public cloud resources based on the availability of the on-premise cluster.

5. Future Works​

Recently, we noticed that there are use cases utilizing AWS EKS Anywhere to combine on-premise and public nodes in a single EKS cluster. We are planning to proof the concept and migrate our on-premise cluster to EKS Anywhere to simplify the management of the hybrid cluster.

6. Used Skills​

Highlighted skills used in this project:

• AWS EKS
• Kubernetes
• Kubespray
• Argo Workflows
• FastAPI
• Prometheus SDK