Back to results

Bybit

Senior Principal Site Reliability Engineer (Chaos Engineering)

Abu Dhabi, UAE; Hong Kong SAR; Kuala Lumpur, Malaysia

Experience
8+ years
Experience level
Senior
Job type
Full-Time
Workplace
Onsite

About the role

About Us

Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem across trading, payments, wealth management, custody, institutional services, and Web3 - connecting users to the future of digital finance.

Our core values define how we build. We listen, care and improve to create products and experiences that put users first. Backed by a global team of ambitious builders, problem-solvers, and innovators, we foster a high-performance and fast-moving environment where talent is empowered to drive real impact at the global scale. Supported by 24/7 multilingual customer service and a strong commitment to innovation, we are shaping the future of finance through technology, collaboration, and bold execution.

Today, Bybit is recognized as one of the most trusted and transparent platforms in the digital asset industry, continuing to expand its global presence while building the infrastructure for the next generation of financial services.

About This Role

As the Core Architect of Bybit’s Chaos Engineering Platform, you will be responsible for designing and building an enterprise-grade chaos engineering platform from the ground up (0 to 1), enabling fault injection and resilience validation in production environments across 2,000+ microservices.

Your mission goes beyond simply building a platform. You will drive continuous system resilience validation, with the ultimate goal of reducing the occurrence of P0/P-1 severity incidents by over 50%.

Core Responsibilities

  • •
  • Chaos Engineering Platform Architecture & Development (50%)
  • •
  • Design and build an enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment (testnet/mainnet) deployments
  • •

Core capability development

•

Fault Injection Engine: Pod-level / Node-level / AZ-level fault simulation, network latency / packet loss / partition, dependency timeout / error injection

•

Production Safety Assurance: Blast radius control, one-click Kill Switch, automatic rollback, real-time impact monitoring

•

Traffic Isolation: Experiment traffic tagging and isolation to ensure fault injection does not impact real users

•

Fault Isolation: Precise impact scoping at service / cluster / AZ granularity

•

Design experiment orchestration capabilities supporting complex fault scenario composition (e.g., simultaneous network latency + downstream timeout + cache invalidation)

•

Deep integration with existing monitoring, alerting, and SLO systems to achieve an automated closed loop: inject fault → observe impact → determine pass/fail

•

Production Resilience Validation Framework (30%)

•

Define safety standards and approval workflows for mainnet fault injection

•

Design and drive routine chaos experiments

•

Daily patrol-level experiments: Low-risk experiments executed automatically on a daily/weekly basis

•

Periodic validation experiments: Monthly/quarterly resilience verification of critical paths

•

Large-scale drills: Cross-AZ / cross-region disaster recovery failover validation

•

Establish a resilience scoring system to quantify system health based on experiment results

•

Deliver improvement recommendations and drive business teams to remediate identified weaknesses

•

Technology Selection & Team Enablement (20%)

•

Evaluate and select the technology foundation (Chaos Mesh / Litmus / custom components - hybrid strategy)

•

Develop chaos engineering best practices and playbooks to enable SRE teams and application developers

•

Mentor and grow the team (2–3 engineers) in chaos engineering capabilities

•

Stay current with industry developments and introduce cutting-edge practices (e.g., AI-driven fault scenario discovery)

Key Requirements

Must-Have

•

8+ years of backend / infrastructure engineering experience, with 3+ years dedicated to chaos engineering or stability engineering

•

Hands-on experience with large-scale fault injection in production environments (not just test environments), with deep understanding of production safety constraints

•

Expert-level proficiency in Kubernetes fault injection (Chaos Mesh / Litmus / custom solutions), familiar with CRD / Operator development

•

Proficient in at least one backend language (Go preferred), with platform-level system architecture design capability

•

Deep understanding of distributed system failure modes (network partitions, split-brain, cascading failures, data inconsistency, etc.)

•

Familiarity with observability tech stack (Prometheus / Grafana / Thanos / OpenTelemetry)

•

Excellent technical documentation and solution design skills

Nice-to-Have

•

Experience in financial / trading system stability (understanding of transaction consistency and fund safety constraints)

•

Experience building SLO / Error Budget frameworks

•

Experience building automated fault recovery (self-healing) systems

•

Familiarity with AWS infrastructure (EC2 / EKS / Multi-AZ / Multi-Region)

•

Knowledge of Netflix Chaos Engineering / AWS Fault Injection Simulator / Gremlin

•

Open-source community contributions (Chaos Mesh / Litmus or similar projects)

Soft Skills

  • •
  • Ability to balance "safety" and "validation depth" - not afraid of production injection, while maintaining strict risk control
  • •
  • Strong cross-team collaboration and influence - chaos engineering requires buy-in from business teams; this role demands persuasion skills
  • •
  • Self-driven, capable of independently planning and executing in ambiguous situations
  • 岗位定位
  • 作为Bybit混沌工程平台的核心架构师,负责从0到1构建企业级混沌工程平台,支撑2000+微服务在生产环境的故障注入和韧性验证能力。你的目标不是"搭个平台",而是通过持续验证系统韧性,将P0/P-1级故障发生率降低50%+。
  • 核心职责
  • •
  • 混沌工程平台架构设计与建设(50%)
  • •
  • 设计并构建企业级混沌工程平台,支持多集群(K8s + EC2混合)、多区域、多环境(testnet/mainnet)
  • •
  • 核心能力建设:
  • •
  • 故障注入引擎: Pod级/Node级/AZ级故障模拟,网络延迟/丢包/分区,依赖超时/错误注入
  • •
  • 生产环境安全保障: 爆炸半径控制、一键停止(Kill Switch)、自动回滚、影响范围实时监控
  • •
  • 流量隔离: 实验流量标记与隔离,确保故障注入不影响真实用户
  • •
  • 故障隔离: 按服务/集群/AZ粒度精准控制影响范围
  • •
  • 设计实验编排能力:支持复杂故障场景的组合编排(如:同时注入网络延迟+下游超时+缓存失效)
  • •
  • 与现有监控、告警、SLO体系深度集成,实现"注入故障→观测影响→判定通过/失败"的自动化闭环
  • •
  • 生产环境韧性验证体系(30%)
  • •
  • 制定主网故障注入的安全规范和审批流程
  • •
  • 设计和推动常态化混沌实验:
  • •
  • 日常巡检级实验(每日/每周自动执行低风险实验)
  • •
  • 定期验证级实验(月度/季度核心链路韧性验证)
  • •
  • 大规模演练(跨AZ/跨Region容灾切换验证)
  • •
  • 建立韧性评分体系:基于实验结果量化系统健康度
  • •
  • 输出改进建议并推动业务方修复发现的弱点
  • •
  • 技术方案选型与团队赋能(20%)
  • •
  • 评估并选择技术底座(Chaos Mesh / Litmus / 自研组件的组合策略)
  • •
  • 制定混沌工程最佳实践和Playbook,赋能SRE团队和业务研发
  • •
  • 培养团队(2-3人)的混沌工程能力
  • •
  • 与业界保持同步,引入前沿实践(如:AI驱动的故障场景发现)
  • 任职要求
  • 必需条件:
  • •
  • 8年+后端/基础设施开发经验,3年+混沌工程或稳定性工程专项经验
  • •
  • 有生产环境大规模故障注入的实操经验(非仅测试环境),深刻理解生产安全约束
  • •
  • 精通Kubernetes故障注入(Chaos Mesh/Litmus/自研方案),熟悉CRD/Operator开发
  • •
  • 精通至少一种后端语言(Go优先),具备平台级系统架构设计能力
  • •
  • 深入理解分布式系统故障模式(网络分区、脑裂、级联故障、数据不一致等)
  • •
  • 熟悉可观测性技术栈(Prometheus/Grafana/Thanos/OpenTelemetry)
  • •
  • 优秀的技术文档和方案设计能力
  • 加分项:
  • •
  • 有金融/交易系统稳定性保障经验(理解交易一致性、资金安全约束)
  • •
  • 有SLO/Error Budget体系建设经验
  • •
  • 有自动化故障恢复(Self-healing)系统建设经验
  • •
  • 熟悉AWS基础设施(EC2/EKS/Multi-AZ/Multi-Region)
  • •
  • 了解Netflix Chaos Engineering / AWS Fault Injection Simulator / Gremlin
  • •
  • 有开源社区贡献经验(Chaos Mesh/Litmus等项目)
  • 软性要求:
  • •
  • 能在"安全"和"验证深度"之间做好平衡 - - 既不畏惧生产注入,也能严格控制风险
  • •
  • 有推动跨团队协作的能力 - - 混沌工程需要业务方配合,这个岗位需要说服力
  • •
  • 自驱型,能在方向模糊时独立规划和推进
  • Why Join Us
  • At Bybit, we are committed to fostering a supportive and enriching work environment.

Our benefits include

  • Study Growth Fund: We support your professional development and continuous learning.
  • Internal Events: Participate in regular team-building activities, workshops, and events designed to promote collaboration and innovation.
  • Global Collaboration: Be part of a diverse, international team, working alongside colleagues from around the world.
  • Career Advancement: Access opportunities for growth and advancement within a rapidly expanding global company.
  • Internal Mobility: Grow with us- Your long-term development is important to us. We offer internal job opportunities to help build your career path.

Job details

Make your next move

You can preview 30 jobs. Sign in to search, filter, save and browse the full collection.

Continue to sign in

We request your basic Google profile, without the email permission. No marketing emails. Privacy policy