At Zyte, we don’t just collect web data—we solve complex web data challenges at scale. We are a globally distributed team that is bold, curious, and dedicated to building innovative ways to deliver clean, reliable web data from across the internet.
- Data is our passion: We unlock access to open web data, empowering organizations—from startups to global enterprises—to power competitive intelligence, analytics, and AI pipelines.
- Remote-first culture: Work from anywhere in the world. With over 260 Zytans across 33 countries, our strength comes from diverse backgrounds, perspectives, and skills.
- Engineering excellence: We love diving deep into complex code bases, evaluating emerging technologies, and pushing technical boundaries. If you thrive on creative problem-solving and reliably delivering production-grade solutions, you will fit right in.
- Established system design framework: We systematically analyze functional and product requirements alongside architectural trade-offs to make informed, sustainable design decisions.
- Real-world distributed systems challenges: Tackle rare engineering problems in custom networking, high-throughput streaming, and large-scale orchestration under real-world operational constraints.
- Strong product ownership: Work on a profitable, market-leading platform with dedicated time and resources allocated for architectural research and strategic feature engineering.
Why This Role?
Zyte API fetches the web through a ladder of routes: plain HTTP, headless browsers, and, for the hardest targets, Edge—our own residential network of real devices. Edge is the tier that must succeed when everything else has failed. Today, a Rust peer on residential Windows PCs long-polls a single command-and-control service (CNC) over mTLS gRPC; CNC schedules fetches onto peers, runs a CONNECT proxy gateway, owns geo-routing, per-site cooldowns, and bandwidth quotas. It must grow from thousands of nodes to tens of millions of requests a day with rock-solid reliability.
The Edge Platform squad owns the control plane and data plane of this residential stack across Windows, Android, and Linux. The next step is building a proper Fleet Manager: a control plane with provider adapters (volunteer PCs, mobile-IP device farms, self-hosted nodes), a reconciler, rollout control, and metrics export sitting beside CNC as the data plane. You will design and operate that system—handling the scheduling, networking, proxying, and reliability work that turns residential devices into infrastructure other squads depend on. You will work directly with the Team Lead, a Principal Engineer, and partner engineering squads across Zyte.
What You’ll Do
- Scale the data plane. Evolve CNC from a single-replica service into a horizontally scalable fetch scheduler: node selection that respects geo, site cooldowns, jail and per-peer quotas while avoiding detectable patterns, at 50–100M requests per day.
- Build the Fleet Manager control plane. Design and ship the inventory, health, configuration and software-rollout plane for the fleet: provider adapters, a reconciler, a rollout controller and an exporter to BigQuery, so a second device provider can be onboarded without bespoke scripts.
- Own the proxy path. Operate and improve the CONNECT proxy gateway and session handling on the edge, and work with the API squad on IP reputation, proxy leases and request scheduling so Edge slots cleanly into the platform’s proxy plane.
- Engineer the networking. mTLS gRPC between control plane and peers, long-poll and streaming semantics over unreliable residential links, NAT and carrier-grade NAT behaviour, and bandwidth accounting that is correct under contention.
- Make it reliable. Define SLOs for availability, success and latency of the Edge route; build the metrics on peer count, drop-off and bandwidth that hold us and our supply partners accountable; run blue/green rollouts of CNC and peers safely.
- Share production ownership. Take part in the squad’s on-call rotation, lead incident post-mortems and turn operational lessons into platform improvements.
- Raise the bar. Write RFDs for scheduling, fleet management and failover; review designs across the squad; and bring distributed-systems rigour (consistency, backpressure, idempotency, blast radius) to a young codebase.
Requirements
Who You Are
- 5+ years of production Rust experience, shipping and operating services you own end to end.
- Distributed systems: you have designed and run schedulers, control planes or stateful services at scale, and can reason about consistency, partitions, backpressure and idempotency without a whiteboard.
- Deep networking: TCP, TLS and HTTP at the handshake and packet level, gRPC and long-lived connections across NAT and lossy links, and what a client looks like on the wire.
- Proxy expertise: you have built or operated forward proxies, proxy gateways or residential/ISP proxy infrastructure, and understand IP reputation, session stickiness and pool health.
- Ops and reliability experience: SLOs and error budgets, observability you built yourself, production incidents you led, and comfort with being on call.
- Cross-platform awareness: comfortable with Rust running on Windows, Linux and Android targets, and with FFI boundaries when the data plane needs them.
- Async communication: strong written communication for a globally distributed, remote-first environment.
- Curious problem solver: you like finding out why a system behaves the way it does under load, and fixing the cause rather than the symptom.
Our Tech Stack
- Languages: Rust (tokio, tonic), C++ (Cronet) at the peer boundary; Python and Java in the services we integrate with.
- Protocols: gRPC and protobuf over mTLS, HTTP CONNECT proxying, the Chromium network stack and our own browserless HTTP client.
- Infrastructure: GCP and Kubernetes for the control plane, Terraform, blue/green deployments; multi-cloud (OCI, GCP) and data centres (Hetzner, Servers.com) across the wider platform.
- Data & Messaging: Valkey, BigQuery, Confluent Kafka.
- Monitoring & Observability: Prometheus, Grafana, Loki, OpenTelemetry.
Bonus Points
- Mobile or desktop Rust: experience shipping Rust on Android, Windows or macOS, or binding Rust to C/C++ via FFI.
- Fleet or device management: reconcilers, rollout controllers, IoT or volunteer-computing fleets.
- Anti-bot and fingerprinting: understanding of how TLS, HTTP and IP signals are used to classify traffic.
- Web scraping & crawling experience: familiarity with the technical challenges of large-scale web data extraction.
- Open source contributions: history of contributing to Rust, networking or distributed-systems open-source projects.
Why Join Us?
- Impact at scale: Build the residential network tier behind web-scale data pipelines for top-tier global enterprises.
- Autonomy and flexibility: Enjoy a genuine remote-first culture with flexible working hours and high organizational trust.
- Continuous innovation: Solve emerging, complex technical challenges as AI and web data extraction rapidly evolve.
- Global community: Collaborate with a passionate, diverse team of distributed systems engineers and data specialists across the world.
If you are a Rust engineer who wants to turn tens of thousands of residential devices into dependable infrastructure, we would love to hear from you. Join us in building the fleet behind web data extraction at Zyte.
Benefits
By joining the Zyte team, you will:
- Become part of a self-motivated, progressive, multi-cultural team
- Have the freedom and flexibility to work from where you do your best work
- Attend conferences and meet with team members from across the globe
- Work with cutting-edge open source technologies and tools
New Zealand