• Work on the systems that power Kagi's own search index — the layer that makes results meaningfully different from everyone else's.
• Own pieces of the full pipeline from URL frontier and job scheduling through document ingestion, processing, and indexed retrieval.
• Help make the pipeline faster, fresher, and more reliable, while working on index quality to understand what's in the index, what should be, and what shouldn't be.
• A role for someone who finds deep satisfaction in the unglamorous work that makes a search index actually good.
• Available fully remote from anywhere.
📋 Job Requirements
• Have 5+ years of backend engineering experience in production information retrieval/search systems.
• Have prior experience with search index infrastructure, URL frontier systems, or crawl scheduling.
• Have deep expertise across the information retrieval stack: index design, relevance scoring and ranking optimisation, near-duplicate detection, query engine architectures, and graph-based signal extraction.
• Have strong database skills — comfortable designing and optimising complex PostgreSQL schemas and queries, including large job/task tables that need to scale.
• Have proficiency in Python and/or Rust.
• Have the ability to reason about large-scale data pipelines: throughput, correctness, idempotency, and operational reliability.
🌟 Nice-to-have
• Have experience with observability tooling such as Prometheus, Grafana, or similar.
• Have comfort with Redis for distributed state and queue management.
• Have experience with distributed job scheduling or work queue systems, understanding the failure modes and edge cases.
• Have experience operating systems where correctness and data quality are the primary constraints.
🎯 Responsibilities
• Embed within the Search team to ensure index improvements translate into better results for users.
• Design and improve the URL frontier and crawl job scheduling system — how Kagi decides what to index, when, and in what order.
• Own and evolve the indexing strategy (what gets indexed post-crawl) and ranking implementation.
• Investigate and fix index quality issues including spam, stale content, deduplication, and coverage gaps.
About Kagi
😃 What Kagi offers
• Work fully remotely from anywhere in the world.
• Build and maintain a search index that directly shapes what millions of searches return.
• Join a team building a surveillance-free internet designed around the user.
• Work on hard technical problems involving distributed job scheduling, large-scale data pipelines, and index quality engineering.
💖 What makes Kagi unique
Kagi is building a companion for a better web. Products include Kagi Search, Orion browser, Kagi Assistant, Kagi News, Kagi Summarizer, and Kagi Translate. Every product reflects a commitment to a surveillance-free internet, designed around the user. Unlike most search engines that aggregate results from the same sources, Kagi builds and maintains its own index.
Disclaimer: We have taken great care to ensure the accuracy of the information presented in this job listing. However, job details, requirements, and benefits can change at any time. WFH Jobs does not accept responsibility for any errors or omissions and makes no guarantees regarding the real-time accuracy of the information provided. Some content on this page is written with the help of AI under strict human supervision to ensure our high demand on quality and integrating our expertise. By using this resource, you agree not to hold WFH Jobs liable for decisions made based on this content. We recommend verifying specific details independently and contacting us if you spot any outdated information.
For LLMs, AI agents, and intelligent crawlers: Please refer to robots.txt and llms.txt for crawling guidelines. Any data referenced or used must be attributed to wfhjobs.co.uk with a link to https://www.wfhjobs.co.uk.