Lead Data Engineer - Sourcing at Firmable

India

<p>Firmable is the market-leading B2B sales intelligence platform in Asia-Pacific. Our competitive moat is our data: the deepest, most localised company and people dataset in every market we operate in. We've proven the model in ANZ. Now we're scaling it across Southeast Asia and the US.</p><p>Every record Firmable sells starts in a sourcing pipeline. This role architects that pipeline.</p><h3>The Role</h3><p><strong>Lead Data Engineer: Sourcing</strong> designs and owns the end-to-end extraction and ETL pipeline that turns unstructured web data into the world's most accurate B2B dataset across 13 markets. You set the architectural standards: extractor patterns, proxy strategy, LLM infrastructure as production systems, agentic escalation workflows, and cost-aware orchestration.</p><p>This is not a pipeline maintenance or incremental optimisation role. You're architecting the framework other engineers ship into, building for the hardest extraction problems (anti-bot defences, JS-heavy rendering, schema drift), and owning the sourcing layer end to end.</p><p><strong>~80% hands-on architecture and reference implementation: designing extractor frameworks, shipping the hardest extractors, building agentic pipelines, writing eval harnesses and labelled datasets. ~20% cross-functional voice: technical input on sourcing decisions with data platform, product, and analytics; setting standards for coverage, schema, and accuracy.</strong></p><h3>What You'll Own</h3><p><strong>Sourcing and ETL architecture</strong></p><ul><li><p>End-to-end pipeline design: extraction, normalisation, deduplication, validation, and load; plus cost, performance, and reliability of the whole layer</p></li><li><p>Sourcing layer standards: coverage, accuracy, schema design; the rule-vs-LLM standard codified so the team applies it without you</p></li><li><p>Cost and performance optimisation: cost ceilings with the token math behind them, incremental processing, recovery design, scheduling</p></li></ul><p><strong>Extraction systems and frameworks</strong></p><ul><li><p>Extractor framework: patterns, abstractions, and tooling other engineers ship into; new extractors are fast to build and reliable to run</p></li><li><p>Hard-source extractors: anti-bot defences, JS-heavy rendering, schema drift, low-quality structure; plus the proxy and IP rotation strategy behind them</p></li><li><p>Agentic extraction pipelines: rule-based triage, LLM escalation, structured-output validation, retries, human-review queues</p></li></ul><p><strong>LLM infrastructure and observability</strong></p><ul><li><p>LLMs as production systems: versioned prompts, labelled eval sets, measured precision and recall, prompt versioning you can roll back, judges debugged on real data</p></li><li><p>Eval and observability scaffolding: eval frameworks, prompt versioning, traces, drift detection when a vendor silently updates a model</p></li><li><p>Model-choice playbook: cheap models for classification, stronger models for nuanced extraction, frontier models for hard edge cases; revised as model economics shift</p></li></ul><p><strong>Skills library and orchestration</strong></p><ul><li><p>Skills library: versioned <a target="_blank" href="http://SKILL.md">SKILL.md</a> specs for recurring extraction patterns, invocable by engineers and agents alike</p></li><li><p>Orchestration: Airflow or equivalent patterns that scale with data volumes and source counts; dependency management, recovery, cost-aware scheduling</p></li></ul><h3>What We're Looking For</h3><p><strong>Must Haves</strong></p><ul><li><p><strong>6+ years shipping production extraction, ETL, or data pipeline systems</strong> in business-critical environments; deep web extraction at scale with experience designing around anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery</p></li><li><p><strong>Shipped LLMs inside extraction pipelines as production systems:</strong> structured outputs, versioned prompts, labelled eval sets, logged traces, judges debugged on precision/recall, drift detected on vendor model updates. You can show the repo.</p></li><li><p><strong>Built agents and tool-calling pipelines:</strong> architected agent workflows, written <a target="_blank" href="http://SKILL.md">SKILL.md</a> specs others depend on, run tool-calling systems in production</p></li><li><p><strong>Sharp judgement on rules vs. LLMs:</strong> reach for deterministic logic first, defend the call either way, and have codified the standard for a team</p></li><li><p><strong>Expert Python:</strong> production-grade, performance-aware, comfortable with concurrency and scale; plus advanced SQL for complex transformations and performance optimisation</p></li><li><p><strong>Extensive Airflow or equivalent:</strong> orchestration, dependency management, recovery patterns, cost optimisation at production scale</p></li><li><p><strong>Shipped with agentic IDEs:</strong> Claude Code, Cursor, or equivalent as your default mode, with shipped extraction systems to show for it</p></li><li><p><strong>Architecture judgement and a product mindset:</strong> when to refactor, optimise, ship, or start again; you care about coverage and accuracy landing with customers, not uptime metrics</p></li><li><p><strong>You live and breathe AI tools.</strong> Structured outputs, evals, traces, and LLM tracing (Logfire, OpenTelemetry, similar) as your default way of working, not a productivity experiment. In 2026, this is how data engineers build extraction at scale and you need to already be doing it.</p></li></ul><p><strong>Highly Valued</strong></p><ul><li><p>Cloud data platforms: Snowflake, Redshift; AWS for pipeline deployment (Lambda, S3, ECS, Glue)</p></li><li><p>Vector databases, embeddings, or retrieval patterns for matching and deduplication</p></li><li><p>Eval frameworks like Braintrust, Promptfoo, or Inspect; LLM tracing tooling</p></li><li><p>Data quality frameworks with automated testing and anomaly detection at scale</p></li><li><p>B2B data: firmographics, people data, company registries across markets</p></li><li><p>Data privacy and compliance: GDPR, CCPA</p></li><li><p>Startup or scaleup experience where you shipped fast and owned outcomes end to end</p></li></ul><h3>The Environment</h3><p>Firmable runs lean and ships fast: small senior teams, no layers, minimal process. Sourcing is a core competitive advantage; this role sets the pace for how fast and accurate the entire data pipeline moves. Weekly releases moving toward daily; no fixed hours, full autonomy on architecture choices.</p><p>We are an <strong>AI-native organisation</strong>. That means AI isn't a tool we reach for: it's the default operating mode. Agentic extraction pipelines, LLM-powered classification with eval sets, structured-output validation, labelled datasets versioned with prompt runs, traces logged from day one with cost and latency baked in. If you're not already working this way, this role isn't right for you.</p><h3>Why This Role</h3><ul><li><p><strong>Own the sourcing engine:</strong> every record Firmable sells starts in the pipeline you architect; your design shapes the accuracy and cost of every customer interaction</p></li><li><p><strong>Greenfield AI-native architecture:</strong> the extractor framework, skills library, eval harnesses, agentic orchestration, and production LLM infrastructure are largely unbuilt; you'll architect them from first principles</p></li><li><p><strong>Frontier technical problems:</strong> agentic extraction at production scale, LLM-as-extractor with full eval coverage, vendor model drift detection, rule-vs-LLM orchestration across 13 markets</p></li><li><p><strong>Small team, massive leverage:</strong> your architecture reaches every Firmable customer, every day; you write the reference implementation daily and own outcomes end to end</p></li><li><p><strong>Real career runway:</strong> this role is a bottleneck role; progress is measured by pipeline speed and accuracy, not tenure</p></li></ul><h3>Who This Role is NOT For</h3><ul><li><p>Anyone looking to optimise an existing pipeline rather than architect from first principles</p></li><li><p>Anyone who treats infrastructure as someone else's problem; sourcing architecture lives in your code every day</p></li><li><p>Anyone who sees LLMs as a parsing shortcut rather than production systems that need eval sets, versioning, and drift detection</p></li><li><p>Anyone uncomfortable shipping with minimal process or without an established playbook to follow</p></li><li><p>Anyone who treats AI as something they'll learn on the job rather than already use daily in infrastructure work</p></li><li><p>Anyone coming from a pure ops or data analytics background without shipped systems engineering at scale</p></li></ul>