Data Engineer

Key Responsibilities

Core Data & AI Pipeline Development

• Build, maintain, and optimize batch and streaming data pipelines to support analytics and AI workloads.
• Ingest structured, semi structured, and unstructured datasets from APIs, databases, SaaS systems, streaming feeds, and file based sources.
• Transform, clean, enrich, and standardize data using Dataflow (Apache Beam), Dataproc (Spark), BigQuery SQL, and Python.
• Deliver high quality curated datasets into BigQuery for analytics, reporting, and machine learning training.
• Build and maintain ML ready feature pipelines supporting Data Scientists and ML Engineers.

Data Quality, Governance & Operations

• Implement data quality checks, schema validation, and automated testing within pipelines.
• Monitor pipeline health and apply observability best practices using Cloud Monitoring and Cloud Logging.
• Apply governance, security, and compliance standards including IAM roles, encryption, data masking, and auditing.
• Enforce schema evolution policies, metadata management, and lineage tracking using Dataplex/Data Catalog.
• Maintain documentation for datasets, transformations, pipeline logic, and operational procedures.

Engineering & Collaboration

• Write efficient, maintainable Python, SQL, Beam, and Spark code.
• Manage ingestion flows using Pub/Sub, GCS, APIs, Datastream, and database connectors.
• Optimize BigQuery tables, partitions, clustering, materialized views, and query performance.
• Implement and maintain DAGs with Cloud Composer (Airflow).
• Troubleshoot pipeline failures, latency issues, and data quality gaps.
• Participate in code reviews, architectural discussions, and agile sprint ceremonies.
• Collaborate with Data Architects, Data Scientists, ML Engineers, and business stakeholders.
• Develop and maintain Infrastructure as Code using Terraform and CI/CD deployment pipelines.

Required Qualifications

• Must be a U.S. Citizen with ability to obtain a Public Trust clearance.
• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, Data Engineering, or related technical field.

• 3–6+ years of hands on experience in data engineering or a similar technical field.
• Minimum three years of experience leading technical teams to achieve outcomes.
• Experience developing and implementing technical standards for cloud and on prem environments.
• Proven experience building production data pipelines on cloud platforms, preferably GCP.
• Hands on experience with BigQuery, GCS, Dataflow (Apache Beam), Dataproc (Spark), and Pub/Sub.
• Experience preparing ML ready datasets for model training.
• Strong background in SQL, Python, distributed data processing, and data modeling.
• Experience with governance, security, and compliance frameworks including IAM, encryption, data masking, and auditing.
• Familiarity with the following tool categories (VAEC Operational Tools):
  • Google Cloud Security tools
  • Google Cloud Monitoring & Logging tools
  • Google Cloud Networking
  • Google Storage services

Preferred Experience

• Master’s degree in a technical field.
• Previous experience in Federal Government environments.
• Knowledge of regulated environments such as FedRAMP, HIPAA, PCI, NIST 800 53, and CIS benchmarks.
• Security certifications such as CISSP or CCSP.
• Experience with Vertex AI workflows or comparable ML platforms.
• Familiarity with Dataplex, data governance frameworks, and metadata management.
• Experience with Apache Kafka or other streaming technologies.
• Experience with Datastream for change data capture (CDC).
• Knowledge of regulated industries such as public sector, healthcare, or finance.
• Strong communication skills and the ability to convey complex data concepts clearly.
• Experience with BI tools such as Looker or Looker Studio.
• Experience with third party tools such as Armis, BigFix, CrowdStrike, Tenable Nessus, Turbot, ServiceNow, Dynatrace, Splunk, and more.
• Hands on experience with DevOps tools and methodologies, including Ansible, GitHub, Jira, Terraform, CI/CD, and cloud migration tools.
• Knowledge in ML enablement, feature stores, and ML pipeline patterns.
• Experience with data quality and testing frameworks such as Great Expectations or dbt tests.