Data Engineer | Enterprise Solutions Delivery
Snoqualmie,WA / Dallas,TX / Overland Park, Kansas
Contract
ID: 133248
Enterprise Solutions Delivery develops innovative closed-loop analytics solutions, enabling insights-driven decision making across the enterprise. We collaborate with multiple functional areas and impact every element of our business: corporate, security, networks, retail, IT, and others. We are looking for a result-oriented engineer to be part of the team, building scalable systems that can handle big data solutions, building and deploying components for container technologies such as Docker, Kubernetes. Will be taking ownership of performance / scalability tuning and building technical infrastructure on-prem and Azure. You will need to understand the business objectives, think like an end user, and know that 90% done is only half done. You love beautiful, simple user interfaces, and you constantly wonder what you could have done to simplify your last project. You are passionate about e2e architecture and database design, focusing on flexibility and scalability. You prioritize API design, as this is necessary to build scalable, future-proof products. You will work closely with data scientists / ML engineers, cloud, and full stack experts to build advanced analytics platform in a hybrid environment integrating on-prem assets, Azure infrastructure and services.
Essential Functions • You will be responsible for collecting and transforming large amounts of data (structured and un[1]structured) into a format that can be used by Data Scientists for model training • Create and maintain optimal data pipeline architecture for complex business requirements, to integrate the real time data streaming with ML model serving • Support building custom images for ML environments including python packages, pickled models. • The role will require extensive experience and knowledge with Apache Spark, Python, Scala, and SQL • Participate in technical design discussions, influencing the team in strategic decisions • Work with data scientists / ML engineers, software developers and interfacing internal customers to translate their requirements to features and develop those solutions • Responsible for other duties/projects as assigned by business management / leadership.
Qualifications
Minimum Required - 5+ years' experience with relevant tools and technologies (Spark, PySpark, Python, Kafka, DScala, SQL) - 3+ years’ experience in building highly distributed pipelines, stream processing solutions, CICD, monitoring and alerting - 3+ years’ experience with data orchestration tools – Airflow/Celery/Apache Spark/Dask - Working experience on cloud technologies (Azure preferred) - Advanced knowledge of performance tuning, scalability, system architecture and engineering best practices with spark clusters - Experience in building end-to-end ETL pipelines on-prem - Strong Linux experience
Desired • Knowledge on Docker, Kubernetes, Databricks or related technologies • Splunk knowledge is a Plus • Graph database experience • Experience in deploying ML models • Shell and YAML scripting experience • Ability to work cross functionally with engineering and operations teams • Bachelor's Degree, preferably from a science or engineering background.