-
Job Description
Spark Architect / SME
Contract Role- 6 months to begin with & its extendable
Location: Leeds, UK (min 3 days onsite)
- Context: Legacy ETL code for example DataStage is being refactored into PySpark using Prophecy low-code no-code and available converters.
- Converted code is causing failures/performance issues.
Skills:
- Spark Architecture – component understanding around Spark Data Integration (PySpark, scripting, variable setting etc.), Spark SQL, Spark Explain plans.
- Spark SME – Be able to analyse Spark code failures through Spark Plans and make correcting recommendations.
- Spark SME – Be able to review PySpark and Spark SQL jobs and make performance improvement recommendations.
- Spark – SME Be able to understand Data Frames / Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations.
- Monitoring – Be able to monitor Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures.
- Cloudera (CDP) – Knowledge of understanding how Cloudera Spark is set up and how the run time libraries are used by PySpark code.
- Prophecy – High level understanding of Low-Code No-Code prophecy set up and its use to generate PySpark code.