Job Description
Responsibilities:

• Build data streams to ingest, load, transform, group, logically join and assemble data ready for data analysis / analytics/ reporting , build data streams to ingest, load, transform, group, logically join and assemble data ready for data analysis / analytics/ reporting.


• Pipeline data using Cloud: Databricks , AWS Big Data Services etc.


• Responsible to write Pyspark code using DataBricks to connect databases, AWS services to transform data.


• Design/Implement QA framework within the Data-Lake. Design and implement test strategies, write test cases, design/implement test automation


• Responsible for maintaining integrity between Data Lake, databases.


• Work on Data Lake/Delta Lake Data Pipeline to take data across from the source all the way to the consumption layer.


• Maintain knowledge and proficiency of current and upcoming hardware/software technologies. Mentor junior staff in ramping up analytical and technical skills.


• SSAS + ETL and Data Engineering experience rather than just Data Engineering skill

• A bachelor's degree from an accredited college in Computer Science or equivalent.


• Strong knowledge of Databricks Data Lake/Delta Lake developments


• Strong knowledge of AWS data related services (DMS, Glue, EMR, S3, Athena, Lambda, Redshift, DynamoDB, KMS).


• Strong knowledge of Python and Pyspark (Hive).


• Strong database/relational/non-relational concepts required.


• Strong analytical and problem solving skills.


• Databricks, oracle, SQL , Hadoop, Kafka, Spark, Scala (preferred).


• Must have 5+ years of Data Systems/Warehouse/Lakes/Equivalent system experience with multiple OS platforms


• Must have 3 + years of Python and Pyspark.


• Must have 3 + years of AWS or alternative cloud systems.


• Must possess or develop business knowledge of how customer transactions reflect the business logic that drive the existing or future code.


• Must possess or develop ability to converse with the business, development, operations, carriers, vendors, etc.


• Strong experience of different architectural components comprising the middle-ware is required.


• 1+ year of Matallion/IBM Data Stage or equivalent ETL tool.