Must have:

• Strong knowledge and working experience with SQL, Databricks
  


• String understanding about the underlying architecture of data components (Access control/Configurations/ Performance blockers)


• Working experience with Azure Data Factory, Azure Data Lake Store, Azure Synapse , Databricks
  


• Hands on experience with Pyspark, Dataframe API, SQL API


• Having very good understanding on Spark, Hadoop Map Reduce framework


• Hands on experience with Data warehousing


• Having prior experience in performance tuning for Big data work load (Spark or Map-reduce framework)


• Having prior experience of handling structure and unstructured data


• Strong knowledge in release management of  azure components


• Must be able to appreciate the process control, change control.


• Must be able to provide platform solution/guidance about various cloud services, database technologies and pipeline structures of Ingestion to consumption layers.

Nice To Have:

• Awareness of data security, DMZ, Encryption mechanism, VPC  etc


• Hands on experience in building devOPS pipeline

Awareness about DataOPS

Synapse, Databricks, Streaming Analytics, Cosmos DB

• 
  Business Understanding
  
  
  1. Understand and define data vison (strategic data requirements) based on business requirement and translate the business requirement into a technology requirement.
  
  
  2. Provide high-level integrated designs to meet business requirements
  
  
  
  


• 
  ELT(Data Processing)
  
  
  
  1. Understand various data sources and source data structure
  
  
  2. Understand data processing requirement – like real time, near real time, batch
  
  
  3. Understand read pattern, write pattern, usage of data, size of datasets to select right data processing tools
  
  
  4. Understand scalability, reliability, maintainability and recoverability   requirement
  
  
  5. Define source to target dataflow and ensure data security in the dataflow diagram by ensuring following – data is secure at rest, at motion and while in use
  
  
  6. Evaluate schemas of various data sources and select right target data format (for analytical or transaction processing) to enable vectorized processing
  
  
  7. Select right compute and storage infrastructure to process data. Perform POC to evaluate tools if necessary.
  
  
  8. Define framework, standards, policies and best practices for data processing
  
  
  9. Collaborate with various stake holders to get feedback on data processing
  
  
  
  


• 
  Data Lake
  
  
  
  1. Define folder structure based on various subject areas and underlying modules
  
  
  2. Define data archival strategy(data lifecycle) based on business requirement
  
  
  3. Classify data according to its sensitivity and define access control
  
  
  4. Define standards, policies and best practices to store and organize data into data lake
  
  
  5. Get feedback from various stake holders/users of data lake store
  
  
  
  

• 
  EDW
  
  
  1. Understand how data should be organized and managed
  
  
  2. Work with data modeler to define data models which meets data vision of the organization
  
  
  3. Identify the EDW solutions which matches the scalability, reliability, recoverability and maintainability needs. Perform POC if necessary to select right tools.
  
  
  4. Classify data according to its sensitivity and define access control
  
  
  5. Define data mapping specifications, data lineage
  
  
  6. Define standards, policies and best practices
  
  
  7. Get continuous feedback from various stake holders/users of EDW solutions to make sure it matches user expectations
  
  
  
  


• 
  Analytical
  
  
  
  1. Understand business use cases for analytics. Analyze and prioritize use cases based on data availability, schedules and current environment.
  
  
  2. Understand usability, security and stability requirement to select right tools
  
  
  
  

Select technologies and tools for analytics by considering current and future needs. Perform POC if require to select right tools