Spark Сonsultant (Part-time)

Why does MWDN rock?

MWDN connects exceptional tech talent with leading companies across Israel, the USA, Great Britain, and Western Europe. We aim to ensure our employees enjoy a rewarding and secure experience while collaborating with prestigious international clients.

Here’s what you can expect when you get employed by MWDN:

• Security first. We vet our clients to eliminate risks, ensuring reliability and timely payments for your hard work—no fraud or unforeseen events here!
• Career support. If a match isn't right, we're here for you. We actively assist our employees in finding new opportunities that fit their skills and aspirations.
• Legal assistance. We provide guidance on legal matters (e.g., opening and administering your private entrepreneur account, taxes, army enrollment, etc.).
• Professional development. We offer English courses and other engaging activities, including team-building events.

Why choose us? MWDN is ranked among the top 5 IT employers in our region by DOU, and we pride ourselves on our transparency and commitment to our team. Curious to learn more? Discover what our employees say about us in their DOU reviews: MWDN reviews.

What is your new project?

Domain: Web data

Location: Florida (USA)

Company size: 11-50 employees

Founded in: 2020

What makes this project exciting?

Our client is an innovative startup that transforms how businesses access and utilize data. Using over 1 million global IPs and a database of more than 650 million verified profiles, they empower companies to make data-driven decisions with precision and ease.

They are looking for creative, driven individuals eager to push boundaries and contribute to a rapidly expanding client base. If you’re passionate about data science, analytics, or tech, this is your chance to join a dynamic team that delivers impactful, real-time data solutions. Their collaborative, innovative culture provides the perfect environment for those looking to grow alongside industry experts.

What makes you a great fit

Deep Spark & PySpark Knowledge:

• Spark Core: Thorough understanding of Spark's architecture (driver, executors, DAG), RDDs, transformations, actions, shuffle operations, and performance tuning.
• PySpark: Proficiency in PySpark API for data manipulation, including DataFrame and SQL operations, UDFs (user-defined functions), and Spark SQL optimizations.
• Spark Streaming: Knowledge of Spark Streaming for real-time data processing, including windowing, stateful operations, and fault tolerance.

Delta Lake Expertise:

• Delta Lake Fundamentals: Solid understanding of Delta Lake's ACID properties, data versioning, time travel, and schema enforcement.
• Delta Lake Operations: Experience with creating, updating, deleting, and querying Delta Lake tables.
• Performance Optimization: Ability to optimize Delta Lake performance, including data partitioning, Z-ordering, and file compaction.

S3 Proficiency:

• S3 Fundamentals: Understanding of S3's object storage model, buckets, keys, and access control.
• S3 Interaction with Spark: Expertise in reading and writing data from/to S3 using PySpark, including handling different file formats (Parquet, CSV, JSON) and compression techniques.
• S3 Optimization: Knowledge of S3 performance optimization techniques, such as data partitioning and using appropriate storage classes.

Bare Metal Environment --> Advantage! 

• Cluster Management: Experience with setting up and managing Spark clusters on bare-metal servers, including resource allocation, configuration, and monitoring.
• Hadoop Ecosystem: Familiarity with Hadoop ecosystem components (HDFS, YARN) and their interaction with Spark in a bare-metal environment.
• Troubleshooting: Ability to diagnose and resolve issues related to Spark, Delta Lake, and S3 in a bare-metal setup.

Soft Skills & Collaboration

• Requirements Gathering: Ability to effectively gather and translate requirements from data analysts and developers, understanding their needs and constraints.
• Communication: Excellent communication skills to explain technical concepts to both technical and non-technical stakeholders.
• Collaboration: Strong teamwork skills to collaborate with data analysts, developers, and other team members to deliver successful projects.
• Problem Solving: Analytical and problem-solving skills to identify and resolve data-related challenges and optimize Spark applications.

Domain Knowledge

• Data Warehousing and ETL: Understanding of data warehousing concepts, ETL (Extract, Transform, Load) processes, and data modeling.
• Data Governance: Knowledge of data quality, data lineage, and data security best practices.
• Business Acumen: Ability to understand business requirements and translate them into technical solutions.

Key Advantages of these skills in a bare-metal environment:

• Cost Optimization: Bare-metal can be more cost-effective than cloud for certain workloads, and these skills enable efficient resource utilization.
• Control & Customization: Bare-metal provides greater control over the environment, allowing for customization and optimization specific to your needs.
• Performance Tuning: A deep understanding of the entire stack allows for fine-grained performance tuning.

Your day-to-day in this position

This comprehensive skill set empowers a Spark expert to effectively design, develop, and optimize data-intensive applications using Delta Lake, S3, and PySpark in a bare-metal environment while collaborating seamlessly with various stakeholders.

Why work with us?

• People-oriented management without bureaucracy.
• The friendly climate inside the company is confirmed by the frequent comeback of previous employees.
• Flexible working schedule.
• Full financial and legal support for private entrepreneurs.
• Free English classes with native speakers or with Ukrainian teachers (for your choice).
• Dedicated HR.