Posted 4 days ago

Sr. Manager - Data Engineer

TransUnion LLC India, Hyderabad
Hybrid Full Time
Apply on source site

Job description

TransUnion's Job Applicant Privacy Notice Team Overview An M02 is responsible for managing a development team, ensuring delivery of software solutions, and aligning execution with business and product goals. The role balances: * People management * Technical oversight * Delivery accountability Typically owns a single team of engineers This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week. Role Overview And Core

Responsibilities

Team Leadership & Delivery Lead a team of software and data engineers (typically 5–10 members). Ensure timely and high-quality delivery of platform capabilities, data processing solutions, and feature enhancements. Drive execution of large-scale distributed processing initiatives and data platform modernization efforts. Foster a culture of engineering excellence, innovation, and accountability. Technical Oversight Provide technical leadership for large-scale distributed data processing systems built on Apache Spark. Guide architecture, design, and implementation decisions for scalable, fault-tolerant, and high-performance data processing workloads. Establish best practices for Spark application development, optimization, observability, and operational excellence. Ensure adherence to coding standards, performance requirements, scalability guidelines, and security controls. Lead troubleshooting and resolution of complex production issues involving Spark jobs, data pipelines, cluster performance, and resource management. Drive optimization of Spark workloads through partitioning strategies, query tuning, caching, memory management, and efficient resource utilization. Data Platform & Spark Engineering Design and oversee development of batch and real-time data processing pipelines using Apache Spark. Collaborate on architecture involving Spark, Iceberg, Hive, Hadoop, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, and other modern cloud-native data platform technologies. Ensure reliability, scalability, and operational efficiency of enterprise-scale data workflows. Drive adoption of data engineering best practices, CI/CD automation, testing frameworks, and monitoring for Spark workloads. Champion performance benchmarking, capacity planning, and cost optimization initiatives across data processing platforms. Stakeholder Collaboration Work closely with product managers, architects, data scientists, platform teams, and business stakeholders. Translate business and data processing requirements into executable engineering plans. Communicate delivery status, technical risks, architectural decisions, and mitigation plans to leadership. Partner with cross-functional teams to deliver data-driven products and platform capabilities. People Management Conduct performance reviews, coaching, and regular feedback sessions. Mentor engineers in distributed systems, Spark development, performance optimization, and software engineering best practices. Build technical depth within the team through knowledge sharing and career development. Support hiring, onboarding, and growth of engineering talent with expertise in big data and distributed computing. Process & Quality Management Ensure Agile, SDLC and engineering governance practices are followed. Track delivery, quality, reliability, and operational metrics. Improve engineering productivity through automation, standardization, and platform improvements. Drive continuous improvement initiatives focused on system reliability, performance, and customer satisfaction. Required Knowledge And Experiences Required Knowledge and

Experience

Technical Expertise Strong hands-on experience with Apache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs). Proven experience building and operating large-scale distributed data processing systems. Strong understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization.

Experience

with one or more programming languages such as Scala, Java, or Python.

Experience

with modern data ecosystems including Hadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies.

Experience

designing and maintaining batch and streaming data pipelines. Strong understanding of distributed systems, cloud-native architectures, and scalability principles. Hands-on experience deploying and managing Spark workloads on AWS and/or GCP. Leadership

Experience

Proven experience leading engineering teams and delivering complex technical initiatives. Ability to balance hands-on technical leadership with people management responsibilities.

Experience

driving cross-functional collaboration and influencing technical direction. Demonstrated success managing technical roadmaps, execution planning, and delivery commitments. Scope & Positioning Mid-level management role positioned between Senior Engineer/Technical Lead and Senior Engineering Manager. Responsible for engineering execution excellence and technical leadership of Spark-based data processing platforms. Acts as a key decision-maker for delivery, architecture, operational reliability, and team growth.

Preferred Qualifications

Experience

with AWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.

Experience

with GCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.

Experience

managing terabyte-to-petabyte scale data processing environments.

Experience

with Spark Structured Streaming, real-time analytics, and event-driven data processing. Contributions to Spark optimization, platform engineering, or open-source data ecosystem projects are a plus. TransUnion Overview: At TransUnion, we encourage and are committed to creating a real, positive impact and shared sense of purpose within our Workforce for Good, which empowers our people to grow, innovate and contribute to a better future for our communities and customers. We strive to build an environment where our associates are in the driver’s seat of their professional development— while having access to help along the way. We recognize that success comes when our associates thrive both professionally and personally; that’s why we prioritize work/life flexibility and offer resources for our teams across the globe to collaborate and drive excellence. Be a part of our Workforce for Good – you’ll work with great people, pioneering products and cutting-edge technology. TransUnion Job Title Manager II, Applications Development Beware of scams tied to potential career opportunities. If you received a link, email, text or social media communication you weren’t expecting from TransUnion, take a moment to confirm the relevant information. Be wary when providing personal information. Contact careers@transunion.com or (312) 258-1717 for assistance. View TransUnion's Job Applicant Privacy Notice TransUnion is a global information and insights company with over 12,000 associates operating in more than 30 countries. We make trust possible by ensuring each person is reliably represented in the marketplace. We do this with a Tru™ picture of each person: an actionable view of consumers, stewarded with care. Through our acquisitions and technology investments we have developed innovative solutions that extend beyond our strong foundation in core credit into areas such as marketing, fraud, risk and advanced analytics. As a result, consumers and businesses can transact with confidence and achieve great things. We call this Information for Good® — and it leads to economic opportunity, great experiences and personal empowerment for millions of people around the world.

Skills and functions

  • Aws
  • Bigquery
  • Data Engineering
  • Gcp
  • Marketing
  • Python
  • Software Engineering
  • Spark
  • Sql