Databricks Platform Administrator

Key Responsibilities

Databricks Administration: Manage workspaces, clusters, jobs, security configurations, libraries, and overall platform health.
Data Engineering: Build and optimize data pipelines and transformations using PySpark and TSQL.
Automation & Development: Create automation and tooling using Python, Bash, and PowerShell.
Governance & Security: Implement and automate data governance, access controls, auditability, and compliance best practices.
Note: A significant portion of our development focuses on governance and security automation.
Solution Architecture: Translate conversational ideas into clear solution architectures; break work into tasks in Planner or Azure DevOps for team execution.
Effort Estimation: Estimate level of effort and forecast delivery dates with reasonable accuracy.
Documentation: Produce high-quality documentation including workflows, gap analyses, design specs, build docs, test plans, and transition documentation.
Testing Oversight: Oversee unit, scenario-based, integrated, capacity, and parallel testing.
On-Call Support: Participate in a 247365 on-call rotation to support Databricks, SQL Server, and other relevant applications supported by the team.
Upgrades & Monitoring: Support system upgrades, patches, performance tuning, observability, and proactive monitoring.
Data Integrity: Maintain accountability for data integrity across applications and pipelines.
Change Management: Complete system build and documentation in adherence to departmental guidelines and change management policy.
Peer Review: Review teammates work and provide constructive feedback.
Communication & Escalation: Communicate effectively with peers, stakeholders, and leadership; provide regular status updates and escalate issues appropriately.
Facilitation & Leadership: Drive discussions, mentor accountable leaders, and coordinate resources across initiatives.
Training & Mentorship: Provide feedback on training materials; conduct training sessions; develop and mentor other team member.
Policies & Vendor Engagement: Help develop department guidelines, standards, procedures, and policies; facilitate vendor reviews and selection processes.
Integrations (as applicable): Demonstrate understanding of integration and technical requirements for peripherals/devices that interface with assigned applications.
Support Coordination: Coordinate remote and/or on-site support as needed for end-users.

Optional Contributions

Create complex AI/BI dashboards.
Contribute to CI/CD pipelines and integrations using Azure DevOps or GitHub.
Apply Terraform for infrastructure as code.
Provide MS SQL Server administrative support.