Middle Data Engineer (Databricks/Python/LLM)

We are looking for a Senior Data Engineer with strong Databricks, Python, and LLM pipeline experience to join a team turning large volumes of clinical-research documents into reliable, AI-ready data on Azure. Company: Data Smart Solutions LLC. Role: Middle Data Engineer. Format: Remote.

Requirements

• Strong hands-on experience with Databricks, especially Databricks Jobs
• Advanced Python skills, writing notebooks daily from ingestion to the gold layer
• Solid understanding of medallion architecture and how to define bronze / silver / gold boundaries
• Experience working with Delta tables
• Experience processing large volumes of PDF documents (100K+ files)
• Experience calling LLM endpoints from notebooks and tracking token usage and cost
• Experience with Azure cloud
• Ability to design and consolidate job orchestration pipelines

Nice to Have

• Experience with Neo4j or other graph databases
• Experience with healthcare or clinical-research data
• Experience building data layers consumed by AI agents

Tasks

• Extracting and structuring data from large collections of clinical-research PDFs
• Building bronze, silver, and gold layers in Databricks with Python notebooks and Delta tables
• Integrating LLM endpoints into data pipelines and monitoring token spend
• Building a Neo4j graph alongside Delta tables for agents to query
• Preparing gold-layer data for an agentic layer used as an API tool
• Consolidating scattered and redundant Databricks job pipelines

Compensation Details

• $600 – $800 (Net) + performance-based bonus

Notice

Ready to apply? Fill out the form.