Job Details

Weekly Hours: 40

Role Number: 200649655-3337

Summary

Do you want to help shape the future of AI at Apple? Our team, part of Apple Services Engineering's (ASE) Human Centered AI Research organization, pioneers new methods and tools for AI evaluation. You will help build tools that accelerate our team's research and empower the entire organization to build and evaluate AI more effectively. As a technical leader, you'll help set engineering standards for evaluation systems across ASE and mentor researchers and engineers on best practices for AI tooling.

Description

This role is for a Machine Learning expert to drive the operational setup, execution, and quality assurance of safety evaluations across languages and markets. You will play a crucial role in collaborative development of canonical evaluation guidelines, with subject matter experts and partners on evaluation task configuration, running pilots, monitoring live evaluations, and ensuring data quality throughout the evaluation lifecycle.

An ideal candidate possesses strong data science fundamentals, and experience managing complex annotation or evaluation tasks.
This role will involve designing evaluations to scale across diverse linguistic contexts, by partnering with subject matter experts and cross-functional partners.

You will play a crucial role in building upon product safety requirements to create taxonomies, compose and curate exemplar safety evaluation datasets, and ensure that evaluation frameworks are culturally and linguistically grounded.

An ideal candidate possesses a strong understanding of sociotechnical evaluation design principles and practices, experiences designing evaluations to support policies and/or product requirements, and classification systems, and annotation and/or study participant guidelines.

Minimum Qualifications

• 3+ years of experience in a data science, applied research, or evaluation operations role, with hands-on experience managing annotation or evaluation pipelines.

• Proficiency in Python and experience with data processing, statistical analysis, and visualization libraries (e.g., pandas, NumPy, scipy, matplotlib, seaborn).

• Experience developing and maintaining annotation guidelines or evaluation protocols for human labeling tasks.

• Comfortable computing and interpreting inter-rater reliability metrics (e.g., Cohen's kappa, Krippendorff's alpha) and other data quality indicators.

• Demonstrated ability to collaborate with annotation operations services, vendor teams, or distributed study participants .

• Able to work independently as well as collaboratively with minimal direction.

• Organized, highly attentive to detail, and manages time well.

• 1+ year of experience working in industry.

Preferred Qualifications

• Advanced degree (MS/PhD) in Data Science, Statistics, Computational Linguistics, Information Science, or a related field.

• Experience operating evaluation or annotation pipelines across multiple languages or markets.

• Familiarity with annotation platforms and task management tools (e.g., Label Studio, Scale AI, or similar).

• Experience with SQL and large-scale data infrastructure (e.g., Spark, Hadoop, or cloud-based analytics platforms).

• Prior experience in AI safety, responsible AI, content moderation, or trust and safety domains.

• Experience designing quality assurance frameworks for crowdsourced or distributed annotation work.

• General familiarity with localization workflows or working with language service providers.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant (https://www.eeoc.gov/sites/default/files/2023-06/22-088_EEOC_KnowYourRights6.12ScreenRdr.pdf) .