Sotabase

Career

· Research Scientist, Apple2022–
· PhD Student, UC Berkeley EECS2016–2022

Publications (20)

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
European Conference on Computer Vision · 2024
157
cited
AXNav: Replaying Accessibility Tests from Natural Language
International Conference on Human Factors in Computing Systems · 2023
70
cited
Drill Sergeant: Supporting Physical Construction Projects through an Ecosystem of Augmented Tools
CHI Extended Abstracts · 2016
56
cited
UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior
International Conference on Human Factors in Computing Systems · 2021
55
cited
HindSight: Enhancing Spatial Awareness by Sonifying Detected Objects in Real-Time 360-Degree Video
International Conference on Human Factors in Computing Systems · 2018
54
cited
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
International Conference on Human Factors in Computing Systems · 2022
41
cited
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
North American Chapter of the Association for Computational Linguistics · 2024
33
cited
Scones: towards conversational authoring of sketches
International Conference on Intelligent User Interfaces · 2020
26
cited
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
International Conference on Intelligent User Interfaces · 2023
25
cited
Never-ending Learning of User Interfaces
ACM Symposium on User Interface Software and Technology · 2023
24
cited
From Interaction to Impact: Towards Safer AI Agent Through Understanding and Evaluating Mobile UI Operation Impacts
International Conference on Intelligent User Interfaces · 2024
7
cited
SCRAM: Simple Checks for Realtime Analysis of Model Training for Non-Expert ML Programmers
CHI Extended Abstracts · 2020
7
cited
From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating UI Operation Impacts
arXiv.org · 2024
4
cited
Imagine it's "Corona" - and no one has noticed
2021
4
cited
AgentBuilder: Exploring Scaffolds for Prototyping User Experiences of Interface Agents
arXiv.org · 2025
1
cited
IMACS: Image Model Attribution Comparison Summaries
arXiv.org · 2022
1
cited
Towards Never-ending Learning of User Interfaces
2023
1
cited
Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
arXiv.org · 2025
Exploratory and Explanatory Tools for ML Application Development
2022
From the face-to-face to the hybrid event. Experiences with the transformation of a conference series to online network research
2021
Sotabase