Future of Driver Education with Yaak
Introduction
Do you have a pen and paper by your side? Yes? No? We are going to need input from your side for this blog, so get them handy. Got it? Good! Let's write down "Hello World".
"Hello World" - Unknown Artist
Do you remember when you first learned to write in your native language? Or in a foreign language with a different script? For most of us it started at an early age with writing the letters of the script in lower and upper cases (if yours had cases!). At the beginning we drew squiggly lines, slowly mastered and controlled the strokes, and gradually got better at it. Your progress was likely driven by two adversarial forces:
Imitating your teacher or literally copying(!) the letters from your textbook; and
Feedback from your teacher on your reproduction of the letters and how to get better at it.
Quite likely 2. Was either verbal, e.g. ‘The middle stroke of E is shorter than the upper and lower ones’, or simply pointing out letters that you need to practice more than others.
Early on, I always got my J’s flipped. The rest of the alphabet was okay, but the J's, I always got them flipped the wrong way. It took more practice, but I got the hang of it, eventually. Most likely you did too, at your own pace, but you made it. Pause a moment for a key observation here. You and I had an expert by our side, always. Vetted with years of experience, constantly providing us with feedback, correction, and explaining why and how our handwriting skills should improve.
Getting there!
Why did I bring up handwriting? Come to think of it, handwriting is incredibly complex, but we make it look so simple — every day. You and I write by moving, rotating, shifting, and pressing our writing tool with six degrees of freedom on a sheet of paper. You and I mastered this with an expert by our side.
One is deemed an expert after years of training in grading, explaining, and correcting the intricacies of handwriting to those who are new to it. This is the litmus test for having mastered any skill.
If we aim to build AI systems for mastering and solving tasks, which seem effortless to us, like handwriting or driving, the litmus test has to be its ability to score, explain, and correct our performance at those tasks, with the proficiency and command of an expert.
Driving instructors have decades of experience at scoring, explaining and correcting these tasks. At Yaak we have tasked ourselves with codifying the collective expertise of driving instructors, contextualized with camera and telemetry data, into AI.
We call it SafetyAI.
SafetyAI will be deemed mature when it can perform the tasks of a driving instructor with high precision. We can measure it, by making it assess a student driver and comparing notes with a driving instructor. To get perfect at this, SafetyAI has to not only learn the difference between good and bad driving (scoring), but also generate explanations for why it flagged the driving (explaining) and what would have been a safer alternative (correcting). Just like a driving instructor does in a driving lesson.
The current paradigm for scoring human driving is centered around rules-based frameworks like RSS[1] and more recently — Tesla's Safety Score[2]. Passive correction of drivers could be done with ADAS features like lane keeping[3] and collision avoidance[4], to name a couple. Explainability is an often forgotten topic and pushed to the sidelines as academic research. These shortcomings and fragmentation of off-the-shelf quasi-AI systems highlight the gap SafetyAI will fill.
Generative models
The next section discusses the inspiration for the modeling of SafetyAI, largely inspired by recent leaps in ML. The ML landscape shifts quickly, and quite likely we will end up with completely new ideas for modeling and architectures in a couple of years. Despite that, the must-have properties from SafetyAI would not shift. Feel free to skip it.
TL;DR: Since at least the early 2020s, machine learning models trained on web-scale textual and image data can summarize books[9], generate/complete sentences and images[7] given context, and explain factually correct and incorrect information[8]. These models would be foundational to building SafetyAI.
In the last few years, circa 2018, natural language processing (NLP) has seen unparalleled development of large scale language models (LLMs), which have nearly all the properties we expect SafetyAI to have. LLMs can score sentences (or code[10]!) of arbitrary length for correctness when trained on highly specialized and clean data sources (hint hint: expertly curated datasets). LLMs can explain via self-attention which chunk of the input data they weighed heavily while scoring. Further, LLMs can generate sentences (even images[7]!) when prompted with context. While generative models have seen amazing success in computer vision and NLP, they are still in a nascent state in an actor-environment setting, such as robotics and self-driving. This is changing fast, through architecture design consolidation taking place in machine learning.
Baked with generative modeling, SafetyAI can be easily validated for maturity with an expert in the loop while driving lessons are performed. Predictions from SafetyAI must flag all driving mistakes annotated by instructors with Yaak's Coaching Hub app. Yaak goes beyond just making predictions on good and bad driving and provides, in human-readable form, the reason for such flags, just like the driving instructor does during the lessons.
Expert in the loop
A typical driving lesson has a structure similar to handwriting courses you and I had. We get instructions for the task, like ‘Write J’s 10 times’ (or was that just me?), or in the case of a driving lesson, ‘Stop at a 4-way intersection and, if no traffic lights are present, follow the right before left rule’. The driving instructor explains what to look out for at the intersection, where to stop, and which scene actor has priority. Carefully checking for the surroundings and the student’s driving, the instructor then explains to them what they did correctly or incorrectly and why. The ‘why’ is a critical feedback needed for the student to internalize the ‘common sense’ and rules of the road, and for us to bake scoring and explainability into SafetyAI.
Annotating driving mistakes, including near misses, with the Yaak app.
The instructor also uses Yaak's Coaching Hub app to annotate in real time any driving mistake that may happen: from unsafe vehicle proximity to checking for incoming traffic at unprotected left turns. SafetyAI learns to flag and explain these mistakes. The feedback from SafetyAI is accepted by the instructor if, when at its application, students improve their driving and make fewer mistakes. If the feedback is rejected it serves as an important reward signal to further optimize SafetyAI. Our customers’ successes drive the path to an optimal SafetyAI.
Learning to drive
How we learn to drive has seen little to no innovation in the past two decades. Instructors track student lessons with a sheet of paper, and have no data on their progress or shortcomings. This is despite the fact that cameras and low-power computers are nearly ubiquitous in the automotive space. With the videos and annotations generated by driving instructors during the lessons as well as scoring from SafetyAI, both instructors and students can discuss, share, and review their lessons and plan the next one on routes and topics on which the student needs to improve the most (my J's eventually flipped the right way).
Driving instructor in the loop
With SafetyAI metrics, we can predict an important and key milestone in learning to drive — is the student ready to take the driving test, and how likely is it that they will pass? Comparing the student performance with the historical data from students who have gotten their driving license, we can estimate the performance gap they need to bridge, and in which topics. Be it unprotected left turns, merging in and out of the autobahn, 5-way roundabouts, or pedestrian crossings. We track a student’s performance on these and more curriculum topics sourced from OpenStreetMap[5] as well as crafted from the vehicle’s context, e.g. proficiency at overtaking parked vehicles.
A driving student's 's view of incidents (red) and their context (blue)
Points of interest (blue) and incident annotations generated by driving instructor / SafetyAI (red)
Yaak goes a step beyond and brings driving education into the 21st century with an autonomous-driving-grade, in-vehicle simulator. Too much information? Ok, let's break it down. Say you get feedback with Yaak's Coaching Hub app and SafetyAI that you haven't yet mastered your unprotected left turns. Now you want to practice more of them (here comes my flipped J's analogy, again). What if you could do it on your own time at a fraction of the cost of an on-road driving lesson by unlocking a Yaak car with our app, putting on a VR headset, and using the vehicle’s steering wheel and pedals to practice your unprotected left turns to your heart's content? And don’t worry, the Yaak vehicle does not physically move.
The simulation in which you practice is a procedurally generated[6], exact digital twin of the city where your driving school operates. Not just that, Yaak customizes your routes based on the feedback the instructor/SafetyAI provided during your prior on-road lessons.
Yaak's procedural generation pipeline
Planet-scale fleet learning
Yaak's driving school customers can run their lessons in any city in the world with a vehicle up-fitted with our sensor and compute kit. The Yaak kit continuously uploads 360° video, telemetry data and any driving mistakes annotated by the instructors. To bake the gold standard of scoring safe driving into SafetyAI, instructors also perform expert data collection in their region. They know the intricacies of easy to difficult routes in their neighborhood from their driving lessons, hence Yaak collects data on routes on a spectrum of easy to difficult in order to master routes. The route difficulty embodies the topology of the road network, geographical and traffic challenges.
Annotations generated via the Yaak app
Yaak scales expert and student (counterfactual) data collection to major cities world-wide with our driving school customers. Our customer fleet gets the best-of-all worlds when it comes to scaling, cost, and data quality. Large scale commercial fleets like ride-hailing and logistics have low operational costs, but unknown data quality, and require setting up triage and annotation teams to filter for quality. On the other hand, an autonomous driving fleet has high operational cost, is usually small scale (<1000 vehicles, <10 cities), but is run by vetted safety drivers.
Event annotations on a timeline with 4/8 cameras in view
Yaak's fleet easily scales to multi-city with our customers, operates at negative cost, and is run by expert driving instructors with driving annotations logged by our Coaching Hub app during the lessons.
Testing readiness
With operational overhead affecting on-road development/testing of autonomous vehicles (AVs) in unseen environments, AV developers have pivoted to using simulations and scenario description languages instead. Simulation is a great tool (we built an in-vehicle simulator 👀), but not a replacement for at-scale, on-road testing for a safety-critical product like autonomous driving.
Yaak bridges the gap of an important, unmet infrastructure need for testing AVs at scale. With a fleet operated by driving instructors in major cities world-wide, Yaak provides a test bench for AV developers to validate their tech stack at scale in unseen geographies without the large operational overhead of owning and running local garages.
How do we do it? It's quite straightforward. We simply ask our network of driving instructors to conduct driving lessons in a test AV, and annotate its driving for safety infringement with the Yaak app. Starting from easy and then gradually testing on harder to master routes, just like they did with a driving student, all the while SafetyAI generates a detailed assessment of its shortcomings, and recommends where to improve on in order to pass the driving test.
These annotations go a step further than the current paradigm of binary hit-or-miss disengagements, which don’t present a full picture. Near misses, like a narrow turn radius at corners, don’t cause an accident, but fall outside the ‘common sense’ for a safe turning radius. Driving instructors pay special attention to these and flag them.
You made it all the way! If there’s anything in the blog that perks your imagination and is something you’d like to work on, good news — Yaak is hiring.
Like to be a part of the beta fleet for testing SafetyAI? Part of a driving school or an autonomous developer? Get in touch for more details.
References
RSS: https://www.mobileye.com/responsibility-sensitive-safety/
Tesla's Safety Score: https://www.tesla.com/support/safety-score
Lane assists: https://en.wikipedia.org/wiki/Lane_departure_warning_system
Collision avoidance: https://en.wikipedia.org/wiki/Collision_avoidance_system
OpenStreetMap: https://www.openstreetmap.org
Procedural generation: https://en.wikipedia.org/wiki/Procedural_generation
