That argument carries enough weight that EURONCAP safety rating requires physical buttons for key features:

"Under the Safe Driving section of Euro NCAP’s safety rating, the key driving controls – indicators, windscreen wipers and washer, hazard warning lights, horn and eCall emergency button – should be operated by a direct physical input rather than hidden within a menu on a touchscreen. "

https://www.euroncap.com/press-media/buttons-are-back-for-basic-functions-euro-ncap-confirms-in-testing/

The “it doesn’t effect me” crowd is of the same kind that says, that controlling a phone during driving, doesn’t effect them or that they can drive just as safe after non-excessive alcohol consumption.