Smoothness of the metric projection onto nonconvex bodies in Hilbert spaces

Abstract

Based on a fundamental work of R. B. Holmes from 1973, we study differentiability properties of the metric projection onto prox-regular sets. We show that if the set is a nonconvex body with a Cp+1-smooth boundary, then the projection is C-p-smooth near suitable open truncated normal rays, which are determined only by the function of prox-regularity. A local version of the same result is established as well, namely, when the smoothness of the boundary and the prox-regularity of the set are assumed only near a fixed point. Finally, similar results are derived when the prox-regular set is itself a Cp+1-submanifold. (C) 2016 Elsevier Inc. All rights reserved.