Several approaches used for inpainting of images take advantage of sparse representations. Some of these seek to learn a dictionary that will adapt the sparse representation to the available data. A further refinement is to adapt the learning process to the task itself. In this paper, we formulate a task-driven approach to inpainting as an optimization problem, and derive an algorithm for solving it. We demonstrate via numerical experiments that a purely task-driven approach gives superior results to other dictionary-learning approaches.