let Q dot P = 0 which means Q is perpendicular to P

to choose between R = Q or -Q you want R dot (-p) > 0 because -p points from the line containing p and S(-p) toward the origin so you can just dot product again