U(2) flavor physics without U(2) symmetry: Searching Inside