I don't really think he as to improve to be a good reliever. Stepping back from the stats, if you have a 100-mph fastball, a lights-out curve, and mediocre control, that's the makeup of a good closer IMO.
edit: OK, I guess the "metrics" have to improve, but that happens with a move the the pen. Things are different for hitters seeing a guy once, who's throwing a lot faster than your previous 3 or 4 trips to the plate.