Diaz was optioned to Triple-A Norfolk on Wednesday.
Diaz was called up Monday and went 0-for-1 with a strikeout as a pinch hitter Tuesday against the Rangers. However, he'll head back to the minors after Brett Phillips was added to the Orioles' active roster Wednesday.