Hell, try it. But a smaller diameter pulley may not have enough "wrap" to drive the alternator at full output.
Or set the idle higher.