If the main purpose is offshore fishing, then definately the 23. Not much difference in total cost for a rigged 20' or 23' with single outboard power.
fixed cost for the 20 and 23 are the same (just about) for either boat: batteries, steering, fuel tank, rigging, trailer, t-top, rod holders, pumps, etc.
The only thing that is marginally different is the cost of the bigger motor for the 23. So for just about the same time and money, you can have the 23'
23' (to me) is still a "big little boat", it has the same basic systems as a skiff. Once you go to a 25'+ with twin motors, the costs jump a lot, "little big boats" are expensive