From a programming standpoint, I would wager the maximum is limited (at a bare minimum) to a signed 8-bit integer (2^7-1 = 127), but is more likely an unsigned 8-bit integer (2^8-1 = 255). At the other end of the spectrum, it is possible the engine allots as much as an unsigned 32-bit integer (2^32-1 = 4294967295), but IMO this is unlikely. At any rate, you would really have to try in order to reach any sort of limit.
It is more than 127 as there are dialogues (e.g.Finsol01, Fatesp) that have more responses in some states (nearly every NPC mod adds there bit of talk here, even if only those of party members are triggered for display). That means the game can handle such a number, even if conditional responses are not displayed if the conditions are not met. Technically it should be the same.
I have not seen any single state that had more than 255 responses to a single state.
Edited by Roxanne, 15 September 2015 - 12:00 AM.