One thing you may want to check is that tcpser is using the same baud rate (the -s option if memory serves) as your modem setting in Vice. From what I recall tcpser's baud rate setting doesn't seem to matter when using an ACIA modem, but it did matter when using the user port modems and I got garbage on any userport modem if the tcpser -s speed didn't match the modem speed in Vice.
Full disclosure - I never did test the UP9600 mode... I pretty much assumed that if you were using speeds over 2400 you'd be using an ACIA modem. I will test that out next time I sit down to work on it again. I wasn't 100% happy with the way the user port system reconnected to tcpser if you killed and restarted tcpser... It worked, but unlike the ACIA modem you had to send two bytes of data to the modem before Vice actually detected the loss of the socket (first byte) and then initiated a new connection to tcpser (second byte) whereupon it was back online by the time you typed the third byte onward.
Racer - You're running both the modified tcpser and vice in Windows, right? On one of the bulletin boards someone said they could not get the Win64 compiled Vice to run, saying it gave an error stating that it was not a 32-bit application. (Technically it is a 64-bit application, so maybe they were trying to run it on a 32-bit version of Windows?) Brent also mentioned getting a similar error, though I he said it claimed Vice was a 16-bit application. I've tried it on two separate clean builds of Windows 10 Pro and apart from the warning about the software not being signed, it's worked fine on both so I am not sure what is going on for those who are having trouble getting it to start, unless they are running a 32-bit version of Windows or are running it from the Program Files (x86) folder, which assumes programs located there are all 32-bit compiled. I could always try building a 32-bit version and see if that helps...
