Before trying everything else, try to capture your screen, rather than game capture. Hafen behaves extremely retarded when apps are focusing on it.
For example, trying to stream the game window (rather than your entire screen) on Discord will literally crash Discord. If your PC is old enough, it will fucking freeze. Don't ask me why, ask loftar.If the problem persists after changing from game capture to screen capture:
Keep in mind that the preview window will sometimes act stupid to preserve resources, especially when not focused. That happens for me too, with a 3080-ti
Have you also checked the result video? Does that also lag?
My recommendation is to also use the advanced output settings.
File -> Settings -> Output -> At the top of the screen, change "Output Mode" from Simple to Advanced.
In the Recording tab, try using something like this (I adjusted them for GTX 1080 ti, 1920x1080):
![Image](https://i.imgur.com/YrtCsSA.png)
If your recorded video (not preview) still lags, you can try lowering the bitrate.
I used to record 60 fps videos on my old PC, with a GTX 750, so you can definitely record videos. OBS is literally the best quality-performance wise.
Also, as lordgrunt mentioned, the client can affect performance by half.
Ardennes' bootleg of a bootleg whatever client is using the old render code. Ender's client for instance, is using the new render.
New render clients literally have twice as much FPS than old render clients.
Let me know if it still lags for you after trying this stuff.