Figured it out. These sockets are ending up in TIME_WAIT state and apparently we're hitting some OS X 32k internal limit on the number of outstanding sockets. Linux either has a higher upper bound, or it's avoiding hitting this situation.

It appears that Async::<TcpListener>::accept may have a bug on OS X where it can miss connection attempts. I was able to reduce this bug down to this:

#[test]
fn tcp_connect_bug() {
    for i in 1.. {
        println!("attempt {i}");

        future::block_on(async {
            let listener = Async::<TcpListener>::bind(([127, 0, 0, 1], 8081)).unwrap();
            let addr = listener.get_ref().local_addr().unwrap();

            let task_client = spawn(async move { Async::<TcpStream>::connect(addr).await });
            let task_server = spawn(async move { listener.accept().await });

            let stream_client = task_client.await.unwrap();
            let stream_server = task_server.await.unwrap().0;
            let stream_server = listener.accept().await.unwrap().0;
            let stream_client = task_client.await.unwrap();

            assert_eq!(
                stream_client.get_ref().peer_addr().unwrap(),
                stream_server.get_ref().local_addr().unwrap(),
            );
            assert_eq!(
                stream_client.get_ref().peer_addr().unwrap(),
                stream_server.get_ref().local_addr().unwrap(),
            );
        })
    }
}

On my OS X laptop, it will succeeds for about approximately ~16000 loops before I get:

...
attempt 16277
thread 'tcp_connect_bug' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 60, kind: TimedOut, message: "Operation timed out" }', tests/async.rs:48:51
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test tcp_connect_bug ... FAILED

I don't get any time out running this on a Linux machine after ~100000ish cycles.

Welp, it looks like I'm hitting some unknown system limit or issue? Removing async altogether hits the same limit around ~16000:

use std::thread;
use std::net::{TcpListener, TcpStream};

fn main() {
    let addr = "127.0.0.1:8081";
    let listener = TcpListener::bind(&addr).unwrap();

    for i in 1.. {
        println!("attempt {i}");

        let task_client = thread::spawn(move || TcpStream::connect(addr));

        let stream_server = listener.accept().unwrap().0;
        let stream_client = task_client.join().unwrap().unwrap();

        assert_eq!(
            stream_client.peer_addr().unwrap(),
            stream_server.local_addr().unwrap(),
        );
        assert_eq!(
            stream_client.peer_addr().unwrap(),
            stream_server.local_addr().unwrap(),
        );
    }
}

Really not sure what's going on here...

Also, according to Activity Monitor, and my process is using roughly a constant amount of memory and ports.

I don't appear to be exhausting file descriptors, at least according to lsof. Here's the output while it's actively running:

➜  ~ lsof -p 22149
COMMAND     PID       USER   FD     TYPE             DEVICE SIZE/OFF      NODE NAME
async-402 22149 etryzelaar  cwd      DIR                1,7      544  24338112 /Users/etryzelaar/projects/rust/async-io
async-402 22149 etryzelaar  txt      REG                1,7  3746136 449816186 /Users/etryzelaar/projects/rust/async-io/target/debug/deps/async-402523331f3d7ebd
async-402 22149 etryzelaar    0u     CHR               16,8 0t304391      2695 /dev/ttys008
async-402 22149 etryzelaar    1u     CHR               16,8 0t304391      2695 /dev/ttys008
async-402 22149 etryzelaar    2u     CHR               16,8 0t304391      2695 /dev/ttys008
async-402 22149 etryzelaar    3u    IPv4 0x6f2bde6253958dc5      0t0       TCP localhost:sunproxyadmin (LISTEN)
async-402 22149 etryzelaar    4u  KQUEUE                                       count=0, state=0x8
async-402 22149 etryzelaar    5u    IPv4 0x6f2bde62523d82a5      0t0       TCP localhost:50641->localhost:sunproxyadmin (ESTABLISHED)

And here's the output when it's hung:

➜  ~ lsof -p 22149
COMMAND     PID       USER   FD     TYPE             DEVICE SIZE/OFF      NODE NAME
async-402 22149 etryzelaar  cwd      DIR                1,7      544  24338112 /Users/etryzelaar/projects/rust/async-io
async-402 22149 etryzelaar  txt      REG                1,7  3746136 449816186 /Users/etryzelaar/projects/rust/async-io/target/debug/deps/async-402523331f3d7ebd
async-402 22149 etryzelaar    0u     CHR               16,8 0t309305      2695 /dev/ttys008
async-402 22149 etryzelaar    1u     CHR               16,8 0t309305      2695 /dev/ttys008
async-402 22149 etryzelaar    2u     CHR               16,8 0t309305      2695 /dev/ttys008
async-402 22149 etryzelaar    3u    IPv4 0x6f2bde62504ee785      0t0       TCP localhost:sunproxyadmin (LISTEN)
async-402 22149 etryzelaar    4u  KQUEUE                                       count=0, state=0xa
➜  ~ lsof -p 22149
COMMAND     PID       USER   FD     TYPE             DEVICE SIZE/OFF      NODE NAME
async-402 22149 etryzelaar  cwd      DIR                1,7      544  24338112 /Users/etryzelaar/projects/rust/async-io
async-402 22149 etryzelaar  txt      REG                1,7  3746136 449816186 /Users/etryzelaar/projects/rust/async-io/target/debug/deps/async-402523331f3d7ebd
async-402 22149 etryzelaar    0u     CHR               16,8 0t345617      2695 /dev/ttys008
async-402 22149 etryzelaar    1u     CHR               16,8 0t345617      2695 /dev/ttys008
async-402 22149 etryzelaar    2u     CHR               16,8 0t345617      2695 /dev/ttys008
async-402 22149 etryzelaar    3u    IPv4 0x6f2bde6250d69625      0t0       TCP localhost:sunproxyadmin (LISTEN)
async-402 22149 etryzelaar    4u  KQUEUE                                       count=0, state=0xa

I've got the default limit descriptors of 256, and I get the same error around ~16000 if I bump the descriptor count up to 1024.

What's weird is the hang always seems to happen around ~16000 cycles after multiple runs of this test, but never precisely the same amount. I wonder if there's some internal limit we're hitting, but the error from the system might be getting ignored?

It appears that Async::<TcpListener>::accept may have a bug on OS X where it can miss connection attempts. I was able to reduce this bug down to this:

#[test]
fn tcp_connect_bug() {
    for i in 1.. {
        println!("attempt {i}");

        future::block_on(async {
            let listener = Async::<TcpListener>::bind(([127, 0, 0, 1], 8081)).unwrap();
            let addr = listener.get_ref().local_addr().unwrap();

            let task_client = spawn(async move { Async::<TcpStream>::connect(addr).await });
            let task_server = spawn(async move { listener.accept().await });

            let stream_client = task_client.await.unwrap();
            let stream_server = task_server.await.unwrap().0;
            let stream_server = listener.accept().await.unwrap().0;
            let stream_client = task_client.await.unwrap();

            assert_eq!(
                stream_client.get_ref().peer_addr().unwrap(),
                stream_server.get_ref().local_addr().unwrap(),
            );
            assert_eq!(
                stream_client.get_ref().peer_addr().unwrap(),
                stream_server.get_ref().local_addr().unwrap(),
            );
        })
    }
}

On my OS X laptop, it will succeeds for about approximately ~16000 loops before I get:

...
attempt 16277
thread 'tcp_connect_bug' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 60, kind: TimedOut, message: "Operation timed out" }', tests/async.rs:48:51
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test tcp_connect_bug ... FAILED

I don't get any time out running this on a Linux machine after ~100000ish cycles.

Polling 2.6.0 depends on windows-sys 0.45.0, but async-io 1.12.0 depends on windows-sys 0.42.0. Would it be possible to make a new release of async-io, which has already migrated from windows-sys to rustix? That would let us use async-io on some of the new platforms supported by polling.