I'm running a fairly simple bit of test code using libevent2 with epoll and
openssl bufferevents and I've run into a 100% cpu usage problem.
Looking into it 100% usage was caused by epoll_wait constantly
returning write events on the openssl socket when it shouldn't really have
been looking for write events at all (N_ACTIVE_CALLBACKS() was returning 0
also).
Looking a bit deeper eventbuffer_openssl socket seems to be requesting
that the EV_WRITE event be removed when it should, but the event isn't
actually being removed from epoll.
Continuing to follow this I think I've found a bug in
event_changelist_del.
For evpoll event_del calls event_changelist_del which caches the change
which is then actioned later when evpoll_dispatch is called.
In event_changlist_del there is a check so that if the currently changed
action is an add then the cached action is changed to a no-op rather than a
delete (which makes sense). The problem arises if there are more than
two add or delete operations between calls to dispatch, in this case it's
possible that the delete is turned into a no-op when it shouldn't have
been.
For example starting with the event on, a delete followed by an add and
then another delete results in a no-op when it should have been a delete (I
added a fair bit of debug output that seems to confirm this behaviour).
I've applied a small change that checks the original old_event stored with
the change and only converts the delete to a no-op if the event isn't on in
old_event. This seems to have fixed my problem.
On all the backends on this little mac laptop, that behavior is to
report a remote socket close as both EV_READ and EV_WRITE.
Historically, we had problem for some of these behaviors on some
backends, so let's make sure that such behaviors don't come back.
The default behavior of test.sh was to suppress all output from
test/regress, and say nothing but OKAY or FAILED. This wasn't so good
for getting bugs reported, since lots of people didn't know to set
TEST_OUTPUT_FILE, or re-run ./test/regress on its own.
Now, when you don't specify an output file for test.sh, it runs
regress with the --quiet option. This option makes the unit tests
only print output on failure, which is what we probably wanted.
The new options let you specify a maximum deviation of bandwidth used
from expected bandwidth used, and make test-ratelim.c exit with a
nonzero status when those deviations are violated.
This patch also adds a test-ratelim.sh script to run test-ratelim with
a few sensible options for testing.
This attribute tells gcc (and anything else that understands gcc
attributes) that the functions will never return control, and helps
the optimizer a little. With luck, it will also tell
less-than-full-program dataflow analysis tools that they don't need to
worry about any code path that involves calling one of these functions
and then returning.
This patch also forces event_exit() to always exit, no matter what the
user-supplied fatal_callback does. This means that the old unit tests
for the event_err* functions don't work any more, since they assume it
is safe to call event_err* if you've given it a bogus fatal_callback
that doesn't exit. Instead, we have to make the unit tests fork
before calling event_err(), and have the main unit test process wait
for the event_err() test to exit with a sane exit code. On unix,
that's trivial. On windows, let's not bother and just assume that
event_err* works.
It turns out that in all conformant shells, "unset FOO" removes FOO
both from the shell's variables and from the exported environment.
(I've tested this on msys, opensolaris, linux, osx, and freebsd.)
And in nearly every shell I can find, "unset FOO; export FOO" does
the same as unset FOO... except in my FreeBSD VM, where the "export
FOO" sets the exported value of FOO equal to "". This broke test.sh
for us.
The fix is simple: remove the needless exports!
This required:
- Adding another WIN32 section in test.sh
- not running "touch /dev/null"
- calling WSAStartup in all the test binaries
- Fixing a dumb windows-only bug in test-time.c
Previously, the be5_outcome field for the dns error would be set to
something dependent on our system resolver. It turns out that you
can't rely on nameservers to really give you an NEXIST answer for
xyz.example.com nowadays: too many of them are annoyingly broken and
like to redirect you to their locked-in portals. This patch changes
the bufferevent_connect_hostname test so that it makes sure that the
dns_error of be5_outcome is "whatever you would get from resolving
the target hostname"
Importantly, we don't actually want to call evbuffer_write() when
the buffer is empty. This makes it an error to ever get a -1 return
value from evbuffer_add_file(), which makes it safe for us to test
the return value.
The "current_base" symbol was never actually declared in an exported
header; it's hideously deprecated, and it was the one remaining
exported symbol (fwict) that was prefixed with neither ev nor
bufferevent nor _ev nor _bufferevent.
codesearch.google.com turns up no actual attempts to use our
current_base from outside libevent.
Of the backends that support edge-triggered IO, most (all?) do not
support attempts to mix edge-triggered and level-triggered IO on the
same FD. With debugging mode enabled, we now detect and refuse attempts
to add a level-triggered IO event to an fd that already has an
edge-triggered IO event, and vice versa.
The old logging code was littered with places where we stored messages in
static char[] fields. This is fine in a single-threaded program, but if you
ever tried to log evdns messages from two threads at once, you'd hit a race.
This patch also refactors evdns's debug_ntop function into a more useful
evutil_sockaddr_port_format() function, with unit tests.
The EVUTIL_CLOSESOCKET() macro required you to include unistd.h in your
source for POSIX. We might as well turn it into a function: an extra
function call is going to be cheap in comparison with the system call.
We retain the EVUTIL_CLOSESOCKET() macro as an alias for the new
evutil_closesocket() function.
(commit message from email by Nick and Sebastian)
The previous evbuffer_expand was not only incorrect; it was
inefficient too. On all questions of time vs memory tradeoffs, it
chose to burn time in order to avoid wasting memory. The new code
tries to be a little more balanced: it only resizes an existing chain
when doing so doesn't require too much copying, and when failing to do
so would waste a lot of the chain's space.
This patch also rewrites evbuffer_chain_insert to work properly with
last_with_datap, and adds a few convenience functions to buffer.c.
To implement evbuffer_expand() properly, you need to be able to
replace the last chunk that has data, which means that we need to keep
track of the the next pointer pointing to the last_with_data chunk,
not the last_with_data chunk itself.
Calling event_base_loop on a base from inside a callback invoked by
that same base, or from two threads at once, has long been a way to
get exceedingly hard-to-diagnose errors. This patch adds code to
detect such reentrant invocatinos, and exit quickly with a warning
that should explain what went wrong.
This is mainly intended to ensure that we don't get hung up on
the 64-handle limit that lots of O(n) Windows functions (but FWICT
not select) like to enforce.
We were using the same bufferevent as the child of two filtering parents,
orphaning another. This made one get freed twice, and the other not at all.
Possible fix for bug 2963306 spotted by Doug Cuthbertson.
This actually makes some of the code a lot simpler. The only
ones that actually used previous_to_last for anything were reserving
and committing space.
This is the first patch in a series to replace previous_to_last with
last_with_data. Currently, we can only use two partially empty chains
at the end of an evbuffer, so if we have one with 511 bytes free, and
another with 512 bytes free, and we try to do a 1024 byte read, we
can't just stick another chain on the end: we need to reallocate the
last one. That's stupid and inefficient.
Instead, this patch adds a last_with_data pointer to eventually
replace previous_to_last. Instead of pointing to the penultimated
chain (if any) as previous_to_last does, last_with_data points to the
last chain that has any data in it, if any. If all chains are empty,
last_with_data points to the first chain. If there are no chains,
last_with_data is NULL.
The next step is to start using last_with_data everywhere that we
currently use previous_to_last. When that's done, we can remove
previous_to_last and the code that maintains it.
