[Oisf-users] Hunting down performance bugs - ioctls

[Victor Julien]

On 12/17/2013 03:35 PM, Mark Ashley wrote:
> 
> Yup, removing --enable-debug made a fair bit of a difference. I don't
> think I need to use it in anger any more now that suricata is ported and
> running.
> 
> What about the --enable-profiling and --enable-profiling-locks options?
> Do they have a significant hit on performance?
> 
> I discovered via their profiling functionality that removing the ioctl
> issue dropped my packet loss from 67% to 64%, and now removing
> --enable-debug dropped it further to between 14% to 33%, depending which
> thread you ask. It's almost usable on this platform!
> 
> I'll shuffle more threads onto the busy interface and see how much I can
> improve on that. Nope, going from 4 to 6 threads raised the drops from
> the previous 14%-33% up to 30%-65%. Going back to one thread only on the
> busy interface, after the initial boot up drops, the drops are almost
> zero and the thread is ticking away at up to 100K packets/second. That
> leads me to believe the locking isn't working so well on this system,
> I'm not sure they are actually using atomics.
> 
> Another issue which hits the O.S. scheduler is top(1M) is pretty bad on
> resources, it increases my dropped packets by 20% - 50%. Using
> prstat(1M) is much nicer on the system. It's the way it's written.

We use atomics in a few critical paths. If they aren't supported (we
only use them if __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 and friends are
defined), we fall back to spinlocks.

The spinlocks wrapper itself can fall back to mutex if spinlocks aren't
available.

On linux, mutexes are cheap and mostly handled in userspace. I have no
idea about how Solaris deals with them.

I suspect that if you can implement the atomics for your architecture in
util-atomics.h you'll see better numbers.

Btw, in master I have added some more info about this to --build-info:
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture

Cheers,
Victor


> ta,
> Mark.
> 
> 
> 
> On Wed, Dec 18, 2013 at 12:57 AM, Victor Julien <lists at inliniac.net
> <mailto:lists at inliniac.net>> wrote:
> 
>     On 12/17/2013 02:42 PM, Mark Ashley wrote:
>     >               suricata`PrintList+0x4e4
>     >               suricata`StreamTcpReassembleAppLayer+0x301
>     >               suricata`StreamTcpReassembleHandleSegmentUpdateACK+0x17b
>     >               suricata`StreamTcpReassembleHandleSegment+0x1a0
>     >               suricata`HandleEstablishedPacketToServer+0x1888
>     >               suricata`StreamTcpPacketStateEstablished+0x20c6
>     >               suricata`StreamTcpPacket+0x82a
>     >               suricata`StreamTcp+0x461
>     >               suricata`TmThreadsSlotVarRun+0xada
>     >               suricata`TmThreadsSlotProcessPkt+0x61
>     >               suricata`PcapCallbackLoop+0xd0b
>     >               libpcap.so.1.5.2`pcap_read_bpf+0x13d
>     >              2843
> 
>     PrintList above suggests you've compiled with --enable-debug. This will
>     be slow no matter what you do. The PrintList above for example iterates
>     the TCP segment list multiple times per packet I think, or maybe just on
>     segment insert. Anyhow: slow.
> 
>     --
>     ---------------------------------------------
>     Victor Julien
>     http://www.inliniac.net/
>     PGP: http://www.inliniac.net/victorjulien.asc
>     ---------------------------------------------
> 
>     _______________________________________________
>     Suricata IDS Users mailing list:
>     oisf-users at openinfosecfoundation.org
>     <mailto:oisf-users at openinfosecfoundation.org>
>     Site: http://suricata-ids.org | Support:
>     http://suricata-ids.org/support/
>     List:
>     https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>     OISF: http://www.openinfosecfoundation.org/
> 
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------