[Oisf-devel] [RFC, PATCH 0/2] lighten Packet structure size

[Eric Leblond]

Hi,

This patchset is a real RFC. Code is not finished (and not enough tested) but
in its current state it will give you a good idea of my proposal.

Working on some performance testing, I've found that the max-pending-packets
can have a great influence on speed of suricata. But increasing this value to
a high value has shown me an explosion in memory usage. The explanation was
easy. This value increase the number of preallocated Packet structures and
the Packet structure contains a char[65535+] field which contains the payload.

Preallocation is one of the key of speed but in this case, this huge structure
is dramatically increasing the memory usage of Suricata.

Considering the fact that most of the packets are smaller than 1500 bytes, it
appears that this huge array is almost unused and could be resized to 1500 if
we manage to found a way to treat bigger packet separately.

This patchset proposes an implementation of this idea. The Packet structure
has been modified to contains a 1500 bytes array which is dynamically converted
to allocated array. Some getter and setter functions permit to have a quasi 
transparency of the modification from a "user" point of view.

Given the scope of the modification, the patchset (even unfinished) is really
huge. I thus decide to send an e-mail to the list with this RFC version to check
if I'm going in a direction oisf fondation will be ready to go.

Patchset statistics:
 src/alert-debuglog.c       |   12 +++++-----
 src/alert-fastlog.c        |    2 +-
 src/alert-unified-log.c    |    8 +++---
 src/alert-unified2-alert.c |   10 ++++----
 src/decode-gre.c           |    8 +++---
 src/decode-ipv4.c          |    4 +-
 src/decode-ipv6.c          |    2 +-
 src/decode-raw.c           |    4 +-
 src/decode.c               |   51 ++++++++++++++++++++++++++++++++++++++++++-
 src/decode.h               |   19 ++++++++++++++-
 src/defrag.c               |   36 +++++++++++++++---------------
 src/detect-bytetest.c      |    6 ++--
 src/detect-engine-tag.c    |    6 ++--
 src/flow.c                 |    2 +-
 src/source-erf-dag.c       |   20 ++++++++--------
 src/source-erf-file.c      |   16 +++++++-------
 src/source-ipfw.c          |   22 +++++++++---------
 src/source-nfq.c           |   27 +++++++++++++----------
 src/source-pcap-file.c     |   18 +++++++-------
 src/source-pcap.c          |   25 +++++++++++----------
 src/source-pfring.c        |   16 +++++++-------
 src/tmqh-packetpool.c      |   11 +++++++++
 src/util-unittest-helper.c |    8 +++---
 23 files changed, 205 insertions(+), 128 deletions(-)

BR,
--
Eric Leblond <eleblond at edenwall.com>