[Oisf-users] Suricata Startup Takes 8 to 10 Minutes with Hyperscan, High detect.profile, and Full detect.sgh-mpm-context

[Peter Manev]

On Thu, Mar 8, 2018 at 8:44 PM, Eric Urban <eurban at umn.edu> wrote:
> I am noticing that Suricata is taking about 8-10 minutes to fully start up
> under certain conditions and am wondering if anyone can say whether or not
> they feel this is expected?

Yes  it is expected -  https://redmine.openinfosecfoundation.org/issues/1770

You should try switching sgh-mpm-context to "auto" with hyperscan - it
will improve the start time.


>
> This happens on versions 4.0.3-4 and also a few different versions of 3.x
> that I have tested.  Suricata is being compiled with Hyperscan support (and
> is using for mpm- and spm-algo since set to auto) and the config is close to
> vanilla with only the HOME_NET value changed, detect.profile set to high,
> and sgh-mpm-context set to full.  If Hyperscan is not used or not setting
> sgh-mpm-context to full, the start up time is greatly reduced so that it is
> a minute or less.  Setting the detect.profile option lower does affect the
> startup time but not as much as changing the previously mentioned settings.
> No alerts are generated until Suricata is fully started.
>
> In this case there are 27,000 rules being loaded and Suricata is running on
> CentOS 7.  Loading fewer rules also does lower the start up time.
>
> I have looked at top for high CPU/load, free for excessive memory usage, and
> vmstat for any significant disc activity and none of these show indications
> of high system utilization.  These 8 to 10 minute start up times occur on
> both a VM (1 core, 2GB memory) and on physical hardware (40 cores, 128 GB
> memory).
>
>
> Here is an example from the suricata.log where there was a 10 minute start
> up time:
> 8/3/2018 -- 08:37:46 - <Notice> - This is Suricata version 4.0.3 RELEASE
> 8/3/2018 -- 08:37:46 - <Info> - CPUs/cores online: 40
> ...
> 8/3/2018 -- 08:38:05 - <Info> - 27542 signatures processed. 250 are IP-only
> rules, 12336 are inspecting packet payload, 18268 inspect application layer,
> 0 are decoder event only
> 8/3/2018 -- 08:38:05 - <Perf> - TCP toserver: 76 port groups, 71 unique
> SGH's, 5 copies
> 8/3/2018 -- 08:38:05 - <Perf> - TCP toclient: 76 port groups, 46 unique
> SGH's, 30 copies
> 8/3/2018 -- 08:38:05 - <Perf> - UDP toserver: 76 port groups, 43 unique
> SGH's, 33 copies
> 8/3/2018 -- 08:38:05 - <Perf> - UDP toclient: 15 port groups, 9 unique
> SGH's, 6 copies
> 8/3/2018 -- 08:38:05 - <Perf> - OTHER toserver: 254 proto groups, 3 unique
> SGH's, 251 copies
> 8/3/2018 -- 08:38:05 - <Perf> - OTHER toclient: 254 proto groups, 0 unique
> SGH's, 254 copies
> 8/3/2018 -- 08:47:35 - <Perf> - Unique rule groups: 172
> 8/3/2018 -- 08:47:35 - <Perf> - Builtin MPM "toserver TCP packet": 57
> ...
> 8/3/2018 -- 08:47:46 - <Info> - RunModeIdsPcapWorkers initialised
> 8/3/2018 -- 08:47:46 - <Perf> - Setting prio 0 for thread "FM#01", thread id
> 8371
> 8/3/2018 -- 08:47:47 - <Perf> - Setting prio 0 for thread "FR#01", thread id
> 8372
> 8/3/2018 -- 08:47:47 - <Perf> - Setting prio 0 for thread "CW", thread id
> 8373
> 8/3/2018 -- 08:47:47 - <Perf> - Setting prio 0 for thread "CS", thread id
> 8374
> 8/3/2018 -- 08:47:47 - <Notice> - all 9 packet processing threads, 4
> management threads initialized, engine started.
>
> --
> Eric Urban
> University Information Security | Office of Information Technology |
> it.umn.edu
> University of Minnesota | umn.edu
> eurban at umn.edu
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/



-- 
Regards,
Peter Manev