5 ways ElastiFlow 7.11 Transforms OpenSearch into a Next Gen Observability Solution

If you’re already using ElastiFlow, you’re used to lightning-fast queries and reduced storage costs. For users leveraging OpenSearch, the release of ElastiFlow 7.11 marks a significant leap forward, transforming OpenSearch into a next-generation observability solution. This isn't just an update; it's a paradigm shift that empowers your teams with real-time insights, enhanced collaboration, and optimized resource utilization. 

Let's dive into five key ways ElastiFlow 7.11 achieves this transformation.

1. Better Teamwork and Collaboration

Bringing the network team and their data to OpenSearch enables more teams with valuable connectivity data. Everyone looking at the same data leads to less finger-pointing and faster MTTR. Custom dashboards and extensible views will also make all teams more productive. ElastiFlow delivers the enrichment (see above) that makes custom dashboards possible.

2. Better Context

Raw flow data can be overwhelming. ElastiFlow enriches this data with valuable context, transforming simple IP addresses and port numbers into meaningful insights. This includes:

• Geolocation: Pinpointing the geographic location of each client and each server they connect to 

• Autonomous System (AS) Information: Identifying the organizations behind IP addresses.

• Application and Service Information: Revealing which applications and services are generating traffic.

• Threat Intelligence: Integrating real-time threat intelligence to identify malicious activity and reduce alert noise.

This enriched data, stored in OpenSearch, empowers your network operations (NetOps) and security operations (SecOps) teams to quickly identify anomalies, investigate threats, and understand network behavior with a depth that raw data alone cannot provide.

3. Better Storage

Storing vast network flow data can quickly consume disk space, driving up operational costs for your OpenSearch cluster. ElastiFlow's latest versions (7.11 and higher) introduce significant storage optimization capabilities for OpenSearch. By intelligently configuring how indexes are handled, ElastiFlow can reduce storage requirements by up to 65%. This is achieved by optimizing index sorting for the specific characteristics of ElastiFlow’s datasets. Less storage means lower infrastructure costs and a more sustainable OpenSearch deployment.

4. Better Queries 

ElastiFlow's storage optimization also directly translates to improved query performance in OpenSearch, with reported reductions in query times by up to 30%. This is critical when troubleshooting real-time network issues or investigating security incidents. By optimizing how data is indexed and stored, ElastiFlow allows OpenSearch to retrieve and process network flow information more efficiently, giving you faster access to the answers you need to diagnose problems and make informed decisions.

5. Better Visibility

See what matters, regardless of where your assets reside.  ElastiFlow combined with OpenSearch provides a unified platform for comprehensive network observability. You can:

• Accelerated Troubleshooting: Rapidly pinpoint the root cause of congestion, high latency, or packet loss.

• Capacity Planning: Analyze historical network usage to optimize resources and prevent bottlenecks.

• Real-Time Detection + Response: Identify suspicious traffic patterns, DDoS attacks, data exfiltration, and unauthorized access attempts.

• Forensic Analysis: In a breach, reconstruct timelines and understand attack methods.

• Compliance: Maintain detailed network activity logs for regulatory requirements

ElastiFlow champions an open data approach, allowing you to retain full control over your network observability data within your OpenSearch deployment. This avoids vendor lock-in and provides the flexibility to integrate with other open-source tools or leverage additional AI and ML models on your network data. This open ecosystem empowers you to tailor your observability solution to your unique needs, ensuring scalability and adaptability as your network evolves.

ElastiFlow's Flow dashboard in OpenSearch

ElastiFlow's Top N dashboard in OpenSearch

Learn More About the ElastiFlow NetObserv 7.11 Storage Optimization Update

In 2023, we introduced TSDS (time series data streams) support for Elasticsearch, reducing the storage requirements to store flow data by up to 70%. 

With the release of Elastic 8.17, the synthetic _source feature (an essential part of how TSDS works) was removed from the free version and made only available with an Elastic Enterprise license, leaving many ElastiFlow users without access to these storage savings.

With NetObserv 7.11 we’re bringing back the storage savings of TSDS and many other benefits. While the storage savings are comparable to TSDS, this new storage optimization reduces query times significantly (up to 30% in our tests). Since a synthetic _source is not required, it is available to all Elastic users.

This storage optimization is also available for OpenSearch, bringing reduced storage (up to 60%) and reduced query times (up to 30%) to all OpenSearch users.

How to get started

1. For OpenSearch uses:
   Just update to NetObserv 7.11 and see the storage savings begin to ramp up. Storage optimization is enabled by default. Please review the changelog for NetObserv 7.11 for more details.

2. For Elastic users who don’t have TSDS enabled
   The same things are true as outlined above. Storage optimization is enabled by default. Let the storage savings begin. Please review the changelog for NetObserv 7.11 for more details.

3. For Elastic users who have TSDS enabled
   Although storage optimization is not enabled by default, we recommend enabling it to gain extra storage savings. Storage optimization works even better when synthetic _source is enabled.

Here are the steps to enable both storage optimization and synthetic _source after upgrading to 7.11 or higher:

• Stop your NetObserv instance.

• In Kibana, delete the existing Elastiflow data streams

• In Kibana, delete the existing ElastiFlow index templates, as new ones will automatically be created once TSDS is disabled.

• Open flowcoll.yaml and set:

  • EF_OUTPUT_ELASTICSEARCH_TSDS_ENABLE to false.

  • EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SYNTHETIC_SOURCE_ENABLE to true.

• Restart your NetObserv instance.

What will change

NetObserv already uses the API of Elasticsearch or OpenSearch to create indexes when NetObserv starts up. 

With this storage optimization feature enabled, NetObserv Flow will now change how it configures indexes in the downstream data store to use disk storage more efficiently.

Note: EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD will automatically be set to ‘rollover’ when storage optimization is enabled, no matter what a user configures. This is necessary for storage optimization to work properly.

When to expect improvements

Full storage savings will be realized after the rolled-over indexes are fully "segment merged" by the ILM/ISM policies. There will be storage savings when the index is being actively written to, however, not as much as after the final segment merge.

• Read the Docs here

• Contact our sales team here

• Request a free demo here