Logo-black

Data Horizon: Innovation Recap & Top Trends for 2024

Data Horizon: Innovation Recap & Top Trends for 2024

Written by

Ajit Monteiro, CTO & Co-Founder

Published

December 15, 2023

Data & AI Strategy

Data Technology Innovation Recap

In 2023, “Artificial Intelligence (AI)” transitioned from a mere buzzword to a tangible reality. Snowflake’s focus on Apps and AI led to strategic acquisitions and significant product updates, while Matillion introduced its cloud-native Data Productivity Cloud and Power BI introduced new features like CoPilot and DAX Query View. Explore how these advancements are transforming data analytics and intelligence.

Snowflake Updates

This year, Snowflake made several acquisitions to improve their AI, data sharing, migration, and python offerings.

Myst AI

Myst AI for its time series forecasting abilities; crucial for optimizing business operations like supply chain management and financial planning

SnowConvert

SnowConvert to bolster Snowflake’s migration toolkit, significantly easing the transition of legacy databases to Snowflake’s cloud data platform

LeapYear

LeapYear to incorporate differential privacy into Snowflake’s services, enabling secure data collaboration with mathematically proven privacy protection

Ponder

Ponder to enrich Snowflake’s data capabilities with Ponder’s expertise in bridging data science libraries and scalable data operations, fostering enhanced analytics and machine learning functionalities

Neeva

Neeva to integrate advanced, privacy-focused search capabilities into Snowflake’s data cloud, enhancing user experience and competitive edge in data analytics

Samooha

Samooha to accelerate our vision for removing the technical, collaboration, and financial barriers to unlocking value with data clean rooms

Improved Python Support

(Generally Available): Snowflake has notably enhanced its Python support, allowing data scientists and engineers to integrate Python tools and libraries with Snowflake’s platform seamlessly. This enhancement facilitates efficient data analysis and machine learning directly within Snowflake, leveraging Python’s widespread popularity in the data science community to maximize the platform’s data analytics capabilities.

Why it matters: Allows you to use Python inside of Snowflake, including in user-defined functions, and workbooks.

Snowflake Native Apps Framework

(Available on AWS, Private Preview on Azure): Snowflake has expanded its ecosystem by introducing native applications, a move that significantly enriches the functionality and flexibility of its cloud data platform. These native apps, developed either by Snowflake or third-party developers, are designed to operate seamlessly within Snowflake’s environment, offering users a range of tools and services directly accessible within the platform.

Why it matters: Allows for developing and publishing data applications inside of Snowflake.

Streaming and Dynamic Tables

(Open Preview): Dynamic Tables allow users to define transformation logic with a simple, declarative SELECT statements, and Snowflake will automatically keep the table up-to-date, with cost-efficient incremental updates, a custom-defined refresh schedule.

Why it matters: Dynamic tables removes the need for some customers to maintain their own incremental update ETL framework.

Support for Git Integration
(Open Preview): You can now maintain your entire database schema in git and deploy automatically via the Snowpark CLI. Snowflake’s new CREATE OR ALTER statement will handle DDL modifications automatically without dropping objects entirely. Check out our blog post to learn more.

Why it matters: Allows development teams to work simultaneously on a codebase while tracking and managing changes efficiently.

Notifications for Better Observability

(Generally Available): The new SYSTEM$SEND_EMAIL() system stored procedure can be used to send- email notifications. You can call this stored procedure to send an email notification from a task, your own stored procedure, or an interactive session.

Why it matters: Allows for data-driven email notifications for use-cases like KPI thresholds, data observability, access monitoring, etc.

External Network Access

(Open Preview): You can create secure access to specific network locations external to Snowflake, then use that access from within the handler code for user-defined functions (UDFs) and stored procedures. You can enable this access through an external access integration.

Why it matters: Allow or block specific network access to Snowflake, as well as call external APIs from Snowflake.

Snowpark Container Services

(Expected Release in Summer 2024): Enables developers to deploy, manage, and scale generative AI and full-stack apps, including infrastructure options like GPUs, securely within Snowflake. This service broadens access to third-party services like LLMs, Notebooks, MLOps tools, and more.

Why it matters: Allows you to host code from any language inside of Snowflake, and make sure your data that is used by the code doesn’t leave the Snowflake cloud.

Large Language Models and Document AI

(Expected Release in Summer 2024): A new LLM developed by Snowflake, built from the acquisition of Applica’s generative AI technology, helps customers extract insights from unstructured data without needing machine learning expertise

Why it matters: Easy access to document-based machine learning.

Matillion Updates

(Expected Release in Summer 2024): A new LLM developed by Snowflake, built from the acquisition of Applica’s generative AI technology, helps customers extract insights from unstructured data without needing machine learning expertise

Data Productivity Cloud

Matillion released its Data Productivity Cloud product, which is a cloud-native version of its software.

Why it matters: In addition to enhanced functionality, moving to the SaaS version of Matillion will remove the need to host and manage their own instance, allowing more time to focus on business value.

LLM-enabled Data Pipelines

In Q1 2024, Matillion expects to launch an AI Prompt component, allowing users to augment their data pipelines with GenAI.

Why it matters: Companies with an existing Matillion implementation may find this feature to be an easy route to proof-of-concept, before investing heavily in a custom LLM integration.

PipelineOS

Announced in July 2023, PipelineOS is the key innovation in Matillion’s new Data Productivity Cloud. The stateless, microservice architecture allows for both horizontal and vertical scaling.

Why it matters: Understanding the concept of PipelineOS will better equip customers to transition to the SaaS platform.

Microsoft Updates

Power BI continues to redefine the landscape of data visualization with its new features, including CoPilot and the DAX Query View. Additionally, Microsoft released new branding for all of their data-related tools: Microsoft Fabric.

CoPilot in Power BI

Using Copilot, you can simply describe the visuals and insights you’re looking for, and Copilot will do the rest. Users can create and tailor reports in seconds, generate and edit DAX calculations, create narrative summaries, and ask questions about their data, all in conversational language.

Why it matters: Allows you to create dashboards using natural language, minimizing the need for a analytics engineer, especially for ad-hoc use cases.

DAX Query View

A new feature that provides powerful querying capabilities, enhancing data analysis and insights derivation.

Why it matters: Gives you access to writing DAX queries inside of PowerBI without any external tools.

Microsoft Fabric

Announced in May 2023, Microsoft Fabic is the new branding for all data-related tools in the Microsoft ecosystem (e.g. Data Factory, PowerBI, Synapse, etc.).

Why it matters: Currently, Snowflake and Databricks are considered to be the industry leading data cloud platforms, and Fabric is Microsoft’s response to those tools.

Data Trends and Predictions for 2024

In the ever-evolving landscape of business intelligence, several key trends are reshaping how organizations perceive, manage, and leverage their data assets. From the imperative of AI-ready data to the paradigm shift of treating data as a product, these trends encapsulate transformative approaches driving the future of business analytics. As businesses head into 2024, understanding and adapting to these trends is paramount for sustained growth and innovation. Let’s delve into each trend, uncovering their significance and impact for modern data organizations.

Trend 1: AI-ready data

If your data isn’t ready for AI, then your organization isn’t ready for AI. Before jumping on the AI bandwagon, companies need to take a step back and assess their data landscape. An AI algorithm’s effectiveness is tied to the quality of data it processes.

While 79% of executives plan to boost investments in AI/ML, a mere 24% consider themselves truly data-driven (Wavestone). As Forrester aptly notes, while AI may be ready for the spotlight, it’s the quality of data and analytics that truly determine if it shines. Enhancing data quality isn’t just a peripheral task; it’s a strategic move that can boost AI/ML model accuracy by a significant 20% (Forrester). So what exactly does it mean for data to be AI-ready? According to Gartner, for data to meet the AI-ready criteria, it must be secure, enriched, fair, accurate, and governed.

To help your organization navigate the complexities of preparing your data for AI, OneSix has built some free tools and resources:

Check out our comprehensive Roadmap to AI-ready Data to get familiar with the core steps
Get the guide
Take our 5-minute AI Readiness Assessment to evaluate your organization’s data maturity; this will be the basis for determining how to accelerate your journey to AI readiness
Take the assessment

Trend 2: Unified business visibility

In the pursuit of unified business visibility, data analytics teams encounter a significant hurdle in finding data within the organization, with a notable 24% citing this challenge as their foremost struggle in executing data strategies during 2023 (Forrester). To address this, Generative AI will increase the trend toward centralized data that creates a single source of truth for large language models (LLMs) and everything else (Snowflake). This shift aligns with the evolving need for a consolidated and authoritative data repository to streamline organizational data.
The explosion of unstructured and semi-structured data further underscores the necessity for organizations to embrace all-in-one unified data platforms. Forrester highlights that these platforms are essential for effective cost management, support for diverse multi-structured data analytics, and enabling broader use cases and workloads.
Embedded analytics is going to play a pivotal role in building a data-driven culture. By integrating analytics capabilities and data visualizations directly into user workflows, applications, or portals, embedded analytics streamlines access to insights and provides users with a highly interactive and user-friendly data engagement experience. OneSix can help you get there, leveraging BI platforms like Power BI, Sigma, Tableau, Sisense, and Pyramid Analytics. Get in touch with us to learn more.

Trend 3: Data as a product

The evolving market trend emphasizes a transformative shift in how data is perceived— treating data as a product. This involves ensuring datasets are discoverable, addressable, self-describing, interoperable, trustworthy, and secure. The consumers of these datasets can then be other departments and organizations through data sharing technologies. This shift presents an opportunity for organizations to enrich their internal repositories by augmenting them with externally published datasets.

These external sources, such as economic indicators, population statistics, or weather data, offer a chance to enhance the depth and breadth of insights, empowering businesses with a more comprehensive and diverse informational landscape to drive informed decision-making and innovation.

Trend 4: Vector databases

Vector databases are rapidly emerging as a highly efficient storage solution tailored for the demands of AI, big data applications, and the long-term memory requirements of Large Language Model use cases. They excel at similarity-based searches, proving invaluable for critical functions like image recognition, recommendation systems, and intricate data comparisons.

What sets vector databases apart is their remarkable scalability and rapid data processing capabilities, making them ideal for real-time operations and large-scale applications. Their adaptability and speed render them not just as a viable option but as a crucial asset in the evolving landscape of data storage and utilization.

Gear up for a transformative year ahead

As we head into 2024, the world of data and AI technology is poised for significant transformation. The trends witnessed in 2023—AI-ready data, unified business visibility, cloud-first approaches, data-as-a-product paradigm, and real-time analytics—carry profound implications for businesses across all industries. Understanding and jumping into these trends is a must for any business aiming to stay ahead in this fast-moving world of data and AI.

Unlocking the Power of AI: Why Data Readiness Matters

Unlocking the Power of AI: Why Data Readiness Matters

Written by

Kwon Lee, Senior Manager

Published

July 31, 2023

Data & AI Strategy

Artificial intelligence (AI) has captured the imagination of businesses worldwide, promising revolutionary insights and transformative outcomes. However, amidst the AI frenzy, a critical aspect often gets overlooked: data readiness

A recent Wavestone report reveals that 79% of executives will be increasing their investment in AI/ML in 2023. But the truth is, most companies are far from having their data AI-ready, and this can hinder their AI initiatives.

The report also sheds light on the alarming state of data understanding among businesses. It reveals that a staggering 81% of companies lack a comprehensive understanding of the data they collect. This lack of awareness poses a significant hurdle when it comes to leveraging AI effectively. 

Transitioning to a data-driven culture is another challenge faced by organizations. According to the same report, only 24% of companies have successfully made this cultural shift. Without a data-driven mindset, organizations struggle to derive meaningful insights from their data and make informed decisions.

For most organizations, poor data quality comes at a high cost. Gartner estimates that subpar data quality costs enterprise businesses an average of $15 million annually. AI algorithms are only as effective as the quality of data they process. If the data is flawed or of low quality, even the most advanced AI models will fail to deliver reliable results.

It is essential for companies to take a step back and assess their data landscape before jumping on the AI bandwagon. To build a strong foundation, companies need:

1. To establish a robust data governance strategy

This should include processes for data management, privacy, and security. By implementing effective data governance practices, organizations can ensure the reliability, integrity, and accessibility of their data. 

2. To integrate disparate data sources

Many companies struggle with data silos, where data is scattered across different systems and departments. Consolidating and harmonizing these data sources is vital to create a comprehensive view that can fuel AI algorithms effectively. 

3. To prioritize data quality.

This involves conducting data cleansing, addressing missing values, removing duplicates, and resolving inconsistencies. By investing in data quality, companies can improve the accuracy and reliability of their AI models and drive meaningful outcomes.

Once companies have laid the groundwork for data readiness, they can begin to explore the exciting possibilities of AI. AI-powered automation, predictive analytics, personalized recommendations, and improved decision-making are just a few of the benefits that await organizations that have prioritized data readiness. 

The Roadmap to AI-Ready Data

Defining AI use cases, assessing data quality, and embracing integration are essential pillars of successful AI implementation. Organizations that strategically combine these aspects can unlock the true potential of AI, making informed decisions, identifying opportunities, and gaining a competitive edge in the data-driven era. 

 

To help you navigate the complexities of preparing your data for AI, OneSix has authored a comprehensive roadmap to AI-ready data. Our goal is to empower organizations with the knowledge and strategies needed to modernize their data platforms and tools, ensuring that their data is optimized for AI applications. 

 

Read our step-by-step guide for a deep understanding of the initiatives required to develop a modern data strategy that drives business results.

Get Started

OneSix helps companies build the strategy, technology and teams they need to unlock the power of their data.

Contact Us

Benefits of a Modern Data Organization for Logistics Companies

Benefits of a Modern Data Organization for Logistics Companies

Published

February 1, 2023

Data & AI Strategy

One Six has worked with an array of logistics companies over the years, and there are several common challenges that we hear when consulting with our logistics clients.

For those of you in the logistics industry, the below challenges probably seem all too familiar.

Relying on multiple systems to operate your business (Order Management Systems, Warehouse Management Systems, Yard Management Systems, Transportation Management Systems, CRM, Invoicing/Finance Systems) that are not ideally integrated.
Massive amounts of transactional data from the above systems that can get out of hand quickly (especially during peak shipping periods) and are rarely standardized across systems and operations.
Large complex data sets, from lack of data governance, that make it difficult to streamline processes and automate operations.
Operating on extremely thin margins while needing to navigate rising costs in fuel prices, labor costs, and other expenses.
IT teams spending the majority of their time resolving customer issues versus proactively developing enhancements for the business.
Increasing competition and mergers and acquisitions across the industry.

And without a modern solution in place, these challenges were resulting in inefficient operations that negatively affected their already tight margins, low customer satisfaction due to late, missed, or incorrect shipments, and decreased market share from competitors offering enhanced features and under-cutting them on price.

Fortunately, modern data solutions are available to address all of these challenges and avoid encountering the issues our logistics clients were dealing with. However, many logistics companies are still operating as legacy data organizations versus modern data organizations.

There are several key differences between modern and legacy data organizations, including the following:

Technology

Modern data organizations typically use advanced technologies such as cloud computing, big data platforms, and machine learning to collect, store, and analyze data. Legacy data organizations, on the other hand, may still be using older, less advanced technologies that are less capable of handling large amounts of data and providing insights.

Data sources

Modern data organizations typically collect data from a wide variety of sources, including internal systems, external sources, and IoT devices. Legacy data organizations may only collect data from a few sources, such as internal systems or customer interactions, which can limit their ability to gain insights.

Data management and analytics

Modern data organizations typically use advanced analytics and machine learning techniques to gain insights from their data, and they integrate these insights into their business processes and decision-making. Legacy data organizations may still rely on traditional methods such as manual analysis and reporting, which can be time-consuming and may not provide as much value. Additionally, legacy data organizations often keep their data siloed in separate systems across their organization, preventing a unified understanding of it.

How to start building a Modern Data Organization:

Start with a clear data strategy.

The enterprise data strategy defines the organization’s data objectives and priorities and outlines a plan for implementing and managing data management and analysis processes. This ensures that the organization is using its data in a way that aligns with its business goals and objectives, and that the data is managed and analyzed in a consistent and effective manner.

The organization should then select and implement modern data platforms and tools that are suitable for their specific data needs.

This involves choosing platforms and tools that are scalable, flexible, cost-effective, and easy to use. They should also offer a range of powerful features and capabilities for data processing, analysis, and visualization.

Once those tools are in place, the organization should establish modern data processes and teams that are well-versed in data management and analysis best practices.

They should define clear roles and responsibilities for data management and analysis and provide training and support so that their teams are equipped to handle the organization’s data effectively.

The organization should then develop native-data applications that are designed to leverage the organization’s data in a practical and meaningful way. 

This can involve building applications that help to improve decision-making, streamline processes, and drive business value, using the organization’s data as a key input.

Finally, the organization should invest in advanced analytics capabilities—such as machine learning and artificial intelligence—to help them uncover insights and patterns in their data that wouldn’t be possible with traditional methods. 

These analytics capabilities improve decision-making, streamline processes and product innovation, and drive business value.

Making the transition to a modern data organization can be a challenge for logistics companies since managing and analyzing large amounts of data can be complex and time-consuming, especially since logistics companies typically have a diverse range of data sources and systems.

This is where One Six can help! We have extensive experience helping logistics companies navigate the complexities and simplify the path to data value.

And making the transition to a modern data organization can provide several benefits for logistics companies, including:

Improved operational efficiency: By using data to optimize routes, delivery schedules, and inventory levels, logistics companies can improve their operational efficiency and reduce costs.
Increased visibility: A modern data organization can provide logistics companies with real-time visibility into their operations, allowing them to quickly identify and respond to any issues or opportunities.
Better decision-making: With access to real-time data and analytics, a modern data organization can provide logistics companies with actionable insights that can help them make better decisions.
Predictive analytics: A modern data organization can use machine learning and predictive analytics to anticipate future trends and plan accordingly.
Improved customer service: By using data to track and monitor shipments, a modern data organization can provide logistics companies with the ability to provide real-time updates on delivery status, resulting in improved customer service.
Competitive advantage: A modern data organization can provide logistics companies with a competitive advantage by helping them to improve their operations, reduce costs, and make better decisions.
Digital Transformation: A modern data organization also helps logistics companies to be more digital-savvy and stay ahead of the competition by leveraging new technologies and platforms.

Get started transitioning to a Modern Data Organization

If you would like to see how we can help transition your business to a modern data organization, contact us today for a Free Consultation.

Contact Us

A Modern BI Primer (And Why Less is More)

A Modern BI Primer (And Why Less is More)

Published

March 5, 2020

Data & AI Strategy
Snowflake
Matillion
Power BI
Tableau

With a data technology landscape that is ever changing, it is at times a challenge even for technology firms to sort through the latest key terms and platforms that pervade the marketplace, each touting its advantage over the status quo. How much more a challenge, then, for organizations where technology is not their primary expertise? From a myriad of choices to build a modern data architecture for a competitive edge, what must companies buy and leverage to start or pivot their data journeys?

This primer is intended to educate a growing number of clients we see in the Small and Medium-Size Business (SMB) segment who are looking for a good framework to consider as they think through their future data capabilities.

So what makes a business intelligence (BI) or data architecture? And what do we mean when we say less is more as it relates to this architecture? Regardless of the technology stack that eventually gets approved and implemented, our team at One Six likes to speak of tools and platforms in terms of four big buckets:

Data Acquisition

What is it? This bucket is the piece of the architecture where we consider the movement of data from source systems to a centralized location for holistic analytics.

Why less is more: There are products in this space today that frankly provide faster and robust capabilities for clients than connectors custom built by consulting firms or in-house developers. With a few clicks of a button, platforms such as Fivetran or Matillion’s Data Loader provide full data replication of a wide array of applications (e.g., Salesforce, Google Ads) and databases (e.g., SQL Server, MongoDB) into a cloud data warehouse. Fivetran, in particular, detects changes to the source system automatically and moves data over seamlessly. Subscribing to managed services by leaders in this space means less maintenance activities for your internal IT group and more time to focus on value-added tasks downstream.

Data Warehouse

What is it? Whether you hear terms like data lake, data warehouse, or data lakehouse, here we are discussing a part of the architecture that stores information from disparate systems for historical, current, or forecasted analyses.

Why less is more: Rather than trying to estimate the right-sized on-premise hardware and software that is required to house all current and future data, consider a cloud data warehouse (CDW) that scales storage and computing power up and down as you need within minutes. Unless you are an organization at the scale of Netflix or Airbnb, a complex architecture is not necessary. A simple yet powerful cloud data warehouse like Snowflake provides a fully managed, pay-as-you-go service that is secure and costs are based only on the storage and compute you use. Again, less burden on your internal team, and more flexibility to build the structures you need to analyze your data efficiently. Plus, as your company grows and data volume grows as a result, the CDW can grow with you.

Data Transformation

What is it? When it comes to data, there have always been and will still be a need to clean, wrangle, and structure data in a way that makes sense for reporting. This bucket relates to the architecture piece where this transformation takes place.

Why less is more: Putting this as the third bucket is intentional. In the old world, a separate staging server was required to process data transformations prior to loading into a data warehouse. In addition, not all data from the source moved to the data warehouse due to cost of storage and compute. Today, because of the cost effectiveness and power of the CDW, data transformations can happen after the data is fully replicated from the source to the warehouse on a repeatable schedule by leveraging tools like Matillion. This means the transformations happen in the data warehouse itself so you can remove the maintenance and costs of a staging server. Even more, the low cost of storage allows the data warehouse to have all data from the source system immediately available for any changes to reporting needs.

Data Analytics

What is it? This bucket is the most visible component to a BI or data architecture. Here we are looking at product capabilities for building standardized dashboards as well as ad-hoc analyses that will be consumed by a wide audience in an organization.

Why less is more: Whether it is Tableau, Power BI, Qlik Sense, or any number of tools in this space, cloud offerings provide a fully managed instance that provides a browser-based, single access point for dashboards and reports. Less management and increased accessibility. In general, while each tool has its advantages, we continue to see tools moving towards parity. Here, a data governance strategy is key is reducing the number of data sets and reports while promoting the reusability of well-structured, certified data sets. This reusability increases the trust of the data and also empowers business users to find answers to questions on their own not found in standardized dashboards. And with tools like Ki and DataRobot that can augment established BI tools with its artificial intelligence and machine learning capabilities, we see increasing ways for analysts to solve business problems that we can provide guidance on.

Final Thoughts

So there you have it – Data Acquisition, Data Warehouse, Data Transformation, Data Analytics – four big buckets to easily see what you need to consider in a modern BI architecture. We hope that this modern BI primer has been helpful as you take the next steps into your organization’s data journey. Please note that the above technology platforms listed were used as examples only. The team here at One Six Solutions works with a wide array of technologies in the data world. Our goal is to design and build an architecture that works best for you based on your industry and business needs. Let us know how we can help.