Data Analytics

Financial Services Companies Need All the Data Analytics They Can Get

Actian Corporation

April 8, 2020

There are times when financial markets trend up. There are times when markets trend down. And then there are times when the markets go crazy and the only thing predictable is volatility. When markets encounter these volatile periods, financial services companies become highly reliant on data analytics to determine where the real market forces are coming from and determine how best to react. There are five key analytics capabilities that financial services companies need to perform effectively in a volatile market environment.

Connected Data

The more data you have available to analyze, the more accuracy you can gain. There is a whole science around correlation analysis that we won’t go into here, but in general, adding more diverse data sets to your analysis gives you more variables to analyze. This, in turn, increases the likelihood of discovering strong correlations between market forces and market performance. The challenge that financial services companies encounter (and the capability they need to develop) is integrating many data sources together to feed their analytics algorithms.

Robust Analytics

Modern financial analytics isn’t done manually; they leverage advanced technology. Financial services companies must fully harness Artificial Intelligence (AI) and Machine Learning (ML) with real-time connected data warehousing of data from a disparate range of customer, business partner, and even governmental sources to fully optimize growth, profitability, and business risk. Your analytics algorithms are what comb through the data, looking for trends, relationships, and meaningful outliers that can then be transformed into actionable market insights. More robust analytics capabilities enable you to analyze more data and discover more meaningful insights.

Engagement Tools

Most financial services companies don’t operate in isolation. They are part of a greater services value chain, building on the work of upstream suppliers, providing value-add services, and providing capabilities to a group of downstream customers. During volatile market times, it is essential that these companies have the capabilities to deliver critical news, information, and analytics to the global financial community and their customers – enabling transactions and connecting communities of trading, investing, financial and corporate professionals.

Fraud Detection and Prevention

Market turmoil is distracting for financial services companies and their customers. Hackers and thieves know this and won’t hesitate to exploit the opportunity to attack. Distractions increase the risk of fraud, so ensuring robust fraud detection and prevention mechanisms is essential. The key is making your fraud systems adaptive, leveraging core Artificial Intelligence and Machine Learning capabilities, and feeding them the right data for training and query. AI and ML systems have powerful capabilities for identifying data anomalies, unusual behavior, and executing automated responses. With AI guarding your operations, equipped with robust, real-time data, fraudsters don’t stand a chance.

Data Processing at Enterprise Scale

The most important capability that financial services companies need to navigate through the murky waters of a volatile market is high-performance data processing. You can have access to all the data in the world, the best algorithms, tools for communicating with your customers, and state of the art fraud monitoring, but if you don’t have the processing power to support these things, you have a real problem. That is where Actian comes in.

Actian Data Platform is a connected data warehouse that is designed for massively parallel processing of streaming data in real-time. With the Actian Data Platform, you can monitor news, social feeds, market performance, customer transactions, competitor actions, and more – analyzing these data sources in real-time to determine what is noise and what is important.

Because you’re dealing with streaming data about the market forces and the market conditions that are changing rapidly, it is essential that your analytics system can operate at an enterprise scale with near-zero latency. Actian can deliver.

Learn more about Actian solutions for the financial services industry.

About Actian Corporation

Actian empowers enterprises to confidently manage and govern data at scale. Actian data intelligence solutions help streamline complex data environments and accelerate the delivery of AI-ready data. Designed to be flexible, Actian solutions integrate seamlessly and perform reliably across on-premises, cloud, and hybrid environments. Learn more about Actian, the data division of HCLSoftware, at actian.com.

Data Management

Making the Transition From Flat Files to SQLite

Actian Corporation

April 7, 2020

puzzle piece missing to depict going from flat files to sqlite

In December, January, and February, I posted a short series of blogs on Flat file systems. The series of blogs focuses on the continued use of flat files and why they are no longer viable for use in the future.

The first installment focused on flat files and why embedded software application developers readily adopted them. Then in the second installment, I discussed why embedded developers are reluctant to use databases. The third installment looked at why developers cling to the flat file systems. The final installment provided a checklist for embedded developers migrating off flat files.

For this next series, I’d like to turn my attention to the next stepping-stone in many embedded developers’ paths away from Flat Files, SQLite. While it’s progress, I’d like to convince you that you should step right over this stone or jump off it if you’re still teetering on it.

Perhaps we should start with the point of why and how it’s a positive step up from flat files. That, of course, would have to be with respect to where and how it’s applied to real-world requirements. According to the SQLite folks themselves, as stated on the SQLite website, Appropriate uses for SQLite: “SQLite does not compete with client/server databases. SQLite competes with fopen().” This statement is tied to their fundamental tenant that they aren’t meant to be compared with SQL database engines because SQL database engines are intended to be a shared repository for Enterprise data. To be a shared repository, according to the SQLite site and therefore the community, an SQL database engine needs: Scalability, Concurrency, Centralization, and Control. SQLite, on the other hand, is focused on local data storage for individual applications and devices where there is a requirement for zero database administration (they then provide the standard laundry list of typical IoT devices).

The SQLite folks also say it would be great in embedded environments if you could have a single compact file format to support cross-platform data transfer and as well as ad-hoc, home-grown data sets of multiple data types. In fact, SQLite has done benchmarking to show they are 35% faster than file systems and reading and writing this type of data through fread() or fwrite()¹.

We agree with most of what SQLite folks are saying: yes, SQL database engines should meet the requirements listed above. Yes, there is a need for local data management that is portable across platforms and can support multiple data types; and, finally, there is definitely no way to avoid a zero-administration environment at the edge for mobile and IoT.

So, with that being said, “U” know what’s coming next, that three-letter word that begins with a “B” and ends with a “T”: BUT. But what if you could have your cake and eat it too? But what if you could get all the features and advantages of Enterprise shared datastore characteristics from an SQL database engine, yet have the ability to scale it down and embed it directly into a local Mobile or IoT application, supporting multiple data types, portability across platforms and required zero database administration?

The answer you should give is yes. However, you may push back with the following reasonable positions:

Firstly, SQLite and file systems are free and popular. Open source SQL database engines (MySQL, Postgres, MariaDB, etc.) aren’t able to run scaled down to an IoT or Mobile footprint. Neither can I run them embedded in my applications.

Secondly, why do people use such a stupid analogy like you can have your cake and eat it too? It’s more like I need a car, not a truck, that’s overkill, so why bother?

Well, my response to you would be that’s so 2015 (by the way, that’s the last time SQLite updated their “appropriate use” page). The fact is, in 2020 and definitely, in 2025, you will need the functionality of a car and a truck with any application you build anyway. Yes, you need to store and analyze far more data locally, even with WiFi-6 and 5G bandwidth. Yet, the sheer increase in the volume of data and the need to handle peer-to-peer and edge-to-cloud device management, sharing of contextual data for governance, common operational pictures, and the like will dictate as much as possible and will still need to take place locally to avoid latency.

Furthermore, many peer-to-peer and edge-to-cloud operations – not to mention gateway operations where you’d take in data from multiple downstream sources – require concurrency for, and control of, those downstream data sources. Gateways and edge datastores will also require scalability such that you can use the same architecture and data portability across platforms. Finally, as you move more functionality to that gateway that would have been in the data center or the cloud, what was considered centralization functionality needs to move there as well.

So, think of this as your car needs two-wheel drive in most instances, but it needs the option of all-wheel drive in the rest. Your car also may need towing capacity. But, also think of this in the reverse, your truck isn’t always towing a boat or being used for off-roading, and maybe it’s carrying additional passengers, and you want many of the comforts of a luxury car.

To summarize, with respect to data management and this analogy, the edge in 2020 and into the next five years requires everything that was needed in the data center for an SQL Data Store (your truck requirements). But also, all that was needed for local device data management when it was standalone (your car requirements).

In essence, you need an SUV that scales from a very small CRV up to a monster size one. This is, unfortunately, not what SQLite is capable of doing, and invariably, when you use SQLite, you’re forced to bolt it on to some other database on the other end. We’ll discuss the drawbacks of this forced marriage in the next blog.

Ready to reconsider SQLite, learn more about Actian Zen. Or, you can just kick the tires for free with Zen Core which is royalty-free for development and distribution.

About Actian Corporation

Data Analytics

Healthcare Analytics Improve Operational Efficiency

Actian Corporation

April 6, 2020

Healthcare is a data-driven industry with data analytics needs that make “big data” seem tiny. For healthcare companies to operate efficiently, they need a high-power data engine to crunch the numbers, analyze the data, and get it into the hands of decision-makers quickly. The healthcare industry is the epitome of business agility, and data analytics is what makes that possible. Here are four examples of how healthcare analytics can help healthcare companies improve operational efficiency.

Healthcare Analytics for Diagnosis and Patient Care

If a patient walks into a clinic with a fever, dry cough, and respiratory symptoms, do they have seasonal allergies that require some antihistamine and a box of tissues or do they have a highly contagious disease like COVID-19 requiring quarantine?

To enable the clinic to diagnose and treat the patient effectively, they need the ability to combine the observations being made with this patient against available information from other providers around the world. They also need to follow guidelines from government organizations, research from drug companies, and many other sources to systematically work through a diagnostic process and determine what the patient’s real problem is and how best to care for them. Healthcare analytics, running behind the scenes at the clinic, in healthcare networks, and in government organizations provide clinicians with the tools they need to do their job effectively.

The diagram below provides a glimpse of how disparate data sources feed the healthcare analytics systems that are used to improve operations and outcomes:
healthcare analytics diagram

Hospital Operations

Hospitals are massive logistics operations with lots of moving parts. From staffing to patient scheduling to stocking of drugs and supplies, ensuring these operations run smoothly requires high-power data analytics. How many nurses are needed for next Tuesday’s day shift? Are there ICU beds available to handle post-operative care for the people undergoing surgery tomorrow? Does the hospital have enough ventilators and beds to treat COVID-19 patients in addition to the typical patient load? Does the hospital blood bank have enough supply on hand to support the demand for the next couple of weeks?

These are the real questions that hospital administration and operations staff need to answer every day. The answers to these questions come from careful analysis of past trends, current stock levels and patient loads, seasonal forecasts of demand, and modeling of potential “unexpected events.” Failure isn’t an option. You can’t just say to a patient, “I’m sorry, we’re temporarily out of stock of ventilators, but don’t worry, we’ll have more in next month.” Healthcare doesn’t work that way. Healthcare companies use data analytics to plan for the unknown so they can be adequately prepared.

Clinical Trials

The development of new drugs, medical devices, and treatment procedures is a collaborative effort between manufacturers, CROs, healthcare providers, and governmental organizations. Clinical trials are a necessary part of the healthcare process to ensure the safety and efficacy of medical care. Clinical trials involve robust data collection and protocol adherence to ensure the confidence of the results and data that are produced. Data analytics plays an essential part in the clinical trial process. It is used to aggregate data being collected in the trial to analyze and interpret findings and share results with the stakeholders participating in the clinical trials. Data analytics are also employed to investigate adverse reactions, isolate testing anomalies, and quantify the risks of new procedures, drugs, and devices.

Insurance Claim Processing

Processing insurance claims accurately and claims accuracy efficiently leads to higher reimbursement rates and faster payments to hospitals and providers. For small clinics and independent providers, efficient claims processing with insurance companies is essential for maintaining cash flow to keep your business operating. For hospitals and large organizations, the complexity of reconciling the activities taking place over many departments with claims approval and processing spanning multiple insurance companies can be a logistical nightmare.

What do all healthcare companies have in common? Frustration. Data integration and analytics can help healthcare providers to improve their claims processing accuracy and reduce the processing time. Automating the flow of data and performing real-time analysis on patient records to identify potential errors and issues in the insurance reporting process so they can be addressed at the time of treatment instead of needing to be reconciled and fixed days (or months) later.

If you want to do healthcare analytics right, you will need a high-power analytics engine designed for the scale, speed, and performance that the healthcare industry demands. Actian Data Platform provides the analytics capabilities healthcare companies need at an enterprise scale at an affordable price. To learn more about Actian’s solutions for the healthcare industry, visit https://www.actian.com/solutions/by-industry/healthcare/

About Actian Corporation

Data Architecture

Data Warehouse Appliances are Becoming History

Actian Corporation

April 6, 2020

The era of data warehouse appliances is coming to an end, rapidly being replaced with a new generation of cloud services. The first data warehouse appliance was introduced onto the market in 2003. For nearly 15 years, this market grew with several large technology players, including IBM, Oracle, and Teradata developing or acquiring database appliance offerings.

These self-contained hardware devices enabled companies to scale their data warehouse infrastructure to support the expanding needs of business analytics. We are now entering a new era when the hardware solution to the data warehouse scaling problem is being replaced with cloud services – enabling even greater elasticity, scale, speed, and performance while lowering the capital costs of hardware infrastructure.

Data Warehouse Appliances Served Their Purpose

A data warehouse appliance is a stand-alone set of hardware (servers, memory, storage, and I/O channels), loaded with an operating system, database management system, and administrative tools) designed to support a multi-node database deployment. They were sold as a self-contained unit that could be installed in a data center, came pre-configured for redundancy and high availability, and often included service and support from the vendor that manufactured them.

Data warehouse appliances did an excellent job of solving the data warehouse scaling problem in the early 2000s for companies that wanted to run enterprise-scale analytics on their OLTP datasets but didn’t want to design, build and operate data warehouse infrastructure themselves.

Data warehouse appliances gave companies a “ready to install” solution instead of a “box of parts” with some assembly required.

The big challenge with these systems is that they were expensive to acquire and costly to operate. Scaling via hardware isn’t something that can be done quickly. There are lead times to procure, install and configure new equipment, and capacity increases couldn’t be made in small increments – meaning companies often had to purchase (in advance) capacity that they wouldn’t need for a few quarters. At the time, however, this was the best option on the market, and companies were happy to have data warehouse appliances available as their time-to-value was faster than a do-it-yourself approach.

Hardware Refresh Cycles and the Shift to the Cloud

Technology changes quickly, and (fortunately) new capabilities are being introduced every day that offer companies computing options with increased performance, capacity, and scalability along with decreasing costs.

These new developments cause existing systems to lose relative value, driving the need (and desire) for hardware refresh cycles. Data warehouse appliances (because they are hardware products) have a finite useful life before it is advantageous for companies to replace them with faster, cheaper alternatives. In most cases, the expected life of a hardware data warehouse appliance is about ten years. That is important because the peak of the database appliance market was the period from 2008-2013, and those systems are now due for refresh and replacement.

Over the past decade (since many database appliances were installed), companies have adopted and embraced cloud services as a viable enterprise computing option. Cloud data warehouses (like Actian Data Platform) go beyond the “ready to install” capabilities of database appliances and provide “ready to use” services that lower operating and maintenance costs even further. Instead of simply upgrading to a newer version of their hardware data warehouse appliances, many companies are evaluating cloud data warehouses as a preferable alternative. In addition to ease of administration, cloud data warehouse solutions also provide three key benefits over hardware data warehouse appliances.

No capital outlay costs – you pay for what you need when you need it.
Dynamic scalability – with hardware, you have the capacity you have. With the cloud, you can scale up or down, depending on your business needs.
Continuous technology refresh – you don’t have to wait ten years to upgrade when new capabilities are available, you can quickly adopt them.

Data warehouse appliances are becoming history. It is time to move to a better, cheaper, faster alternative. If you are a company that is using data warehouse appliances to support your data warehouse today, you should consider a shift to the cloud or hybrid-cloud at your next refresh cycle. Actian provides the next generation of hybrid cloud data warehouse capabilities designed for the needs of modern business. Actian provides cloud-scale performance, availability, and resiliency with a cost model that better aligns with the needs of today’s companies.

To learn more, visit www.actian.com/data-platform.

About Actian Corporation

Data Intelligence

Data Ops Rules to Avoid Data Oops

Actian Corporation

April 6, 2020

Data Ops is a new way to address the deployment of data and analytics solutions.

The success of this methodology is based on techniques that promote faster, more flexible, and more reliable data delivery. To deliver on this promise, let’s take a moment and analyze this sentence: “The focus is not just on building the systems right, but also building the right systems”.

Many different definitions, interpretations, and publications address DataOps as a concept, but it is much more than just that. It is a way of understanding, discovering, producing analysis, and creating actionable intelligence with data. In a changing world that revolves around data, latencies in data products or their analysis are no longer acceptable.

The entire organization must be put to work to support the deployment and improvement of data and analysis projects!

Data Oops Definition

The concept of DataOps emerged in response to the challenges of failing data systems and failing data project implementations, but also the fragility, friction, or even fear when it comes to the use of data. If you are experiencing this situation, then don’t look too far…you are in the middle of a Data Oops!

In this context of Data Oops, you will agree that your data teams are struggling to achieve the speed and reliability of directed projects.

The main reasons are that companies have too many roles, are too complex, and have constantly changing requirements or objectives, making tasks difficult to frame and deliver.

This complexity is exacerbated by a lack of confidence in data, even to the point of “fearing” it. This occurs when we observe limited or inconsistent coordination between the different roles involved in the construction, deployment, and maintenance of data flows. We are convinced that an organization that does not know its data is doomed to fail.

How to Succeed in Your DataOps

Simply put, DataOps is a collaborative data management practice that aims to improve communication, integration and automation of data flows between data managers and data consumers within an organization. It is based on the alignment of objectives confronted by results. DataOps accepts failure and is built through continuous experimentation.

Here’s a list of principles for successful DataOps:

Learn from DevOps, through their techniques for developing and deploying agile applications in your data and analysis work.
Identify quantifiable, measurable and achievable business objectives. You will then be able to communicate more regularly, progress towards a common goal and adjust more easily.
Start by identifying and mapping your data (type, format, who, when, where, why, etc.) using data catalog solutions.
Encourage collaboration between different data stakeholders by providing communication channels and solutions for sharing metadata.
Take care of your data, as it may produce value at any given time. Clean it, catalog it, and make it a part of your enterprise’s key assets, whether it is valuable now or not.
A model may work well once, but not on the next batch of data. Over-specifying and over-engineering a model will likely not be applicable to previously unseen data or for new circumstances in which the model will be deployed.
Maximize your chances of success of introducing a DataOps approach by selecting data and analysis projects that are struggling due to a lack of collaboration or are struggling to keep pace. They will allow you to better demonstrate its value.
Keep it agile, short designed, develop, test, release, and repeat! Keep it lean and build on incremental changes. Continuous improvement is found when a culture of experimentation is encouraged and when people learn from their failures. Remember, data science is still science!

What are the Benefits of DataOps?

DataOps helps your business move at the speed of data – keeping pace to deliver the right data. It focuses data activities to be aligned with business objectives, and not on the analytic inputs (big data hype). DataOps also focuses on delivering value from all your data activities, from even the smallest of these can inspire cultural changes needed for other implementations to come.

Adopting DataOps in a culture of experimentation is good data practice and empowers the innovators across the organization to start small and scale fast. It is the path to good business practices.

About Actian Corporation

Data Architecture

The More You Refine Stored Data, the More Valuable it Becomes

Actian Corporation

April 1, 2020

Having a lot of data available to you is a good thing, right? That depends. Data is a raw material, like a mineral mined from the ground. It contains a potential for creating value, but that potential is only realized through refinement.

Your company produces a lot of data every day (every second really). Merely creating and/or possessing this data doesn’t mean it is generating value for you. Harvesting value from your company data requires a transformation process to convert data into information, actionable insights, to decisions, and then to action.

Data management professionals, IT staff, and business analysts are the people responsible for guiding data transformation. They employ a series of refinement steps to convert the raw materials that your operations generate into meaningful and actionable insights that decision-makers across the company use to direct your staff, processes, and resources. Here is an overview of the refinement steps your data goes through, and the value addition that takes place along the way.

Collection

Data exists in your operations whether you collect it or not. The first step in data refinement is collection. This takes place within your operational systems, embedded sensors, and transactional workflows being executed throughout your company. Some data is collected in real-time through sensors, telemetry, and monitoring, while other data is collected periodically (perhaps hourly or at the end of the day). Data collection is all about measurement. The data management adage goes, “If you don’t measure it, you can’t manage it.” Extending this a step further, if you don’t collect the data, you can’t use it for decision-making.

Aggregation

There are many data sources across your organization, and no single source contains all the information that is needed for effective decision-making. Why is this?… because each data source provides a point of view on your operations. Using a single data source is like walking around a sports arena at night with only the light of a flashlight – you only see a very narrow view of your environment, not the big picture. Data aggregation brings the data from various sources together in one place, like illuminating a bunch of lightbulbs in that sports arena. Some data will overlap, so it can be filtered out, and there will be some gaps and shadows, but aggregation gets you one step closer to seeing the bigger picture of your operations.

Reconciliation

Once you have your data aggregated in one place, the next step in the refinement is reconciling the different data sets together to address gaps, overlaps, and conflicting information. This is also sometimes called data harmonization. A way to image this is considering the days before digital cameras when people took photos on film. To create a panoramic image, you took multiple pictures of adjacent scenes and then (after waiting for them to be developed) aligned the images by overlapping the frames into a panoramic view. Data reconciliation is similar, although considerably more complex. Some of the factors used in data reconciliation are data source, data quality when the data was captured (because you’re not viewing a still image, business data is a moving target). The result of data reconciliation is a unified data set that includes inputs from all your data sources.

Categorization

Categorization (often called cataloging) is the first step in understanding the content of your data. The purpose of categorization is to help you understand “what your data is.” Note, this is different from understanding “what your data means,” which is addressed in the next step. The best way to understand data categorization is a library full of books. Individual books represent different pieces of data. Librarians use a cataloging system (Dewey decimal system, the library of congress, etc.) to sort and organize books according to their content.

In the business world, companies have data metamodels which provide the cataloging structure. Categorization is all about aligning operational data (from whatever source it was collected) to these metamodels, so like concepts (such as customer data) can be analyzed together. This is when data is transformed into information.

Analysis

Data analysis is all about understanding what your information means. Data and business professionals are summarizing, sorting, filtering, correlating, projecting, performing trend analysis to refine categorized information into meaningful and actionable insights about your business. It’s interesting that the data showed a specific process step took 2.385 seconds. It’s informative to know that the process measurement was the time it took to authorize a credit card transaction. But is that number good or bad? Is it relevant? Does it indicate something is wrong? Does someone need to initiate action because of it? Data analysis is the refinement step that converts information into insights about your business.

Presentation

Possessing data, information, and insights does not create value for your organization. Value comes from the decisions you make and the actions that result from interpreting the data. The last step in the data refinement process is taking the insights that you’ve generated and presenting them to decision-makers, system operators, and making them available for automation systems. Just as you aggregated data earlier in the process, this step involves disseminating, publishing, and visualizing the insights for consumption.

The quality of data insights available for presentation is directly related to the effectiveness of your collection, aggregation, reconciliation, categorization, and analysis processes. Actian provides a set of data management capabilities to help your staff in orchestrating the refinement process – enabling not only these necessary steps but also the implementation of robust activities within these steps. Data becomes more valuable, the more you refine it.

With the right tools, you will be able to develop better insights faster. This will lead to better decisions and greater value realization for your company. The Actian Data Platform Data Warehouse includes connectors to hundreds of data sources and functions to refine and transform raw data into information.

You can learn more about Actian’s Real-Time Connected Data Warehouse at https://www.actian.com/solutions/connected-data-warehouse/

About Actian Corporation

Data Integration

Five Reasons Why You Need an Enterprise Data Integration Strategy

Traci Curran

March 31, 2020

Modern businesses don’t run on a single application – they leverage many IT systems to provide the capabilities to enable operational processes and users to be effective. To ensure complex IT environments run smoothly, companies are placing increased focus on how systems are integrated, and the activities required to manage integrations across the ecosystem. While system integration is a multi-layered challenge, the essential part to get right is your enterprise data integration strategy.

An enterprise data integration strategy is a set of policies, processes, design guidelines, and governance controls that you put in place to ensure data integrations are implemented consistently, controlled centrally, and are fully supported by your IT function. If you think about it, data integrations are applications in themselves.

They are a set of functions that move data from one system to another, monitor the flow of data, enforce security rules, and enable business processes. When you consider how many integrations your company has (typically 3-5x the number of applications), it is clear why you need a holistic enterprise data integration strategy to ensure these critical IT components are well managed.

The following is a list of 5 reasons why you need an enterprise data integration strategy for your company

1. Save Time and Resources in Building Integrations

How many ways are there to connect two systems together? If you don’t have any standardized design patterns for data integration, the answer is infinite. More importantly, how much time will your expensive IT staff spend brainstorming and designing novel ways of doing integrations that result in increased complexity and risk a few years down the line when your IT service management (ITSM) team is trying to support them? An enterprise data integration strategy should include a set of design patterns for integrating IT systems that your developers can choose from. If you have a data integration platform like Actian DataConnect in your environment, you can use the integration templates as a starting point.

2. Ensure Business Continuity With Fewer Disruptions to Business Processing

You have Service Level Agreements (SLAs) for application performance and availability, do you have SLAs for your data integrations? Most companies don’t. When an integration fails, how quickly are you able to respond and repair it? An enterprise data integration strategy should include a set of guidelines for how to monitor the availability and performance of your data integrations to ensure any failures that could impact business services and operational processes are identified quickly and resolved. Most companies have ITSM functions in place, with incident management teams ready to respond to application and hardware failures. With the right data integration strategy, you can enable enterprise support for integrations as well.

3. Centralized Governance and Management Lowers Risk

Your enterprise data integration strategy should include a plan for how you will manage integrations once they have been developed. This includes things like access control (who can see the integrated data), change management processes, rules for extending integrations (re-using them for other purposes), management of system credentials, and data encryption. The strategy must go beyond a set of published guidelines and be supported by a data management system that can help you enforce controls and monitor compliance across the organization. Centralized governance and visibility is a critical part of managing both the risk of business disruption and ensuring efficient use of organizational resources.

4. Faster Response to Threats

Data security threats are a real problem for organizations that have a high dependency on IT systems. Your enterprise data integration strategy provides your company with an extra set of tools for defending against intrusion and responding to data security threats. Managing your integrations in a consistent way (with the governance controls discussed earlier) not only enables you to control the free-flow of information across your organization, but it also means that you have control points to shut-off that flow if a threat is identified. With consistent integration patterns and central control plane, you can also (confidently) shut any backdoors that could enable an intruder to bypass your security. Once a threat is contained, you can then update credentials, secure the environment, and restore business processing to normal.

5. Increased Agility in Times of Change

Recently there has been a lot of discussion about enterprise business agility – the need for companies to adapt their processes, systems, and strategies in response to forces and events (both inside and external to the company like COVID-19). An agile enterprise data integration strategy that is consistently applied across the organization and supported by the right set of data integration tooling can help your company reduce the technical debt (the spaghetti pile of system integrations) that prevent you from safely implementing changes. It can also provide you the tools to change your existing integrations and create new ones faster – meaning you can change your overall IT systems more quickly to respond to evolving business needs and strategic opportunities.

Actian DataConnect is an industry-leading data integration platform, enabling you to connect anything, anytime anywhere. DataConnect can help bring your enterprise data integration strategy to life – providing a centralized set of tools for implementing and managing the data integrations across your company.

To learn more, visit DataConnect.

About Traci Curran

Traci Curran is Director of Product Marketing at Actian, focusing on the Actian Data Platform. With 20+ years in tech marketing, Traci has led launches at startups and established enterprises like CloudBolt Software. She specializes in communicating how digital transformation and cloud technologies drive competitive advantage. Traci's articles on the Actian blog demonstrate how to leverage the Data Platform for agile innovation. Explore her posts to accelerate your data initiatives.

Data Analytics

Real-Time Reporting Platform for Operational Systems

Actian Corporation

March 30, 2020

interface showing future computer technology for operational systems

Digital transformation has drastically expanded the use of operational systems, IoT, and other technologies within companies’ business processes. This is a good thing. It means that end-users, whether they be employees, partners, or customers, are more productive. The introduction of modern operational technologies and integrating them into business processes is only the first step in the digital transformation journey. Next comes a focus on managing operational data and developing real-time reporting platforms that give operators and decision-makers better control over digitally transformed processes.

Digital Transformation has Created More Real-Time Data

A farmer with a tractor is more productive than a farmer with a hoe and rake. That was the story 20 years ago when companies embraced the first wave of IT in things like CRM and ERP. Today there is a new story “A farmer with a drone, a robotic tractor, and data at his disposal can increase crop yield and bigger profits.” The difference here isn’t the technology – technology is continuously improving and improves efficiency/productivity incrementally. The change is how the modern farmer uses data. New farm technology generates real-time data, but it is the reporting platform that enables the farmer to make better-informed decisions, directing his/her time and the farm’s resources in more effective ways.

Data-Driven Decision-Making Drives Value and Profits

Most people reading this article aren’t farmers – this situation holds true for other businesses too. The digital transformation brought with it a technology upgrade and the introduction of new operational systems in all industries. Retailers are using new point-of-sale (POS) technology for tracking each individual item that is sold and RFID on shopping carts to track customer flow throughout their stores. This new technology is excellent, but it is the reporting platform that enables them to do dynamic inventory management, optimize stock levels to maximize inventory turnover and store layouts and product offerings to changing customer preferences.

Hospitals and healthcare companies are using recently installed healthcare management systems to capture patient data, share information between providers, and orchestrate orders for tests and medication. These systems make doctors, nurses, and other staff more effective in directing patient care. What you don’t see as a patient is the additional layer of technology on top of the operational system. A real-time reporting platform is used by clinic managers, scheduling staff, and others (similar to the retail example) to ensure that the provider’s time is optimized, exam rooms, and other facilities are fully utilized, supplies, and medications are stocked and available. The data collected from the operational systems and operational processes are aggregated in a reporting platform to enable real-time management decisions.

Telecommunications companies have implemented monitoring within their networks, in mobile apps, and integrated into customer devices (like cable set-top boxes) to capture data about customer viewing preferences and usage of their services. This data is streamed into a real-time reporting platform where it can be analyzed for patterns and opportunities to increase revenue, calculate costs, and identify system issues impacting user experience. Operators and decision-makers then use these insights to optimize networks for performance and peak utilization, provide personalized content offerings/ads to customers, and direct repair and maintenance efforts.

Real-Time Reporting Platforms Unlock the Value Potential of Operational Systems

If you are like many companies, you’ve spent the past few years doing technology upgrades focused on business processes and end-user experience. Are you harvesting the full value from these operational system investments? Real-time reporting platforms like Actian can help you unlock the unrealized potential of your operational systems. Actian can help you aggregate streaming data from all your data sources into one place, analyze trends/patterns, convert data into meaningful insights, and get those insights into the hands of operators and decision-makers faster.

This isn’t about enabling your users and processes; it’s about optimizing your business. Real-time operational analytics and reporting help you identify opportunities, risks, and issues that need to be addressed faster. By seeing the problem sooner, you can address it sooner. By addressing the problem sooner, you can maximize the opportunities, mitigate the risks, and address the issues to optimize both your costs and revenue. The end result… more profits for your business. This is the goal, and a real-time reporting platform from Actian can help you achieve it.

Learn more about Actian’s real-time connected data warehouse at https://www.actian.com/solutions/connected-data-warehouse/.

About Actian Corporation

Data Intelligence

Everything You Need to Know About Data Ops

Actian Corporation

March 26, 2020

“Within the next year, the number of data and analytics experts in business units will grow at three times the rate of experts in IT departments, which will force companies to rethink their organizational models and skill sets” – Gartner, 2020.

Data and Analytics teams are becoming more and more essential in supporting various complex business processes, and many are challenged with scaling the work they do in delivering data to support their use cases. The pressure to deliver faster and with higher quality is causing data & analytics leaders to rethink how their teams are organized.

Where traditional waterfall models were implemented and used in enterprises in the past, these methodologies are now proving to be too long, too siloed, and too overwhelming.

This is where Data Ops steps in: a more agile, collaborative, and change-friendly approach for managing data pipelines.

Data Ops Definition

Gartner defines Data Ops as being a “collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization”. Basically, it makes life easier for data users.

Similar to how DevOps, a set of practices that combines software development (Dev) and information-technology operations (Ops), changed the way we deliver software, DataOps uses the same methodologies for teams building data products.

While both agile frameworks, DataOps requires the coordination of data and anyone that works with data across the entire enterprise.

Specifically, data and analytics leaders should implement these key approaches proven to deliver significant value for organizations:

Deployment Frequency Increase: Shifting towards a more rapid and continuous delivery methodology enables organizations to reduce the time to market.
Automated Testing: Removing time-consuming, manual testing enables higher quality data deliveries.
Metadata Control: Tracking and reporting metadata across all consumers in the data pipeline ensures better change management and avoids errors.
Monitoring: tracking data behavior and the usage of the pipeline enables more rapid identification on both flawed – that needs to be corrected – and good quality data for new capabilities.
Constant Collaboration: communication between data stakeholders on data is essential for faster data delivery.

Who is Involved in Data Ops?

Given the importance of data and analytics use cases today, the roles involved in successful data project delivery are more numerous and more distributed than ever before. Ranging from data science teams to people outside of IT, a large number of roles are involved:

Business analysts
Data architects.
Data engineers.
Data stewards.
Data scientists.
Data product managers.
Machine Learning developers.
Database administrators.

As mentioned above, a Data Ops approach requires fluid communication and collaboration across these roles. Each collaborator needs to understand what others expect of them, what others produce, and must have a shared understanding of the goals of the data pipelines they are creating and evolving.

Creating channels through which these roles can work together, such as a collaboration tool, or metadata management solution, is the starting point.

About Actian Corporation

Data Intelligence

How You’re Going to Fail Your Data Catalog Project (or Not…)

Actian Corporation

March 25, 2020

Many solutions on the data catalog market offer an overview of all enterprise data, all thanks to the efforts conducted by data teams.

However, after a short period of use, due to the approaches undertaken by enterprises and the solutions that were chosen, data catalog projects often fall into disuse.

Here are some of the things that can make a data catalog project fail…or not:

Your Objectives Were Not Defined

Many data catalog projects are launched using a Big Bang approach to document assets, but without truly knowing their objectives.

Fear not! In order to avoid bad project implementation, we advocate a model based on iteration and value generation. Conversely, this approach allows for better risk control and the possibility of a faster return on investment.

The first effects should be observable at the end of each iteration. In other words, the objective must be set to produce concrete value for the company, especially for your data users.

For example, if your goal is data compliance, start documentation focused on these properties and target a particular domain, geographic area, business unit, or business process.

Your Troop’s Motivation Will Wear Off Over Time

While it is possible to gain adherence and support regarding your company’s data inventory efforts in its early stages, it is impossible to maintain this support and commitment over time without automation capabilities.

We believe that descriptive documentation work should be kept to a minimum to keep your teams motivated. The implementation of a data catalog must be a progressive project and will only last if the effort required by each individual is greater than the value they will get in the near future.

You Won’t Have the Critical Mass of Information Needed

For a data catalog to bring value to your organization, it must be richly populated.

In other words, when a user searches for information in a data catalog, they must be able to find it for the most part.

At the start of your data catalog implementation project, the chances that the information requested by a user is not available are quite high.

However, this transition period should be as short as possible so that your users can quickly see the value generated by the data catalog. By choosing a tactical solution, based on its technology and connectivity to information sources, a pre-filled data catalog will be available as soon as it is implemented.

Does Not Reflect Your Operational Reality

In addition to these challenges, data catalogs must have a set of automated features that are useful and effective over time. Surprisingly, many solutions do not have offer these minimum requirements for a viable project, and are unfortunately destined for a slow and painful death.

Connecting data catalogs to your sources will ensure that your data consumers:

Reliability as to the information made available in the data catalog for analysis and use in their projects.
Fresh information: Are they up to date, in real time?

About Actian Corporation

Actian Life

A Letter Regarding COVID-19 From Actian CEO Rohit De Souza

Actian Corporation

March 20, 2020

A letter to our customers and partners:

At Actian, our hearts and thoughts go out to the people who have been affected by this unprecedented global event and we appreciate the healthcare workers, local communities, and governments around the world who are on the front line working to contain this pandemic. Our focus is on the health and safety of our employees, family and the communities we engage with socially and professionally.

Please know that we are vigilantly monitoring the COVID-19 (coronavirus) situation around the clock and we are confident we have taken the necessary precautions to ensure our ability to continue running our business, operating our platform, and providing continuous high quality support to all our customers globally.

To date, Actian does not have any confirmed cases of COVID-19 and we are dedicated to minimizing the risk of exposure. We have not had any disruptions in our services and offerings, and as part of our pandemic response we have implemented the following measures — with the goal of ensuring our service to you will be uninterrupted, while protecting in every way possible the health and well-being of our personnel.

We are utilizing social distancing as a worldwide policy. All Actian employees around the world have been strongly encouraged to work from home. With our highly distributed workforce, and many of our employees typically working remotely, this shift has been extremely smooth, and we will continue to deliver the highest levels of service and support to our customers. We have global collaboration tools and a weekly company-wide “All Hands Call” (along with numerous regular team calls) for continuous feedback and updates to our employees on any enhanced guidelines. We have also mandated travel restrictions and visitor guidelines to reduce the risk of infection.

Our teams have been instructed to work with customers through digital channels as much as possible in support of social distancing and keeping in-person interactions to a minimum. We have postponed our in-person marketing events and implemented a shift to virtual events to keep our customers updated and connected with the technology community. To assist our customers with the increased needs for remote expertise during these challenging times, we are offering our Remote DBA and Customer Success services at a discounted rate.

Actian is committed to doing our part to stem the spread of the COVID-19 virus, and to heed the best practices directed by public health authorities and government guidelines. This dynamic and rapidly moving health crisis will present challenges for everyone at home and in the workplace. Our early preparedness efforts give us confidence that we will continue providing excellent services to you without interruption or compromise. We at Actian are wishing you and your families and your colleagues good health and well-being.

With the actions we have taken, we remain confident and committed to supporting all our customers and partners as they work through these trying times.

Rohit de Souza
President & CEO

About Actian Corporation

Data Intelligence

How Spotify Improved Their Data Discovery for Their Data Scientists

Actian Corporation

March 19, 2020

As the world leader in the music streaming market, it is without question that the huge firm is driven by data.

Spotify has access to the biggest collections of music in the world, along with podcasts and other audio content.

Whether they’re considering a shift in product strategy or deciding which tracks they should add, Spotify says that “data provides a foundation for sound decision-making”.

Spotify in Numbers

Founded in 2006 in Stockholm, Sweden, by Daniel Ek and Martin Lorentzon, the leading music app’s goal was to create a legal music platform in order to fight the challenge of online music piracy in the early 2000s.

Here are some statistics & facts about Spotify in 2020:

248 million active users worldwide.
20,000 songs are added per day on their platform.
Spotify has a 40% share of the global music streaming market.
20 billion hours of music were streamed in 2015.

These numbers not only represent Spotify’s success, but also the colossal amounts of data that is generated each year, let alone each day! To enable their employees, or as they call them, Spotifiers, to make faster and smarter decisions, Spotify developed Lexikon.

Lexikon is a library of data and insights that helps employees find and understand the data and knowledge generated by their expert community.

What Were the Data Issues at Spotify?

In their article How We Improved Data Discovery for Data Scientists at Spotify, Spotify explains that they started their data strategy by migrating data to the Google Cloud Platform, and saw an explosion of their datasets. They were also in the process of hiring many data specialists such as data scientists, analyst, etc. However, they explain that datasets lacked clear ownership and had little-to-no documentation, making it difficult for these experts to find them.

The next year, they released Lexikon, as a solution for this problem.

Their first release allowed their Spotifiers to search and browse through available BigQuery tables as well as discover past researches and analysis. However, months after the launch, their data scientists were still reporting data discovery as a major pain point, spending most of their time trying to find their datasets therefore delaying informed decision-making.

Spotify decided then to focus on this specific issue by iterating on Lexikon, with the unique goal to improve data discovery experience for data scientists.

How Does Lexikon Data Discovery Work?

In order for Lexikon to work, Spotify started out by conducting research on their users, their needs as well as their pain points. In doing so, the firm was able to gain a better understanding of their users intent and use this understanding to drive product development.

Low Intent Data Discovery

For example, you’ve been in a foul mood so you’d like to listen to music to lift your spirits. So, you open Spotify, browse through different mood playlists and put on the “Mood Booster” playlist.

Tah-dah! This is an example of low-intent data discovery, meaning your goal was reached without extremely strict demands.

To put this into Spotify’s data scientists context, especially new ones, their low intent data discovery would be:

Find popular datasets used widely across the company.
Find datasets that are relevant to the work my team is doing.
Find datasets that I might not be using, but I should know about.

So in order to satisfy these needs, Lexikon has a customizable homepage to serve personalized recommendations to users. The homepage recommends potentially relevant, automatically generated suggestions for datasets such as:

Popular datasets used within the company.
Dataset recently used by the user.
Datasets widely used by the team the user belongs to.

High Intent Data Discovery

To explain this in simple terms, Spotify uses the example of hearing a song, and researching it over and over in the app until you finally find it, and listen to it on repeat. This is high intent data discovery.

A data scientist at Spotify with high intent has specific goals and is likely to know exactly what they are looking for. For example they might want to:

Find a dataset by its name.
Find a dataset that contains a specific schema field.
Find a dataset related to a particular topic.
Find a dataset that a colleague used of which they can’t remember the name.
Find the top datasets that a team has used for collaborative purposes.

To fulfill their data scientists needs, Spotify focused first on their search experience.

They built a search ranking algorithm based on popularity. By doing so, data scientists reported that their search results were more relevant, and had more confidence in the datasets they discovered because they were able to see which dataset was more widely-used by the company.

In addition to improving their search rank, they introduced new types of properties (schemas, fields, contact, team, etc.) to Lexikon to better represent their data landscape.

These properties are able to open up new pathways for data discovery. In the example down below, a data scientist is searching for a “track_uri”. They are able to navigate through the “track_uri” schema field page and see the top tables containing this information. Since adding this new feature, it has proven to be a critical pathway for data discovery, with 44% of Lexikon users visiting these types of pages.”

Final Thoughts on Lexikon

Since making these improvements, the use of Lexikon amongst data scientists has increased from 75% to 95%, putting it in the top 5 tools used by data scientists!

Data discovery is thus, no longer a major pain point for their Spotifiers.

Sources:

Spotify Usage and Revenue Statistics (2019): https://www.businessofapps.com/data/spotify-statistics/
How We Improved Data Discovery for Data Scientists at Spotify: https://labs.spotify.com/2020/02/27/how-we-improved-data-discovery-for-data-scientists-at-spotify/
75 Amazing Spotify Statistics and Facts (2020): https://expandedramblings.com/index.php/spotify-statistics/

Financial Services Companies Need All the Data Analytics They Can Get

Connected Data

Robust Analytics

Engagement Tools

Fraud Detection and Prevention

Data Processing at Enterprise Scale

About Actian Corporation

Related Tags

Subscribe to the Actian Blog

Subscribe

Thank you for subscribing to the Actian Blog!

Ready to Get Started?

Making the Transition From Flat Files to SQLite

About Actian Corporation

Related Tags

Subscribe to the Actian Blog

Subscribe

Thank you for subscribing to the Actian Blog!

Ready to Get Started?

Healthcare Analytics Improve Operational Efficiency

Healthcare Analytics for Diagnosis and Patient Care

Hospital Operations

Clinical Trials

Insurance Claim Processing

About Actian Corporation

Related Tags

Subscribe to the Actian Blog

Subscribe

Thank you for subscribing to the Actian Blog!

Ready to Get Started?

Data Warehouse Appliances are Becoming History

Data Warehouse Appliances Served Their Purpose

Hardware Refresh Cycles and the Shift to the Cloud

About Actian Corporation

Related Tags

Subscribe to the Actian Blog

Subscribe

Thank you for subscribing to the Actian Blog!

Ready to Get Started?

Data Ops Rules to Avoid Data Oops

Data Oops Definition

How to Succeed in Your DataOps

What are the Benefits of DataOps?

About Actian Corporation

Subscribe to the Actian Blog

Subscribe

Thank you for subscribing to the Actian Blog!

The More You Refine Stored Data, the More Valuable it Becomes

Collection

Aggregation

Reconciliation

Categorization

Analysis

Presentation

About Actian Corporation

Related Tags

Subscribe to the Actian Blog

Subscribe

Thank you for subscribing to the Actian Blog!

Ready to Get Started?

Five Reasons Why You Need an Enterprise Data Integration Strategy

1. Save Time and Resources in Building Integrations

2. Ensure Business Continuity With Fewer Disruptions to Business Processing

3. Centralized Governance and Management Lowers Risk

4. Faster Response to Threats

5. Increased Agility in Times of Change

About Traci Curran

Related Tags

Subscribe to the Actian Blog

Subscribe

Thank you for subscribing to the Actian Blog!

Ready to Get Started?

Real-Time Reporting Platform for Operational Systems

Digital Transformation has Created More Real-Time Data

Data-Driven Decision-Making Drives Value and Profits

Real-Time Reporting Platforms Unlock the Value Potential of Operational Systems

About Actian Corporation

Related Tags

Subscribe to the Actian Blog

Subscribe