Data engineering is a fascinating and fulfilling career – you are at the helm of every business operation that requires data, and as long as users generate data, businesses will always need data engineers. In other words, job security is guaranteed. But, with such great power comes great responsibility. The journey to becoming a successful data engineer […]
How a Modern Data Engineering Stack Can Help Create a Data-Driven Culture
Creating a collaborative, data-driven culture is one of the most important goals of many modern organizations. However, access to the data and data processing tools remains restricted to the selected few technical users or upper management echelons. Data-driven culture cannot exist without the democratization of the data. Data democratization certainly does not mean unrestricted access to all […]
Data Architecture Trends in 2022
The primary purpose of implementing a Data Architecture is to standardize the methods and protocols, as well as the systems for acquiring, storing, managing, and sharing data across the enterprise for improved decision-making. In modern businesses, most decisions are made in real time, and to facilitate an efficient and real-time friendly Data Management infrastructure, data […]
The Future of Data Engineering
A Data Engineering Guide reveals that while people often rely on the work of data engineers — depending on Siri for quick solutions or being enchanted by custom recommendations or promos — they often do not realize that these advanced tools can provide accurate results only because of the hard work put in by data […]
Data Architect vs. Data Engineer
Data careers are becoming increasingly important and popular all across the globe, simply because “data” is the new currency of the data economy. The pandemic gave the needed push to accelerate the digital transformation of global businesses, and currently, the primary market differentiator is an enterprise’s data infrastructure readiness. This data infrastructure comprises systems, processes, […]
Data Architect vs. Data Modeler vs. Data Engineer
Michael Bowers, author and Chief Data Architect at FairCom Corporation, initially set out to research three careers in his presentation titled Data Architect vs. Data Modeler vs. Data Engineer for the DATAVERSITY® Data Architecture Online 2019 Conference. The process brought him to a wealth of information he would have appreciated much earlier in his career, […]
Case Study: Deriving Spark Encoders and Schemas Using Implicits
Click to learn more about author Dávid Szakallas. In recent years, the size and complexity of our Identity Graph, a data lake containing identity information about people and businesses around the world, begged the addition of Big Data technologies in the ingestion process. We used Apache Pig initially, and then migrated to Apache Spark a […]
Building the Future of Women in Data
Careers involving data expertise are soaring. Data skills have seen significant growth in recent years, according to The Quant Crunch report. By 2020, the report predicts, the number of positions for data and analytics talent in the United States will increase by 364,000 openings, to 2,720,000. Clinical Data Analysis (+54 percent) Data Science (+40 […]
Jun 29 Webinar: Data Quality, Data Engineering, and Data Science
DATE: June 29, 2017 This webinar has passed. The recording will be made available within two US business days. TIME: 2 PM Eastern / 11 AM Pacific PRICE: Free to all attendees About the Webinar This webinar explores the organizational constructs and processes for enabling business to build better insights through Data Quality, Data […]
The Tech Industry Is in the Middle of a Data Engineering Talent Shortage
by Angela Guess According to a new press release, “A new study, The State of Data Engineering released today by Stitch, an ETL service; and Galvanize, a learning community for technology, reveals a shortage in data engineering talent. There are only 6,500 self-reported data engineers on LinkedIn, but the San Francisco Bay area alone has […]