Chariot Solutions – Medium

Chariot Solutions

Chariot Solutions
in
Thoughts on Data Engineering

Aggregating Files in Your Data Lake: Part 3

Several years ago, I wrote the post Friends Don’t Let Friends Use JSON (in their data lakes). The crux of that post was that JSON doesn’t…

9 min readMar 27, 2024

--

Aggregating Files in Your Data Lake: Part 3

--

Chariot Solutions
in
Thoughts on Data Engineering

Aggregating Files in your Data Lake: Part 2

In my last post, I developed a data pipeline to aggregate CloudTrail log files. When I ran this pipeline against Chariot’s CloudTrail…

7 min readFeb 29, 2024

--

Aggregating Files in your Data Lake: Part 2

--

Chariot Solutions
in
Thoughts on Data Engineering

Aggregating Files in your Data Lake: Part 1

As I’ve written in the past, large numbers of small files make for an inefficient data lake. But sometimes, you can’t avoid small files…

10 min readFeb 15, 2024

--

Aggregating Files in your Data Lake: Part 1

--

Chariot Solutions
in
Thoughts on Data Engineering

Data Engineering: More SRE Than SQL

Following my post about the Chariot Data Engineering interview, I received some comments along the lines of “wait, you don’t test their SQL…

5 min readJan 9, 2024

--

Data Engineering: More SRE Than SQL

--

Chariot Solutions
in
Thoughts on Data Engineering

Developing A Coding Test for Data Engineering

Hiring good candidates is difficult. After nearly 40 years in this business, and interviewing hundreds of candidates, I’m not going to…

6 min readNov 21, 2023

--

Developing A Coding Test for Data Engineering

--

Chariot Solutions
in
Thoughts on Data Engineering

Small Data: A Pipeline for Low-Latency Decision Support

In my last post, I said that I didn’t think Postgres was a good choice for a decision support database, versus a task-specific DBMS such as…

5 min readAug 14, 2023

--

Small Data: A Pipeline for Low-Latency Decision Support

--

Chariot Solutions
in
Thoughts on Data Engineering

Why Not Just Use Postgres?

My last few posts have focused on Redshift and Athena, two specialized tools for managing and querying Big Data. But there’s a meme that’s…

12 min readJul 31, 2023

--

Why Not Just Use Postgres?

--

Chariot Solutions
in
Thoughts on Data Engineering

A Deep Dive on Redshift Execution Plans

Execution plans are one of the primary tools to optimize your database queries, but they can be daunting to read and understand. In this…

12 min readJul 17, 2023

--

A Deep Dive on Redshift Execution Plans

--

Chariot Solutions
in
Thoughts on Data Engineering

Unbalanced Data in Redshift

I first experienced unbalanced data in a data warehouse thirty years ago. I was working for a mutual fund company, and something like a…

8 min readJul 3, 2023

--

Unbalanced Data in Redshift

--

Chariot Solutions
in
Thoughts on Data Engineering

Rightsizing Data for Athena

Amazon Athena is a service that lets you run SQL queries against structured data files stored in S3. It takes a “divide and conquer”…

7 min readJun 19, 2023

--

Rightsizing Data for Athena

--

Chariot Solutions

Chariot Solutions

We solve your most complex business problems with our strategic, high-touch approach and extensive tech experience. Learn more at chariotsolutions.com

Help
Status
About
Careers
Blog
Privacy
Terms
Text to speech
Teams