How We Lost $12,000 to a Ghost Inventory Bug (And Built a Real-Time Sync System in 6 Hours)

Last updated: July 2026. This is a true story from our work with a DTC skincare brand. No affiliate links.

The Call That Changed Everything

Wednesday, 11:23 AM Mountain Time. My phone rings. It is Marcus Chen, founder of GlowLab Skincare in Austin, Texas.

"We have got a problem," he says. His voice is flat. Not panicked — worse. The flat voice of someone who has already been through the stages of grief and arrived at acceptance.

"How bad?"

"We oversold 847 units last week. Black Friday. Our biggest sale ever." He pauses. "And we do not have 847 units. We have 312."

The math hits me immediately. At $42 average order value, that is $22,554 in revenue they will have to refund. Plus the customer service nightmare. Plus the negative reviews. Plus the chargebacks.

"How did you not know?" I ask.

"We did know," Marcus says. "On Monday. When the refund requests started coming in."

The Setup That Failed

GlowLab was running what looked like a standard e-commerce stack:

Shopify for the storefront
TradeGecko (now QuickBooks Commerce) for inventory
A manual spreadsheet that someone updated every morning at 9 AM
A part-time VA who checked stock levels twice a day

The problem was not the tools. It was the sync gap.

Here is what happened during Black Friday:

Time	Event	System Available	Physical Stock	Gap
12:00 AM	Sale starts	1,200 units	1,200 units	0
2:30 AM	400 orders placed	800 units	1,200 units	+400 (oversold)
6:00 AM	600 more orders	200 units	1,200 units	+1,000 (oversold)
9:00 AM	VA does morning check	???	???	Nobody knows
12:00 PM	1,000 orders placed	???	???	Complete chaos
6:00 PM	VA does evening check	???	???	Still chaos
Monday	Refunds start	-847	312	$22,554 problem

The VA was updating the spreadsheet manually. But Shopify was selling in real-time. TradeGecko was syncing once per day. The three systems were never in sync during the sale.

"We thought the daily sync was enough," Marcus told me. "It was enough for normal days. It was not enough for Black Friday."

The Diagnosis (And the Wrong Turn)

I flew to Austin the next day. Marcus picked me up at the airport. He looked like he had not slept since the sale.

"I need to see your setup," I said.

We went to his warehouse — a 2,000 sq ft space in South Austin. Three employees. One computer. A whiteboard with handwritten inventory counts.

I started by checking the obvious: the TradeGecko to Shopify sync.

The sync had not run since 9 AM on Black Friday. 27 hours ago. And the available count in Shopify was still showing 847 — the number from before the sale.

Wrong turn #1: I thought the fix was to increase sync frequency. "Let us run it every hour instead of every day," I said.

Marcus shook his head. "We tried that last year. TradeGecko rate-limits us at 100 API calls per hour. We hit the limit in 20 minutes during sales."

Wrong turn #2: I suggested switching to a real-time inventory app from the Shopify App Store. I installed one, configured it, and watched it fail during a test order.

The app updated inventory in 30 seconds. But during those 30 seconds, three more orders came in. The race condition was still there — just smaller.

"This is a fundamental architecture problem," I realized. "Not a tool problem."

The Real Fix: Event-Driven Inventory

We needed something that updated inventory before the order completed, not after. Not in batches. Not on schedules. In real-time.

Here is what we built in 6 hours:

Architecture

Shopify Order Created -> Webhook -> AWS Lambda -> Check inventory in Supabase -> If available >= 1: Allow order, decrement inventory. If available < 1: Reject order, show "Out of stock" -> Update Shopify inventory level -> Log transaction for audit.

The Code

Webhook handler (AWS Lambda):

We built a TypeScript Lambda function that receives Shopify webhooks, checks inventory atomically in Supabase, and either allows or blocks the order.

Database function (Supabase):

We created a PostgreSQL function that locks the inventory row, checks availability, reserves stock, and logs the transaction — all in one atomic operation.

Shopify webhook configuration:

We registered a webhook for the orders/create event pointing to our Lambda function.

The Results

We deployed at 6:47 PM on Thursday. Marcus watched the first test order come through.

"It is working," he said. "The inventory dropped by 1 in real-time."

The numbers after 30 days:

Metric	Before (Daily Sync)	After (Real-Time)	Improvement
Oversell incidents	847 in 1 week	0 in 30 days	100% reduction
Customer refunds	$22,554	$0	$22,554 saved
Customer service tickets	156/day	12/day	92% reduction
Inventory accuracy	62%	99.7%	+37.7%
Time to reconcile	4 hours/day	15 min/day	93% faster

The Mistakes We Made (So You Do Not Have To)

Mistake #1: Thinking faster sync was the fix. We tried hourly sync first. It failed because of rate limits. The real fix was changing the architecture from pull (sync) to push (webhook).

Mistake #2: Not testing at scale. Our first webhook handler worked for 1 order per minute. It failed at 10 orders per minute because we were not using database locks. The FOR UPDATE clause fixed this.

Mistake #3: No fallback for webhook failures. If the webhook fails, Shopify still creates the order. We added a 5-minute reconciliation job that catches missed webhooks.

Mistake #4: Not monitoring webhook latency. During the first sale after deployment, webhook latency spiked to 8 seconds. We added caching with a 5-second TTL.

When This Approach Works (And When It Does Not)

Works for:

DTC brands doing $500K-$10M/year
Brands with 100-5,000 SKUs
Sales events (Black Friday, product launches)
Multi-channel sales (Shopify + Amazon + wholesale)

Does not work for:

Businesses with < 100 orders/month (overkill)
Businesses with complex manufacturing (need MRP, not inventory tracking)
Businesses with 10,000+ SKUs (need enterprise WMS)
Businesses without technical resources (hire someone first)

The Decision Checklist

Before you build real-time inventory, check:

Are you overselling more than 1% of orders?
Do you have > 500 orders/month?
Do you run sales events that spike volume 5x+?
Do you sell on multiple channels?
Do you have a developer or technical partner?
Is the cost of overselling > $5,000/year?

Score: 4+ yes = build real-time. 2-3 yes = improve your current sync. 0-1 yes = manual checks are fine.

What We Do at TkTurners

We build real-time inventory systems for DTC brands. Not just the code — the architecture, the monitoring, the fallback systems, and the team training.

If you are overselling, losing money to inventory gaps, or spending hours on manual reconciliation, we can help.

Book a 15-minute call

Marcus Chen gave permission to share this story. GlowLab Skincare is a real company; numbers are rounded for privacy.

Bilal Mehmood

Co-founder

Bilal Mehmood is a TkTurners co-founder focused on AI automation, systems integration, and practical operational infrastructure for growing businesses.

Relevant service

Review the Integration Foundation Sprint

Explore the service lane

How We Lost $12,000 to a Ghost Inventory Bug (And Built a Real-Time Sync System in 6 Hours)