Postmortem Report: Delivery Issue on Dec 21-22, 2024

Update: The issue has been resolved. Sorry for the inconvenience.

Summary

Some users noticed articles weren’t arriving on their Kindles. Investigations showed emails from KTool were bouncing at Amazon’s server. Here’s what happened and how we fixed it.

Timeline

• Dec 21: About 20% of articles weren’t delivered.
• Dec 22, 11:00 UTC: About 95% of articles bounced. Multiple users reported the issue.
• Dec 22, 14:00 UTC: The issue was resolved.

Root Cause

KTool uses Amazon's send-to-kindle email service under the hood. On Dec 21, Amazon suddenly started soft-bouncing some of our emails. By Dec 22, it escalated to 95%. A recent spike in suspicious article sends (due to bad actors) likely triggered Amazon to put KTool “on hold.”

Resolution & Recovery

• I switched to Resend as our email service provider, which resolved the delivery problem.
• I continue to work with MailerSend to diagnose the bounce issue.
• Total downtime and investigation took about 3 hours.

Preventative Measures

• Use multiple email providers instead of relying on one.
• Implement rate limiting to prevent abuse.
• Strengthen delivery monitoring so we can catch these problems faster.

Customer Impact

A total of 439 articles failed to send. Please resend your past articles because Amazon’s system doesn’t let KTool know which ones ultimately made it through.

You can do this at https://ktool.io/app/history.

Thank you for your patience and ongoing support!

—Daniel